Monday, February 10, 2014

Installing Yara into IDA Pro 64-bit Linux

tl;dr Install a 32-bit VM, compile Yara, copy files over. See link below for files to just install.

Last Friday, pnX posted that he updated his awesome IDA plug-in, IDAScope, to include Yara support. This means that you can now run Yara sigs against files you are reversing to help in the analysis process.

After I installed the new version of IDAScope into IDA Pro, however, I received errors stating that Yara could not be imported. I thought this was odd as I had Yara installed on my system, until I remembered how IDA works on a 64-bit Linux system.

The following is based off my observations and experiences. If I am incorrect on this, please forgive me and let me know in the comments.

IDA is a 32-bit program. Even the 64-bit version of IDA is compiled as a 32-bit program.

$ file idaq idaq64
idaq:   ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), BuildID[sha1]=0xcb635dd38de5c73f050de37a0f2e492688b3ab9a, stripped
idaq64: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), BuildID[sha1]=0x1f03dcff4bfd776b23df71c8d9d471fb63b0bf48, stripped

This causes a number of interesting issues on 64-bit Linux systems, especially with Python. Hex Rays has gotten the majority of these fixed in the default install so you don't worry about them, and the way it does this with Python is by allowing you to install a bundled Python into the IDA Pro directory. (There are other ways, but I have not done them.) This gives you a working "out of the box" product.

This also means that when you want to install a new Python library and use it in IDA, you have to install it into the IDA's bundled Python directory as well. If this is a pure Python module, then no problem. Just copy and it should work. Yara is different.

Since Yara compiles as a 64-bit library on a 64-bit system, and yara-python does the same, we can't just install it directly into the IDA Python directory. If you do, you'll receive errors that IDA is unable to load a 64-bit module.

In order to get Yara working, we'll need to compile it as a 32-bit library. The easiest way, IMO, to do this is to load a 32-bit Linux system into a VM, compile Yara, then copy the files into your IDA installation. I did this in a Debian 6.0.3 and it worked without a problem. Just to be safe, make sure you are using a system with Python 2.7 as well since that is what IDA bundles.

There are two files you will need: the Yara library libyara.so.0 and the Yara Python library yara.so (located in the Python dist-packages directory after installation). Follow the instructions to compile and install Yara in your 32-bit VM, and copy the files onto your 64-bit system. libyara.so.0 goes into your base IDA install directory, and yara.so goes into the python directory underneath that.

After you do that, Yara-python will be installed and will work great!



Don't want to go through all the trouble of installing a 32-bit VM, compiling, and copying? I don't blame you. I uploaded the version I compiled to my Google Drive here.

yara-ida-libs.tgz (SHA256: 38674b584adf3932e5cd1cafbd0bb288b7db3302304a83041bad9295472aa064)

Just untar this into your base install dir for IDA and you should be good to go.

Hex Rays has published instructions on how to install Python packages from Pip on a 64-bit system. I recommend checking them out. This time, my way just felt easier.

Wednesday, September 4, 2013

Installing BinDiff on Linux Mint 14

I recently upgraded my system to Linux Mint 14 and went about re-installing all my software. When I got to Zynamics/Google BinDiff, I found I had an issue:
$ sudo dpkg -i bindiff401-debian50-amd64.deb

Selecting previously unselected package bindiff.
Unpacking bindiff (from bindiff401-debian50-amd64.deb) ...
dpkg: dependency problems prevent configuration of bindiff:
 bindiff depends on sun-java6-jre; however:
  Package sun-java6-jre is not installed.

Unfortunately, BinDiff requires sun-java6-jre, which is not in the Linux Mint repository, nor any other repository I could find. I could circumvent this by installing BinDiff by using the --ignore-depends=sun-java6-jre option to dpkg. However, every time I went to install updates I would get an error message that BinDiff was broken, and be prompted to uninstall it before I could continue.

However, I found a work-around - create a dummy package named sun-java6-jre using the tool equivs. (There are some docs out there on this, but I was unable to find a non-Google cached copy, so here was what I did.)

Linux Mint has equivs in its repository, so if its not already installed, apt-get it.

Next, run equivs-control sun-java6-jre and this will create a file named sun-java6-jre that you will need to modify.

At minimum, you'll need to uncomment and/or fill out the following fields:
  • Package
  • Version
  • Maintainer
I also filled out the description fields so I would remember what it was.

After the file is modifoed, run equivs-build sun-java6-jre and you should see something similar to below:
$ equivs-build sun-java6-jre
dh_testdir
dh_testroot
dh_prep
dh_testdir
dh_testroot
dh_install
dh_installdocs
dh_installchangelogs
dh_compress
dh_fixperms
dh_installdeb
dh_gencontrol
dh_md5sums
dh_builddeb
dpkg-deb: building package `sun-java6-jre' in `../sun-java6-jre_6.0_all.deb'.

The package has been created.
Attention, the package has been created in the current directory,
not in ".." as indicated by the message above!
Once that has successfully completed, you should have a sun-java6-jre_6.0_all.deb file in your directory. If that failed, you probably forgot to modify one of the fields in the file.

Finally, just dpkg -i the new deb file and BinDiff, and you should be ready to go!
$ sudo dpkg -i sun-java6-jre_6.0_all.deb
Selecting previously unselected package sun-java6-jre.
(Reading database ... 237677 files and directories currently installed.)
Unpacking sun-java6-jre (from sun-java6-jre_6.0_all.deb) ...
Setting up sun-java6-jre (6.0) ...
$ sudo dpkg -i bindiff401-debian50-amd64.deb
Selecting previously unselected package bindiff.
(Reading database ... 237681 files and directories currently installed.)
Unpacking bindiff (from bindiff401-debian50-amd64.deb) ...
bindiff license has already been accepted
Setting up bindiff (4.0.1) ...

$
Then you are good to go!

Sunday, May 19, 2013

My Take on the City of Akron Hack

On Thursday, May 16, 2013, a Turkish hacking group called Turkish Ajan hacked into the City of Akron and released a number of files that contain personal information on a number of Akron citizens. According to the city, the attackers were able to gain access into some internal systems where they obtained tax information.

The news has died down on this for the moment, but from the information that has been released, there are some things we can infer:

1. The attackers compromised the city's public website. From the errors that were being displayed on the site, information that has been released from the city, and the way this group works, it was likely through SQL injection (although this has not been specifically stated yet).

2. The attackers compromised the city's internal systems and obtained access to tax systems. It is unknown if they were able to do from the city's public website, through the tax paying system, or some other server. In any case, this appears to be where the attackers got the files they posted.

3. Around 25K people are affected.

4. The FBI is involved and was called in quickly after the compromise was discovered. IMO, this is good.

Any additional information on what happened is pretty much speculation. Trust me, I've been speculating alot and have a pretty good idea of what happened, but I have no proof. Hopefully whoever is doing the forensics for the city will have their findings released at some point. As an Akron citizen, and tax payer, I want to know this information.

However, there is one thing that I think needs brought up. Why was this information stored unencrypted? If it was encrypted, how did the attackers obtain access to the keys to decrypt it?

The information that was released contains social security numbers of both the taxpayer and their spouse, and credit card numbers. According to PCI standards (and my understanding), the credit card numbers should have been encrypted. The federal government is required to comply with PCI, what about the city of Akron government?

As for the SSNs, I don't know of any specific regulations that requires that information to be encrypted (please let me know if there is), but I can't imagine that there is any reason it shouldn't be. I have a feeling there are at least 25,000 people who agree with me.

One final item of note. The press has been getting quotes from Deputy Mayor Rick Merolla. With all due respect sir, shut up. I can only imagine your IT and information security people are cringing whenever they read your quotes pertaining to the security of the city of Akron systems.

I'm sure you are very smart, but its obvious you are not familiar with information security. Quotes such as "Our systems are all, all our virus protection, intrusion protection systems, all of our virus software is still up to date so we are still not sure how they got in" show this. Let those performing the investigation or the talented IT personnel you employ speak on these things.

If you like, I am personally offering to give you a training course on information security, hackers, and how attacks take place. This will at least give you an idea on why the things you have been quoted as saying are cringe-worthy.

Friday, April 19, 2013

MASTIFF 0.6.0 Released!

The latest version of MASTIFF, 0.6.0, has just been released! Run over to the download site and grab the latest version!

The official changelog is located here, but the major improvements are described below.

Upgrading MASTIFF to the latest version is easy. You can follow this process:
  1. Download and install pydeep.
  2. Download MASTIFF 0.6.0 and untar it.
  3. Run "make test" to ensure you are not missing any dependencies.
  4. Run "sudo make install" to install the latest version.
  5. Copy the analysis plug-ins (the plugins directory in the tarball) to your location of choice and ensure the config file is pointing to that directory.
  6. Add any new options to your MASTIFF config file. The easiest way may be to use sdiff.

Queue

MASTIFF now has a queueing system so multiple files can be analyzed by the framework. To utilize this, give MASTIFF a directory instead of a file to analyze. It will find all files in that directory and its subdirectories, add them to the queue, and begin processing.

The queue is maintained within the MASTIFF database. So, if you have to stop MASTIFF in the middle of its run, it will begin re-processing the queue when its restarted. Some additional options have been added to allow you to work with the queue:
  • --clear-queue: This will clear the current queue.
  • --ignore-queue: This will ignore the queue and just process the file you give it.
Analysis plug-ins are also taking advantage of the queue. The pdf-parser and ZipExtract plug-ins have a new option ("feedback") which allow you to feed files from the plug-ins back into the queue for processing. For example, the ZipExtract plug-in will add all files that were extracted from the archive into the queue for processing.

Fuzzy Hashing

Fuzzy hashing is not something new within MASTIFF. However, we have changed the Python library used for it. Previously, we used pyssdeep but found that there were a number of stability issues with it on OSX and when processing large amounts of files.

Therefore, we have switched to pydeep (https://github.com/kbandla/pydeep). Our testing has shown it to be much more stable thus far.

libmagic

There was some confusion on which Python libmagic libraries to use when installing MASTIFF. To help alleviate some of that, the framework has been modified to use two different libmagic libraries:
If either library is installed, MASTIFF will utilize them.

Other Changes

A number of other bug fixes and improvements have been made. Please see the changelog file for a complete list.

As always, if you have any questions, please email mastiff-project@korelogic.com.

We have alot of great things coming down the pipe for MASTIFF, but if you have any suggestions, enhancements or plug-ins, let us know!

Thursday, February 21, 2013

MASTIFF: Automated Static Analysis Framework

Malware analysis is a process that begs to be automated. Messing up one step or running one tool incorrectly can cause you to have to restart the entire process. Fortunately, there are a number of automation frameworks or systems, such as Cuckoo or Threat Expert, that exist to help automate malware analysis.

While these automation frameworks are great, they tend to focus on dynamic analysis (behavioral analysis); static analysis (characteristic analysis) is mostly left out. The static analysis techniques that the frameworks do perform vary, but typically include hashing, strings extraction, some file-type specific tools, along with a couple other techniques. Additional static analysis programs or techniques usually have to be implemented on their own.

To do this, analysts typically create a master static analysis script that runs all of the tools desired against a file. However, if an analysis tool is run against a file type that it cannot analyze, such as a PE header analysis tool on a PDF, you run the risk of crashing the analysis program and, in turn, your automation script.

As an incident responder and malware analyst, I came up against these issues all the time, so I started to look for a solution. Nothing existed to automate the entire static analysis process and allow you to add in your own techniques.

That is why MASTIFF, an open source automated static analysis framework, was created. MASTIFF performs two functions for the analyst:
  • The file type of the file being analyzed is automatically determined.
  • Only those techniques which work on that file type are applied.
By automatically determining the file type for the analyst and ensuring that only the static analysis techniques that work on that file type are run, analysts can be assured that the risk of crashing the automated process is lessened, and that only relevant data is returned.


MASTIFF works by utilizing plug-ins for both file-type detection and static analysis techniques. The decision to utilize plug-ins was two-fold:
  • The types of files analyzed and the techniques available within MASTIFF can be easily expanded by adding new plug-ins.
  • MASTIFF is able to be "crowd-sourced".
The last reason was especially important. Anyone can create a new plug-in to add a new file type or analysis technique. As more people add plug-ins, the more useful the framework becomes. To facilitate easier plug-in development, template, or skeleton, plug-ins have been included with the project. In just a few minutes, someone can modify a few fields in the template and have a new plug-in ready to go.

In the coming weeks, I'll be posting information and tutorials related to MASTIFF, how to use it, how to create plug-ins for it, etc. Please let me know any questions you have on the framework or there is something specific that should be focused on.

Finally, I want to state that MASTIFF was funded through KoreLogic, the company I work for, and the DARPA Cyber Fast Track (CFT) program. If you are unfamiliar with CFT, I highly recommend looking at their site and submitting a proposal. Its a great program, but you only have until April 1, 2013 to do so and then no further submissions will be taken.

Tuesday, February 19, 2013

ShmooCon 2013

This past weekend I went to my first ShmooCon in Washington D.C. I have to say this was an experience that I was not expecting. I've been to many security conferences in the past, included RECon, BlackHat, GFIRST, and some SANS and OWASP conferences. ShmooCon ranks up there in the top 2 spots, if not one of the best that I've been to.

The best thing about ShmooCon is that it has a small con feel to it, while having everything the big cons have (e.g. big name speakers, contests, prizes, lots of smart people). It also has a small con price - if you can get a ticket, its only going to cost you around $150.

I was also lucky enough to be selected as a speaker this year, presenting a talk on my newly open-sourced tool MASTIFF. As a speaker, they one of the best run CFP processes I have ever used. After selection, they are constantly available for questions, have excellent moderators and are great in making sure you have what you need.

The talks at the conference were amazing. They are of the highest quality and even the ones I didn't like were full of good information. Since I was releasing MASTIFF the first day I was there, and I was freaking out about my talk (I was in the last speaking slot of the tracks), I didn't get to see all that I would have liked. However, these stood out:

  • NSM and more with Bro Network Monitor by Liam Randall - This was the best talk of the conference IMO. Liam gave an excellent talk about what Bro is, how it works, and even how easy it is to extend it. His presentation was how all presentations should be - easy to follow and good at explaining a relatively complicated concept.
  • Crypto: You're doing it wrong by Ron Bowes -  Ron gave an excellent talk about some crypto attacks, how they can be performed, and even did 3 live demos (that didn't fail) that performed these attacks. I'm not a crypto guy, but Ron's explanations of everything were easy to follow and entertaining. Plus he used The Call of Cthulhu as some of his encrypted text.
There were alot more that I saw that were excellent, and some that I unfortunately missed. Luckily, ShmooCon makes all their recordings available online for free and should be up in a couple of weeks. I look forward to next year!

Friday, November 9, 2012

2008 Malware Challenge


In 2008, Greg Feezel and I published the following malware analysis challenge. The goal was to answer the questions below and submit them back to us for prizes. While the challenge is no longer going on, we wanted to publish it again so those that wished to try it could.

The malware is contained within a password protected zip file named malware.zip. The password is “infected”. The MD5 hash of the files are:
  • 59a95f668e1bd00f30fe8c99af675691 malware.exe
  • 31d2ec3b312d0fd27940aae5c89e3787 malware.zip

Situation:

A system administrator within your organization has come to you because a user's PC was infected with malware. Unfortunately, anti-virus is unable to remove the malware. However, the administrator was able to recover the suspected malware executable. Your job is to analyze the malware.
Participants should download the malware sample and analyze it. The end result should be a document containing details on the analysis performed. The analysis document can be written in any form, but the following questions and statements should be answered within it. Participants should note when questions are being answered.
  • Describe your malware lab.
  • What information can you gather about the malware without executing it?
  • Is the malware packed? If so, how did you determine what it was?
  • Describe the malware's behavior. In other words - what files does it drop, what registry keys does it modify, what network connections does it create, how does it auto-start, etc?
  • What type of command and control server does the malware use? Describe the server and interface this malware uses.
  • What commands are present within the malware and what do they do? If possible, take control of the malware and run some of these commands, documenting how you did it.
  • How would you classify this malware? Why?
  • What do you think the purpose of this malware is?
Bonus questions:
  • Is it possible to find the malware's source code? If so, how did you do it?
  • How would you write a custom detection and removal tool to determine if the malware is present on the system and remove it?