Friday, April 19, 2013

MASTIFF 0.6.0 Released!

The latest version of MASTIFF, 0.6.0, has just been released! Run over to the download site and grab the latest version!

The official changelog is located here, but the major improvements are described below.

Upgrading MASTIFF to the latest version is easy. You can follow this process:
  1. Download and install pydeep.
  2. Download MASTIFF 0.6.0 and untar it.
  3. Run "make test" to ensure you are not missing any dependencies.
  4. Run "sudo make install" to install the latest version.
  5. Copy the analysis plug-ins (the plugins directory in the tarball) to your location of choice and ensure the config file is pointing to that directory.
  6. Add any new options to your MASTIFF config file. The easiest way may be to use sdiff.

Queue

MASTIFF now has a queueing system so multiple files can be analyzed by the framework. To utilize this, give MASTIFF a directory instead of a file to analyze. It will find all files in that directory and its subdirectories, add them to the queue, and begin processing.

The queue is maintained within the MASTIFF database. So, if you have to stop MASTIFF in the middle of its run, it will begin re-processing the queue when its restarted. Some additional options have been added to allow you to work with the queue:
  • --clear-queue: This will clear the current queue.
  • --ignore-queue: This will ignore the queue and just process the file you give it.
Analysis plug-ins are also taking advantage of the queue. The pdf-parser and ZipExtract plug-ins have a new option ("feedback") which allow you to feed files from the plug-ins back into the queue for processing. For example, the ZipExtract plug-in will add all files that were extracted from the archive into the queue for processing.

Fuzzy Hashing

Fuzzy hashing is not something new within MASTIFF. However, we have changed the Python library used for it. Previously, we used pyssdeep but found that there were a number of stability issues with it on OSX and when processing large amounts of files.

Therefore, we have switched to pydeep (https://github.com/kbandla/pydeep). Our testing has shown it to be much more stable thus far.

libmagic

There was some confusion on which Python libmagic libraries to use when installing MASTIFF. To help alleviate some of that, the framework has been modified to use two different libmagic libraries:
If either library is installed, MASTIFF will utilize them.

Other Changes

A number of other bug fixes and improvements have been made. Please see the changelog file for a complete list.

As always, if you have any questions, please email mastiff-project@korelogic.com.

We have alot of great things coming down the pipe for MASTIFF, but if you have any suggestions, enhancements or plug-ins, let us know!

6 comments:

Anonymous said...

Not working for me. I run linux mint 15 x64 at work and I can't get this installed. pydeep complains that python.h can't be found, and when I try to install python-dev or any variation thereof, I get either the error "no installation candidate", "package not found" or " is already installed and at its latest version".

Tyler said...

Somehow I missed this comment (a while ago). If you are still having issues, let me know. This is typically seen because pydeep is missing a dependency.

Anonymous said...

Anon: I had the exact same error with mint 15 x64 - can you install the package 'python2.7-dev'? that worked for me

Anonymous said...

I am also having the same problem with pydeep

Tyler said...

Ah the joys of not being notified of comments. Sorry for the delay in responses.

pydeep requires python-dev to be installed. On Mint, you should be able to do "sudo aptitude install python-dev". If that doesn't work, do "sudo aptitude search python | grep dev" and see what pulls up.

Anonymous said...

I have a question about "The file type of the file being analyzed is automatically determined.",the file type of the file is not sure before analyzing.