# Getting Started with md5deep

This document provides an introduction to using md5deep and was last updated on
The current version of this document can be found on the md5deep web site at http://md5deep.sourceforge.net/. sha1deep, sha256deep, tigerdeep, and whirlpooldeep work just like md5deep; all of the command line options are the same.

- Installing md5deep
- Basic Operation
- Error Messages
- Recursive Mode
- Time Estimation Mode
- File Size Mode
- Matching Mode
- Advanced Matching Mode
- Expert Mode

## Installing md5deep

### Microsoft Windows

Users running Microsoft Windows are strongly encouraged to download the precompiled binaries from http://md5deep.sourceforge.net/. Please note that these binaries are created using a Mingw cross compiler. Compiling the programs directly from Windows is not supported.

### Automatic Installation

Before you try to install md5deep manually, see if your operating system supports the programs via an automatic installation method. Some operating systems that provide this feature for md5deep are:

Mac OS X: Brew, DarwinPorts, Fink
Linux: Ubuntu, Fedora, Gentoo, Slackware
BSD: FreeBSD

### Manual installation

If your operating system does not support the automatic installation methods described above, you will have to download the source code and compile the programs yourself. First download the latest tarball of the program from http://md5deep.sourceforge.net/. This file should be named something like `md5deep-3.7.tar.gz`. Uncompress the file with the following command:

        $ tar zxvf md5deep-4.1.tar.gz

Change into the decompressed directory

        $ cd md5deep-4.1

and configure the program.

        $ ./configure

The configure script can accept lots of options. Run `./configure --help` for the complete list. The most common option used is the `prefix` option which installs the program in a location other than the default, `/usr/local/bin`. If you wanted to install the program elsewhere, for example, `/tmp/md5deep`, you would run `./configure --prefix=/tmp/md5deep` instead.

You can now compile the program using the make command:

        $ make

and install it:

        $ make install

Note that you must be root on most operating systems to install the program to its default location, `/usr/local/bin`. The tool `sudo` may help:

        $ sudo make install

## Basic Operation

### Opening a command prompt

md5deep is a command line program. You cannot run the program by double clicking on it! On Microsoft Windows, click on the Start button and choose "Run..." from the menu. In this dialog box, type `cmd.exe` and hit enter. A command prompt should appear. In this window, type the full path to md5deep.exe and then the files you want to hash. For example:

     c:\Documents and Settings\jessek\Desktop\md5deep.exe c:\Windows\*

Note that you can drag the md5deep icon into this window and the operating system will fill in the path information for you. You can also install the programs in a directory that is included in your `PATH` environment variable.

### Computing hashes

By default, md5deep generates MD5 hashes for each input file, displaying the full path information for each file:

    $ md5deep config.h INSTALL README
    b08b18e0a3d2440feb0b321ea8080b36  /home/jessek/coreutils-5.2.0/config.h
    9f3e20fdff9c78aa8e3f9b42be166769  /home/jessek/coreutils-5.2.0/INSTALL
    4aa18483f73fc56c907f0ae9025e2582  /home/jessek/coreutils-5.2.0/README

If no input files are specified, standard input is hashed. You can either pipe the output of other programs into md5deep or type manually at the command line. To end input from the command line, most shells use Control-D, except for Microsoft Windows, which uses Control-Z.

    $ uname -a | md5deep
    f54c5e12c13791b67d299221424a5d80

    $ md5deep
    This is a test
    [Enter and Control-D hit]
    ff22941336956098ae9a564289d1bf1b

You can also have md5deep print relative filenames instead of absolute ones. That is, omit all of the path information except that specified on the command line. To enable relative paths, use the `-l` flag. Repeating our first example with the `-l` flag:

    $ md5deep -l ../*
    b08b18e0a3d2440feb0b321ea8080b36  ../config.h
    9f3e20fdff9c78aa8e3f9b42be166769  ../INSTALL
    4aa18483f73fc56c907f0ae9025e2582  ../README

You can have md5deep only print out the basename of each file it processes. That is, all directory information will be stripped off. To enable basename mode, use the `-b` flag:

    $ md5deep -b config.h INSTALL README
    b08b18e0a3d2440feb0b321ea8080b36  config.h
    9f3e20fdff9c78aa8e3f9b42be166769  INSTALL
    4aa18483f73fc56c907f0ae9025e2582  README

Finally, if you need md5deep to insert an asterisk before the filename, similar to how md5sum does when you use the -b flag with that program, you can use the -k flag on md5deep.

    $ md5deep -k config.h INSTALL README
    b08b18e0a3d2440feb0b321ea8080b36 */home/jessek/coreutils-5.2.0/config.h
    9f3e20fdff9c78aa8e3f9b42be166769 */home/jessek/coreutils-5.2.0/INSTALL
    4aa18483f73fc56c907f0ae9025e2582 */home/jessek/coreutils-5.2.0/README

## Error messages

If an input file can't be found, an error message is normally printed. These, and all other error messages, can be surpressed by using the `-s` flag.

    $ md5deep doesnotexist.txt
    md5deep: /home/jessek/doesnotexist.txt: No such file or directory

    $ md5deep -s doesnotexist.txt
    $

## Recursive Mode

Normally, attempting to process a directory will generate an error message. Under recursive mode, md5deep will hash files in the current directory and file in subdirectories. Recursive mode is activated by using the `-r` flag.

    $ md5deep *
    md5deep: /home/jessek/archives: Is a directory
    md5deep: /home/jessek/bin: Is a directory

    $ md5deep -r *
    ea024eaf04ee7a2a4270655d584445d2  /home/jessek/archives/coreutils-5.2.0.tar.bz2
    a07715c3344524da1270e9eb39f9b9e1  /home/jessek/archives/md5deep-0.16.tar.gz
    ef62d2a26c266df6151ae35447bfd2ad  /home/jessek/archives/dcfldd-1.0.tar.gz
    fb3dab239dd0baa0f4a1ccb10c3a5b0a  /home/jessek/bin/hex2dec

## Time Estimation Mode

When processing large files, it is sometimes helpful to have an estimate of the time remaining in the operation. md5deep can generate an estimate of how long it will take to finish processing the current file. The `-e` flag prints this estimate to standard error, like this:

    $ md5deep -e /dev/hda1
    hda1: 1MB of 47MB done, 00:00:46 left

When the file is completed, the last time estimate is removed and the hash is displayed:

    $ md5deep -e /dev/hda1
    ca1b8297dbceaa14682d889483320a1a  /dev/hda1

## File Size Mode

md5deep can optionally display a ten digit representation of each file's size along with the hash. The file size comes before the hash on the line. This mode is activated with the -z flag, like this:

    $ md5deep -z *.h
          1666  3a5353527e28b1cb5b844d602094c25a  /home/jessek/md5deep/src/algorithms.h
          1232  a27d2799c36c7c64e370bf480dd463c0  /home/jessek/md5deep/src/hashTable.h
          1880  d8defb61898fe255c6d66c4d880a8536  /home/jessek/md5deep/src/md5.h

Note that ten spaces are used, even if the file size doesn't require that much space. This is done to make sorting easier.

If the file size is larger than 9,999,999,999 bytes (about 9.3 GB), the program will display the size as 9999999999.

## Matching mode

One of the more powerful features of md5deep is the ability to match the hashes of input files against a list of known hashes. You can do both postive matching, which displays those files that do match the list of known hashes, or negative matching, which displays those files that do not match the list of known hashes.

### Positive Matching

Let's say that we have a text file `known-hashes.txt` which contains a few hashes:

    65e7d67f9dbc831d4334f23c7fb9cfb1  foo.doc
    406f25e49e9e08bb859b574a96746177  bar.gif

You can use hash files generated by md5deep, md5sum, md5 (found on *BSD systems), Hashkeeper, iLook, and the National Software Reference Library. We can specify to use this file for positive matching by using the `-m` flag. Then, any input files that match either of these hashes will be displayed.

    $ md5deep -m known-hashes.txt *
    /home/jessek/tmp/a-matching-file.doc
    /home/jessek/tmp/some-other-file.gif

If you want to see the hashes along with the filenames, you can use the `-M` flag instead.

    $ md5deep -M known-hashes.txt *
    65e7d67f9dbc831d4334f23c7fb9cfb1 /home/jessek/tmp/a-matching-file.doc
    406f25e49e9e08bb859b574a96746177 /home/jessek/tmp/some-other-file.gif

If you need to match against a single hash, or would like to add a single hash to the set of known hashes, you can use the `-a` flag. This flag enables matching mode and add a single hash.

    $ md5deep -a 65e7d67f9dbc831d4334f23c7fb9cfb1 *
    /home/jessek/tmp/a-matching-file.doc

If you would like to see filename of the known file that generated the match, use the `-w` flag. Continuing our example:

    $ md5deep -wM known-hashes.txt *
    65e7d67f9dbc831d4334f23c7fb9cfb1 /home/jessek/tmp/a-matching-file.doc matched foo.doc
    406f25e49e9e08bb859b574a96746177 /home/jessek/tmp/some-other-file.gif matched bar.gif

### Negative Matching

Negative matching is the same as positive matching, above, but displays those files that are not in the list of known hashes. Negative matching can be enabled using the `-x` flag, or the `-X` flag if you want to see the hashes along with the filenames. The `-A` flag will add a single hash to the set of hashes.

## Advanced Matching Modes

### Which File Matched

First, the -w mode can be used to see which file an input matched. For example, let's say we have a file of known hashes, `known.txt` that we're matching against. Normally we see a list of input files that matched any known file:

    $ md5deep -m known.txt *
    /home/jessek/an-input-file.txt
    /home/jessek/another-matching-file.txt

When we add the -w mode to the command line, we see the filename in the list of known files that matched.

    $ md5deep -wm known.txt *
    /home/jessek/an-input-file.txt matched EVILEVIL.EXE
    /home/jessek/another-matching-file.txt matched san-dimas-high-school.jpg

The -w flag only works with positive matching, or -m mode. Attempting to use -w mode with negative matching produces no more information that negative matching does normally:

    $ md5deep -wx known.txt *
    /home/jessek/unknown-file does NOT match

### Unused Hashes

The second advanced matching mode, -n, indicates which known hashes were not matched by any of the input files. Using our example from above, let's say our file of known hashes, `known.txt` looked like this:

    3668422d5b728776cb8720de8496abc9  san-dimas-high-school.jpg
    b85444c8b82f208ad24e41f20c2b1831  EVILEVIL.EXE
    6e43e96bde5c9f8abdbc47661cf4975b  never-seen-again.dat

If we run md5deep with the -n flag, we will get a listing of any hashes that were not matched to an input file.

    $ md5deep -nm known.txt *
    never-seen-again.dat

## Expert Mode

md5deep's expert mode allows you to specify which and only which types of files are processed. The available file types are:

- Regular files, such as text, graphics, and executables.
- Block files, such as devices, hard drives, tape drives, CDROMs, etc.
- Character devices, such as /dev/tty.
- Named Pipes
- Symbolic Links - Soft links. Note that this option tells the program to follow a symbolic link provided that you have also specified the parameter for whatever the link is pointing to. That is, if the symlink points to a directory, you must have also specified -r. If the symbolic links points to a regular file, you must have also specified the -o f mode.
- Socket - Network connections
- Solaris Door - Available only on Solaris

To use expert mode, use the `-o` flag followed by the letter or letters corresponding to the types of files you want to process.

File type | Letter
Regular | f
Block | b
Character | c
Named Pipe | p
Symbolic Link | l
Socket | s
Solaris Door | d

Let's say that in the current directory there are files hda (a block device), my-link, a symbolic link to a block device, and data.txt, a regular file.

    $ md5deep -o f *
    2e1582d37db445402faee0fc331e9032  /home/jessek/data.txt
    $

Note that only the regular files are hashed. Conversely:

    $ md5deep -o lb *
    ee1615f01210aa22c27058db161d2160  /home/jessek/hda
    7fa749a2ea2cc635c77eb0a5370b93b0  /home/jessek/my-link
    $

Note that the recursive mode can be used in conjunction with the expert mode. Directories are ignored without the recursive flag.