Md5deep

Last updated
md5deep (and hashdeep)
Original author(s) Jesse Kornblum
Developer(s) Jesse Kornblum
Stable release
4.4 / January 29, 2014 (2014-01-29)
Repository
Written in C++
Operating system Cross-platform
License Public domain software (except for GPLv2-licensed Tiger hash function from Libgcrypt)
Website github.com/jessek/hashdeep/

md5deep is a software package used in the computer security, system administration and computer forensics communities to run large numbers of files through any of several different cryptographic digests. It was originally authored by Jesse Kornblum, at the time a special agent of the Air Force Office of Special Investigations. As of 2017, he still maintains it.

Contents

The name md5deep is misleading. Since version 2.0, the md5deep package contains several different programs able to perform MD5, SHA-1, SHA-256, Tiger192 and Whirlpool digests, each of them named by the digest type followed by the word "deep". Thus, the name may confuse some people into thinking it only provides the MD5 algorithm when the package supports many more.

md5deep can be invoked in several different ways. Typically users operate it recursively, where md5deep walks through one directory at a time giving digests of each file found, and recursing into any subdirectories within. Its recursive behavior is approximately a depth-first search, which has the benefit of presenting files in lexicographical order. On Unix-like systems, similar functionality can be often obtained by combining find with hashing utilities such as md5sum, sha256sum or tthsum.

md5deep exists for Windows and most Unix-based systems, including OS X. It is present in OS X's Fink , Homebrew and MacPorts projects. Binary packages exist for most free Unix systems. Many vendors initially resist including md5deep as they mistakenly[ citation needed ] believe its functions can be reproduced with one line of shell scripting. [1] The matching function of the program, however, cannot be done easily in shell.[ citation needed ]

Because md5deep was written by an employee of the U.S. government, on government time, it is in the public domain. Other software surrounding it, such as graphical front-ends, may be copyrighted.

See also

Related Research Articles

The MD5 message-digest algorithm is a cryptographically broken but still widely used hash function producing a 128-bit hash value. Although MD5 was initially designed to be used as a cryptographic hash function, it has been found to suffer from extensive vulnerabilities. It can still be used as a checksum to verify data integrity, but only against unintentional corruption. It remains suitable for other non-cryptographic purposes, for example for determining the partition for a particular key in a partitioned database.

In cryptography, SHA-1 is a cryptographic hash function which takes an input and produces a 160-bit (20-byte) hash value known as a message digest – typically rendered as a hexadecimal number, 40 digits long. It was designed by the United States National Security Agency, and is a U.S. Federal Information Processing Standard.

Shell script Script written for the shell, or command line interpreter, of an operating system

A shell script is a computer program designed to be run by the Unix shell, a command-line interpreter. The various dialects of shell scripts are considered to be scripting languages. Typical operations performed by shell scripts include file manipulation, program execution, and printing text. A script which sets up the environment, runs the program, and does any necessary cleanup, logging, etc. is called a wrapper.

Unix security refers to the means of securing a Unix or Unix-like operating system. A secure environment is achieved not only by the design concepts of these operating systems, but also through vigilant user and administrative practices.

Cryptographic hash function Hash function that is suitable for use in cryptography

A cryptographic hash function (CHF) is a mathematical algorithm that maps data of arbitrary size to a bit array of a fixed size. It is a one-way function, that is, a function which is practically infeasible to invert or reverse the computation. Ideally, the only way to find a message that produces a given hash is to attempt a brute-force search of possible inputs to see if they produce a match, or use a rainbow table of matched hashes. Cryptographic hash functions are a basic tool of modern cryptography.

In cryptanalysis and computer security, password cracking is the process of recovering passwords from data that has been stored in or transmitted by a computer system in scrambled form. A common approach is to repeatedly try guesses for the password and to check them against an available cryptographic hash of the password.

md5sum is a computer program that calculates and verifies 128-bit MD5 hashes, as described in RFC 1321. The MD5 hash functions as a compact digital fingerprint of a file. As with all such hashing algorithms, there is theoretically an unlimited number of files that will have any given MD5 hash. However, it is very unlikely that any two non-identical files in the real world will have the same MD5 hash, unless they have been specifically created to have the same hash.

passwd Tool to change passwords on Unix-like OSes

passwd is a command on Unix, Plan 9, Inferno, and most Unix-like operating systems used to change a user's password. The password entered by the user is run through a key derivation function to create a hashed version of the new password, which is saved. Only the hashed version is stored; the entered password is not saved for security reasons.

Simple file verification (SFV) is a file format for storing CRC32 checksums of files to verify the integrity of files. SFV is used to verify that a file has not been corrupted, but it does not otherwise verify the file's authenticity. The .sfv file extension is usually used for SFV files.

File verification is the process of using an algorithm for verifying the integrity of a computer file. This can be done by comparing two files bit-by-bit, but requires two copies of the same file, and may miss systematic corruptions which might occur to both files. A more popular approach is to generate a hash of the copied file and comparing that to the hash of the original file.

CRUX Linux distribution

CRUX is a lightweight, x86-64-optimized Linux distribution targeted at experienced Linux users and delivered by a tar.gz-based package system with BSD-style initscripts. It is not based on any other Linux distribution. It also utilizes a ports system to install and upgrade applications.

Digest access authentication is one of the agreed-upon methods a web server can use to negotiate credentials, such as username or password, with a user's web browser. This can be used to confirm the identity of a user before sending sensitive information, such as online banking transaction history. It applies a hash function to the username and password before sending them over the network. In contrast, basic access authentication uses the easily reversible Base64 encoding instead of hashing, making it non-secure unless used in conjunction with TLS.

SHA-2 is a set of cryptographic hash functions designed by the United States National Security Agency (NSA) and first published in 2001. They are built using the Merkle–Damgård construction, from a one-way compression function itself built using the Davies–Meyer structure from a specialized block cipher.

dar is a computer program, a command-line archiving tool intended as a replacement for tar in Unix-like operating systems.

In cryptography, key stretching techniques are used to make a possibly weak key, typically a password or passphrase, more secure against a brute-force attack by increasing the resources it takes to test each possible key. Passwords or passphrases created by humans are often short or predictable enough to allow password cracking, and key stretching is intended to make such attacks more difficult by complicating a basic step of trying a single password candidate. Key stretching also improves security in some real-world applications where the key length has been constrained, by mimicking a longer key length from the perspective of a brute-force attacker.

Metalink

Metalink is an extensible metadata file format that describes one or more computer files available for download. It specifies files appropriate for the user's language and operating system; facilitates file verification and recovery from data corruption; and lists alternate download sources.

sha1sum is a computer program that calculates and verifies SHA-1 hashes. It is commonly used to verify the integrity of files. It is installed by default on most Linux distributions. Typically distributed alongside sha1sum are sha224sum, sha256sum, sha384sum and sha512sum, which use a specific SHA-2 hash function.

SHA-3 is the latest member of the Secure Hash Algorithm family of standards, released by NIST on August 5, 2015. Although part of the same series of standards, SHA-3 is internally different from the MD5-like structure of SHA-1 and SHA-2.

Unix Family of computer operating systems that derive from the original AT&T Unix

Unix is a family of multitasking, multiuser computer operating systems that derive from the original AT&T Unix, whose development started in the 1970s at the Bell Labs research center by Ken Thompson, Dennis Ritchie, and others.

crypt is a POSIX C library function. It is typically used to compute the hash of user account passwords. The function outputs a text string which also encodes the salt, and identifies the hash algorithm used. This output string forms a password record, which is usually stored in a text file.

References

  1. #438753 – ITP: md5deep – Recursing file hash calculator – Debian Bug report logs