Deduplication is a technique commonly used in the computer storage industry to reduce the storage footprint by removing common portions of files. Deduplication is typically performed by scanning entire files, creating chunks of smaller size within the scanned files, and thereafter creating a fingerprint for the byte stream in each of the chunks within a file. The fingerprint for each chuck of a file is then compared against a fingerprint database. The fingerprint database stores fingerprints for previously processed chunks of files.
If a particular fingerprint created for a chunk (“chunk X”) is not present in the fingerprint database, then that particular fingerprint is unique (i.e., it has not been previously encountered) and the fingerprint, and information about its association to chunk X, is stored in the fingerprint database. On the other hand, if the fingerprint is already present in the fingerprint database, then this means that chunk X is considered to be a duplicate of another chunk (“chunk Y”) already represented by another fingerprint stored in the fingerprint database; consequently, a pointer reference is saved that associates the previously stored fingerprint (which is also associated with chunk Y) to chunk X.