WIM format automatically detects duplicate files and archives without duplication.Is there any alternative on UNIX, Linux or Mac?
An effective archive method for many duplicate files
tar;archive
If the archive is small enough, most archive formats will do a decent job, with the exception of zip. Zip compresses each file independently, but other popular formats (tar.anything, 7z, rar) compress the archive as a whole. If identical files are close enough in the archive then the second occurrence can be compressed down to a few bytes. How close is close enough depends on the archive format.A method that works for any archiver that understands hard links, such as tar, is to first replace the identical files by hard links. This is only applicable if you don't want the files with duplicate contents to have different metadata (permissions, timestamps, etc.). You use fdupes to look for duplicates and then a bit of post-processing to replace duplicates by hard links, assuming that the file names don't contain newlines:fdupes -q -r . | awk '    $0 ==  {first = ; next}    {        gsub(/\047/, \047\\\047\047, $0);        if (first == ) first = $0;        else system(ln -f \047 first \047 \047 $0 \047);    }'