Can tarring a bunch of files together improve compression with the standard tools, e.g. gzip, bzip2, xz?I've long thought this to be the case but never tested it out. If we have 2 copies of the same 20Mb file of random bytes tarred together, a clever compression program that realizes this could compress the entire tarball down to almost 20Mb.I just tried this experiment using gzip, bzip2, and xz to compress 1) a file of random bytes, 2) a tarball of two copies of that file, and 3) a cat of two copies of that file. In all cases the compression did not reduce the file size. This is expected for case 1 but for cases 2 and 3 the optimal result is that a 40Mb file can be shrunk to nearly 20Mb. That's a difficult insight for a compression program to see, especially because the redundancy is distant, so I wouldn't expect a perfect result but I still had figured there would be some compression.Test:dd if=/dev/urandom of=random1.txt bs=1M count=20cp random1.txt random2.txtcat random1.txt random2.txt > random_cat.txttar -cf randoms.tar random1.txt random2.txtgzip -k random* &bzip2 -k random* &xz -k random* &waitdu -sh random*Result:20+0 records in20+0 records out20971520 bytes (21 MB) copied, 1.40937 s, 14.9 MB/s[1]   Done                    gzip -k random*[2]-  Done                    bzip2 -k random*[3]+  Done                    xz -k random*20M random1.txt21M random1.txt.bz221M random1.txt.gz21M random1.txt.xz20M random2.txt21M random2.txt.bz221M random2.txt.gz21M random2.txt.xz40M random_cat.txt41M random_cat.txt.bz241M random_cat.txt.gz41M random_cat.txt.xz41M randoms.tar41M randoms.tar.bz241M randoms.tar.gz41M randoms.tar.xzIs this generally what I should expect?Is there a way to improve compression here?
Can tarring files improve compression?
tar;compression
You're up against the block size of the compressor.  Most compression programs break the input into blocks and compress each block.  It appears the bzip block size only goes up to 900K, so it won't see any pattern that takes longer than 900K bytes to repeat.http://www.bzip.org/1.0.3/html/memory-management.htmlgzip appears to use 32K blocks.With xz you're in luck though!  From the man page:   Preset   DictSize   CompCPU   CompMem   DecMem     -0     256 KiB       0        3 MiB    1 MiB     -1       1 MiB       1        9 MiB    2 MiB     -2       2 MiB       2       17 MiB    3 MiB     -3       4 MiB       3       32 MiB    5 MiB     -4       4 MiB       4       48 MiB    5 MiB     -5       8 MiB       5       94 MiB    9 MiB     -6       8 MiB       6       94 MiB    9 MiB     -7      16 MiB       6      186 MiB   17 MiB     -8      32 MiB       6      370 MiB   33 MiB     -9      64 MiB       6      674 MiB   65 MiBso xz -8 will find up to 32MB patterns, and xz -9 up to 64MB patterns.  But beware how much ram it is requiring to perform the compression (and to decompress)...