<story><title>How are zlib, gzip and Zip related? (2013)</title><url>http://stackoverflow.com/questions/20762094/how-are-zlib-gzip-and-zip-related-what-do-they-have-in-common-and-how-are-they/20765054#20765054</url></story><parent_chain><item><author>lpage</author><text>In other compression news, Apple open sourced their implementation of lzfse yesterday: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;lzfse&amp;#x2F;lzfse&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;lzfse&amp;#x2F;lzfse&lt;/a&gt;. It&amp;#x27;s based on a relatively new type of coding - asymmetric numeral systems. Huffman coding is only optimal if you consider one bit as the smallest unit of information. ANS (and more broadly, arithmetic coding) allows for fractional bits and gets closer to the Shannon limit. It&amp;#x27;s also simpler to implement than (real world) Huffman.&lt;p&gt;Unfortunately, most open source implementations of ANS are not highly optimized and quite division heavy, so they lag on speed benchmarks. Apple&amp;#x27;s implementation looks pretty good (they&amp;#x27;re using it in OS X, err, macOS, and iOS) and there&amp;#x27;s some promising academic work being done on better implementations (optimizing Huffman for x86, ARM, and FPGA is a pretty well studied problem). The compression story is still being written.</text></item></parent_chain><comment><author>twitch_checksum</author><text>History rewritten once again...&lt;p&gt;Apple&amp;#x27;s fse implementation is based on an open-source work from an individual which predates lzfse by about ~20 months.&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;Cyan4973&amp;#x2F;FiniteStateEntropy&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;Cyan4973&amp;#x2F;FiniteStateEntropy&lt;/a&gt;&lt;p&gt;It&amp;#x27;s even less optimized than the original one, due to a few errors that slipped through during lzfse conception.&lt;p&gt;&lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;encode.ru&amp;#x2F;threads&amp;#x2F;2221-LZFSE-New-Apple-Data-Compression?p=48491&amp;amp;viewfull=1#post48491&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;encode.ru&amp;#x2F;threads&amp;#x2F;2221-LZFSE-New-Apple-Data-Compressi...&lt;/a&gt;</text></comment>
<story><title>How are zlib, gzip and Zip related? (2013)</title><url>http://stackoverflow.com/questions/20762094/how-are-zlib-gzip-and-zip-related-what-do-they-have-in-common-and-how-are-they/20765054#20765054</url></story><parent_chain><item><author>lpage</author><text>In other compression news, Apple open sourced their implementation of lzfse yesterday: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;lzfse&amp;#x2F;lzfse&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;lzfse&amp;#x2F;lzfse&lt;/a&gt;. It&amp;#x27;s based on a relatively new type of coding - asymmetric numeral systems. Huffman coding is only optimal if you consider one bit as the smallest unit of information. ANS (and more broadly, arithmetic coding) allows for fractional bits and gets closer to the Shannon limit. It&amp;#x27;s also simpler to implement than (real world) Huffman.&lt;p&gt;Unfortunately, most open source implementations of ANS are not highly optimized and quite division heavy, so they lag on speed benchmarks. Apple&amp;#x27;s implementation looks pretty good (they&amp;#x27;re using it in OS X, err, macOS, and iOS) and there&amp;#x27;s some promising academic work being done on better implementations (optimizing Huffman for x86, ARM, and FPGA is a pretty well studied problem). The compression story is still being written.</text></item></parent_chain><comment><author>legulere</author><text>I followed your link which leads me to rant a bit:&lt;p&gt;LZFSE has been out since one year. Not one mention on wikipedia. The repo lacks a good description what lzfse is. It also contains a LZVN encoder&amp;#x2F;decoder. The only information I found about it is some blog where someone seems to reverse engineer it for some hackintosh purposes.&lt;p&gt;I know documenting and presenting the case why people should use your software&amp;#x2F;file format can be annoying to do, but it&amp;#x27;s really important.</text></comment>