<story><title>Pigz: Parallel gzip for modern multi-processor, multi-core machines</title><url>https://zlib.net/pigz/</url></story><parent_chain><item><author>adamgordonbell</author><text>One wild thing is how much performance wins were available compared to ZLib. Pigz is parrellel, but what if you just had a better way to compress and decompress than DEFLATE?&lt;p&gt;When zstd came out – and Brotli before it to a certain extent – they were 3x faster than ZLib with a slightly higher compression ratio. You&amp;#x27;d think that such performance jumps in something as well explored as data compression would be hard to come by. We weren&amp;#x27;t that close to the efficiency frontier.</text></item><item><author>adamgordonbell</author><text>I heard of pigz in the discussions following my interview of Yann Collet, creator of LZ4 and zstd.&lt;p&gt;If you&amp;#x27;ll excuse the plug, here is the LZ4 story:&lt;p&gt;Yann was bored and working as a project manager. So he started working on a game for his old HP 48 graphing calculator.&lt;p&gt;Eventually, this hobby led him to revolutionize the field of data compression, releasing LZ4, ZStandard, and Finite State Entropy coders.&lt;p&gt;His code ended up everywhere: in games, databases, file systems, and the Linux Kernel because Yann built the world&amp;#x27;s fastest compression algorithms. And he got started just making a fun game for a graphing calculator he&amp;#x27;d had since high school.&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;corecursive.com&amp;#x2F;data-compression-yann-collet&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;corecursive.com&amp;#x2F;data-compression-yann-collet&amp;#x2F;&lt;/a&gt;</text></item></parent_chain><comment><author>twotwotwo</author><text>A part of this was that zstd and Brotli were able to use compression history windows of MBs not KBs, while DEFLATE maxes out at 32KB. RAM was thousands of times more expensive in the early 90s, so a smaller history window made sense.&lt;p&gt;There are also optimizations that only work on today&amp;#x27;s larger cores, and you have to actively update old code to get the advantages (happily some work is going into that): &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=32533061&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=32533061&lt;/a&gt; &amp;#x2F; &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=32537545&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=32537545&lt;/a&gt;&lt;p&gt;That&amp;#x27;s not to minimize the clever ideas and amazing implementation work in new stuff. It&amp;#x27;s more that people were making smart decisions both then and now, more so than you might guess just from comparisons on today&amp;#x27;s hardware.</text></comment>
<story><title>Pigz: Parallel gzip for modern multi-processor, multi-core machines</title><url>https://zlib.net/pigz/</url></story><parent_chain><item><author>adamgordonbell</author><text>One wild thing is how much performance wins were available compared to ZLib. Pigz is parrellel, but what if you just had a better way to compress and decompress than DEFLATE?&lt;p&gt;When zstd came out – and Brotli before it to a certain extent – they were 3x faster than ZLib with a slightly higher compression ratio. You&amp;#x27;d think that such performance jumps in something as well explored as data compression would be hard to come by. We weren&amp;#x27;t that close to the efficiency frontier.</text></item><item><author>adamgordonbell</author><text>I heard of pigz in the discussions following my interview of Yann Collet, creator of LZ4 and zstd.&lt;p&gt;If you&amp;#x27;ll excuse the plug, here is the LZ4 story:&lt;p&gt;Yann was bored and working as a project manager. So he started working on a game for his old HP 48 graphing calculator.&lt;p&gt;Eventually, this hobby led him to revolutionize the field of data compression, releasing LZ4, ZStandard, and Finite State Entropy coders.&lt;p&gt;His code ended up everywhere: in games, databases, file systems, and the Linux Kernel because Yann built the world&amp;#x27;s fastest compression algorithms. And he got started just making a fun game for a graphing calculator he&amp;#x27;d had since high school.&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;corecursive.com&amp;#x2F;data-compression-yann-collet&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;corecursive.com&amp;#x2F;data-compression-yann-collet&amp;#x2F;&lt;/a&gt;</text></item></parent_chain><comment><author>dist-epoch</author><text>&amp;gt; as well explored as data compression&lt;p&gt;What&amp;#x27;s well explored is compression rate, where indeed it&amp;#x27;s difficult to improve, and true innovations, like arithmetic coding, are rare.&lt;p&gt;Compressing speed on the other hand it&amp;#x27;s not very interesting to academics, it&amp;#x27;s more of an engineering problem. And there is plenty of work to do here, starting with stuff as simple as multi-threading and SIMD.&lt;p&gt;ZLib and ZStandard are probably in the same complexity class, but with different constant factors, which academics don&amp;#x27;t care about but which have massive practical consequences.</text></comment>