<story><title>Honey, I shrunk the NPM package</title><url>https://jamiemagee.co.uk/blog/honey-i-shrunk-the-npm-package/</url></story><parent_chain><item><author>bhouston</author><text>What to cut total NPM traffic by 50% or more overall?  Easy:&lt;p&gt;Create a shared Brotli dictionary (or zstandard or whatever) based on the top NPM packages by download bandwidth and then have all npm packages compressed using it.&lt;p&gt;I think this can be done server side by npmjs.org, where NPM packages are recompressed in this fashion after upload using the shared dictionary, thus it is an optional feature and fully backwards compatible.&lt;p&gt;Riffing on this idea because of this new chrome feature, which does this in a flexible fashion:
&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;chromestatus.com&amp;#x2F;feature&amp;#x2F;5124977788977152&quot; rel=&quot;nofollow noreferrer&quot;&gt;https:&amp;#x2F;&amp;#x2F;chromestatus.com&amp;#x2F;feature&amp;#x2F;5124977788977152&lt;/a&gt;&lt;p&gt;EDIT: This recompression of packages may be insecure as the digital signature of the package no longer aligns, but then the trick is to sign the contents of the package rather than package itself.</text></item></parent_chain><comment><author>shanemhansen</author><text>It would be a fun test to run. But I&amp;#x27;m not encouraged by the fact that the existing brotli dictionary already contains a bunch of javascript specific stuff:&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;gist.github.com&amp;#x2F;klauspost&amp;#x2F;2900d5ba6f9b65d69c8e&quot; rel=&quot;nofollow noreferrer&quot;&gt;https:&amp;#x2F;&amp;#x2F;gist.github.com&amp;#x2F;klauspost&amp;#x2F;2900d5ba6f9b65d69c8e&lt;/a&gt;&lt;p&gt;brotli literally already has a tokens for function&amp;#x2F;return&amp;#x2F;throw&amp;#x2F;indexOf(&amp;#x2F;.match&amp;#x2F;.length&amp;#x2F;etc.&lt;p&gt;Also verify after decompress is not without tradeoffs. On one hand we have folks like github who can&amp;#x27;t change the version of zlib because people rely on identical .tar.gz. &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=34586917&quot;&gt;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=34586917&lt;/a&gt;&lt;p&gt;On the other hand we have a whole lot of iffy stuff you can do to make programs decompressing content use large amounts of resources &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Zip_bomb&quot; rel=&quot;nofollow noreferrer&quot;&gt;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Zip_bomb&lt;/a&gt; which makes &amp;quot;decompress this potentially untrusted file so that I can validate it&amp;#x27;s safe to use&amp;quot; hard.</text></comment>
<story><title>Honey, I shrunk the NPM package</title><url>https://jamiemagee.co.uk/blog/honey-i-shrunk-the-npm-package/</url></story><parent_chain><item><author>bhouston</author><text>What to cut total NPM traffic by 50% or more overall?  Easy:&lt;p&gt;Create a shared Brotli dictionary (or zstandard or whatever) based on the top NPM packages by download bandwidth and then have all npm packages compressed using it.&lt;p&gt;I think this can be done server side by npmjs.org, where NPM packages are recompressed in this fashion after upload using the shared dictionary, thus it is an optional feature and fully backwards compatible.&lt;p&gt;Riffing on this idea because of this new chrome feature, which does this in a flexible fashion:
&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;chromestatus.com&amp;#x2F;feature&amp;#x2F;5124977788977152&quot; rel=&quot;nofollow noreferrer&quot;&gt;https:&amp;#x2F;&amp;#x2F;chromestatus.com&amp;#x2F;feature&amp;#x2F;5124977788977152&lt;/a&gt;&lt;p&gt;EDIT: This recompression of packages may be insecure as the digital signature of the package no longer aligns, but then the trick is to sign the contents of the package rather than package itself.</text></item></parent_chain><comment><author>IshKebab</author><text>50%? I doubt that, unless your dictionary is &lt;i&gt;literally&lt;/i&gt; the top NPM packages, in which case you&amp;#x27;re not saving much bandwidth by sending them all in advance...&lt;p&gt;Maybe a small dictionary would save you 20% though.</text></comment>