<story><title>Ask HN: Why does every package+module system become a Rube Goldberg machine?</title><text>A programming language has a &amp;quot;core language&amp;quot; plus a package&amp;#x2F;module system.
In each successful language, the core language is neat-and-tidy,
but the package&amp;#x2F;module system is a Rube Goldberg machine.
See JavaScript&amp;#x2F;TypeScript, Python, or C&amp;#x2F;C++.&lt;p&gt;Lots of brain cycles are spent on &amp;quot;programming language theory&amp;quot;.
We&amp;#x27;ve roughly figured out the primitives required to express real-world computation.&lt;p&gt;In contrast, we apparently have no &amp;quot;package management theory&amp;quot;.
We have not figured out the primitives required to express dependencies.
As a result, we keep building new variants and features,
until we end up with &amp;lt;script&amp;gt;, require(), import, npm, yarn, pnpm, (py)?(v|virtual|pip)?env, (ana)?conda, easy_install, eggs and wheels ...&lt;p&gt;Is it just a &amp;quot;law of software&amp;quot; that this must happen to any successful language?
Or are there examples of where this it has not happened, and what can we learn from them?
Is there a &amp;quot;theory of package management&amp;quot;, or a &amp;quot;lambda calculus of package management&amp;quot; out there?</text></story><parent_chain><item><author>tazjin</author><text>&amp;gt; In contrast, we apparently have no &amp;quot;package management theory&amp;quot;. We have not figured out the primitives required to express dependencies&lt;p&gt;We have a good hunch. The basic theory behind Nix definitely goes in the right direction, and if we look away from all the surface-level nonsense going on in Nix, it&amp;#x27;s conceptually capable (e.g. [0]) of being a first-class language dependency manager.&lt;p&gt;For this to work at scale we&amp;#x27;d need to overcome a couple of large problems though (in ascending order of complexity):&lt;p&gt;1. A better technical implementation of the model (working on it [1]).&lt;p&gt;2. A mindset shift to make people understand that &amp;quot;binary distribution&amp;quot; is not a goal, but a side-effect of a reasonable software addressing and caching model. Without this conceptual connection, everything is 10x harder (which is why e.g. Debian packaging is completely incomprehensible - their fundamental model is wrong).&lt;p&gt;3. A mindset shift to make people understand that their pet programming language is not actually a special snowflake. No matter what the size of your compilation units is, whether you call modules &amp;quot;modules&amp;quot;, &amp;quot;classes&amp;quot; or &amp;quot;gorboodles&amp;quot;, whether you allow odd features like mutually-recursive dependencies and build-time arbitrary code execution etc.: Your language fits into the same model as every other language. You don&amp;#x27;t have to NIH a package manager.&lt;p&gt;This last one is basically impossible at the current stage. Maybe somewhere down the line, if we manage to establish such a model successfully in a handful of languages and people see for themselves, but for now we have to just hold out.&lt;p&gt;[0]: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;code.tvl.fyi&amp;#x2F;about&amp;#x2F;nix&amp;#x2F;buildGo&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;code.tvl.fyi&amp;#x2F;about&amp;#x2F;nix&amp;#x2F;buildGo&lt;/a&gt;&lt;p&gt;[1]: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;cs.tvl.fyi&amp;#x2F;depot&amp;#x2F;-&amp;#x2F;tree&amp;#x2F;tvix&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;cs.tvl.fyi&amp;#x2F;depot&amp;#x2F;-&amp;#x2F;tree&amp;#x2F;tvix&amp;#x2F;&lt;/a&gt;</text></item></parent_chain><comment><author>jnxx</author><text>And Guix has put that in an beautiful form. There are two things which make Guix special:&lt;p&gt;1. The package definitions are just a normal, battle-proven, very well defined general-purpose, functional-style supporting programming language (Scheme).&lt;p&gt;2. There is no conceptual difference between a package definition in the public Guix system, and a self-written package definition which a developers makes to build and test his own package, or to build and run a specific piece of software. The difference is equally small as between using an Emacs package, and configuring that package in ones .emacs configuration file.</text></comment>
<story><title>Ask HN: Why does every package+module system become a Rube Goldberg machine?</title><text>A programming language has a &amp;quot;core language&amp;quot; plus a package&amp;#x2F;module system.
In each successful language, the core language is neat-and-tidy,
but the package&amp;#x2F;module system is a Rube Goldberg machine.
See JavaScript&amp;#x2F;TypeScript, Python, or C&amp;#x2F;C++.&lt;p&gt;Lots of brain cycles are spent on &amp;quot;programming language theory&amp;quot;.
We&amp;#x27;ve roughly figured out the primitives required to express real-world computation.&lt;p&gt;In contrast, we apparently have no &amp;quot;package management theory&amp;quot;.
We have not figured out the primitives required to express dependencies.
As a result, we keep building new variants and features,
until we end up with &amp;lt;script&amp;gt;, require(), import, npm, yarn, pnpm, (py)?(v|virtual|pip)?env, (ana)?conda, easy_install, eggs and wheels ...&lt;p&gt;Is it just a &amp;quot;law of software&amp;quot; that this must happen to any successful language?
Or are there examples of where this it has not happened, and what can we learn from them?
Is there a &amp;quot;theory of package management&amp;quot;, or a &amp;quot;lambda calculus of package management&amp;quot; out there?</text></story><parent_chain><item><author>tazjin</author><text>&amp;gt; In contrast, we apparently have no &amp;quot;package management theory&amp;quot;. We have not figured out the primitives required to express dependencies&lt;p&gt;We have a good hunch. The basic theory behind Nix definitely goes in the right direction, and if we look away from all the surface-level nonsense going on in Nix, it&amp;#x27;s conceptually capable (e.g. [0]) of being a first-class language dependency manager.&lt;p&gt;For this to work at scale we&amp;#x27;d need to overcome a couple of large problems though (in ascending order of complexity):&lt;p&gt;1. A better technical implementation of the model (working on it [1]).&lt;p&gt;2. A mindset shift to make people understand that &amp;quot;binary distribution&amp;quot; is not a goal, but a side-effect of a reasonable software addressing and caching model. Without this conceptual connection, everything is 10x harder (which is why e.g. Debian packaging is completely incomprehensible - their fundamental model is wrong).&lt;p&gt;3. A mindset shift to make people understand that their pet programming language is not actually a special snowflake. No matter what the size of your compilation units is, whether you call modules &amp;quot;modules&amp;quot;, &amp;quot;classes&amp;quot; or &amp;quot;gorboodles&amp;quot;, whether you allow odd features like mutually-recursive dependencies and build-time arbitrary code execution etc.: Your language fits into the same model as every other language. You don&amp;#x27;t have to NIH a package manager.&lt;p&gt;This last one is basically impossible at the current stage. Maybe somewhere down the line, if we manage to establish such a model successfully in a handful of languages and people see for themselves, but for now we have to just hold out.&lt;p&gt;[0]: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;code.tvl.fyi&amp;#x2F;about&amp;#x2F;nix&amp;#x2F;buildGo&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;code.tvl.fyi&amp;#x2F;about&amp;#x2F;nix&amp;#x2F;buildGo&lt;/a&gt;&lt;p&gt;[1]: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;cs.tvl.fyi&amp;#x2F;depot&amp;#x2F;-&amp;#x2F;tree&amp;#x2F;tvix&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;cs.tvl.fyi&amp;#x2F;depot&amp;#x2F;-&amp;#x2F;tree&amp;#x2F;tvix&amp;#x2F;&lt;/a&gt;</text></item></parent_chain><comment><author>noobermin</author><text>I think 3 is probably true for language developers, but for users, language repos feel like they are needed because like none of the linux distros really ship everything you need, or even a reliable fraction without installing a bunch of dumb hacks to get it to work. It&amp;#x27;s much easier just to do &amp;quot;pip install&amp;quot; so that&amp;#x27;s where the demand is.&lt;p&gt;And sure, may be you&amp;#x27;re right that distro packaging is the &amp;quot;wrong model,&amp;quot; again, that is the problem then for distros, users are stuck using ubuntu or whatever so they don&amp;#x27;t have the option to do the &amp;quot;right&amp;quot; thing, so they do use the mishmash of packaging&amp;#x2F;repo systems as just the cost of doing business.</text></comment>