<story><title>Open source AI is the path forward</title><url>https://about.fb.com/news/2024/07/open-source-ai-is-the-path-forward/</url></story><parent_chain><item><author>stavros</author><text>&amp;quot;Open weights&amp;quot; means you can use the weights for free (as in beer). &amp;quot;Open source&amp;quot; means you get the training dataset and the methodology. ~Nobody does open source LLMs.</text></item><item><author>causal</author><text>&amp;quot;Open weights&amp;quot; is a more appropriate term but I&amp;#x27;ll point out that these weights are also largely inscrutable to the people with the code that trained it. And for licensing reasons, the datasets may not be possible to share.&lt;p&gt;There is still a lot of modifying you can do with a set of weights, and they make great foundations for new stuff, but yeah we may never see a competitive model that&amp;#x27;s 100% buildable at home.&lt;p&gt;Edit: mkolodny points out that the model code is shared (under llama license at least), which is really all you need to run training &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;meta-llama&amp;#x2F;llama3&amp;#x2F;blob&amp;#x2F;main&amp;#x2F;llama&amp;#x2F;model.py&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;meta-llama&amp;#x2F;llama3&amp;#x2F;blob&amp;#x2F;main&amp;#x2F;llama&amp;#x2F;model.p...&lt;/a&gt;</text></item><item><author>the8thbit</author><text>&amp;quot;Eventually though, open source Linux gained popularity – initially because it allowed developers to modify its code however they wanted ...&amp;quot;&lt;p&gt;I find the language around &amp;quot;open source AI&amp;quot; to be confusing. With &amp;quot;open source&amp;quot; there&amp;#x27;s usually &amp;quot;source&amp;quot; to open, right? As in, there is human legible code that can be read and modified by the user? If so, then how can current ML models be open source? They&amp;#x27;re very large matrices that are, for the most part, inscrutable to the user. They seem akin to binaries, which, yes, can be modified by the user, but are extremely obscured to the user, and require enormous effort to understand and effectively modify.&lt;p&gt;&amp;quot;Open source&amp;quot; code is not just code that isn&amp;#x27;t executed remotely over an API, and it seems like maybe its being conflated with that here?</text></item></parent_chain><comment><author>larodi</author><text>Indeed, since when the deliverable being a jpeg&amp;#x2F;exe, which is similar to what the model file is, is considered the source? it is more like open result or freely available vm image, which works, but has its core FS scrambled or crypted.&lt;p&gt;Zuck knows this very well and it does him no honour to speak like, and from his position this equals attempt ate trying to change the present semantics of open source. Of course, others do that too - using the notion of open source to describe something very far from open.&lt;p&gt;What Meta is doing under his command can better be desdribed as releasing the resulting...build, so that it can be freely poked around and even put to work. But the result cannot be effectively reversed engineered.&lt;p&gt;Whats more ridiculous is that precisely because the result is not the source in its whole form, that these graphical structures can made available. Only thanks to the fact it is not traceable to the source, which makes the whole game not only closed, but like... sealed forever. An unfair retell of humanity&amp;#x27;s knowledge tossed around in very obscure container that nobody can reverse engineer.&lt;p&gt;how&amp;#x27;s that even remotely similar to open source?</text></comment>
<story><title>Open source AI is the path forward</title><url>https://about.fb.com/news/2024/07/open-source-ai-is-the-path-forward/</url></story><parent_chain><item><author>stavros</author><text>&amp;quot;Open weights&amp;quot; means you can use the weights for free (as in beer). &amp;quot;Open source&amp;quot; means you get the training dataset and the methodology. ~Nobody does open source LLMs.</text></item><item><author>causal</author><text>&amp;quot;Open weights&amp;quot; is a more appropriate term but I&amp;#x27;ll point out that these weights are also largely inscrutable to the people with the code that trained it. And for licensing reasons, the datasets may not be possible to share.&lt;p&gt;There is still a lot of modifying you can do with a set of weights, and they make great foundations for new stuff, but yeah we may never see a competitive model that&amp;#x27;s 100% buildable at home.&lt;p&gt;Edit: mkolodny points out that the model code is shared (under llama license at least), which is really all you need to run training &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;meta-llama&amp;#x2F;llama3&amp;#x2F;blob&amp;#x2F;main&amp;#x2F;llama&amp;#x2F;model.py&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;meta-llama&amp;#x2F;llama3&amp;#x2F;blob&amp;#x2F;main&amp;#x2F;llama&amp;#x2F;model.p...&lt;/a&gt;</text></item><item><author>the8thbit</author><text>&amp;quot;Eventually though, open source Linux gained popularity – initially because it allowed developers to modify its code however they wanted ...&amp;quot;&lt;p&gt;I find the language around &amp;quot;open source AI&amp;quot; to be confusing. With &amp;quot;open source&amp;quot; there&amp;#x27;s usually &amp;quot;source&amp;quot; to open, right? As in, there is human legible code that can be read and modified by the user? If so, then how can current ML models be open source? They&amp;#x27;re very large matrices that are, for the most part, inscrutable to the user. They seem akin to binaries, which, yes, can be modified by the user, but are extremely obscured to the user, and require enormous effort to understand and effectively modify.&lt;p&gt;&amp;quot;Open source&amp;quot; code is not just code that isn&amp;#x27;t executed remotely over an API, and it seems like maybe its being conflated with that here?</text></item></parent_chain><comment><author>nine_k</author><text>Linux is open source and is mostly C code. You cannot run C code directly, you have to compile it and produce binaries. But it&amp;#x27;s the C code, not binary form, where the collaboration happens.&lt;p&gt;With LLMs, weights are the binary code: it&amp;#x27;s how you run the model. But to be able to train the model from scratch, or to collaborate on new approaches, you have to operate at a the level of architecture, methods, and training data sets. They are the source code.</text></comment>