<story><title>FineWeb: Decanting the web for the finest text data at scale</title><url>https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1</url></story><parent_chain></parent_chain><comment><author>nyyp</author><text>I&amp;#x27;m always happy to see the proliferation of open-source resources for the next generative models. But I strongly suspect that OpenAI and friends are all using copywritten content from the wealth of shadow book repositories available online [1]. Unless open models are doing the same, I doubt they will ever get meaningfully close to the quality of closed-source models.&lt;p&gt;Related: I also suspect that this is one reason we get so little information about the exact data used to train Meta&amp;#x27;s Llama models (&amp;quot;open weights&amp;quot; vs &amp;quot;open source&amp;quot;).&lt;p&gt;[1]: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.annas-archive.org&amp;#x2F;llm&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.annas-archive.org&amp;#x2F;llm&lt;/a&gt;</text></comment>
<story><title>FineWeb: Decanting the web for the finest text data at scale</title><url>https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1</url></story><parent_chain></parent_chain><comment><author>nmstoker</author><text>Am continuing to reflect on this, so may have more to say later, but what struck me first was the commitment to openness with FineWeb, it is impressive and thorough (eg all the scripts they actually used are available and linked to, not just the finished data)</text></comment>