<story><title>FineWeb: Decanting the web for the finest text data at scale</title><url>https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1</url></story><parent_chain></parent_chain><comment><author>nyyp</author><text>I&amp;#x27;m always happy to see the proliferation of open-source resources for the next generative models. But I strongly suspect that OpenAI and friends are all using copywritten content from the wealth of shadow book repositories available online [1]. Unless open models are doing the same, I doubt they will ever get meaningfully close to the quality of closed-source models.&lt;p&gt;Related: I also suspect that this is one reason we get so little information about the exact data used to train Meta&amp;#x27;s Llama models (&amp;quot;open weights&amp;quot; vs &amp;quot;open source&amp;quot;).&lt;p&gt;[1]: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.annas-archive.org&amp;#x2F;llm&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.annas-archive.org&amp;#x2F;llm&lt;/a&gt;</text></comment>
<story><title>FineWeb: Decanting the web for the finest text data at scale</title><url>https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1</url></story><parent_chain></parent_chain><comment><author>ipython</author><text>Reading the section on [synthetic data](&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;huggingfacefw-blogpost-fineweb-v1.static.hf.space&amp;#x2F;dist&amp;#x2F;index.html#synthetic_data&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;huggingfacefw-blogpost-fineweb-v1.static.hf.space&amp;#x2F;di...&lt;/a&gt;) was eye-opening for me. The hockey-stick growth of words associated with common ChatGPT output in the common crawl database over the past ~18 months is worrying.</text></comment>