<story><title>The Pile is a 825 GiB diverse, open-source language modelling data set (2020)</title><url>https://pile.eleuther.ai/</url></story><parent_chain><item><author>Ninjinka</author><text>I raised a concern about the inclusion of books3 in the Pile back in 2020, and this is what the head of Eleuther (Stella Biderman) told me:&lt;p&gt;&amp;quot;So here’s the big picture. There are three sets of datasets:
1. Data exists out there in the world. It has been collected into datasets and posted online. I’ll call this raw data.
2. We take that data, clean it, and process it for language modeling.  I’ll call this per-set data.
3. We combine those per-set data into one massive dataset, the Pile. This is heavily processed, including weighing the components.&lt;p&gt;We created 2 and 3 and put them online. We put 2 online so that people can reweigh and remix the data if they wish, but we expect most people to just download 3 and use it out of the box. Access to 3 will be provided in several forms, including HuggingFace and from our website.&lt;p&gt;2 and 3 are not copyright violations, even if the data is copyrighted, because they fall under fair use (at least in the US).&lt;p&gt;The Pile contains code that turns 1 into 2 and code that turns 2 into 3.&lt;p&gt;When you download Maroon 5 from a website, you are creating a dataset corresponding to 2. That &lt;i&gt;can be&lt;/i&gt; copyright violation depending on what you do with it, but our use is not a copyright violation.&amp;quot;</text></item></parent_chain><comment><author>otterley</author><text>&amp;gt; 2 and 3 are not copyright violations, even if the data is copyrighted, because they fall under fair use (at least in the US).&lt;p&gt;This cannot be known until it is litigated. Fair Use is not something you can unilaterally declare and have it be so, just like you can&amp;#x27;t be like Michael Scott in the Office shouting &amp;quot;I declare bankruptcy!&amp;quot; OpenAI is currently defending itself against the New York Times for this very reason.&lt;p&gt;There&amp;#x27;s a multi-factor test that courts weigh the facts against in making a determination as to whether a &lt;i&gt;prima facie&lt;/i&gt; copyright violation would be protected under a Fair Use defense:&lt;p&gt;Factor 1: The Purpose and Character of the Use&lt;p&gt;Factor 2: The Nature of the Copyrighted Work&lt;p&gt;Factor 3: The Amount or Substantiality of the Portion Used&lt;p&gt;Factor 4: The Effect of the Use on the Potential Market for or Value of the Work&lt;p&gt;See &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;copyright.columbia.edu&amp;#x2F;basics&amp;#x2F;fair-use.html&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;copyright.columbia.edu&amp;#x2F;basics&amp;#x2F;fair-use.html&lt;/a&gt; for a pretty good overview of what the analysis entails.</text></comment>
<story><title>The Pile is a 825 GiB diverse, open-source language modelling data set (2020)</title><url>https://pile.eleuther.ai/</url></story><parent_chain><item><author>Ninjinka</author><text>I raised a concern about the inclusion of books3 in the Pile back in 2020, and this is what the head of Eleuther (Stella Biderman) told me:&lt;p&gt;&amp;quot;So here’s the big picture. There are three sets of datasets:
1. Data exists out there in the world. It has been collected into datasets and posted online. I’ll call this raw data.
2. We take that data, clean it, and process it for language modeling.  I’ll call this per-set data.
3. We combine those per-set data into one massive dataset, the Pile. This is heavily processed, including weighing the components.&lt;p&gt;We created 2 and 3 and put them online. We put 2 online so that people can reweigh and remix the data if they wish, but we expect most people to just download 3 and use it out of the box. Access to 3 will be provided in several forms, including HuggingFace and from our website.&lt;p&gt;2 and 3 are not copyright violations, even if the data is copyrighted, because they fall under fair use (at least in the US).&lt;p&gt;The Pile contains code that turns 1 into 2 and code that turns 2 into 3.&lt;p&gt;When you download Maroon 5 from a website, you are creating a dataset corresponding to 2. That &lt;i&gt;can be&lt;/i&gt; copyright violation depending on what you do with it, but our use is not a copyright violation.&amp;quot;</text></item></parent_chain><comment><author>dougb5</author><text>I don&amp;#x27;t know what the right answer is to the copyright questions, but I hope that in 2024 we&amp;#x27;ll have a better attitude about the human labor that went into these models than &amp;quot;Data exists out there in the world&amp;quot; and the passive-voice &amp;quot;It has been collected into datasets&amp;quot;</text></comment>