<story><title>RedPajama: Reproduction of LLaMA with friendly license</title><url>https://www.together.xyz/blog/redpajama</url></story><parent_chain><item><author>doctoboggan</author><text>I am a little concerned that they have only about 60% of the code tokens (GitHub and stackexchange). Given that so far the only concrete use case I have for LLMs is coding assistance I wouldn&amp;#x27;t want this open source model to be and less quality in that area.&lt;p&gt;In your opinion do you think this will hamper the model at all? Or is it still more than enough to get good coding assistance?</text></item><item><author>simonw</author><text>The training data - all 1.2 trillion tokens - can be downloaded by grabbing each of the 2,084 URLs listed here: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;data.together.xyz&amp;#x2F;redpajama-data-1T&amp;#x2F;v1.0.0&amp;#x2F;urls.txt&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;data.together.xyz&amp;#x2F;redpajama-data-1T&amp;#x2F;v1.0.0&amp;#x2F;urls.txt&lt;/a&gt;&lt;p&gt;I ran a HEAD request against them all to sum up the total file size, and it&amp;#x27;s 2.67TB total.&lt;p&gt;Here&amp;#x27;s a Datasette Lite URL that lets you explore the size metadata about those files: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;lite.datasette.io&amp;#x2F;?json=https:&amp;#x2F;&amp;#x2F;gist.github.com&amp;#x2F;simonw&amp;#x2F;73d15c0dd1025d1196829740bacf4464#&amp;#x2F;data&amp;#x2F;raw?_facet=top_folder&amp;amp;_facet=top_folders&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;lite.datasette.io&amp;#x2F;?json=https:&amp;#x2F;&amp;#x2F;gist.github.com&amp;#x2F;simo...&lt;/a&gt;&lt;p&gt;And a SQL query that shows the breakdown across the different sources:&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;lite.datasette.io&amp;#x2F;?json=https:&amp;#x2F;&amp;#x2F;gist.github.com&amp;#x2F;simonw&amp;#x2F;73d15c0dd1025d1196829740bacf4464#&amp;#x2F;data?sql=select+top_folder%2C+sum%28size_gb%29+from+raw+group+by+top_folder+order+by+sum%28size_gb%29+desc&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;lite.datasette.io&amp;#x2F;?json=https:&amp;#x2F;&amp;#x2F;gist.github.com&amp;#x2F;simo...&lt;/a&gt;&lt;p&gt;Sizes here are in GB:&lt;p&gt;&lt;pre&gt;&lt;code&gt;    common_crawl  1341.6166818914935
    c4  806.7667234372348
    github  212.1786002581939
    wikipedia  111.89125544670969
    book  100.43162744678557
    arxiv  87.35323827341199
    stackexchange  74.54870238155127
&lt;/code&gt;&lt;/pre&gt;
Common Crawl is in there a few times - they have the following folders:&lt;p&gt;&lt;pre&gt;&lt;code&gt;    common_crawl&amp;#x2F;2020-05 198 files
    common_crawl&amp;#x2F;2021-04 176 files
    common_crawl&amp;#x2F;2023-06 175 files
    common_crawl&amp;#x2F;2022-05 157 files
    common_crawl&amp;#x2F;2019-30 153 files
&lt;/code&gt;&lt;/pre&gt;
And then C4 as well, which is &amp;quot;a colossal, cleaned version of Common Crawl&amp;#x27;s web crawl corpus. It was based on Common Crawl dataset&amp;quot;: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;paperswithcode.com&amp;#x2F;dataset&amp;#x2F;c4&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;paperswithcode.com&amp;#x2F;dataset&amp;#x2F;c4&lt;/a&gt;</text></item></parent_chain><comment><author>csris</author><text>Nice catch! We sampled the github dataset to match the total # tokens seen by LLaMA during training: ~64B tokens (they only pass through 0.64 of their total Github dataset according to the paper). We have a lot of Github data and will make them available soon. Note, we also have not built this for compute optimal training. We are following LLaMA&amp;#x27;s lead and are training on more data for longer to optimize for quality, not compute.</text></comment>
<story><title>RedPajama: Reproduction of LLaMA with friendly license</title><url>https://www.together.xyz/blog/redpajama</url></story><parent_chain><item><author>doctoboggan</author><text>I am a little concerned that they have only about 60% of the code tokens (GitHub and stackexchange). Given that so far the only concrete use case I have for LLMs is coding assistance I wouldn&amp;#x27;t want this open source model to be and less quality in that area.&lt;p&gt;In your opinion do you think this will hamper the model at all? Or is it still more than enough to get good coding assistance?</text></item><item><author>simonw</author><text>The training data - all 1.2 trillion tokens - can be downloaded by grabbing each of the 2,084 URLs listed here: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;data.together.xyz&amp;#x2F;redpajama-data-1T&amp;#x2F;v1.0.0&amp;#x2F;urls.txt&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;data.together.xyz&amp;#x2F;redpajama-data-1T&amp;#x2F;v1.0.0&amp;#x2F;urls.txt&lt;/a&gt;&lt;p&gt;I ran a HEAD request against them all to sum up the total file size, and it&amp;#x27;s 2.67TB total.&lt;p&gt;Here&amp;#x27;s a Datasette Lite URL that lets you explore the size metadata about those files: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;lite.datasette.io&amp;#x2F;?json=https:&amp;#x2F;&amp;#x2F;gist.github.com&amp;#x2F;simonw&amp;#x2F;73d15c0dd1025d1196829740bacf4464#&amp;#x2F;data&amp;#x2F;raw?_facet=top_folder&amp;amp;_facet=top_folders&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;lite.datasette.io&amp;#x2F;?json=https:&amp;#x2F;&amp;#x2F;gist.github.com&amp;#x2F;simo...&lt;/a&gt;&lt;p&gt;And a SQL query that shows the breakdown across the different sources:&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;lite.datasette.io&amp;#x2F;?json=https:&amp;#x2F;&amp;#x2F;gist.github.com&amp;#x2F;simonw&amp;#x2F;73d15c0dd1025d1196829740bacf4464#&amp;#x2F;data?sql=select+top_folder%2C+sum%28size_gb%29+from+raw+group+by+top_folder+order+by+sum%28size_gb%29+desc&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;lite.datasette.io&amp;#x2F;?json=https:&amp;#x2F;&amp;#x2F;gist.github.com&amp;#x2F;simo...&lt;/a&gt;&lt;p&gt;Sizes here are in GB:&lt;p&gt;&lt;pre&gt;&lt;code&gt;    common_crawl  1341.6166818914935
    c4  806.7667234372348
    github  212.1786002581939
    wikipedia  111.89125544670969
    book  100.43162744678557
    arxiv  87.35323827341199
    stackexchange  74.54870238155127
&lt;/code&gt;&lt;/pre&gt;
Common Crawl is in there a few times - they have the following folders:&lt;p&gt;&lt;pre&gt;&lt;code&gt;    common_crawl&amp;#x2F;2020-05 198 files
    common_crawl&amp;#x2F;2021-04 176 files
    common_crawl&amp;#x2F;2023-06 175 files
    common_crawl&amp;#x2F;2022-05 157 files
    common_crawl&amp;#x2F;2019-30 153 files
&lt;/code&gt;&lt;/pre&gt;
And then C4 as well, which is &amp;quot;a colossal, cleaned version of Common Crawl&amp;#x27;s web crawl corpus. It was based on Common Crawl dataset&amp;quot;: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;paperswithcode.com&amp;#x2F;dataset&amp;#x2F;c4&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;paperswithcode.com&amp;#x2F;dataset&amp;#x2F;c4&lt;/a&gt;</text></item></parent_chain><comment><author>bkm</author><text>Relevant: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;twitter.com&amp;#x2F;abacaj&amp;#x2F;status&amp;#x2F;1647999551964323844&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;twitter.com&amp;#x2F;abacaj&amp;#x2F;status&amp;#x2F;1647999551964323844&lt;/a&gt;</text></comment>