license:apache-2.0 region:us
---
license: apache-2.0
---
Cleaned dataset for experiments and fast pretokenizing. The SQL query to create the dataset is the following:
```python
SELECT
  f.repo_name,
  f.path,
  c.copies,
  c.size,
  c.content,
  l.license
FROM
 (select f.*, row_number() over (partition by id order by path desc) as seqnum
      from `bigquery-public-data.github_repos.files` AS f) f
JOIN
  `bigquery-public-data.github_repos.contents` AS c
ON
  f.id = c.id AND seqnum=1
JOIN
  `bigquery-public-data.github_repos.licenses` AS l
ON
  f.repo_name = l.repo_name
WHERE
  NOT c.binary
  AND ((f.path LIKE '%.vhdl'
      OR f.path LIKE '%.vhd'
    AND (c.size BETWEEN 0
      AND 1048575)))
```