In our evaluation of data sources, we consider three dimensions: public availability, licensing, and scale. We use two sources of data for the financial corpus: the financial news common crawl and SEC filings. Financial News CommonCrawl is curated by filtering out financial news from the public CommonCrawl data. We follow the de-duplication procedure of Pythia suite [4] to remove duplicate training data. While there is conflicting evidence of duplication hurting the performance [4; 14], there is no evidence of the benefits of duplication in the training data. Hence, for a more efficient training, we use de-duplication following [4]. Using these two sources, we create a combined dataset of 23.9 billion tokens (16.5 billion words). Details of curation steps can be found in Appendix E.