Mikko Aulamo, Nikolay Bogoychev, Shaoxiong Ji, Graeme Nail, Gema Ramírez-Sánchez, Jörg Tiedemann, Jelmer Van Der Linde, Jaume Zaragoza &ndash; University of Helsinki, ; University of Edinburgh, United Kingdom; Prompsit Language Engineering
HPLT: High Performance Language Technologies
We describe the High Performance Language Technologies project (HPLT), a 3-year EU-funded project started in September 2022. HPLT will build a space combining petabytes of natural language data with large-scale model training. It will derive monolingual and bilingual datasets from the Internet Archive and CommonCrawl and build efficient and solid machine translation (MT) as well as large language models (LLMs). HPLT aims at providing free, sustainable and reusable datasets, models and workflows at scale using high-performance computing (HPC).
Datasets: Starting from 7 PB of web-crawled data from the Internet Archive3 and 5 from CommonCrawl,4 we will derive monolingual and bilin- gual datasets for systematic LLM and MT building with a large language coverage.