News
The original one billion word language model benchmark data [1]}, produced from News Crawl data. Popular reporting usually involves one or several writers, and a large number of readers. The text is usually edited several times for easy understanding.
Books
A corpus of 11,038 books in 16 different genres, e.g., Romance, Fantasy, Science fiction, etc., collected by . There books are all free books written by yet unpublished authors. Fiction books usually involves one writer, and a moderate size of readers. The text is usually edited many times, reflecting writer personality.
MIMIC
A clinical database comprising over 58,000 hospital admissions for intensive care unit (ICU) patients [2]}. Clinical notes are usually written by doctors and nurses under time pressure, and read by their colleagues. The text is seldom carefully edited, so it is syntactically noisy and usually contains a lot of jargon for efficient communication.
PubMed
Titles and abstracts of biomedical scholar articles. Scholar articles are usually written by a small groups of writers, and the readers usually have similar knowledge background with authors. The text is usually edited many times for more comprehensible and less ambiguous.
Yelp
Crowd-sourced reviews about local businesses, including restaurants, hotels, barbers, mechanics, etc. Online review is usually written by a single customer, and read by a small group of people who are interested in the business. The text is usually edited once, and writer tends to use descriptive language to share their experiences.
Wikipedia
A free online encyclopedia. Online encyclopedia is created and edited by volunteers around the world, and it has around 250 million page views every day Siteviews Analysis: shorturl.at/ayQR5. Accessed data: 2021-Jan-31.. Another important feature of Wikipedia as an open-collaborative website is that articles are edited all the time by human users and bots, reflecting the newest development of the world knowledge.

