Michael Conover, Matthew Hayes, Scott Blackburn, Pete Skomoroch, Sam Shah &ndash; Workday, Inc., San Francisco, CA, USA
Pangloss: Fast Entity Linking in Noisy Text Environments
Workday, Inc., San Francisco, CA, USA
ir/information-extraction
The Common Crawl datasets represents a sample of web crawl data containing raw web page data, metadata and text extracts overseen by a 501(c)(3) nonprofit of the same name. Facilitating ease of access for industrial practitioners, the dataset is hosted for free on Amazon Web Services’ Public Data Set repository in addition to academic hosts the world over. As part of a batch Hadoop job run on a monthly basis we filter the Common Crawl data (∼70TB) down to records which contain at least one hyperlink that points to English Wikipedia. This corpus has proven particularly valuable as a source of signal for associating tokens with knowledge base entries in the context of domain-specific, messy natural language.