Colin Lockard, Xin Luna Dong, Arash Einolghozati, Prashant Shiralkar &ndash; amazon.com
CERES: Distantly Supervised Relation Extraction from the Semi-Structured Web
amazon.com
ir/information-extraction, ir/relation-extraction
The CommonCrawl corpus consists of monthly snapshots of pages from millions of websites [1] on the Web. We started with a few well-known sites, including rottentomatoes.com, boxofficemojo.com, and themoviedb.org. Based on a Wikipedia list of the largest global film industries by admissions, box office, and number of productions⁸, we then issued Google searches for terms corresponding to these countries, such as “Nigerian film database” and recorded resulting sites that had detail pages related to movies. We also issued a few additional searches related to specific genres we thought may not be well-represented in mainstream sites, including “animated film database” and “documentary film database”. After compiling our list of sites, we then checked CommonCrawl⁹ and kept all sites with more than one hundred pages available. Our final list contains a broad mix of movie sites, including sites based around national film industries, genres, film music, and screen size. Most are in English, but the set also includes sites in Czech, Danish, Icelandic, Italian, Indonesian, and Slovak. ⁸https://en.wikipedia.org/wiki/Film_industry ⁹For each site, we scanned the CommonCrawl indices for all monthly scrapes prior to January 2018 and downloaded all pages for the site from the scrape with the largest number of unique webpages. Note that these scrapes do not necessarily obtain all pages present on a site, so the retrieved pages represent only a subset of the full site.