Ralph Peeters, Reng Chiz Der, Christian Bizer &ndash; University of Mannheim, Germany
WDC Products: A Multi-Dimensional Entity Matching Benchmark
The first step of the pipeline is the extraction of large amounts of product offers from the Common Crawl⁴ [⁴https://commoncrawl.org/] using schema.org annotations. Some product offers contain product identifiers like MPNs and GTINs which allow us to group offers into [...] The Web Data Commons6 project regularly extracts schema.org annotations from the Common Crawl, the largest web corpus available to the public, in order to monitor the adoption of semantic annotations on the Web and to provide the extracted data for public download. The WDC Products benchmark uses product offers from the WDC Product Data Corpus V2020 (PDC2020)7. The corpus was created by extracting schema.org product data from the September 2020 version of the Common Crawl. The extracted data goes through a pipeline of cleansing steps such as removing offers from listing pages as well as advertisements that are contained in a page in addition to the main offer [31]. The resulting PDC2020 corpus consists of ∼98 million product offers originating from 603,000 websites.