Marius Løvold Jørgensen &ndash; UiT, The Arctic University of Norway, Norway
BacklinkDB: A Purpose-Built Backlink Database Management System
In order to compile a list of all the backlinks for a given webpage, we need knowledge about all the outgoing links on the web. Traversing the web and storing all the backlink data in a database allows us to efficiently retrieve the list of backlinks for a web page on demand. However, the web consists of billions of backlinks which translates to terabytes of data. As the web is continuously evolving, the database needs to be rebuilt periodically in order for it to closely resemble the current state of the web. This thesis presents BacklinkDB, a purpose-built database management system designed for managing a backlink database. Using a series of in-memory hash indices allows for high insert throughput when building the database. The backlink data for a given domain is stored together in sections throughout the database file. This allows for the requested backlink data to be easily located. With a simple sql-inspired query language, the users can both insert and retrieve backlink data. The evaluation shows that building a purpose-built database management sys- tem allows us to make the trade-offs between which performance metrics that is important. In this thesis, we will focus on creating a scalable backlink database management system with high insert performance
5.1.3 Data¶ The link data used in the experiments is downloaded from the Common Crawls website1. Common Crawl is a non-profit organization that periodically crawls the web and publicizes data. For the experiments described in this chapter, data from the August 2022 crawl² [²https://commoncrawl.org/2022/08/august-2022-crawl-archive-now-available/] is used.¶ Data prepossessing¶ Common Crawl provides data on all the indexable webpages. This data is provided in a series of warc files found in their public repository. Common Crawl also provide WAT files which are produced by processing the warc files and extracting the metadata for each webpage. The WAT files contain a list of all the outgoing links for each of the webpages.¶ All external links from the WAT file are extracted to their own link file so that they can be directly inserted into a database. Each link is stored on a separate line in the file using spaces to separate the source domain, source path, destination domain, and destination path. All the backlinks containing urls longer than 2048 characters are discarded. A link file is created for each of the WAT files. These link files contain all the information needed to build a backlink database.