Wang Tongjing, Zhao Yin, Ziyu Bao, Evert Meijers &ndash; Utrecht University, The Netherlands; Delft University of Technology, The Netherlands
Although the importance of intercity relationships is theoretically acknowledged for cities’ socioeconomic development, the availability of such relational data often limits relevant urban studies. One of the new approaches of collecting city relational data is to extract the co-appearance of their place names from web texts. However, dealing with a gigantic web corpus is difficult for domain researchers given the complexities of processing terabytes of raw data. This paper develops an efficient and easy-to-follow method to extract a dataset of intercity relationships between 293 large Chinese cities applying the toponym co-occurrence method to a web archive. Our method successfully filters a 6.98 TB CC data set into a 202 GB single language text corpus. A highly-scalable Hadoop- based framework processes the full CC corpus utilizing a 1080 CPU cluster on the Amazon Elastic Map/Reduce infrastructure. To reveal more details of the intercity relationships, the intercity relationships are further classified into six categories: industry, information technology (IT), finance, research, culture, and government.
The data was retrieved from a Common Crawl raw corpus through a series of data processing. The web pages in this corpus that do not contain Chinese characteristics or Chinese placenames were filtered out based on keyword selection. The filtered Chinese corpus was 202 GB and the filtered Chinese corpus with placenames was about 139.5GB. Then we count the number of web pages where two city names co-appear. These intercity relationships were further classified into six categories using a lexicon-based classification method.