This paper presents twAwler, a lightweight twitter crawler that targets language-specific communities of users. twAwler takes advantage of multiple endpoints of the twitter API to explore user relations and quickly recognize users belonging to the targetted set. It performs a complete crawl for all users, discovering many standard user relations, including the retweet graph, mention graph, reply graph, quote graph, follow graph, etc. twAwler respects all twitter policies and rate limits, while able to monitor large communities of active users. twAwler was used between August 2016 and March 2018 to generate an extensive dataset of close to all Greek-speaking twitter accounts (about 330 thousand) and their tweets and relations. In total, the crawler has gathered 750 million tweets of which 424 million are in Greek; 750 million follow relations; information about 300 thousand lists, their members (119 million member relations) and subscribers (27 thousand subscription relations); 705 thousand trending topics; information on 52 million users in total of which 292 thousand have been since suspended, 141 thousand have deleted their account, and 3.5 million are protected and cannot be crawled. twAwler mines the collected tweets for the retweet, quote, reply, and mention graphs, which, in addition to the follow relation crawled, offer vast opportunities for analysis and further research.
TwitterEcho @cite_15 is a Twitter crawler focused on discovering and crawling small communities. The authors design a targeted crawler and use it to recognize and track Portuguese accounts. TwitterEcho also prioritizes account crawling by ordering users according to their activity patterns, opting to crawl active users more frequently. TwitterEcho is a distributed system requiring multiple clients to crawl and aggregating the results into a single server. can also be used in this fashion for large communities, but we found that for communities having on the order of 100,000 to 200,000 active users like the Greek-speaking twitter users, one computer suffices. TwitterEcho collected more than 14 million tweets from 100,000 users within 11 months. manages to use many more available endpoints from the twitter REST API, crawling, in addition to tweets and user information, the follow graph, the favorites of all tracked users, as well as list ownership, subscription, and membership information. uses the fact that the Greek alphabet suffices to recognize the language, while TwitterEcho resorts to a more complex language classifier to separate Portuguese from Brazilian.
@cite_15
2030559659
Modern social network analysis relies on vast quantities of data to infer new knowledge about human relations and communication. In this paper we describe TwitterEcho, an open source Twitter crawler for supporting this kind of research, which is characterized by a modular distributed architecture. Our crawler enables researchers to continuously collect data from particular user communities, while respecting Twitter's imposed limits. We present the core modules of the crawling server, some of which were specifically designed to focus the crawl on the Portuguese Twittosphere. Additional modules can be easily implemented, thus changing the focus to a different community. Our evaluation of the system shows high crawling performance and coverage.