Xiaojing Liao, Sumayah Alrwais, Kan Yuan, Luyi Xing, XiaoFeng Wang, Shuang Hao, Raheem Beyah &ndash; Indiana University Bloomington, USA; King Saud University, Saudi Arabia; University of Texas at Dallas, USA; Georgia Institute of Technology, USA
Cloud repository as a malicious service: challenge, identification and implication
Indiana University Bloomington, USA; King Saud University, Saudi Arabia; University of Texas at Dallas, USA; Georgia Institute of Technology, USA
computer-security/malicious-hosting-service, computer-security/internet-security
[...], we developed BarFinder, a scanner that automatically detects Bars through inspecting the topological relations between websites and the cloud bucket they use, in an attempt to capture Bars based on the external features of the websites they serve. [...] Running the scanner over all the data collected by the Common Crawl (Crawl 2015), which indexed five billion web pages, for those associated with all major cloud storage providers (including Amazon S3, Cloudfront, Google Drive, etc.), we found around 1 million sites utilizing 6885 repositories hosted on these clouds. [...] We built the site list with the help of Common Crawl (Crawl 2015), a public big data project that crawls about 5 billion webpages each month through a large-scale Hadoop-based crawler and maintains lists of the crawled websites and their embedded links. Searching the Common Crawl (Crawl 2015) dataset, collected in February 2015, for the websites loading content from the 400 clean and malicious buckets identified above, we found 141,149 websites, were used by our crawler. [...] We further developed a tool in Python to recover cloud URLs from the web content gathered by Common Crawl.