<commit_before>Please refer to the [License file](/LICENSE.txt) for the licensing of the project.

**Project** - DotaWp.com Wallpaper Crawler  
**Creator** - <NAME> aka !mmorta!  
**Created** - 19th of February, 2014  


###Project Summary

* Using Scrapy's CrawlSpider, we crawl the [DotaWP](http://www.dotawp.com) website.
* This fetches all images in the [images/full](/DotaWp/images/full) directory.
* The [rename_files.py](/DotaWp/rename_files.py) then renames all images with their corresponding title.


###Instructions for Crawling

* Either [download](https://github.com/ashishnitinpatil/dotawp_crawler/archive/master.zip) [this repository]() or clone it on your machine  
    `git clone https://github.com/ashishnitinpatil/dotawp_crawler.git <local_dir>`
* Install python 2.7 or later on your machine  
    Refer to the official [Beginner's guide](https://wiki.python.org/moin/BeginnersGuide/Download) if you don't know how.
* Install scrapy on your machine  
    Refer to the [scrapy docs](http://scrapy.readthedocs.org/en/latest/intro/install.html) for that.
* Now `cd` to your downloaded DotaWP project directory.  
    `cd <path_to_dotawp_dir>/DotaWp`
* Execute the crawl  
    `python -m scrapy crawl dotawp`
* Let the crawling finish. (time taken depends on number of images, project settings)
* Rename the files by running the [rename_files](/DotaWp/rename_files.py) script.  
    You can double click the file or run the below command.  
    `python rename_files.py`

That's it! Now you have all the DotaWp wallpapers in the images directory!<commit_msg>++Progress viewing note<commit_after>Please refer to the [License file](/LICENSE.txt) for the licensing of the project.

**Project** - DotaWp.com Wallpaper Crawler  
**Creator** - <NAME> aka !mmorta!  
**Created** - 19th of February, 2014  


###Project Summary

* Using Scrapy's CrawlSpider, we crawl the [DotaWP](http://www.dotawp.com) website.
* This fetches all images in the [images/full](/DotaWp/images/full) directory.
* The [rename_files.py](/DotaWp/rename_files.py) then renames all images with their corresponding title.


###Instructions for Crawling

* Either [download](https://github.com/ashishnitinpatil/dotawp_crawler/archive/master.zip) [this repository]() or clone it on your machine  
    `git clone https://github.com/ashishnitinpatil/dotawp_crawler.git <local_dir>`
* Install python 2.7 or later on your machine  
    Refer to the official [Beginner's guide](https://wiki.python.org/moin/BeginnersGuide/Download) if you don't know how.
* Install scrapy on your machine  
    Refer to the [scrapy docs](http://scrapy.readthedocs.org/en/latest/intro/install.html) for that.
* Now `cd` to your downloaded DotaWP project directory.  
    `cd <path_to_dotawp_dir>/DotaWp`
* Execute the crawl  
    `python -m scrapy crawl dotawp`
* Let the crawling finish. (time taken depends on number of images, project settings)  
    You can see the progress in the [logs](/DotaWp/logs.txt)
* Rename the files by running the [rename_files](/DotaWp/rename_files.py) script.  
    You can double click the file or run the below command.  
    `python rename_files.py`

That's it! Now you have all the DotaWp wallpapers in the images directory!
{'fraction_non_alphanumeric': 0.10599937830276655, 'fraction_numerical': 0.004973577867578489, 'mean_word_length': 4.207119741100324, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3238531', 'n_tokens_mistral': 1083, 'n_tokens_neox': 1032, 'n_words': 341}