<commit_before>Rotto-Links-Scraper
=======================

A web crawler/scraper to find the broken links in the targeted seed url based on the keywords matched around that broken links.

##Installation
1. Redis
2. Python 2.7+

##Commands

To run the worker :

    $ python worker.py

To run Flask frontend and api :

    $ python run.py

##Developer
1. [<NAME>](https://www.facebook.com/AKSHAYPRATAP007)
2. [<NAME>](https://www.facebook.com/sunnyLA.Gupta)<commit_msg>edited REAME.md<commit_after>Rotto-Links-Scraper
=======================

A web crawler/scraper to find the broken links in the targeted seed url based on the keywords matched around that broken links.

##Installation
1. Redis
2. Python 2.7+

##Commands

To run the worker :

    $ python rottoscraper/worker.py

To run Flask frontend and api :

    $ python rottoscraper/run.py

##Developer
1. [<NAME>](https://www.facebook.com/AKSHAYPRATAP007)
2. [<NAME>](https://www.facebook.com/sunnyLA.Gupta)
{'fraction_non_alphanumeric': 0.15424430641821946, 'fraction_numerical': 0.018633540372670808, 'mean_word_length': 4.171122994652406, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 4, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6550898', 'n_tokens_mistral': 342, 'n_tokens_neox': 319, 'n_words': 98}