<commit_before># WormScraper
A web scraper to obtain a text file for Wildbow's Worm which can be found [here](https://parahumans.wordpress.com/).

## Dependencies
1. [BeautifulSoup4](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)
2. [lxml parser](http://lxml.de/)

You can install both with pip.
If you cloned the repo you can simply run pip install -r requirements.txt

## Instructions

1. Clone the repo whereever required with
```
git clone https://github.com/gopal131072/WormScraper
```
2. Make sure all dependencies are installed.

3. Run bot.py with
```
python bot.py
```
This generates url.txt which contains all the links to parse.

4. Run bot2.py with
```
python bot2.py
```
This generates worm.txt which the final text file.

5. Convert the text file with calibre into desired format. <commit_msg>Small changes<commit_after># WormScraper
A web scraper to obtain a text file for Wildbow's Worm which can be found [here](https://parahumans.wordpress.com/).

## Dependencies
1. [BeautifulSoup4](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)
2. [lxml parser](http://lxml.de/)

Included in requirements.txt
Can be installed with
```
pip install -r requiremennts.txt
```

## Instructions

1. Clone the repo whereever required with
```
git clone https://github.com/gopal131072/WormScraper
```
2. Make sure all dependencies are installed.

3. Run bot.py with
```
python bot.py
```
This generates url.txt which contains all the links to parse.

4. Run bot2.py with
```
python bot2.py
```
This generates worm.txt which the final text file.

5. Convert the text file with calibre into desired format.
{'fraction_non_alphanumeric': 0.11124845488257108, 'fraction_numerical': 0.021013597033374538, 'mean_word_length': 4.13968253968254, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4163473', 'n_tokens_mistral': 554, 'n_tokens_neox': 521, 'n_words': 195}