<commit_before># Crawly
... a crawler that retrieves news articles via RSS feeds.

# Description
_Crawly_ retrieves RSS feeds in a pre-defined interval and checks whether there are any new RSS items present. 
If so it immediately downloads the news articles referred to in the new RSS items (link tag).

Since some articles may be modified / corrected later on, _crawly_ can download articles in a fixed interval for a fixed number of times. 


# Getting started

Clone the repository
```bash
git clone https://github.com/le1nux/crawly.git
```
and create the following directories: 
```bash
cd crawly
mkdir logs
mkdir resources
mkdir dump
```
Install the [required packages](https://github.com/le1nux/crawly/blob/master/requirements.txt) by running
```bash
cd crawly
pip install requirements.txt
```<commit_msg>fix in Gettting started<commit_after># Crawly
... a crawler that retrieves news articles via RSS feeds.

# Description
_Crawly_ retrieves RSS feeds in a pre-defined interval and checks whether there are any new RSS items present. 
If so it immediately downloads the news articles referred to in the new RSS items (link tag).

Since some articles may be modified / corrected later on, _crawly_ can download articles in a fixed interval for a fixed number of times. 


# Getting started

Clone the repository
```bash
git clone https://github.com/le1nux/crawly.git
```
and create the following directories: 
```bash
cd crawly
mkdir logs
mkdir resources
mkdir dump
```
Install the [required packages](https://github.com/le1nux/crawly/blob/master/requirements.txt) by running
```bash
cd crawly
pip install -r requirements.txt
```
{'fraction_non_alphanumeric': 0.072782874617737, 'fraction_numerical': 0.0024464831804281344, 'mean_word_length': 4.5646258503401365, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1132503', 'n_tokens_mistral': 488, 'n_tokens_neox': 443, 'n_words': 214}