<commit_before>Article-Collector takes a source URL for a news site, extracts all the individual article URLs it can find, and sends each one as a message to RabbitMQ for later processing.


## Config

Article-Collector uses a config file, written in YAML, that contains two properties:
* `source` - the URL for a news site
* `rabbit_url` - the connection URL for a RabbitMQ server

### Example
```yaml
source: http://cnn.com/
rabbit_url: https://user:password@somerabbitserver.com/whatever
```

## Run with Docker
The Article-Collector is easy to run using [Docker](https://www.docker.com/), first you need to build an image using the provided `Dockerfile`. From inside the project directory:

```
docker build -t give_it_a_name .
```

After the build process completes, you can launch a container to run the process. When launching the container, you will need to mount a volume containing the config file you wish to use:

### Run in Interactive Mode

```
docker run -it -v /path/to/config.yml:/src/config.yml give_it_a_name
```

### Run in Detached Mode<commit_msg>Forgot this commit a long time ago<commit_after>Article-Collector takes a source URL for a news site, extracts all the individual article URLs it can find, and sends each one as a message to RabbitMQ for later processing.


## Config

Article-Collector uses a config file, written in YAML, that contains two properties:
* `source` - the URL for a news site
* `rabbit_url` - the connection URL for a RabbitMQ server

#### Example
```yaml
source: http://cnn.com/
rabbit_url: https://user:password@somerabbitserver.com/whatever
```

## Run with Docker
The Article-Collector is easy to run using [Docker](https://www.docker.com/), first you need to build an image using the provided `Dockerfile`. From inside the project directory:

```
docker build -t give_it_a_name .
```

After the build process completes, you can launch a container to run the process. When launching the container, you will need to mount a volume containing the config file you wish to use:

#### Interactive Mode

```
docker run -it -v /path/to/config.yml:/src/config.yml give_it_a_name
```

#### Detached Mode
{'fraction_non_alphanumeric': 0.09078212290502793, 'fraction_numerical': 0.0, 'mean_word_length': 4.385964912280702, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1576337', 'n_tokens_mistral': 649, 'n_tokens_neox': 589, 'n_words': 285}