<commit_before># <%= name %>

TODO: Write description.

## Requirements

- Ruby
- RabbitMQ
- RDB
  - PostgreSQL (default)
  - MySQL
  - SQLite3

## Usage

1. Install dependencies

```
$ bundle install
```

2. Create database

```
$ bundle exec rake db:create
$ bundle exec rake db:migrate
```

3. Open new terminal and run crawler/processor

```
$ bin/crawler   # on new terminal
$ bin/processor # on new terminal
```

4. Enqueue task

```
$ bin/enqueue url http://example.com/
```

5. You'll see `It works with 'http://example.com'` on your terminal which runs your processor!

6. You can re-enqueue task for processor

```
$ bin/enqueue response http://example.com/
```

Display `It works with 'http://example.com'` again on your terminal which runs your processor.

## Usage with docker-compose
<commit_msg>Update generated README.md<commit_after># <%= name %>

TODO: Write description.

## Requirements

- Ruby
- RabbitMQ
- RDB
  - PostgreSQL (default)
  - MySQL
  - SQLite3

## Usage

1. Install dependencies

```
$ bundle install
```

2. Create database

```
$ bundle exec rake db:create
$ bundle exec rake db:migrate
```

3. Open new terminal and run crawler/processor

```
$ bin/crawler   # on new terminal
$ bin/processor # on new terminal
```

4. Enqueue task

```
$ bin/enqueue url http://example.com/
```

5. You'll see `It works with 'http://example.com'` on your terminal which runs your processor!

6. You can re-enqueue task for processor

```
$ bin/enqueue response http://example.com/
```

Display `It works with 'http://example.com'` again on your terminal which runs your processor.

## Usage with docker-compose

1. Build docker images

```
$ docker-compose build
```

2. Run docker containers

```
$ docker-compose up -d
```

3. Run a command on docker containers

```
$ docker-compose exec <service name> <command>
```

For example,

```
$ docker-compose exec <%= name %>-db bash
$ docker-compose exec <%= name %>-crawler sh
```

4. Shutdown docker containers

```
$ docker-compose down
$ docker-compose down --rmi all # Remove all related images
```
{'fraction_non_alphanumeric': 0.13796430294259526, 'fraction_numerical': 0.008683068017366137, 'mean_word_length': 3.4891774891774894, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3107172', 'n_tokens_mistral': 740, 'n_tokens_neox': 671, 'n_words': 235}