<commit_before># seo-duplication

### Running the application

To execute the app, please run the following:

`TODO`

### Testing

To test the application, please run the following:

`./env/bin/py.test`<commit_msg>chore(README): update documentation<commit_after># seo-duplication

This app is intended for python3 use only.


### Installation

Clone this repo using the command

`$ git clone https://github.com/Falinor/seo-cartographer.git`

or download a ZIP archive and unzip it.

Because this project is a student project, it is not referenced in the
pip registry. Thus, you cannot install it by running
`$ pip install cartodup`. In order to install the module locally, go to
your installation folder and run:

`$ pip install .`

The module itself is named *cartodup*. Assuming you are still in the
installation folder, you may run `$ mkdir test && cd test` to create a
directory where you will run the app.


### Running the application

The main goal of the application is to compare pages 2 by 2, displaying
their similarity ratio at the end of the process. The `--file` option
allows one to pass a text file containing URLs to fetch data from.

A sample file is present in `resources/pages.txt`. To test the app, you
may run the following:

`$ cartodup -f <REPO_ROOT>/resources/pages.txt`

This command will run the module and output a CSV file containing
results. You can add any number of resource URLs you want.


### Testing

To test the application, you may run the following:

`$ python3 setup.py test`

This command will run the tests contained in the `tests/` folder.
{'fraction_non_alphanumeric': 0.07075173720783323, 'fraction_numerical': 0.002526847757422615, 'mean_word_length': 4.12621359223301, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2510247', 'n_tokens_mistral': 452, 'n_tokens_neox': 435, 'n_words': 234}