<commit_before>## CitationHunt

CitationHunt is a simple tool for finding unsourced statements on
Wikipedia in different languages. It is hosted at
[https://tools.wmflabs.org/citationhunt/](https://tools.wmflabs.org/citationhunt/).

This repository contains the full server and client code. The
[scripts/](https://github.com/eggpi/citationhunt/tree/master/scripts)
directory contains all the scripts used for processing Wikipedia dumps.
Hopefully they will be illustrative and reusable for similar applications.

### Developing

The `run-tests.sh` script will run all tests for you. You may want to install it
as a pre-commit hook on git:

```
ln -s ../../run-tests.sh .git/hooks/pre-push
```

### Running

It's highly recommended that you use a
[virtualenv](https://pypi.python.org/pypi/virtualenv) for running CitationHunt:

```
$ virtualenv ch-venv
$ cd ch-venv<commit_msg>Fix name of hook.<commit_after>## CitationHunt

CitationHunt is a simple tool for finding unsourced statements on
Wikipedia in different languages. It is hosted at
[https://tools.wmflabs.org/citationhunt/](https://tools.wmflabs.org/citationhunt/).

This repository contains the full server and client code. The
[scripts/](https://github.com/eggpi/citationhunt/tree/master/scripts)
directory contains all the scripts used for processing Wikipedia dumps.
Hopefully they will be illustrative and reusable for similar applications.

### Developing

The `run-tests.sh` script will run all tests for you. You may want to install it
as a pre-push hook on git:

```
ln -s ../../run-tests.sh .git/hooks/pre-push
```

### Running

It's highly recommended that you use a
[virtualenv](https://pypi.python.org/pypi/virtualenv) for running CitationHunt:

```
$ virtualenv ch-venv
$ cd ch-venv
{'fraction_non_alphanumeric': 0.10775370581527936, 'fraction_numerical': 0.0, 'mean_word_length': 4.43343653250774, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1609088', 'n_tokens_mistral': 550, 'n_tokens_neox': 513, 'n_words': 190}