<commit_before># Test suite for patent preprocessing

We're intending on conforming to PEP guidelines,
please note where implementation is not meeting
a relevant PEP.

## Running integration tests

The integration tests require two types of databases:

1. A set of sqlite databases located in the test directory as a result
   of a succesful parse, and
2. Databases `loctbl` and `NBER_asg` linked from elsewhere like so:
    * `ln -s /data/patentdata/NBER/NBER_asg .`
    * `ln -s /data/patentdata/location/loctbl.sqlite3/NBER_asg loctbl`

(Your links may be different.)

<commit_msg>Bad instructions for linking loctbl<commit_after># Test suite for patent preprocessing

We're intending on conforming to PEP guidelines,
please note where implementation is not meeting
a relevant PEP.

## Running integration tests

The integration tests require two types of databases:

1. A set of sqlite databases located in the test directory as a result
   of a succesful parse, and
2. Databases `loctbl` and `NBER_asg` linked from elsewhere like so:
    * `ln -s /data/patentdata/NBER/NBER_asg .`
    * `ln -s /data/patentdata/location/loctbl.sqlite3 loctbl`

(Your links may be different.)


{'fraction_non_alphanumeric': 0.06689246401354784, 'fraction_numerical': 0.005080440304826418, 'mean_word_length': 4.139130434782609, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7270000', 'n_tokens_mistral': 370, 'n_tokens_neox': 355, 'n_words': 154}