<commit_before># Python scripts for processing USPTO inventor and patent data

The following collection of scripts performs pre- and
post-processing on patent data as part of the patent
inventor disambiguation process.

Follow development, subscribe to
[RSS
feed](https://github.com/funginstitute/patentprocessor/commits/master.atom).

## Processing patents

There are two ways to get started:

* Run `preprocess.sh <path-to-config-file> <number-of-threads>`

  See `process.cfg` for an example of a configuration file. The options in the
  `[process]` section will be used to determine which data is parsed, which
  steps are run, and where the data will be located after the process finishes.
  This process requires [`IPython`](http://ipython.org/install.html) to be
  installed.

* run `parse.py` directly to customize which directories are processed and
  which regex is used to process the files. Run `parse.py -h` to see the
  relevant command-line options. Follow with `clean.py` then
  `consolidate.py` to obtain a full set of tables.

To run the `clean.py` script, the
[location](https://s3.amazonaws.com/funginstitute/geolocation_data.sqlite3)
table must exist in the `patentprocessor/lib` directory.

## Configuring the Preprocessing Environment

The python-based preprocessor is tested on Ubuntu 12.04 and MacOSX 10.6.  Any
flavor of Unix with the following installed should work, though it is possible
to get the toolchain running on Windows.

If you have [`pip`](http://www.pip-installer.org/en/latest/index.html)
installed, you can simplify the installation process by just running `sudo pip
install -r requirements.txt` from within the patentprocessor directory.

* Python 2.7.3
* scipy package for Python
* sqlite3 --> Note: you need version 3.7.15.1 or higher
* IPython
* pyzmq
* sqlalchemy
* python-Levenshtein
* python-mysqldb
* More? Please [file an
  issue](https://github.com/funginstitute/patentprocessor/issues) if you find another dependency.

For Ubuntu, apt-get install the following packages

```
sudo apt-get install -y python-Levenshtein
sudo apt-get install -y python-mysqldb<commit_msg>Update readme.md with new AWS link<commit_after># Python scripts for processing USPTO inventor and patent data

The following collection of scripts performs pre- and
post-processing on patent data as part of the patent
inventor disambiguation process.

Follow development, subscribe to
[RSS
feed](https://github.com/funginstitute/patentprocessor/commits/master.atom).

## Processing patents

There are two ways to get started:

* Run `preprocess.sh <path-to-config-file> <number-of-threads>`

  See `process.cfg` for an example of a configuration file. The options in the
  `[process]` section will be used to determine which data is parsed, which
  steps are run, and where the data will be located after the process finishes.
  This process requires [`IPython`](http://ipython.org/install.html) to be
  installed.

* run `parse.py` directly to customize which directories are processed and
  which regex is used to process the files. Run `parse.py -h` to see the
  relevant command-line options. Follow with `clean.py` then
  `consolidate.py` to obtain a full set of tables.

To run the `clean.py` script, the
[location](https://s3.amazonaws.com/fungpatdownloads/geolocation_data.7z)
table must exist in the `patentprocessor/lib` directory. File requires [7zip](http://www.7-zip.org/) to unpack.

## Configuring the Preprocessing Environment

The python-based preprocessor is tested on Ubuntu 12.04 and MacOSX 10.6.  Any
flavor of Unix with the following installed should work, though it is possible
to get the toolchain running on Windows.

If you have [`pip`](http://www.pip-installer.org/en/latest/index.html)
installed, you can simplify the installation process by just running `sudo pip
install -r requirements.txt` from within the patentprocessor directory.

* Python 2.7.3
* scipy package for Python
* sqlite3 --> Note: you need version 3.7.15.1 or higher
* IPython
* pyzmq
* sqlalchemy
* python-Levenshtein
* python-mysqldb
* More? Please [file an
  issue](https://github.com/funginstitute/patentprocessor/issues) if you find another dependency.

For Ubuntu, apt-get install the following packages

```
sudo apt-get install -y python-Levenshtein
sudo apt-get install -y python-mysqldb
{'fraction_non_alphanumeric': 0.08368103847936949, 'fraction_numerical': 0.00880853036624942, 'mean_word_length': 4.441361916771752, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 9, 'https://': 6, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2386284', 'n_tokens_mistral': 1315, 'n_tokens_neox': 1237, 'n_words': 517}