<commit_before># Python scripts for disambiguating patent data

The following collection of scripts performs pre- and
post-processing on patent data as part of the patent
inventor disambiguation process.

## Contributing to the Patent Processor Project

Pull requests are welcome. Here are a few pointers which will make everything easier:

* Small, tightly constrained commits.
* New files should be in their own commit, and committed before they are used in subsequent commits.
* Commits should tell a story in a logical sequence. It should be possible to understand the gist
  of the development just from reading the commits (hard, but worthwhile goal).
* The ideal commit:
    * Unit (or similar) test for a single functionality.
    * Implementation to pass the unit test.
    * Documentation (the "why") of the function/method in the appropriate location (platform dependent).
    * 0 or 1 use of the new functionality in production.
    * Further uses of functionality should go in future commits.
* Formatting updates, code cleanup and renaming should go into independent commits.
* Submit only code which is covered by *working* unit tests.
* Testing scripts, including unit tests, integration tests and functional tests go in the `test` directory.
* Code which does work goes in the lib directory.
* Code which provides a workflow (i.e., processing patents or building necessary
  infrastructure) goes in the top level directory. In the future, much of this code may
  be put into a `bin` directory.
* Test code should follow the pattern `test/test_libfile.py`. This pattern may change in
  the future, whence this documentation will change at that time.


## Configuring the Preprocessing Environment

In order to properly configure the preprocessing environment, the end user must
manually perform the following:

* Download the relevant XML files which need to be processed. These can be
  placed in any directory, but `parse.py` assumes the root directory `/`.

* Set the PATENTROOT environment variable (assuming a UNIX environment). This
  can be customized later using command line options, but for the purposes of
  running unit tests, it is necessary to actually export the environment
  variable, e.g.

  `export PATENTROOT = /path/to/xml/files`

* You *do not* need to have the `.sqlite3` files already in the project root 
  directory, as they will be automatically generated by `parse.py`.
<commit_msg>added a note about rebasing to the readme<commit_after># Python scripts for disambiguating patent data

The following collection of scripts performs pre- and
post-processing on patent data as part of the patent
inventor disambiguation process.

## Contributing to the Patent Processor Project

Pull requests are welcome. Here are a few pointers which will make everything easier:

* Small, tightly constrained commits.
* New files should be in their own commit, and committed before they are used in subsequent commits.
* Commits should tell a story in a logical sequence. It should be possible to understand the gist
  of the development just from reading the commits (hard, but worthwhile goal).
* The ideal commit:
    * Unit (or similar) test for a single functionality.
    * Implementation to pass the unit test.
    * Documentation (the "why") of the function/method in the appropriate location (platform dependent).
    * 0 or 1 use of the new functionality in production.
    * Further uses of functionality should go in future commits.
* Formatting updates, code cleanup and renaming should go into independent commits.
* Submit only code which is covered by *working* unit tests.
* Testing scripts, including unit tests, integration tests and functional tests go in the `test` directory.
* Code which does work goes in the lib directory.
* Code which provides a workflow (i.e., processing patents or building necessary
  infrastructure) goes in the top level directory. In the future, much of this code may
  be put into a `bin` directory.
* Test code should follow the pattern `test/test_libfile.py`. This pattern may change in
  the future, whence this documentation will change at that time.

** Please rebase for issuing pull request: `git pull --rebase <upstream> master**.

## Configuring the Preprocessing Environment

In order to properly configure the preprocessing environment, the end user must
manually perform the following:

* Download the relevant XML files which need to be processed. These can be
  placed in any directory, but `parse.py` assumes the root directory `/`.

* Set the PATENTROOT environment variable (assuming a UNIX environment). This
  can be customized later using command line options, but for the purposes of
  running unit tests, it is necessary to actually export the environment
  variable, e.g.

  `export PATENTROOT = /path/to/xml/files`

* You *do not* need to have the `.sqlite3` files already in the project root 
  directory, as they will be automatically generated by `parse.py`.

{'fraction_non_alphanumeric': 0.050171267378601656, 'fraction_numerical': 0.0012089462018940156, 'mean_word_length': 4.230769230769231, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '178076', 'n_tokens_mistral': 1205, 'n_tokens_neox': 1166, 'n_words': 716}