<commit_before>Change Log
==========

This project uses `semantic versioning <http://semver.org/>`_ to
track version numbers, where backwards incompatible changes
(highlighted in **bold**) bump the major version of the package.


latest changes in development for next release
----------------------------------------------

.. THANKS FOR CONTRIBUTING; MENTION WHAT YOU DID IN THIS SECTION HERE!

2.0.0
-----

There have been some changes in the scrubadub API, but few breaking changes.
The changes include:

* Several new detectors have been added (spacy, stanford NER, tax reference number, credit card, ...)
* Added ability to easily evaluate a `Detector`\ 's performance, see :ref:`accuracy`.
* Started to localise detectors to function for more than one language/location.
* Support for scrubbing multiple documents together.
* Introduced the concept of a `PostProcessor`.
  This will allow more complex groupings of `Filth`\ s and new types of tokenization.

Scrubber
^^^^^^^^

* `Detector`\ s and `PostProcessor`\ s can be added and removed using a string containing their default name, their class or an instance.
* You can clean multiple documents with one `Scrubber().clean_documents(docs)` call
* **A default set of `Detector`\ s are loaded instead of all `Detector`\ s.**
  This is particularly useful for detectors that are slow or have complex dependencies, as they dont need to be loaded each time.
  However, this might need an explicit `Scrubber().add_detector(detector)` call for the same behaviour as before.
* Added a `locale` parameter to the `Scrubber` initialiser.
* A Scrubber will only auto-load detectors that support a given `Scrubber` `locale`.

Detectors
^^^^^^^^^

* The the name of the detector has been separated from the type of filth found.
  This means multiple instances of the same detector (configured differently) can be in the same `Scrubber` instance and one `Detector` can return multiple types of `Filth`.
* **`Detector`\ s now required to define an attribute called name**, which should be unique within a `Scrubber` instance.
* **`Detector`\ s are now passed a `locale` argument to the `Detector` initialiser**.
* `Detector`\ s have an optional `supported_locale(locale)` function that returns a bool to indicate if a given `Detector` supports a `locale`.
* **Regular expressions used by the `RegexDetector` class have been moved from `RegexFilth.regex` to `RegexDetector.regex`**.
* **Renamed `SSNDetector` to `SocialSecurityNumberDetector`**.
* Added `AddressDetector`, which detects US, CA and GB addresses.
* Added `CreditCardDetector`, which detects credit card numbers (based on the Detector in the alphagov scrubadub fork).
* Added `DateOfBirthDetector`, which detects dates of birth.
* Added `DriversLicenceDetector`, which detects GB drivers licence numbers.
* Added `KnownFilthDetector`, which is used to find bits of known Filth, such as human tagged Filth.
* Added `PostalCodeDetector`, which detects GB post codes.
* Added `SpacyEntityDetector`, which detects a `range of named entities <https://spacy.io/api/annotation#named-entities>`_, including names.
* Added `StanfordEntityDetector`, which also detects `slightly different range of named entities <https://nlp.stanford.edu/software/CRF-NER.html#Models>`_, including names.
* Added `NationalInsuranceNumberDetector`, which detects GB National Insurance Numbers (NINO).
* Added `TaxReferenceNumberDetector`, which detects GB Tax Reference Numbers (TRN).
* Added `VehicleLicencePlateDetector`, which detects number plates on GB cars.
* Added `RegionLocalisedRegexDetector`, which derived from the convenience class `RegexDetector` to allow for quickly creating regional regex based detectors.

Filth
^^^^^

* **Introduced three parameters in the constructor `detector_name`, `document_name` and `locale`**.
  These keep track of the `Detector` that found the `Filth`, the document it came from and the documents locale.
  This results in `Filth` objects being passed additional parameters on initialisation.
  If you have defined custom `Filth`\ s they will need to be updated so that `Filth.__init__` accepts the `detector_name`, `document_name` and `locale` keywords and call the base class constructor.

PostProcessors
^^^^^^^^^^^^^^

* Introduction of simple `PostProcessors`:
   * `FilthTypeReplacer`: Replace the filth with the type of filth ``<EMAIL> -> EMAIL``
   * `HashReplacer`: Replace the filth with a configurable hash ``<EMAIL> -> 196aa39e9f8159ec``
   * `NumericReplacer`: Replace the filth with a monotonically increasing number for each unique piece of filth, optionally including the filth type ``<EMAIL> -> EMAIL-1``.
   * `PrefixSuffixReplacer`: Add a prefix and/or suffix onto the replacement ``EMAIL-1 -> {{EMAIL-1}}``
* It is envisioned that other more complex operations can be done here too such as grouping filth (e.g. "John", "<NAME>" and "<NAME>" could be grouped together).

1.2.2
-----

`LeapBeyond <http://leapbeyond.ai/>`_ are now supporting scrubadub with maintanance and development.

* bug fixes:
<commit_msg>add attribution for new detectors, tweak formatting<commit_after>Change Log
==========

This project uses `semantic versioning <http://semver.org/>`_ to
track version numbers, where backwards incompatible changes
(highlighted in **bold**) bump the major version of the package.


latest changes in development for next release
----------------------------------------------

.. THANKS FOR CONTRIBUTING; MENTION WHAT YOU DID IN THIS SECTION HERE!

2.0.0
-----

There have been some changes in the scrubadub API, but few breaking changes.
The changes include:

* Several new detectors have been added (spacy, stanford NER, tax reference number, credit card, ...)
* Added ability to easily evaluate a `Detector`\ 's performance, see :ref:`accuracy`.
* Started to localise detectors to function for more than one language/location.
* Support for scrubbing multiple documents together.
* Introduced the concept of a `PostProcessor`.
  This will allow more complex groupings of `Filth`\ s and new types of tokenization.

Scrubber
^^^^^^^^

* `Detector`\ s and `PostProcessor`\ s can be added and removed using a string containing their default name, their class or an instance.
* You can clean multiple documents with one `Scrubber().clean_documents(docs)` call
* **A default set of Detectors are loaded instead of all Detectors.**
  This is particularly useful for detectors that are slow or have complex dependencies, as they dont need to be loaded each time.
  However, this might need an explicit `Scrubber().add_detector(detector)` call for the same behaviour as before.
* Added a `locale` parameter to the `Scrubber` initialiser.
* A Scrubber will only auto-load detectors that support a given `Scrubber` `locale`.

Detectors
^^^^^^^^^

* The the name of the detector has been separated from the type of filth found.
  This means multiple instances of the same detector (configured differently) can be in the same `Scrubber` instance and one `Detector` can return multiple types of `Filth`.
* **Detectors now required to define an attribute called name**, which should be unique within a `Scrubber` instance.
* **Detectors are now passed a locale argument to the Detector initialiser**.
* `Detectors have an optional `supported_locale(locale)` function that returns a bool to indicate if a given `Detector` supports a `locale`.
* **Regular expressions used by the `egexDetector class have been moved from RegexFilth.regex to RegexDetector.regex**.
* **Renamed SSNDetector to SocialSecurityNumberDetector**.
* New `AddressDetector`, which detects US, CA and GB addresses.
* New `CreditCardDetector`, which detects credit card numbers (based on the Detector in the `alphagov scrubadub fork <https://github.com/alphagov/scrubadub>`_).
* New `DateOfBirthDetector`, which detects dates of birth (thanks to `@mirandachong <https://github.com/mirandachong>`_).
* New `DriversLicenceDetector`, which detects GB drivers licence numbers.
* New `KnownFilthDetector`, which is used to find bits of known Filth, such as human tagged Filth.
* New `PostalCodeDetector`, which detects GB post codes.
* New `SpacyEntityDetector`, which detects a `range of named entities <https://spacy.io/api/annotation#named-entities>`_, including names (thanks to `@aCampello <https://github.com/aCampello>`_).
* New `StanfordEntityDetector`, which also detects `slightly different range of named entities <https://nlp.stanford.edu/software/CRF-NER.html#Models>`_, including names.
* New `NationalInsuranceNumberDetector`, which detects GB National Insurance Numbers (NINO) (thanks to `@mirandachong <https://github.com/mirandachong>`_).
* New `TaxReferenceNumberDetector`, which detects GB Tax Reference Numbers (TRN) (thanks to `@mirandachong <https://github.com/mirandachong>`_).
* New `VehicleLicencePlateDetector`, which detects number plates on GB cars (based on the Detector in the `alphagov scrubadub fork <https://github.com/alphagov/scrubadub>`_).
* New `RegionLocalisedRegexDetector`, which derived from the convenience class `RegexDetector` to allow for quickly creating regional regex based detectors.

Filth
^^^^^

* **Introduced three parameters in the constructor `detector_name`, `document_name` and `locale`**.
  These keep track of the `Detector` that found the `Filth`, the document it came from and the documents locale.
  This results in `Filth` objects being passed additional parameters on initialisation.
  If you have defined custom `Filth`\ s they will need to be updated so that `Filth.__init__` accepts the `detector_name`, `document_name` and `locale` keywords and call the base class constructor.

PostProcessors
^^^^^^^^^^^^^^

* Introduction of simple `PostProcessors`:
   * `FilthTypeReplacer`: Replace the filth with the type of filth ``<EMAIL> -> EMAIL``
   * `HashReplacer`: Replace the filth with a configurable hash ``<EMAIL> -> 196aa39e9f8159ec``
   * `NumericReplacer`: Replace the filth with a monotonically increasing number for each unique piece of filth, optionally including the filth type ``<EMAIL> -> EMAIL-1``.
   * `PrefixSuffixReplacer`: Add a prefix and/or suffix onto the replacement ``EMAIL-1 -> {{EMAIL-1}}``
* It is envisioned that other more complex operations can be done here too such as grouping filth (e.g. "John", "<NAME>" and "<NAME>" could be grouped together).

1.2.2
-----

`LeapBeyond <http://leapbeyond.ai/>`_ are now supporting scrubadub with maintanance and development.

* bug fixes:

{'fraction_non_alphanumeric': 0.10267175572519084, 'fraction_numerical': 0.0036259541984732823, 'mean_word_length': 4.77147577092511, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 35, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1411634', 'n_tokens_mistral': 2957, 'n_tokens_neox': 2797, 'n_words': 1300}