<commit_before>.. _api_scrubadub_detectors:

scrubadub.detectors
===================

``scrubadub`` consists of several ``Detector``'s, which are responsible for
identifying and iterating over the ``Filth`` that can be found in a piece of
text.

Base classes
------------

Every ``Detector`` that inherits from ``scrubadub.detectors.base.Detector``.

scrubadub.detectors.base.Detector
---------------------------------

.. autoclass:: scrubadub.detectors.base.Detector

scrubadub.detectors.base.RegexDetector
--------------------------------------

For convenience, there is also a ``RegexDetector``, which makes it easy to
quickly add new types of ``Filth`` that can be identified from regular
expressions:

.. autoclass:: scrubadub.detectors.base.RegexDetector

scrubadub.detectors.base.RegionLocalisedRegexDetector
-----------------------------------------------------

.. autoclass:: scrubadub.detectors.base.RegionLocalisedRegexDetector


Detectors enabled by default
----------------------------

These are the detectors that are enabled in the scrubber by default.

scrubadub.detectors.CredentialDetector
--------------------------------------

.. autoclass:: scrubadub.detectors.CredentialDetector

scrubadub.detectors.CreditCardDetector
--------------------------------------

.. autoclass:: scrubadub.detectors.CreditCardDetector

scrubadub.detectors.DriversLicenceDetector
------------------------------------------

.. autoclass:: scrubadub.detectors.DriversLicenceDetector

scrubadub.detectors.EmailDetector
---------------------------------

.. autoclass:: scrubadub.detectors.EmailDetector

scrubadub.detectors.NationalInsuranceNumberDetector
---------------------------------------------------

.. autoclass:: scrubadub.detectors.NationalInsuranceNumberDetector

scrubadub.detectors.PhoneDetector
---------------------------------

.. autoclass:: scrubadub.detectors.PhoneDetector

scrubadub.detectors.PostalCodeDetector
--------------------------------------

.. autoclass:: scrubadub.detectors.PostalCodeDetector

scrubadub.detectors.SocialSecurityNumberDetector
------------------------------------------------

.. autoclass:: scrubadub.detectors.SocialSecurityNumberDetector

scrubadub.detectors.TaxReferenceNumberDetector
----------------------------------------------

.. autoclass:: scrubadub.detectors.TaxReferenceNumberDetector

scrubadub.detectors.TwitterDetector
-----------------------------------

.. autoclass:: scrubadub.detectors.TwitterDetector

scrubadub.detectors.UrlDetector
-------------------------------

.. autoclass:: scrubadub.detectors.UrlDetector

scrubadub.detectors.VehicleLicencePlateDetector
-----------------------------------------------

.. autoclass:: scrubadub.detectors.VehicleLicencePlateDetector


Detectors that need to be enabled
---------------------------------

These detectors need to be imported first before they they can be added to a ``Scrubber``.

scrubadub.detectors.AddressDetector
-----------------------------------

.. autoclass:: scrubadub.detectors.AddressDetector

scrubadub.detectors.DateOfBirthDetector
---------------------------------------

.. autoclass:: scrubadub.detectors.DateOfBirthDetector

scrubadub.detectors.SkypeDetector
---------------------------------

.. autoclass:: scrubadub.detectors.SkypeDetector

scrubadub.detectors.SpacyEntityDetector
---------------------------------------

.. autoclass:: scrubadub.detectors.SpacyEntityDetector

scrubadub.detectors.SpacyNameDetector 
---------------------------------------

.. autoclass:: scrubadub.detectors.spacy_name_title.SpacyNameDetector

scrubadub.detectors.StanfordEntityDetector
------------------------------------------

.. autoclass:: scrubadub.detectors.StanfordEntityDetector

scrubadub.detectors.TaggedEvaluationFilthDetector
-------------------------------------------------

.. autoclass:: scrubadub.detectors.TaggedEvaluationFilthDetector

scrubadub.detectors.TextBlobNameDetector
----------------------------------------

.. autoclass:: scrubadub.detectors.TextBlobNameDetector<commit_msg>update the documentation with new paths within the package<commit_after>.. _api_scrubadub_detectors:

scrubadub.detectors
===================

``scrubadub`` consists of several ``Detector``'s, which are responsible for
identifying and iterating over the ``Filth`` that can be found in a piece of
text.

Base classes
------------

Every ``Detector`` that inherits from ``scrubadub.detectors.base.Detector``.

scrubadub.detectors.base.Detector
---------------------------------

.. autoclass:: scrubadub.detectors.base.Detector

scrubadub.detectors.base.RegexDetector
--------------------------------------

For convenience, there is also a ``RegexDetector``, which makes it easy to
quickly add new types of ``Filth`` that can be identified from regular
expressions:

.. autoclass:: scrubadub.detectors.base.RegexDetector

scrubadub.detectors.base.RegionLocalisedRegexDetector
-----------------------------------------------------

.. autoclass:: scrubadub.detectors.base.RegionLocalisedRegexDetector


Detectors enabled by default
----------------------------

These are the detectors that are enabled in the scrubber by default.

scrubadub.detectors.CredentialDetector
--------------------------------------

.. autoclass:: scrubadub.detectors.CredentialDetector

scrubadub.detectors.CreditCardDetector
--------------------------------------

.. autoclass:: scrubadub.detectors.CreditCardDetector

scrubadub.detectors.DriversLicenceDetector
------------------------------------------

.. autoclass:: scrubadub.detectors.DriversLicenceDetector

scrubadub.detectors.EmailDetector
---------------------------------

.. autoclass:: scrubadub.detectors.EmailDetector

scrubadub.detectors.NationalInsuranceNumberDetector
---------------------------------------------------

.. autoclass:: scrubadub.detectors.NationalInsuranceNumberDetector

scrubadub.detectors.PhoneDetector
---------------------------------

.. autoclass:: scrubadub.detectors.PhoneDetector

scrubadub.detectors.PostalCodeDetector
--------------------------------------

.. autoclass:: scrubadub.detectors.PostalCodeDetector

scrubadub.detectors.SocialSecurityNumberDetector
------------------------------------------------

.. autoclass:: scrubadub.detectors.SocialSecurityNumberDetector

scrubadub.detectors.TaxReferenceNumberDetector
----------------------------------------------

.. autoclass:: scrubadub.detectors.TaxReferenceNumberDetector

scrubadub.detectors.TwitterDetector
-----------------------------------

.. autoclass:: scrubadub.detectors.TwitterDetector

scrubadub.detectors.UrlDetector
-------------------------------

.. autoclass:: scrubadub.detectors.UrlDetector

scrubadub.detectors.VehicleLicencePlateDetector
-----------------------------------------------

.. autoclass:: scrubadub.detectors.VehicleLicencePlateDetector


Detectors that need to be enabled
---------------------------------

These detectors need to be imported first before they they can be added to a ``Scrubber``.

scrubadub.detectors.address.AddressDetector
-------------------------------------------

.. autoclass:: scrubadub.detectors.address.AddressDetector

scrubadub.detectors.date_of_birth.DateOfBirthDetector
-----------------------------------------------------

.. autoclass:: scrubadub.detectors.date_of_birth.DateOfBirthDetector

scrubadub.detectors.skype.SkypeDetector
---------------------------------------

.. autoclass:: scrubadub.detectors.skype.SkypeDetector

scrubadub.detectors.spacy.SpacyEntityDetector
---------------------------------------------

.. autoclass:: scrubadub.detectors.spacy.SpacyEntityDetector

scrubadub.detectors.spacy_name_title.SpacyNameDetector
------------------------------------------------------

.. autoclass:: scrubadub.detectors.spacy_name_title.SpacyNameDetector

scrubadub.detectors.stanford.StanfordEntityDetector
---------------------------------------------------

.. autoclass:: scrubadub.detectors.stanford.StanfordEntityDetector

scrubadub.detectors.TaggedEvaluationFilthDetector
-------------------------------------------------

.. autoclass:: scrubadub.detectors.TaggedEvaluationFilthDetector

scrubadub.detectors.text_blob.TextBlobNameDetector
--------------------------------------------------

.. autoclass:: scrubadub.detectors.text_blob.TextBlobNameDetector
{'fraction_non_alphanumeric': 0.3108416376515788, 'fraction_numerical': 0.0, 'mean_word_length': 1.8400954653937946, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5979489', 'n_tokens_mistral': 2181, 'n_tokens_neox': 2090, 'n_words': 341}