<commit_before>*****
About
*****

Introduction
============

The **recordlinkage** toolkit is a library to link records in or between data
sources. The library provides tools needed for record linkage and
deduplication. The package contains indexing methods, functions to compare
attributes and classifiers. The package is developed for research and the
linkage of small or medium sized files.

The *recordlinkage* project is inspired by the `Freely Extensible Biomedical
Record Linkage (FEBRL) <https://sourceforge.net/projects/febrl/>`__ project,
which is a great project. In contrast with FEBRL, the recordlinkage project makes extensive use of data
manipulation tools like `pandas <http://pandas.pydata.org/>`__ and `numpy <http://www.numpy.org/>`__. The use of *pandas*, a flexible and powerful data analysis and
manipulation library for Python, makes the record linkage process much easier
and faster. The extensive *pandas* library can be used to integrate your
record linkage directly into existing data manipulation projects.

One of the aims of this project is to make an extensible record linkage
framework. It is easy to include your own indexing algorithms,
comparison/similarity measures and classifiers. The main features of the Python Record Linkage Toolkit are:

-  Clean and standardise data with easy to use tools
-  Make pairs of records with smart indexing methods such as
   **blocking** and **sorted neighbourhood indexing**
-  Compare records with a large number of comparison and similarity
   measures for different types of variables such as strings, numbers and dates.
-  Several classifications algorithms, both supervised and unsupervised
   algorithms.
-  Common record linkage evaluation tools
-  Several built-in datasets. 

<commit_msg>Rename toolkit<commit_after>*****
About
*****

Introduction
============

The **Python Record Linkage Toolkit** is a library to link records in or
between data sources. The toolkit provides most of the tools needed for 
record linkage and deduplication. The package contains indexing methods, 
functions to compare records and classifiers. The package is developed 
for research and the linking of small or medium sized files.

The project is inspired by the `Freely Extensible Biomedical
Record Linkage (FEBRL) <https://sourceforge.net/projects/febrl/>`__ project,
which is a great project. In contrast with FEBRL, the recordlinkage project makes extensive use of data
manipulation tools like `pandas <http://pandas.pydata.org/>`__ and `numpy <http://www.numpy.org/>`__. The use of *pandas*, a flexible and powerful data analysis and
manipulation library for Python, makes the record linkage process much easier
and faster. The extensive *pandas* library can be used to integrate your
record linkage directly into existing data manipulation projects.

One of the aims of this project is to make an extensible record linkage
framework. It is easy to include your own indexing algorithms,
comparison/similarity measures and classifiers. The main features of the Python Record Linkage Toolkit are:

-  Clean and standardise data with easy to use tools
-  Make pairs of records with smart indexing methods such as
   **blocking** and **sorted neighbourhood indexing**
-  Compare records with a large number of comparison and similarity
   measures for different types of variables such as strings, numbers and dates.
-  Several classifications algorithms, both supervised and unsupervised
   algorithms.
-  Common record linkage evaluation tools
-  Several built-in datasets. 


{'fraction_non_alphanumeric': 0.05923836389280677, 'fraction_numerical': 0.0, 'mean_word_length': 4.784665579119086, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6561073', 'n_tokens_mistral': 888, 'n_tokens_neox': 827, 'n_words': 475}