<commit_before>===========================
fastchunking Python library
===========================

What it is
----------

`fastchunking` is a Python library that contains efficient and easy-to-use
implementations of string chunking algorithms.

It has been developed as part of the work [LS16]_ at CISPA, Saarland University.

Installation
------------

::

    $ pip install fastchunking

.. note:: For performance reasons, parts of this library are implemented in C++.
	Installation from a source distribution, thus, requires availability of a
	correctly configured C++ compiler.

Usage and Overview
------------------
<commit_msg>Included badges: travis, pypi and readthedocs<commit_after>===========================
fastchunking Python library
===========================

.. image:: https://travis-ci.org/netleibi/fastchunking.svg?branch=master
    :target: https://travis-ci.org/netleibi/fastchunking

.. image:: https://badge.fury.io/py/fastchunking.svg
    :target: https://badge.fury.io/py/fastchunking

.. image:: https://readthedocs.org/projects/fastchunking/badge/?version=latest
:target: http://fastchunking.readthedocs.io/en/latest/?badge=latest
:alt: Documentation Status

What it is
----------

`fastchunking` is a Python library that contains efficient and easy-to-use
implementations of string chunking algorithms.

It has been developed as part of the work [LS16]_ at CISPA, Saarland University.

Installation
------------

::

    $ pip install fastchunking

.. note:: For performance reasons, parts of this library are implemented in C++.
	Installation from a source distribution, thus, requires availability of a
	correctly configured C++ compiler.

Usage and Overview
------------------

{'fraction_non_alphanumeric': 0.1858562244301578, 'fraction_numerical': 0.0023378141437755697, 'mean_word_length': 3.795518207282913, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '40386', 'n_tokens_mistral': 479, 'n_tokens_neox': 458, 'n_words': 158}