<commit_before>URLExtract
----------

URLExtract is python class for collecting (extracting) URLs from given
text.

How does it work
~~~~~~~~~~~~~~~~

It tries to find any occurrence of TLD in given text. If TLD is found it
starts from that position to expand boundaries to both sides searching
for "stop character" (usually whitespace, comma, single or double
quote).

NOTE: List of TLDs is downloaded from iana.org to keep you up to date with new TLDs.

Installation
~~~~~~~~~~~~

Package is available on PyPI - you can install it via pip.

::

   pip install urlextract

Documentation
~~~~~~~~~~~~~

Online documentation is published at http://urlextract.readthedocs.io/


Requirements
~~~~~~~~~~~~

- IDNA for converting links to IDNA format
- uritools for domain name validation

   ::

       pip install idna
       pip install uritools

Example
~~~~~~~

You can look at command line program *bin/urlextract*.
But everything you need to know is this:

.. code:: python

    from urlextract import URLExtract

    extractor = URLExtract()
    urls = extractor.find_urls("Text with URLs. Let's have URL janlipovsky.cz as an example.")
    print(urls) # prints: ['janlipovsky.cz']

License
~~~~~~~

This piece of code is licensed under The MIT License.
<commit_msg>Adding travis-ci build icon<commit_after>URLExtract
----------

URLExtract is python class for collecting (extracting) URLs from given
text.

.. image:: https://travis-ci.org/lipoja/URLExtract.svg?branch=master
    :target: https://travis-ci.org/lipoja/URLExtract

How does it work
~~~~~~~~~~~~~~~~

It tries to find any occurrence of TLD in given text. If TLD is found it
starts from that position to expand boundaries to both sides searching
for "stop character" (usually whitespace, comma, single or double
quote).

NOTE: List of TLDs is downloaded from iana.org to keep you up to date with new TLDs.

Installation
~~~~~~~~~~~~

Package is available on PyPI - you can install it via pip.

::

   pip install urlextract

Documentation
~~~~~~~~~~~~~

Online documentation is published at http://urlextract.readthedocs.io/


Requirements
~~~~~~~~~~~~

- IDNA for converting links to IDNA format
- uritools for domain name validation

   ::

       pip install idna
       pip install uritools

Example
~~~~~~~

You can look at command line program *bin/urlextract*.
But everything you need to know is this:

.. code:: python

    from urlextract import URLExtract

    extractor = URLExtract()
    urls = extractor.find_urls("Text with URLs. Let's have URL janlipovsky.cz as an example.")
    print(urls) # prints: ['janlipovsky.cz']

License
~~~~~~~

This piece of code is licensed under The MIT License.

{'fraction_non_alphanumeric': 0.11364485981308411, 'fraction_numerical': 0.0, 'mean_word_length': 3.711267605633803, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6660778', 'n_tokens_mistral': 794, 'n_tokens_neox': 776, 'n_words': 329}