<commit_before>URLExtract
----------

URLExtract is python class for collecting (extracting) URLs from given
text based on locating TLD.

.. image:: https://img.shields.io/github/tag/lipoja/URLExtract.svg
    :target: https://github.com/lipoja/URLExtract/tags
    :alt: Git tag
.. image:: https://img.shields.io/pypi/pyversions/urlextract.svg
    :target: https://pypi.python.org/pypi/urlextract
    :alt: Python Version Compatibility


How does it work
~~~~~~~~~~~~~~~~

It tries to find any occurrence of TLD in given text. If TLD is found it
starts from that position to expand boundaries to both sides searching
for "stop character" (usually whitespace, comma, single or double
quote).

A dns check option is available to also reject invalid domain names.

NOTE: List of TLDs is downloaded from iana.org to keep you up to date with new TLDs.

Installation
~~~~~~~~~~~~

Package is available on PyPI - you can install it via pip.

.. image:: https://img.shields.io/pypi/v/urlextract.svg
    :target: https://pypi.python.org/pypi/urlextract
.. image:: https://img.shields.io/pypi/status/urlextract.svg
    :target: https://pypi.python.org/pypi/urlextract
<commit_msg>Show build status<commit_after>URLExtract
----------

URLExtract is python class for collecting (extracting) URLs from given
text based on locating TLD.

.. image:: https://img.shields.io/github/workflow/status/lipoja/URLExtract/Upload%20Python%20Package
    :alt: Build Status
.. image:: https://img.shields.io/github/tag/lipoja/URLExtract.svg
    :target: https://github.com/lipoja/URLExtract/tags
    :alt: Git tag
.. image:: https://img.shields.io/pypi/pyversions/urlextract.svg
    :target: https://pypi.python.org/pypi/urlextract
    :alt: Python Version Compatibility


How does it work
~~~~~~~~~~~~~~~~

It tries to find any occurrence of TLD in given text. If TLD is found it
starts from that position to expand boundaries to both sides searching
for "stop character" (usually whitespace, comma, single or double
quote).

A dns check option is available to also reject invalid domain names.

NOTE: List of TLDs is downloaded from iana.org to keep you up to date with new TLDs.

Installation
~~~~~~~~~~~~

Package is available on PyPI - you can install it via pip.

.. image:: https://img.shields.io/pypi/v/urlextract.svg
    :target: https://pypi.python.org/pypi/urlextract
.. image:: https://img.shields.io/pypi/status/urlextract.svg
    :target: https://pypi.python.org/pypi/urlextract

{'fraction_non_alphanumeric': 0.13017031630170317, 'fraction_numerical': 0.0016220600162206002, 'mean_word_length': 3.771760154738878, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 17, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6554208', 'n_tokens_mistral': 813, 'n_tokens_neox': 789, 'n_words': 249}