<commit_before>lurk
====

Tiny python script which converts HTML from web pages that match a certain CSS pattern into JSON.
::

    $ pip install lurk

=====
usage
=====

**in python**

::

    from lurk import lurk

    for link in lurk('http://en.wikipedia.org/wiki/en', 'a'):
        if 'href' in link:
            print link

**in terminal**

Familiarize yourself with [CSS attribute selectors](https://developer.mozilla.org/en-US/docs/Web/CSS/Attribute_selectors).
::

    $ ./lurk \
    http://www.gnu.org/software/libc/manual/html_node/Function-Index.html \
    'a[href*="#index-"]' \
    > links.json

This command saves a JSON object containing an array of links to all GNU C functions into **links.json**:

::<commit_msg>readme fix<commit_after>lurk
====

Tiny python script which converts HTML from web pages that match a certain CSS pattern into JSON.
::

    $ pip install lurk

=====
usage
=====

**in python**

::

    from lurk import lurk

    for link in lurk('http://en.wikipedia.org/wiki/en', 'a'):
        if 'href' in link:
            print link

**in bash**

Familiarize yourself with `CSS attribute selectors <https://developer.mozilla.org/en-US/docs/Web/CSS/Attribute_selectors>`_.

::

    $ lurk \
    http://www.gnu.org/software/libc/manual/html_node/Function-Index.html \
    'a[href*="#index-"]' \
    > links.json

This command saves a JSON object containing an array of links to all GNU C functions into **links.json**:

::
{'fraction_non_alphanumeric': 0.14285714285714285, 'fraction_numerical': 0.0, 'mean_word_length': 3.069832402234637, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 6, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4276390', 'n_tokens_mistral': 482, 'n_tokens_neox': 456, 'n_words': 140}