<commit_before>.. _tutorial:

########
Tutorial
########

Introduction
~~~~~~~~~~~~

This tutorial shows you how to extract data from an HTML web page using
Wextracto.

To work through the tutorial you need to download and install
`Python <https://www.python.org/downloads/>`_.

You also need to install Wextracto.  If you can, you should install it
into a
`virtual environment <http://virtualenv.readthedocs.org/en/latest/virtualenv.html>`_
because this makes things easier to manage.
The recommended way to install Wextracto is using
`pip <https://pip.pypa.io/en/latest/installing.html>`_:

.. code-block:: shell

    $ pip install Wextracto

This will install the ``wex`` command:

.. code-block:: shell

    $ wex --help

You are now ready to begin the tutorial.


Writing A Minimal Extractor
~~~~~~~~~~~~~~~~~~~~~~~~~~~

An extractor is a function that takes an HTTP response and returns values 
extracted from it.  Our extractor is going to return the URL of the response.
Write or copy the following into a file called ``tutorial.py``:

.. literalinclude:: samples/tutorial/step1/tutorial.py

The ``response`` parameter here is
file-like object of the type used by the standard library
`urllib2 <https://docs.python.org/2/library/urllib2.html#urllib2.urlopen>`_.

Now we need to tell the ``wex`` command about our new extractor.  We do this
by creating a file called ``entry_points.txt`` with the following contents:

.. literalinclude:: samples/tutorial/step1/entry_points.txt

Now run ``wex`` with the following URL:

.. code-block:: shell

    $ wex http://gilessbrown.github.io/cheeses/cheddar.html
    "http://gilessbrown.github.io/cheeses/cheddar.html"

Congratulations, you have just written an extractor!

.. _selecting-elements:

Selecting Elements
~~~~~~~~~~~~~~~~~~

Python has a great library for processing XML and HTML data called 
`lxml <http://lxml.de/>`_.  We can use this library in our extractor.

Let's use a simple `XPath <http://en.wikipedia.org/wiki/XPath>`_ expression 
to get some text from our chosen web page.

Edit ``tutorial.py`` to look like this:

.. literalinclude:: samples/tutorial/step2/tutorial.py

Now re-run ``wex`` with the same URL we used previously:

.. code-block:: shell

    $ wex http://gilessbrown.github.io/cheeses/cheddar.html
    ["\n\t\t\tCheddar\n\t\t"]

You may be wondering about the square brackets around the text.  That is 
because ``wex`` serializes values using 
`JSON <http://en.wikipedia.org/wiki/JSON>`_.  
Our XPath expression returns a Python list which gives us the square brackets
in JSON.

You may also have noticed the leading and trailing whitespace.  We'll look
at how to get rid of that in the next section.


Extracting Text
~~~~~~~~~~~~~~~

We normally want text we extract HTML elements to be space-normalized.
This means runs of whitespace are converted into a single space 
character and leading and trailing whitespace is trimmed.

Wextracto provides the :func:`text <wex.etree.text>` function to return 
the space-normalized text for each selected element.

Here is what our extractor now looks like:

.. literalinclude:: samples/tutorial/step3/tutorial.py

Let's run ``wex`` with the usual URL again to check the result:

.. code-block:: shell

    $ wex http://gilessbrown.github.io/cheeses/cheddar.html
    "Cheddar"

That is much tidier.

You may be wondering why we don't just use the XPath
`normalize-space <https://developer.mozilla.org/en-US/docs/Web/XPath/Functions/normalize-space>`_
function.  There actually several reasons why we do not want to do this, most of which are specific to extracting text from HTML as opposed to XML:

The :func:`text <wex.etree.text>` function:
   * understands `<br>` tags
   * uses a unicode definition of whitespace (e.g. non breaking spaces)
   * can work with multiple nodes in an node-set

.. _multiple-values:

Multiple Values
~~~~~~~~~~~~~~~

Often we want to extract multiple values from our web page.  This is done by 
`yield`-ing values instead `return`-ing a single value.

So that we know which value is which we also label the values by yielding
a name for the value at the same time.

Modify ``tutorial.py`` to yield the names and values:

.. literalinclude:: samples/tutorial/step4/tutorial.py

Now re-run ``wex``:

.. code-block:: shell

    $ wex http://gilessbrown.github.io/cheeses/cheddar.html
    "name"	"Cheddar"
    "country"	"England"
    "region"	"Somerset"

Wextracto uses the tab character to separate the label from the value.

Errors
~~~~~~

Yielding multiple values from an extractor is ok if all the values<commit_msg>Simplify some text in tutorial.rst<commit_after>.. _tutorial:

########
Tutorial
########

Introduction
~~~~~~~~~~~~

This tutorial shows you how to extract data from an HTML web page using
Wextracto.

To work through the tutorial you need to download and install
`Python <https://www.python.org/downloads/>`_.

You also need to install Wextracto.  If you can, you should install it
into a
`virtual environment <http://virtualenv.readthedocs.org/en/latest/virtualenv.html>`_
because this makes things easier to manage.
The recommended way to install Wextracto is using
`pip <https://pip.pypa.io/en/latest/installing.html>`_:

.. code-block:: shell

    $ pip install Wextracto

This will install the ``wex`` command:

.. code-block:: shell

    $ wex --help

You are now ready to begin the tutorial.


Writing A Minimal Extractor
~~~~~~~~~~~~~~~~~~~~~~~~~~~

An extractor is a function that takes an HTTP response and returns values 
extracted from it.  Our extractor is going to return the URL of the response.
Write or copy the following into a file called ``tutorial.py``:

.. literalinclude:: samples/tutorial/step1/tutorial.py

The ``response`` parameter here is
file-like object of the type used by the standard library
`urllib2 <https://docs.python.org/2/library/urllib2.html#urllib2.urlopen>`_.

Now we need to tell the ``wex`` command about our new extractor.  We do this
by creating a file called ``entry_points.txt`` with the following contents:

.. literalinclude:: samples/tutorial/step1/entry_points.txt

Now run ``wex`` with the following URL:

.. code-block:: shell

    $ wex http://gilessbrown.github.io/cheeses/cheddar.html
    "http://gilessbrown.github.io/cheeses/cheddar.html"

Congratulations, you have just written an extractor!

.. _selecting-elements:

Selecting Elements
~~~~~~~~~~~~~~~~~~

Python has a great library for processing XML and HTML data called 
`lxml <http://lxml.de/>`_.  We can use this library in our extractor.

Let's use a simple `XPath <http://en.wikipedia.org/wiki/XPath>`_ expression 
to get some text from our chosen web page.

Edit ``tutorial.py`` to look like this:

.. literalinclude:: samples/tutorial/step2/tutorial.py

Now re-run ``wex`` with the same URL we used previously:

.. code-block:: shell

    $ wex http://gilessbrown.github.io/cheeses/cheddar.html
    ["\n\t\t\tCheddar\n\t\t"]

You may be wondering about the square brackets around the text.  That is 
because ``wex`` serializes values using 
`JSON <http://en.wikipedia.org/wiki/JSON>`_ and our `XPath` expression
returns a list.

You may also have noticed the leading and trailing whitespace.  We'll look
at how to get rid of that in the next section.


Extracting Text
~~~~~~~~~~~~~~~

We normally want the text we extract to be space-normalized.
This means runs of whitespace are converted into a single space 
character and leading and trailing whitespace are trimmed.

Wextracto provides the :func:`text <wex.etree.text>` function to return 
the space-normalized text for each selected element.

Here is what our extractor now looks like:

.. literalinclude:: samples/tutorial/step3/tutorial.py

Let's run ``wex`` with the usual URL again to check the result:

.. code-block:: shell

    $ wex http://gilessbrown.github.io/cheeses/cheddar.html
    "Cheddar"

That is much tidier.

If you know `XPath` you may be wondering why we don't use the
`normalize-space <https://developer.mozilla.org/en-US/docs/Web/XPath/Functions/normalize-space>`_
function.  There actually several reasons.  The :func:`text <wex.etree.text>` function:
   * converts `<br>` tags to newlines which is what you want when handling HTML
   * uses a unicode definition of whitespace (e.g. non breaking spaces)
   * can work with multiple nodes in an node-set (`lxml` supports `XPath` 1.0)

.. _multiple-values:

Multiple Values
~~~~~~~~~~~~~~~

We normally want to extract more than one value from a web page.  `Wextracto`
allows extractor functions to `yield` multiple values.

So that we know which value is which we also label the values by yielding
a name for the value at the same time.

Modify ``tutorial.py`` to yield the names and values:

.. literalinclude:: samples/tutorial/step4/tutorial.py

Now re-run ``wex``:

.. code-block:: shell

    $ wex http://gilessbrown.github.io/cheeses/cheddar.html
    "name"	"Cheddar"
    "country"	"England"
    "region"	"Somerset"

Wextracto uses the tab character to separate the label from the value.

Errors
~~~~~~

Yielding multiple values from an extractor is ok if all the values
{'fraction_non_alphanumeric': 0.11227067999560585, 'fraction_numerical': 0.0021970778864110734, 'mean_word_length': 3.96943231441048, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 8, 'lorem ipsum': 0, 'www.': 2, 'xml': 5}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '63480', 'n_tokens_mistral': 2862, 'n_tokens_neox': 2714, 'n_words': 1103}