<commit_before>RoboBrowser: Your friendly neighborhood web scraper
===============================================

.. image:: https://badge.fury.io/py/robobrowser.png
    :target: http://badge.fury.io/py/robobrowser
    
.. image:: https://travis-ci.org/jmcarp/robobrowser.png?branch=master
        :target: https://travis-ci.org/jmcarp/robobrowser

.. image:: https://coveralls.io/repos/jmcarp/robobrowser/badge.png?branch=master
        :target: https://coveralls.io/r/jmcarp/robobrowser

Homepage: `http://robobrowser.readthedocs.org/ <http://robobrowser.readthedocs.org/>`_

RoboBrowser is a simple, Pythonic library for browsing the web without a standalone web browser. RoboBrowser
can fetch a page, click on links and buttons, and fill out and submit forms. If you need to interact with services
that don't have APIs, RoboBrowser can help.

.. code-block:: python
    
    import re
    from robobrowser import RoboBrowser
    
    # Browse to Rap Genius
    browser = RoboBrowser(history=True)
    browser.open('http://rapgenius.com/')
    
    # Search for Queen
    form = browser.get_form(action='/search')
    form            # <RoboForm q=>
    form['q'].value = 'queen'
    browser.submit_form(form)

    # Look up the first song
    songs = browser.select('.song_name')
    browser.follow_link(songs[0])
    lyrics = browser.select('.lyrics')
    lyrics[0].text     # \n[Intro]\nIs this the real life...
    
    # Back to results page
    browser.back()

    # Look up my favorite song
    browser.follow_link('death on two legs')

    # Can also search HTML using regex patterns
    lyrics = browser.find(class_=re.compile(r'\blyrics\b'))
    lyrics.text     # \n[Verse 1]\nYou suck my blood like a leech...

RoboBrowser combines the best of two excellent Python libraries: 
`Requests <http://docs.python-requests.org/en/latest/>`_ and 
`BeautifulSoup <http://www.crummy.com/software/BeautifulSoup/>`_. 
RoboBrowser represents browser sessions using Requests and HTML responses 
using BeautifulSoup, transparently exposing methods of both libraries:

.. code-block:: python

    import re
    from robobrowser import RoboBrowser

    browser = RoboBrowser(user_agent='a python robot')<commit_msg>update docs<commit_after>RoboBrowser: Your friendly neighborhood web scraper
===============================================

.. image:: https://badge.fury.io/py/robobrowser.png
    :target: http://badge.fury.io/py/robobrowser
    
.. image:: https://travis-ci.org/jmcarp/robobrowser.png?branch=master
        :target: https://travis-ci.org/jmcarp/robobrowser

.. image:: https://coveralls.io/repos/jmcarp/robobrowser/badge.png?branch=master
        :target: https://coveralls.io/r/jmcarp/robobrowser

Homepage: `http://robobrowser.readthedocs.org/ <http://robobrowser.readthedocs.org/>`_

RoboBrowser is a simple, Pythonic library for browsing the web without a standalone web browser. RoboBrowser
can fetch a page, click on links and buttons, and fill out and submit forms. If you need to interact with web services
that don't have APIs, RoboBrowser can help.

.. code-block:: python
    
    import re
    from robobrowser import RoboBrowser
    
    # Browse to Rap Genius
    browser = RoboBrowser(history=True)
    browser.open('http://rapgenius.com/')
    
    # Search for Queen
    form = browser.get_form(action='/search')
    form                # <RoboForm q=>
    form['q'].value = 'queen'
    browser.submit_form(form)

    # Look up the first song
    songs = browser.select('.song_name')
    browser.follow_link(songs[0])
    lyrics = browser.select('.lyrics')
    lyrics[0].text      # \n[Intro]\nIs this the real life...
    
    # Back to results page
    browser.back()

    # Look up my favorite song
    browser.follow_link('death on two legs')

    # Can also search HTML using regex patterns
    lyrics = browser.find(class_=re.compile(r'\blyrics\b'))
    lyrics.text         # \n[Verse 1]\nYou suck my blood like a leech...

RoboBrowser combines the best of two excellent Python libraries: 
`Requests <http://docs.python-requests.org/en/latest/>`_ and 
`BeautifulSoup <http://www.crummy.com/software/BeautifulSoup/>`_. 
RoboBrowser represents browser sessions using Requests and HTML responses 
using BeautifulSoup, transparently exposing methods of both libraries:

.. code-block:: python

    import re
    from robobrowser import RoboBrowser

    browser = RoboBrowser(user_agent='a python robot')
{'fraction_non_alphanumeric': 0.13111061049785988, 'fraction_numerical': 0.0013516557783284523, 'mean_word_length': 3.4135188866799204, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 10, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4568141', 'n_tokens_mistral': 1385, 'n_tokens_neox': 1356, 'n_words': 367}