<commit_before>Newspaper: Article scraping & curation
======================================

.. image:: https://badge.fury.io/py/textblob.png
    :target: http://badge.fury.io/py/textblob
        :alt: Latest version

.. image:: https://pypip.in/d/textblob/badge.png
    :target: https://crate.io/packages/textblob/
        :alt: Number of PyPI downloads


Homepage: `https://newspaper.readthedocs.org/ <https://newspaper.readthedocs.org/>`_

Inspired by `requests` for its simplicity and powered by `lxml` for its speed; `newspaper` is a Python 2 library
for extracting articles from the web and curating text (NLP) for keywords, summaries, authors, etc.
Newspaper utilizes async io and caching for speed. Everything is in unicode :)

There are two API's available. The low level article API, and the newspaper API.

.. code-block:: pycon

    >>> import newspaper

    >>> cnn_paper = newspaper.build('http://cnn.com')

    >>> for article in cnn_paper.articles: 
    >>>     print article.url
    u'http://www.cnn.com/2013/11/27/justice/tucson-arizona-captive-girls/'
    u'http://www.cnn.com/2013/12/11/us/texas-teen-dwi-wreck/index.html?hpt=hp_t1'
    u'http://www.cnn.com/2013/12/07/us/life-pearl-harbor/?iref=obinsite'
    ...

    >>> print cnn_paper.category_urls
    [u'http://lifestyle.cnn.com', u'http://cnn.com/world', u'http://tech.cnn.com' ...]

    >>> print cnn_paper.feeds_urls     
    [u'http://rss.cnn.com/rss/cnn_crime.rss', ... ] 

    >>> cnn_paper.download() # Add a check, if trying to call html before downloading print msg

    >>> print cnn_paper.articles[0].title
    u'Police: 3 sisters imprisoned in Tucson home, tortured with music'

    >>> print cnn_paper.articles[0].html
    u'<!DOCTYPE HTML><html itemscope itemtype="http://...'

    >>> print cnn_paper.articles[0].top_img   # via reddit's thumbnail alg  
    u'http://some.cdn.com/3424hfd4565sdfgdg436/

    >>> print cnn_paper.articles[0].authors
    [u'<NAME>', u'Some CoAuthor']

    >>> print cnn_paper.brand
    u'cnn'

    >>> for article in cnn_paper.articles:
    >>>     article.parse() # CPU heavy & long step
    
    >>> print cnn_paper.articles[0].keywords
    [u'music', u'Tucson', ... ]

    >>> print cnn_paper.articles[0].text
    u'Three sisters who were imprisoned for possibly ... a constant barrage ...'

    >>> print cnn_paper.articles[0].summary
    u'... imprisoned for possibly ... a constant barrage ...'

Alternatively, you may use newspaper's lower level Article API.

.. code-block:: pycon

    >>> from newspaper import Article

    >>> article = Article('http://cnn.com/2013/11/27/travel/weather-thanksgiving/index.html')
    >>> article.download()

    >>> print article.html 
    u'<!DOCTYPE HTML><html itemscope itemtype="http://...'

    >>> article.parse() # Add a check, if trying to call text, etc before parsing, print msg

    >>> print article.summary
    u'...and so that is how a great Thanksgiving meal is cooked...'

    >>> print article.text
    u'The purpose of this article is to introduce to you all how to...'

    >>> print article.authors
    [u'<NAME>', u'<NAME>']

Newspaper stands on the giant shoulders of `lxml`_, `nltk`_, and `requests`_.

.. _`lxml`: https://textblob.readthedocs.org/en/latest/quickstart.html#quickstart
.. _`nltk`: https://textblob.readthedocs.org/en/latest/quickstart.html#quickstart
.. _`requests`: https://textblob.readthedocs.org/en/latest/quickstart.html#quickstart

Features
--------

- Noun phrase extraction
- Part-of-speech tagging
- Sentiment analysis
- Classification (Naive Bayes, Decision Tree)
- Language translation and detection powered by Google Translate
- Tokenization (splitting text into words and sentences)
- Word and phrase frequencies
- Parsing
- `n`-grams
- Word inflection (pluralization and singularization) and lemmatization
- Spelling correction
- JSON serialization
- Add new models or languages through extensions
- WordNet integration

Get it now
----------
::

    $ pip install newspaper

Examples
--------

See more examples at the `Quickstart guide`_.

.. _`Quickstart guide`: https://newspaper.readthedocs.org/en/latest/quickstart.html#quickstart


Documentation
-------------

Full documentation is available at https://newspaper.readthedocs.org/.

Requirements
------------

- Python >= 2.6 and <= 2.7*

License
-------

MIT licensed. See the bundled `LICENSE <https://github.com/sloria/TextBlob/blob/master/LICENSE>`_ file for more details.
<commit_msg>readme modified<commit_after>Newspaper: Article scraping & curation
======================================

.. image:: https://badge.fury.io/py/textblob.png
    :target: http://badge.fury.io/py/textblob
        :alt: Latest version

.. image:: https://pypip.in/d/textblob/badge.png
    :target: https://crate.io/packages/textblob/
        :alt: Number of PyPI downloads


Homepage: `https://newspaper.readthedocs.org/ <https://newspaper.readthedocs.org/>`_

Inspired by `requests` for its simplicity and powered by `lxml` for its speed; `newspaper` is a Python 2 library
for extracting articles from the web and curating text (NLP) for keywords, summaries, authors, etc.
Newspaper utilizes async io and caching for speed. Everything is in unicode :)

There are two API's available. The low level article API, and the newspaper API.

.. code-block:: pycon

    >>> import newspaper

    >>> cnn_paper = newspaper.build('http://cnn.com')

    >>> for article in cnn_paper.articles: 
    >>>     print article.url
    u'http://www.cnn.com/2013/11/27/justice/tucson-arizona-captive-girls/'
    u'http://www.cnn.com/2013/12/11/us/texas-teen-dwi-wreck/index.html?hpt=hp_t1'
    u'http://www.cnn.com/2013/12/07/us/life-pearl-harbor/?iref=obinsite'
    ...

    >>> print cnn_paper.category_urls
    [u'http://lifestyle.cnn.com', u'http://cnn.com/world', u'http://tech.cnn.com' ...]

    >>> print cnn_paper.feeds_urls     
    [u'http://rss.cnn.com/rss/cnn_crime.rss', ... ] 

    >>> cnn_paper.download() # DON'T FORGET

    >>> print cnn_paper.articles[0].title
    u'Police: 3 sisters imprisoned in Tucson home, tortured with music'

    >>> print cnn_paper.articles[0].html
    u'<!DOCTYPE HTML><html itemscope itemtype="http://...'

    >>> print cnn_paper.articles[0].top_img   # via reddit's thumbnail alg  
    u'http://some.cdn.com/3424hfd4565sdfgdg436/

    >>> print cnn_paper.articles[0].authors
    [u'<NAME>', u'Some CoAuthor']

    >>> print cnn_paper.brand
    u'cnn'

    >>> for article in cnn_paper.articles:
    >>>     article.parse() # CPU heavy & long step
    
    >>> print cnn_paper.articles[0].keywords
    [u'music', u'Tucson', ... ]

    >>> print cnn_paper.articles[0].text
    u'Three sisters who were imprisoned for possibly ... a constant barrage ...'

    >>> print cnn_paper.articles[0].summary
    u'... imprisoned for possibly ... a constant barrage ...'

Alternatively, you may use newspaper's lower level Article API.

.. code-block:: pycon

    >>> from newspaper import Article

    >>> article = Article('http://cnn.com/2013/11/27/travel/weather-thanksgiving/index.html')
    >>> article.download()

    >>> print article.html 
    u'<!DOCTYPE HTML><html itemscope itemtype="http://...'

    >>> article.parse() # DON'T FORGET

    >>> print article.summary
    u'...and so that is how a great Thanksgiving meal is cooked...'

    >>> print article.text
    u'The purpose of this article is to introduce to you all how to...'

    >>> print article.authors
    [u'<NAME>', u'<NAME>']

Newspaper stands on the giant shoulders of `lxml`_, `nltk`_, and `requests`_.

.. _`lxml`: https://textblob.readthedocs.org/en/latest/quickstart.html#quickstart
.. _`nltk`: https://textblob.readthedocs.org/en/latest/quickstart.html#quickstart
.. _`requests`: https://textblob.readthedocs.org/en/latest/quickstart.html#quickstart

Features
--------

- Noun phrase extraction
- Part-of-speech tagging
- Sentiment analysis
- Classification (Naive Bayes, Decision Tree)
- Language translation and detection powered by Google Translate
- Tokenization (splitting text into words and sentences)
- Word and phrase frequencies
- Parsing
- `n`-grams
- Word inflection (pluralization and singularization) and lemmatization
- Spelling correction
- JSON serialization
- Add new models or languages through extensions
- WordNet integration

Get it now
----------
::

    $ pip install newspaper

Examples
--------

See more examples at the `Quickstart guide`_.

.. _`Quickstart guide`: https://newspaper.readthedocs.org/en/latest/quickstart.html#quickstart


Documentation
-------------

Full documentation is available at https://newspaper.readthedocs.org/.

Requirements
------------

- Python >= 2.6 and <= 2.7*

License
-------

MIT licensed. See the bundled `LICENSE <https://github.com/sloria/TextBlob/blob/master/LICENSE>`_ file for more details.

{'fraction_non_alphanumeric': 0.15752493200362647, 'fraction_numerical': 0.012919310970081596, 'mean_word_length': 3.2124105011933173, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 169, 'https://': 22, 'lorem ipsum': 0, 'www.': 6, 'xml': 6}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5640224', 'n_tokens_mistral': 3014, 'n_tokens_neox': 2769, 'n_words': 716}