<commit_before>ScrapyDo
========

Crochet_-based blocking API for Scrapy_.

This module provides function helpers to run Scrapy_ in a blocking fashion. See
the `scrapydo-overview.ipynb <http://nbviewer.ipython.org/github/darkrho/scrapydo/blob/master/notebooks/scrapydo-overview.ipynb>`_
notebook for a quick overview of this module.


Installation
============

Using ``pip``::

  pip install scrapydo


Using ``conda``::

  conda install -c https://conda.anaconda.org/rolando scrapydo


Usage
=====

The function ``scrapydo.setup`` must be called once to initialize the reactor.

Example:

.. code:: python

    import scrapydo
    scrapydo.setup()

    # Fetch a single URL.
    response = scrapydo.fetch("http://example.com")
    # do stuff with response ...


    # Crawl an URL with given callback.
    def callback(response):
        yield {
            'title': response.css('title').extract(),
            'url': response.url,
        }

    items = scrapydo.crawl('http://example.com', callback)
    # do stuff with items ...

    # Run an existing spider class.
    items = scrapydo.run_spider(MySpider)

    # Run an existing spider class with custom arguments.
    formdata={'user': 'john', 'pass': '<PASSWORD>'}
    items = scrapydo.run_spider(MySpider(**formdata), **formdata)

    # do stuff with items ...


Available Functions
===================

``scrapydo.setup()``
    Initialize reactor.

``scrapydo.fetch(url, spider_cls=DefaultSpider, capture_items=True, return_crawler=False, settings=None, timeout=DEFAULT_TIMEOUT)``
    Fetches an URL and returns the response.

``scrapydo.crawl(url, callback, spider_cls=DefaultSpider, capture_items=True, return_crawler=False, settings=None, timeout=DEFAULT_TIMEOUT)``
    Crawls an URL with given callback and returns the scraped items.

``scrapydo.run_spider(spider_cls, capture_items=True, return_crawler=False, settings=None, timeout=DEFAULT_TIMEOUT)``
    Runs a spider and returns the scraped items.

``highlight(code, lexer='html', formatter='html', output_wrapper=None)``<commit_msg>Improved example in readme.<commit_after>ScrapyDo
========

Crochet_-based blocking API for Scrapy_.

This module provides function helpers to run Scrapy_ in a blocking fashion. See
the `scrapydo-overview.ipynb <http://nbviewer.ipython.org/github/darkrho/scrapydo/blob/master/notebooks/scrapydo-overview.ipynb>`_
notebook for a quick overview of this module.


Installation
============

Using ``pip``::

  pip install scrapydo


Using ``conda``::

  conda install -c https://conda.anaconda.org/rolando scrapydo


Usage
=====

The function ``scrapydo.setup`` must be called once to initialize the reactor.

Example:

.. code:: python

    import scrapydo
    scrapydo.setup()

    scrapydo.default_settings.update({
        'LOG_LEVEL': 'DEBUG',
        'CLOSESPIDER_PAGECOUNT': 10,
    })

    # Enable logging display
    import logging
    logging.basicConfig(level=logging.DEBUG)

    # Fetch a single URL.
    response = scrapydo.fetch("http://example.com")

    # Crawl an URL with given callback.
    def parse_page(response):
        yield {
            'title': response.css('title').extract(),
            'url': response.url,
        }
        for href in response.css('a::attr(href)'):
            url = response.urljoin(href)
            yield Request(url, callback=parse_page)

    items = scrapydo.crawl('http://example.com', callback)

    # Run an existing spider class.
    items = scrapydo.run_spider(MySpider)

    # Run an existing spider class with custom arguments.
    formdata={'user': 'john', 'pass': '<PASSWORD>'}
    items = scrapydo.run_spider(MySpider(**formdata), **formdata)


Available Functions
===================

``scrapydo.setup()``
    Initialize reactor.

``scrapydo.fetch(url, spider_cls=DefaultSpider, capture_items=True, return_crawler=False, settings=None, timeout=DEFAULT_TIMEOUT)``
    Fetches an URL and returns the response.

``scrapydo.crawl(url, callback, spider_cls=DefaultSpider, capture_items=True, return_crawler=False, settings=None, timeout=DEFAULT_TIMEOUT)``
    Crawls an URL with given callback and returns the scraped items.

``scrapydo.run_spider(spider_cls, capture_items=True, return_crawler=False, settings=None, timeout=DEFAULT_TIMEOUT)``
    Runs a spider and returns the scraped items.

``highlight(code, lexer='html', formatter='html', output_wrapper=None)``
{'fraction_non_alphanumeric': 0.1322219684859557, 'fraction_numerical': 0.00045672527974423386, 'mean_word_length': 4.0402761795166855, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2578285', 'n_tokens_mistral': 1425, 'n_tokens_neox': 1343, 'n_words': 350}