<commit_before>VidScraper: A simple python library for scraping common video sites

* About

VidScraper is a clean, simple library for a couple of rather messy
issues:
 - Retrieving the source video from a "flash-only" website
 - Finding out contextual data about a pasted url.. title/description/etc

Vidscraper provides a unified api for an issue that requires a lot of
one-off scraping.

* Usage

** Simple usage: auto_scrape

Most use cases will simply require the auto_scrape function.  Usage is
incredibly easy;

: >>> from vidscraper import auto_scrape
: >>> data = auto_scrape(my_url)

That's it!  Couldn't be eaiser.  auto_scrape will figure out what
scraping methods map to your url and return a dictionary full of what
information it can find out about the data.  (If it can't, it'll
return a [[CantIdentifyUrl]] error.. see [[Handling errors]] below.)

auto_scrape returns a dictionary with fields that represent the data
it knows how to figure out for that site (some sites might not support
some fields... in that case, auto_scrape simply won't include those
fields in the dictionary).  Common fields are:
 - *title:* title of the video
 - *description:* description of the video
 - *submit_date:* date the video was submitted
 - *file_url:* The url of the source video file

** Scrape from a particular site: scrape_suite

If for some reason you know the particular site you want to scrape
from, you can use the site's "suite".  The suite is a dictionary that
contains:
 - *regex*: the regular expression that identifies a url as
   processable by this suite
 - *funcs*: a dictionary of field names to methods that retrieve those
   field names
 - *order*: (not required) the order that fields should be processed
   in (for optimization's sake)

But you don't really need to know about any of that.  You can just
pass in the suite to the scrape_suite function.

: >>> from vidscraper import scrape_suite
: >>> from vidscraper.sites import youtube
: >>> data = scrape_suite(url, youtube.SUITE)

(If you want, you can possibly use the 'regex' field to verify that
your url actually will work with said suite first.)

** Retrieving only certain fields

If you only need certain fields (say you only need the "file_url"
and the "title" fields), you can potentially save some unnecessary
work by passing in a list of fields to either the 'auto_scrape' or the
'scrape_suite' function (one you're using).

: >>> data = auto_scrape(url, fields=['file_url', 'title'])
: # or
: >>> data = scrape_suite(url, youtube.SUITE, fields=['file_url', 'title'])

** Calling the methods manually

Nothing magical about it... they just take the url as their first argument.

: >>> from vidscraper.sites import vimeo
: >>> vimeo.scrape_title('http://www.vimeo.com/1084537')
: 'Big Buck Bunny'

** Handling errors
*** BaseUrlLoadFailure
Error if you can't even load the base url
*** ParsingError<commit_msg>A line about exceptions<commit_after>VidScraper: A simple python library for scraping common video sites

* About

VidScraper is a clean, simple library for a couple of rather messy
issues:
 - Retrieving the source video from a "flash-only" website
 - Finding out contextual data about a pasted url.. title/description/etc

Vidscraper provides a unified api for an issue that requires a lot of
one-off scraping.

* Usage

** Simple usage: auto_scrape

Most use cases will simply require the auto_scrape function.  Usage is
incredibly easy;

: >>> from vidscraper import auto_scrape
: >>> data = auto_scrape(my_url)

That's it!  Couldn't be eaiser.  auto_scrape will figure out what
scraping methods map to your url and return a dictionary full of what
information it can find out about the data.  (If it can't, it'll
return a [[CantIdentifyUrl]] error.. see [[Handling errors]] below.)

auto_scrape returns a dictionary with fields that represent the data
it knows how to figure out for that site (some sites might not support
some fields... in that case, auto_scrape simply won't include those
fields in the dictionary).  Common fields are:
 - *title:* title of the video
 - *description:* description of the video
 - *submit_date:* date the video was submitted
 - *file_url:* The url of the source video file

** Scrape from a particular site: scrape_suite

If for some reason you know the particular site you want to scrape
from, you can use the site's "suite".  The suite is a dictionary that
contains:
 - *regex*: the regular expression that identifies a url as
   processable by this suite
 - *funcs*: a dictionary of field names to methods that retrieve those
   field names
 - *order*: (not required) the order that fields should be processed
   in (for optimization's sake)

But you don't really need to know about any of that.  You can just
pass in the suite to the scrape_suite function.

: >>> from vidscraper import scrape_suite
: >>> from vidscraper.sites import youtube
: >>> data = scrape_suite(url, youtube.SUITE)

(If you want, you can possibly use the 'regex' field to verify that
your url actually will work with said suite first.)

** Retrieving only certain fields

If you only need certain fields (say you only need the "file_url"
and the "title" fields), you can potentially save some unnecessary
work by passing in a list of fields to either the 'auto_scrape' or the
'scrape_suite' function (one you're using).

: >>> data = auto_scrape(url, fields=['file_url', 'title'])
: # or
: >>> data = scrape_suite(url, youtube.SUITE, fields=['file_url', 'title'])

** Calling the methods manually

Nothing magical about it... they just take the url as their first argument.

: >>> from vidscraper.sites import vimeo
: >>> vimeo.scrape_title('http://www.vimeo.com/1084537')
: 'Big Buck Bunny'

** Handling errors
Exceptions are defined in the vidscraper.errors module.

*** BaseUrlLoadFailure
Error if you can't even load the base url
*** ParsingError
{'fraction_non_alphanumeric': 0.07706734867860188, 'fraction_numerical': 0.0023870417732310316, 'mean_word_length': 4.352189781021898, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1404698', 'n_tokens_mistral': 1722, 'n_tokens_neox': 1623, 'n_words': 825}