<commit_before>===============================
pyvideo_scraper
===============================

.. image:: https://badge.fury.io/py/pyvideo_scraper.png
    :target: http://badge.fury.io/py/pyvideo_scraper
    
.. image:: https://travis-ci.org/johnnycakes79/pyvideo_scraper.png?branch=master
        :target: https://travis-ci.org/johnnycakes79/pyvideo_scraper

.. image:: https://pypip.in/d/pyvideo_scraper/badge.png
        :target: https://crate.io/packages/pyvideo_scraper?version=latest


What will this do...?

* Free software: BSD license
* Documentation: http://pyvideo_scraper.rtfd.org.

Features
--------

* TODO<commit_msg>Update README.rst<commit_after>===============================
PyVideo Scraper
===============================

Building on the `Gist of Miguel Grinberg`_ and merging with my own fork of <NAME>'s `pytube`_ I've hacked this together to scan `pyvideo.org`_ pages and download the video content from youtube.

To get going grab my fork of `pytube`_ (it's been slightly modified to have a non-verbose mode) run python setup.py install in the pylab directory and then all you have to do is download this repo - it doesn't need to be installed...

.. _Gist of Miguel Grinberg: https://gist.github.com/miguelgrinberg/5f52ceb565264b1e969a
.. _pytube: https://github.com/johnnycakes79/pytube
.. _pyvideo.org: http://pyvideo.org/

Features
--------

* TODO: Fix this repo with proper tests and documentation.
* TODO: Generalise the scirpt to also 'search' youtube and download the results.
{'fraction_non_alphanumeric': 0.19047619047619047, 'fraction_numerical': 0.012566137566137565, 'mean_word_length': 4.2717770034843205, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1712214', 'n_tokens_mistral': 506, 'n_tokens_neox': 475, 'n_words': 133}