<commit_before>The Public Whip Source Code
---------------------------

Hello!  Here's the source code used to generate the Public Whip website.
To see the end product go to http://www.publicwhip.org.uk.  If you don't
know what this is all about, have a look at the FAQ there.

To learn how to use the code look at
http://www.publicwhip.org.uk/project/code.php or locally in
webpage/project/code.php.  A description of the files and folders in
this package follows.

LICENSE.html - Details of open source licensing terms, under the GNU GPL
todo.txt - Things I'm thinking of doing in the short term
ideas.txt - Zillions of ideas of things which could be done
errata.txt - Errors in Hansard that the software has found

scraper   - Screen scrapes Hansard website to fill the database, some analysis
rawdata   - Source data files not previously available on the Internet
cluster   - MP cluster analysis using Multi-dimensional Scaling
website   - Code for www.publicwhip.org.uk, PHP extracts data from database
build     - Scripts I use to upload to www.publicwhip.org.uk

members   - MP list as one fat XML file
pythonexp - New experiments (using Python)


If you need any help, please email me <EMAIL>.<commit_msg>Updated to explain new scraper<commit_after>The Public Whip Source Code
---------------------------

Hello!  Here's the source code used to generate the Public Whip website.
To see the end product go to http://www.publicwhip.org.uk.  If you don't
know what this is all about, have a look at the FAQ there.

To learn how to use the code look at
http://www.publicwhip.org.uk/project/code.php or locally in
webpage/project/code.php.  A description of the files and folders in
this package follows.

LICENSE.html - Details of open source licensing terms, under the GNU GPL
todo.txt - Things I'm thinking of doing in the short term
ideas.txt - Zillions of ideas of things which could be done
errata.txt - Errors in Hansard that the software has found

scraper   - Screen scraper which directly fills the database with divisions
rawdata   - Source data files not previously available on the Internet
cluster   - MP cluster analysis using Multi-dimensional Scaling

members   - MP list as one fat XML file (used by pyscraper)
pyscraper - Hansard screen scraper which makes XML files, does Written Answers
pyindex   - Indexing code for XML made by pyscraper

website   - Code for www.publicwhip.org.uk, PHP extracts data from database/XML
build     - Scripts I use for admin, such as to upload to www.publicwhip.org.uk

If you need any help, please email me <EMAIL>.
{'fraction_non_alphanumeric': 0.07040062232594321, 'fraction_numerical': 0.0, 'mean_word_length': 3.3741496598639458, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 8, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7092301', 'n_tokens_mistral': 715, 'n_tokens_neox': 692, 'n_words': 369}