<commit_before>onsstat
=======

A website for graphing datasets from the Office of National Statistics.

What Does it Do?
================

The ONS has an extensive set of wonderful [datasets](http://www.ons.gov.uk/ons/datasets-and-tables/index.html?content-type=Dataset&pubdateRangeType=allDates&sortBy=pubdate&sortDirection=DESCENDING&newquery=*&pageSize=50&applyFilters=true&content-type-orig=%22Dataset%22+OR+content-type_original%3A%22Reference+table%22) covering a wide range
of economic and social data. Unfortunately, there was way to easily search or 
view this data. Searching for 'unemployment wales' would return a large number of
datasets but no information about the individual data they contained. Data on, for 
instance, manufacturing might be indexed by 'manufacturing' or perhaps simply 'manuf',
but there is no way to view datasets for both together, or to discover 'manuf' without
having studied the 33 thousand CDID names. To view the data, each dataset must be 
downloaded, the CDID identifiers deciphered, and the results plotted. Data is repeated 
over and over for each CDID, but not all data for a CDID is a repeat, and there is no
alternative to examining many - often dozens - of datasets to attain a complete 
picture.

This website is designed to change that.

The data from all ONS datasets has been parsed and aggregated, much redundancy 
has been eliminated, and CDID names can be searched in a fast and intuitive manner.
Data can be plotted and compared instantly.


Code Structure
==============

* webapp.py

--* This is the central flask application. It serves static pages, data for plotting in a 
REST manner, and serves the websocket links for live-search.

* namematch.py

--* Implements a suffix tree for live-search of CDID names. Also initializes the suffix tree
from the database and presents an object-oriented interface for searching the tree, 
including ordering of results.


* datacache.py

--* Object-oriented cache for data (perhaps you guessed that one) needed by live search for each CDID.

* static/datasets.js

--* Implements the clientside interface for querying the backend and logic to render and plot the 
resulting data.

* mydb.py

--* Handles some common database interaction.

*csv data parsing.ipynb

--* An ipython notebook for parsing the raw datasets and updating the database.

* ons-retrieval.py

--* Retrieves the datasets from the ONS website.<commit_msg>enhanced ReadMe<commit_after>onsstat
=======

A website for graphing datasets from the Office of National Statistics.

What Does it Do?
================

The ONS has an extensive set of wonderful [datasets](http://www.ons.gov.uk/ons/datasets-and-tables/index.html?content-type=Dataset&pubdateRangeType=allDates&sortBy=pubdate&sortDirection=DESCENDING&newquery=*&pageSize=50&applyFilters=true&content-type-orig=%22Dataset%22+OR+content-type_original%3A%22Reference+table%22) covering a wide range
of economic and social data. Unfortunately, there was way to easily search or 
view this data. Searching for 'unemployment wales' would return a large number of
datasets but no information about the individual data they contained. Data on, for 
instance, manufacturing might be indexed by 'manufacturing' or perhaps simply 'manuf',
but there is no way to view datasets for both together, or to discover 'manuf' without
having studied the 33 thousand CDID names. To view the data, each dataset must be 
downloaded, the CDID identifiers deciphered, and the results plotted. Data is repeated 
over and over for each CDID, but not all data for a CDID is a repeat, and there is no
alternative to examining many - often dozens - of datasets to attain a complete 
picture.

This website is designed to change that.

The data from all ONS datasets has been parsed and aggregated, much redundancy 
has been eliminated, and CDID names can be searched in a fast and intuitive manner.
Data can be plotted and compared instantly.


Code Structure
==============

* webapp.py

-- This is the central flask application. It serves static pages, data for plotting in a 
REST manner, and serves the websocket links for live-search.

* namematch.py

Implements a suffix tree for live-search of CDID names. Also initializes the suffix tree
from the database and presents an object-oriented interface for searching the tree, 
including ordering of results.


* datacache.py

Object-oriented cache for data (perhaps you guessed that one) needed by live search for each CDID.

* static/datasets.js

Implements the clientside interface for querying the backend and logic to render and plot the 
resulting data.

* mydb.py

Handles some common database interaction.

*csv data parsing.ipynb

An ipython notebook for parsing the raw datasets and updating the database.

* ons-retrieval.py

Retrieves the datasets from the ONS website.
{'fraction_non_alphanumeric': 0.06768784930656178, 'fraction_numerical': 0.005381908507555371, 'mean_word_length': 4.725118483412322, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3150867', 'n_tokens_mistral': 1324, 'n_tokens_neox': 1264, 'n_words': 638}