<commit_before># Strucutre

This is the sructure of the data that the visualizaiton will ultimately use. We need to get our data in to this format.

There are three csv's. One which has the actual data (data_values.csv), one which contains information about each variable (data_variables.csv), and one which contains information about geographic locations (data_geo.csv).

We can denormalize these in to a single table for consumption of the data viz, but these should be kept as three different csv's so, for example, if we change the description of a variable we only have to do it in one place.

## data_values.csv

This actually contains the values of the data

The fields are:

<ol>
	<li>GeoID: this corresponds to Id2 from the census data, like 6001, should match GeoId in data_geo.csv</li>
	<li>VariableID: the id we have given to the variable, should match VariableId in data_variables.csv</li>
	<li>Date: this should be a date of format yyyy-mm-dd</li>
	<li>Value: the value of the variable, for that geo, for that date</li>
</ol>

## data_variables.csv

This contains information about each variable used in our visualization

<ol>
	<li>VariableID: the id we have given the variable</li>
	<li>VariableName: the name of the variable as we want it to appear in the visualizaiton</li>
	<li>VariableExplanation: longer explanation of the variable as it will appear in the explanatory text</li>
</ol>

## data_geos.csv

This contains information about each geographic location

<ol>
	<li>GeoID: this corresponds to Id2 from the census data, like 6001</li>
	<li>LongID: this corresponds to Id1 form the census data, 0500000US06001</li>
	<li>LongName: this corresponds to Geography from the census data, like "Alameda County, California"</li>
	<li>ShortName: this the geography name as we want it displayed in th viz, like "Alameda"</li>
</ol><commit_msg>fix typos<commit_after># Structure

This is the sructure of the data that the visualizaiton will ultimately use. We need to get our data in to this format.

There are three csv's. One which has the actual data (data_values.csv), one which contains information about each variable (data_variables.csv), and one which contains information about geographic locations (data_geos.csv).

We can denormalize these in to a single table for consumption by the data viz, but these should be kept as three different csv's so, for example, if we change the description of a variable we only have to do it in one place.

## data_values.csv

This actually contains the values of the data

The fields are:

<ol>
	<li>GeoID: this corresponds to Id2 from the census data, like 6001, should match GeoId in data_geo.csv</li>
	<li>VariableID: the id we have given to the variable, should match VariableId in data_variables.csv</li>
	<li>Date: this should be a date of format yyyy-mm-dd</li>
	<li>Value: the value of the variable, for that geo, for that date</li>
</ol>

## data_variables.csv

This contains information about each variable used in our visualization

<ol>
	<li>VariableID: the id we have given the variable</li>
	<li>VariableName: the name of the variable as we want it to appear in the visualizaiton</li>
	<li>VariableDescription: longer explanation of the variable as it will appear in the explanatory text</li>
</ol>

## data_geos.csv

This contains information about each geographic location

<ol>
	<li>GeoID: this corresponds to Id2 from the census data, like 6001</li>
	<li>LongID: this corresponds to Id1 form the census data, 0500000US06001</li>
	<li>LongName: this corresponds to Geography from the census data, like "Alameda County, California"</li>
	<li>ShortName: this the geography name as we want it displayed in th viz, like "Alameda"</li>
</ol>
{'fraction_non_alphanumeric': 0.07217883113385402, 'fraction_numerical': 0.012388903851333153, 'mean_word_length': 4.678899082568807, 'pattern_counts': {'":': 0, '<': 59, '<?xml version=': 0, '>': 59, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4959844', 'n_tokens_mistral': 1123, 'n_tokens_neox': 1057, 'n_words': 542}