<jupyter_start><jupyter_text>Data> Original raw data comes in CSV format, every year of data in a separate file. In this module, we construct schema that adheres to [Frictionless Data specificaitons](https://frictionlessdata.io/specs/), validate each table and erase problematic entries.<jupyter_code># export
import os
import subprocess
from pathlib import Path<jupyter_output><empty_output><jupyter_text>Original dataCreate symlinks from data location to "./in".  Preprocessing Convert to UTF-8Original data come in unknown encoding. Make best guess and save in UTF-8. Validate against schema[Schema spec](https://frictionlessdata.io/specs/table-schema/)That code is in a separate repo, and is possibly outdated. For now download validated and cleaned files from GCS.```mkdir -p data/csvgsutil -m cp -r gs://info-group-corr/*.csv data/csv/```The last 2 years (2016 and 2017) are missing.The schema is a separate JSON file, and can be used to assign correct data types when importing into BigQuery, Stata, pandas or any other dtype aware format.<jupyter_code># notest
# hide

# Separate schemas into separate files, one file per year.
# Some columns are missing in certain years, and this is captured in "field_lists" fields of the JSON schema.

import json

def make_schema(y):
    '''Prepare year-specific schema.'''
    with open('data/csv/schema.json') as f:
        schema = json.load(f)
    for fl in schema['field_lists']:
        if y in fl['years']:
            used_fields = fl['fields']
            break
    new_fields = []
    for f in schema['fields']:
        if f['name'] in used_fields:
            new_fields.append(f)
    schema['fields'] = new_fields
    del schema['field_lists']
    return schema

for y in range(1997, 2016):
    with open(f'data/csv/{y}_schema.json', 'w') as f:
        json.dump(make_schema(y), f)<jupyter_output><empty_output><jupyter_text>ExtractsIt is handy to have smaller files for testing. Multiple approaches can be used to extract subsets in longitudinal data.  First 100k recordsThis is probably unneccessary, because `pd.read_csv(nrows=100000)` does just that.<jupyter_code># export
 
def extract_100k():
    """Create extract with header and first 100k records."""
    dir_in = Path('./out/valid')
    dir_out = Path('./out/extracts/100k')
    for fn_in in dir_in.glob('*.csv'):
        fn_out = dir_out / fn_in.name
        with open(fn_out, 'w') as fout:
            subprocess.run(['head', '-n', '100001', fn_in], stdout=fout)
# notest
extract_100k()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10449526813880126, 'fraction_numerical': 0.018138801261829655, 'mean_word_length': 3.9357976653696496, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 14, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '621164', 'license': 'Apache-2.0', 'n_tokens_mistral': 810, 'n_tokens_neox': 755, 'n_words': 277, 'path': 'nbs/00_core.ipynb', 'repo_name': 'antonbabkin/ig_format', 'size': '2536'}