<jupyter_start><jupyter_text>Parse International Census Data from United States Census Bureau Dataset gateway is [here](https://www.census.gov/data-tools/demo/idb/informationGateway.php), and the full dataset can be downloaded [here](https://www2.census.gov/programs-surveys/international-programs/about/idb/idbzip.zip), see [here](https://www.census.gov/programs-surveys/international-programs/about/idb/faq.htmlpar_textimage_2).See also Google's [COVID-19 public datasets](https://console.cloud.google.com/marketplace/details/bigquery-public-datasets/covid19-public-data-program?_ga=2.118817228.-458258001.1586199431&pli=1)<jupyter_code>import pandas as pd
from os.path import join<jupyter_output><empty_output><jupyter_text>Download census data<jupyter_code>%sx wget -O ./site-data/census-data.zip https://www2.census.gov/programs-surveys/international-programs/about/idb/idbzip.zip
%sx unzip ./site-data/census-data.zip -d ./site-data/census-data<jupyter_output><empty_output><jupyter_text>Get population size of every country of the world<jupyter_code>censusPath = join('site-data', 'census-data', 'idbzip')
countryCodes = pd.read_csv(join(censusPath, 'IDBextCTYS.txt'), sep='|', 
            names=['countryCode','countryName','landArea'])
countryCodes.head()
population = pd.read_csv(join(censusPath, 'IDBext001.txt'), sep='|', 
                         names=['countryCode', 'year', 'midYearPopulation'])
currentYear = pd.Timestamp.today().year
populationCurrent = population[population.year == currentYear]
populationCurrent.head()
populationWorld = pd.merge(countryCodes, populationCurrent, on = 'countryCode', how='left')
populationWorld = populationWorld[['countryCode', 'countryName', 'midYearPopulation']]
assert populationWorld['midYearPopulation'].isnull().sum() == 0
populationWorld.head()
populationWorld.shape
populationWorld.to_csv(join('site-data','census-data', 'populationCountries.csv'), index=False)<jupyter_output><empty_output><jupyter_text>Get age & sex distribution of every country of the world<jupyter_code>ageSex = pd.read_csv(join(censusPath, 'IDBext094.txt'), sep='|', 
           names=['countryCode', 'year', 'totalFlag', 'startAge', 'isOpen', 'endAge', 
                  'midYearPopulationAll', 'midYearPopulationMale', 'midYearPopulationFemale'])

ageSex = ageSex[(ageSex.year == currentYear) & (ageSex.totalFlag == 'A')]
ageSexWorld = pd.merge(countryCodes, ageSex, on = 'countryCode', how='left')
ageSexWorld = ageSexWorld[['countryCode', 'countryName', 'startAge', 'isOpen', 
                                 'endAge', 'midYearPopulationMale', 'midYearPopulationFemale']]
assert ageSexWorld['midYearPopulationMale'].isnull().sum() == 0
assert ageSexWorld['midYearPopulationFemale'].isnull().sum() == 0
ageSexWorld.head()
ageSexWorld.shape
ageSexWorld.to_csv(join('site-data','census-data', 'ageSexCountries.csv'), index=False)<jupyter_output><empty_output><jupyter_text>Clean up<jupyter_code>from os import remove
from shutil import rmtree
remove(join('./site-data', 'census-data.zip'))
remove('../.wget-hsts')
rmtree(join('./site-data', 'census-data', 'idbzip'))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15566188197767145, 'fraction_numerical': 0.01467304625199362, 'mean_word_length': 6.448931116389549, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 5, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '396258', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1097, 'n_tokens_neox': 1007, 'n_words': 159, 'path': 'work/census-data.ipynb', 'repo_name': 'VirusTrack/COVIDvu', 'size': '3135'}