<jupyter_start><jupyter_text>Downloading master data from Reference Data Repository**openclean** integrates the [refdata package](https://github.com/VIDA-NYU/reference-data-repository) to provides easy access to several different reference datasets that are available online for download. Reference datasets are for example a great source for lookup tables and mappings that are used in data cleaning for outlier detection and data standardization.<jupyter_code># Setup the environment for this demo. All files will be
# stored in a subfolder data.

import os

from openclean.config import ENV_DATA_DIR
from refdata.config import ENV_BASEDIR

os.environ[ENV_DATA_DIR] = './data/archives'
os.environ[ENV_BASEDIR] = './data/refdata'
# Import the amasterdata and reference data modules.

import openclean.data.archive.base as masterdata
import openclean.data.refdata as refdata
# Print a listing for all dataset in the default
# reference data repository.

for dataset in refdata.repository():
    print('{} ({})'.format(dataset.name, dataset.identifier))<jupyter_output>Company Suffixes (company_suffixes)
Cities in the U.S. (encyclopaedia_britannica:us_cities)
NYC Finance - State Codes (nyc.gov:dof:state_codes)
REST Countries (restcountries.eu)
C1 Street Suffix Abbreviations (usps:street_abbrev)
C2 Secondary Unit Designators (usps:secondary_unit_designators)
States and territories of the U.S. (wikipedia:us_states)<jupyter_text>restcountries.euDataset of countries in the world that is available from the [restcountries.eu project](http://restcountries.eu/).<jupyter_code># Download the countries dataset.

refdata.download('restcountries.eu')
# Print the first rows in the countries dataset.

countries = refdata.load('restcountries.eu').df()

print(countries.head())
# Print entry for 'Venezuela'

countries.loc[countries['name'] == 'Venezuela (Bolivarian Republic of)']<jupyter_output><empty_output><jupyter_text>Modified Mastedata CopiesThe user has the option to modify the downloaded data and store the updated dataset version in the local masterdata repository.<jupyter_code># Create a local masterdata archive for the countries dataset.

archive = masterdata.create('restcountries', source=countries, primary_key=['alpha3Code'], replace=True)
# Rename 'Venezuela (Bolivarian Republic of)' to 'Venezuela'.

from openclean.operator.transform.update import update

countries = update(countries, 'name', {'Venezuela (Bolivarian Republic of)': 'Venezuela'})
countries.loc[countries['name'] == 'Venezuela']
countries.head()
# Store the modified dataset as a new snapshot in the masterdata
# archive.

archive = masterdata.get('restcountries')
archive.commit(countries)
countries = masterdata.get('restcountries').checkout()
countries.loc[countries['name'] == 'Venezuela']
snapshots = masterdata.get('restcountries').snapshots()
for s in snapshots:
    print(s)
prov = masterdata.get('restcountries').diff(0, 1)
prov.describe()
row = prov.rows().update()[0]
old_val, new_val = row.cells[0].values()

print("Country name updated from '{}' to '{}'".format(old_val, new_val))<jupyter_output>Country name updated from 'Venezuela (Bolivarian Republic of)' to 'Venezuela'<jupyter_text>Encyclopaedia BritannicaDatasets that are extracted from Web pages of the Encyclopaedia Britannica.<jupyter_code># Get dataset with US city names. Used the auto_download
# option that will fetch the dataset if it has not been
# downloaded yet.

us_cities = refdata.load(key='encyclopaedia_britannica:us_cities', auto_download=True).df()
us_cities.head()<jupyter_output><empty_output><jupyter_text>Cleanup<jupyter_code># Remove the created data folder.

import shutil

shutil.rmtree('./data')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1020242914979757, 'fraction_numerical': 0.0018893387314439945, 'mean_word_length': 6.704781704781705, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '202669', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1142, 'n_tokens_neox': 1123, 'n_words': 337, 'path': 'examples/notebooks/Masterdata.ipynb', 'repo_name': 'remram44/openclean-core', 'size': '3705'}