<jupyter_start><jupyter_text>This IPython notebook illustrates how to read the CSV files from disk as tables and set their metadata. First, we need to import *py_entitymatching* package and other libraries as follows:<jupyter_code>import py_entitymatching as em
import pandas as pd
import os, sys<jupyter_output>/Users/pradap/miniconda3/lib/python3.5/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)<jupyter_text>Get the Path of the CSV File  the paths of the CSV file in the disk. For the convenience of the user, we have included some sample files in the package. The path of a sample CSV file can be obtained like this:<jupyter_code># Get the datasets directory
datasets_dir = em.get_install_path() + os.sep + 'datasets'

# Get the paths of the input tables
path_A = datasets_dir + os.sep + 'person_table_A.csv'
# Display the contents of the file in path_A
!cat $path_A | head -3<jupyter_output>ID,name,birth_year,hourly_wage,address,zipcode
a1,<NAME>,1989,30,"607 From St, San Francisco",94107
a2,<NAME>,1988,27.5,"1652 Stockton St, San Francisco",94122<jupyter_text>Ways to Read a CSV File and Set Metadata There are three different ways to read a CSV file and set  metadata:1. Read a CSV file first, and then set the metadata2. Read a CSV file and set the metadata together3. Read a CSV file and set the metadata from a file in disk   Read the CSV file First and Then Set the Metadata First, read the CSV files as follows:<jupyter_code>A = em.read_csv_metadata(path_A)

A.head()
# Display the 'type' of A 
type(A)<jupyter_output><empty_output><jupyter_text>Then set the metadata for the table. We see `ID` is the key attribute (since it contains unique values and no value is missing) for the table. We can set this metadata as follows:<jupyter_code>em.set_key(A, 'ID')
# Get the metadata that were set for table A
em.get_key(A)<jupyter_output><empty_output><jupyter_text>Now the CSV file is read into the memory and the metadata (i.e. key) is set for the table.   Read a CSV File and Set Metadata Together In the above, we saw that we first read in the CSV file and then set the metadata. These two steps can be combined into a single step like this:<jupyter_code>A = em.read_csv_metadata(path_A, key='ID')

# Display the 'type' of A
type(A)
# Get the metadata that were set for the table A 
em.get_key(A)<jupyter_output><empty_output><jupyter_text>Read a CSV File and Set Metadata from a File in Disk The user can specify the metadata in a file.This file *MUST* be in the same directory as the CSV file and the file name should be same, except the extension is set to '.metadata'.<jupyter_code># We set the metadata for table A (stored in person_table_A.CSV).

# Get the path where the metadata file must be stored
metadata_file = datasets_dir + os.sep + 'person_table_A.metadata'

# Specify the metadata for table A . Here we specify that 'ID' is the key attribute for the table. Note that this step 
# requires write permission to the datasets directory.
!echo '#key=ID' > $metadata_file

# Read the CSV file for table A
A = em.read_csv_metadata(path_A)
em.get_key(A)
!rm $metadata_file<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06777713957495692, 'fraction_numerical': 0.014359563469270534, 'mean_word_length': 4.933560477001704, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '247167', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1103, 'n_tokens_neox': 1049, 'n_words': 489, 'path': 'notebooks/guides/.ipynb_checkpoints/Reading the CSV Files from Disk-checkpoint.ipynb', 'repo_name': 'kvpradap/py_entitymatching', 'size': '3497'}