<jupyter_start><jupyter_text>This notebooks shows basic techniques for reading data in multiple formats into a Pandas DataFrame. First we will explain basic techniques using data stored in CSV.<jupyter_code>%%writefile data.csv
a,b,c,d,message
1,2,3,4,hello
5,6,7,8,world
9,10,11,12,foo<jupyter_output>Overwriting data.csv<jupyter_text>The simplest thing we can do we a CSV file is reading it as is. In this case:* The first row will be considered the column names* The following rows are directly read as rows in the DataFrame<jupyter_code>filename = "data.csv"
df = pd.read_csv(filename)
df<jupyter_output><empty_output><jupyter_text>Pandas can also read CSV files that don't have a header. In this case, columns will be a numeric Index.<jupyter_code>%%writefile data_no_header.csv
1,2,3,4,hello
5,6,7,8,world
9,10,11,12,foo
filename = "data_no_header.csv"
df = pd.read_csv(filename, header=None)
df<jupyter_output><empty_output><jupyter_text>We can specify the column names as well. Specifying column names implies that no header row is present<jupyter_code>filename = "data_no_header.csv"
df = pd.read_csv(filename, names=['a', 'b', 'c', 'd', 'message'])
df<jupyter_output><empty_output><jupyter_text>We can also use one of the columns in the data as the index. In this case the column used as index is not includes as data as well.<jupyter_code>filename = "data_no_header.csv"
names=['a', 'b', 'c', 'd', 'message']
df = pd.read_csv(filename, names=names, index_col='message')
df<jupyter_output><empty_output><jupyter_text>Unsing this technique it is also possible to create hierachical indexes.<jupyter_code>%%writefile data_hierarchy.csv
1,1,3,4
1,2,3,4
2,4,2,5
2,4,5,6
2,1,1,1
filename = "data_hierarchy.csv"
names=['a', 'b', 'c', 'd']
df = pd.read_csv(filename, names=names, index_col=['a','b'])
df
# See documentation of pd.read_csv for extra arguments
help(pd.read_csv)<jupyter_output>Help on function read_csv in module pandas.io.parsers:

read_csv(filepath_or_buffer:Union[str, pathlib.Path, IO[~AnyStr]], sep=',', delimiter=None, header='infer', names=None, index_col=None, usecols=None, squeeze=False, prefix=None, mangle_dupe_cols=True, dtype=None, engine=None, converters=None, true_values=None, false_values=None, skipinitialspace=False, skiprows=None, skipfooter=0, nrows=None, na_values=None, keep_default_na=True, na_filter=True, verbose=False, skip_blank_lines=True, parse_dates=False, infer_datetime_format=False, keep_date_col=False, date_parser=None, dayfirst=False, cache_dates=True, iterator=False, chunksize=None, compression='infer', thousands=None, decimal:str='.', lineterminator=None, quotechar='"', quoting=0, doublequote=True, escapechar=None, comment=None, encoding=None, dialect=None, error_bad_lines=True, warn_bad_lines=True, delim_whitespace=False, low_memory=True, memory_map=False, float_precision=None)
    Read a comma-separated values (csv) file into D[...]<jupyter_text>With pandas we can also read a file in chunks. This is useful when the contents of the file don't fint in memory at once.<jupyter_code>filename = "big_data.csv"
with open(filename, "wt") as f:
    for i in range(10000):
        f.write("1,2,3\n")
# Accumulate value counts chunk by chunk
tot = pd.Series([])
chunks = pd.read_csv(filename, chunksize=1000)
for chunk in chunks:
    tot = tot.add(chunk['1'].value_counts(), fill_value=0)
tot<jupyter_output>/usr/local/lib/python3.6/site-packages/ipykernel_launcher.py:2: DeprecationWarning: The default dtype for empty Series will be 'object' instead of 'float64' in a future version. Specify a dtype explicitly to silence this warning.
{'fraction_non_alphanumeric': 0.12037809285515708, 'fraction_numerical': 0.019738671114817904, 'mean_word_length': 7.252293577981652, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '453877', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 1266, 'n_tokens_neox': 1231, 'n_words': 346, 'path': 'notebooks/pandas/Data loading with Pandas.ipynb', 'repo_name': 'julitopower/LearnMachineLearning', 'size': '3597'}