<jupyter_start><jupyter_text>Get data from CSVsIn this exercise, you'll create a data frame from a CSV file. The United States makes available CSV files containing tax data by ZIP or postal code, allowing us to analyze income information in different parts of the country. We'll focus on a subset of the data, `vt_tax_data_2016.csv`, which has select tax statistics by ZIP code in Vermont in 2016.To load the data, you'll need to import the `pandas` library, then read `vt_tax_data_2016.csv` and assign the resulting data frame to a variable. Then we'll have a look at the data.Instructions1. Import the `pandas` library as `pd`.2. Use `read_csv()` to load `vt_tax_data_2016.csv` and assign it to the variable `data`.3. View the first few lines of the data frame with the `head()` method. This code has been written for you.<jupyter_code># Import pandas as pd
import pandas as pd

# Read the CSV and assign it to the variable data
data = pd.read_csv('vt_tax_data_2016.csv')

# View the first few lines of data
data.head()<jupyter_output><empty_output><jupyter_text>Get data from other flat filesWhile CSVs are the most common kind of flat file, you will sometimes find files that use different delimiters. `read_csv()` can load all of these with the help of the `sep` keyword argument. By default, `pandas` assumes that the separator is a comma, which is why we do not need to specify `sep` for CSVs.The version of Vermont tax data here is a tab-separated values file (TSV), so you will need to use `sep` to pass in the correct delimiter when reading the file. Remember that tabs are represented as `\t`. Once the file has been loaded, the remaining code groups the `N1` field, which contains income range categories, to create a chart of tax returns by income category.Instructions1. Import `pandas` with the alias `pd`.2. Load `vt_tax_data_2016.tsv`, making sure to set the correct delimiter with the `sep` keyword argument.<jupyter_code># Import pandas with the alias pd
import pandas as pd

# Load TSV using the sep keyword argument to set delimiter
# data = pd.read_csv('vt_tax_data_2016.tsv', sep='\t')
data = pd.read_csv('vt_tax_data_2016.csv')

# Plot the total number of tax returns by income group
counts = data.groupby('agi_stub').N1.sum()
counts.plot.bar()
plt.show()<jupyter_output><empty_output><jupyter_text>Import a subset of columnsThe Vermont tax data contains 147 columns describing household composition, income sources, and taxes paid by ZIP code and income group. Most analyses don't need all these columns. In this exercise, you will create a data frame with fewer variables using `read_csv()`s `usecols` argument.Let's focus on household composition to see if there are differences by geography and income level. To do this, we'll need columns on income group, ZIP code, tax return filing status (e.g., single or married), and dependents. The data uses codes for variable names, so the specific columns needed are in the instructions.`pandas` has already been imported as `pd`.Instructions1. Create a list of columns to use: `zipcode`, `agi_stub` (income group), `mars1` (number of single households), `MARS2` (number of households filing as married), and `NUMDEP` (number of dependents).2. Create a data frame from `vt_tax_data_2016.csv` that uses only the selected columns.<jupyter_code># Create list of columns to use
cols = ['zipcode', 'agi_stub', 'mars1', 'MARS2', 'NUMDEP']

# Create data frame from csv using only selected columns
data = pd.read_csv('vt_tax_data_2016.csv', usecols=cols)

# View counts of dependents and tax returns by income level
data.groupby('agi_stub').sum()<jupyter_output><empty_output><jupyter_text>Import a file in chunksWhen working with large files, it can be easier to load and process the data in pieces. Let's practice this workflow on the Vermont tax data.The first 500 rows have been loaded as `vt_data_first500`. You'll get the next 500 rows. To do this, you'll employ several keyword arguments: `nrows` and `skiprows` to get the correct records, `header` to tell `pandas` the data does not have column names, and `names` to supply the missing column names. You'll also want to use the `list()` function to get column names from `vt_data_first500` to reuse.`pandas` has been imported as `pd`.Instructions1. Use `nrows` and `skiprows` to make a data frame, `vt_data_next50`, with the next 500 rows.2. Set the `header` argument so that `pandas` knows there is no header row.3. Name the columns in `vt_data_next500` by supplying a list of `vt_data_first500`'s columns to the `names` argument.<jupyter_code>vt_data_first500 = pd.read_csv('vt_tax_data_2016.csv', nrows=500)
# Create data frame of next 500 rows with labeled columns
vt_data_next500 = pd.read_csv('vt_tax_data_2016.csv', 
                              nrows=500,
                              skiprows=500,
                              header=None,
                              names=list(vt_data_first500))

# View the Vermont data frames to confirm they're different
display(vt_data_first500.head())
display(vt_data_next500.head())<jupyter_output><empty_output><jupyter_text>Specify data typesWhen loading a flat file, `pandas` infers the best data type for each column. Sometimes its guesses are off, particularly for numbers that represent groups or qualities instead of quantities.Looking at the data dictionary for `vt_tax_data_2016.csv` reveals two such columns. The `agi_stub` column contains numbers that correspond to income categories, and `zipcode` has 5-digit values that should be strings -- treating them as integers means we lose leading 0s, which are meaningful. Let's specify the correct data types with the `dtype` argument.`pandas` has been imported for you as `pd`.Instructions1. Load `vt_tax_data_2016.csv` with no arguments and view the data frame's `dtypes` attribute. Note the data types of `zipcode` and `agi_stub`.2. Create a dictionary, `data_types`, specifying that `agi_stub` is `"category"` data and `zipcode` is string data.3. Reload the CSV with the `dtype` argument and the dictionary to set the correct column data types.4. View the data frame's `dtypes` attribute.<jupyter_code># Load csv with no additional arguments
data = pd.read_csv('vt_tax_data_2016.csv')

# Print the data types
data.dtypes
# Create dict specifying data types for agi_stub and zipcode
data_types = {'agi_stub': 'category',
              'zipcode': str}

# Load csv using dtype to set correct data types
data = pd.read_csv('vt_tax_data_2016.csv', dtype=data_types)

# Print data types of resulting frame
data.dtypes.head()<jupyter_output><empty_output><jupyter_text>Set custom NA valuesPart of data exploration and cleaning consists of checking for missing or NA values and deciding how to account for them. This is easier when missing values are treated as their own data type and there are `pandas` functions that specifically target such NA values. `pandas` automatically treats some values as missing, but we can pass additional NA indicators with the `na_values` argument. Here, you'll do this to ensure that invalid ZIP codes in the Vermont tax data are coded as NA.`pandas` has been imported as `pd`.Instructions1. Create a dictionary, `null_values`, specifying that `0`s in the `zipcode` column should be considered NA values.2. Load `vt_tax_data_2016.csv`, using the `na_values` argument and the dictionary to make sure invalid ZIP codes are treated as missing.<jupyter_code># Create dict specifying that 0s in zipcode are NA values
null_values = {'zipcode': 0}

# Load csv using na_values keyword argument
data = pd.read_csv('vt_tax_data_2016.csv', 
                   na_values=null_values)

# View rows with NA ZIP codes
data[data['zipcode'].isna()]<jupyter_output><empty_output><jupyter_text>Skip bad dataIn this exercise you'll use `read_csv()` parameters to handle files with bad data, like records with more values than columns. By default, trying to import such files triggers a specific error, `pandas.io.common.CParserError`.Some lines in the Vermont tax data here are corrupted. In order to load the good lines, we need to tell `pandas` to skip errors. We also want `pandas` to warn us when it skips a line so we know the scope of data issues.`pandas` has been imported as `pd`. The exercise code will try to read the file. If there is a `pandas.io.common.CParserError`, the code in the `except` block will run.Instructions1. Try to import the file `vt_tax_data_2016_corrupt.csv` without any keyword arguments.2. Import `vt_tax_data_2016_corrupt.csv` with the `error_bad_lines` parameter set to skip bad records.3. Update the import with the `warn_bad_lines` parameter set to issue a warning whenever a bad record is skipped.<jupyter_code>try:
    # Import the CSV without any keyword arguments
    data = pd.read_csv('vt_tax_data_2016_corrupt.csv')
  
    # View first 5 records
    print(data.head())
  
except pd.io.common.CParserError:
    print('Your data contained rows that could not be parsed.')
try:
    # Import CSV with error_bad_lines set to skip bad records
    data = pd.read_csv('vt_tax_data_2016_corrupt.csv', 
                     error_bad_lines=False)
  
    # View first 5 records
    print(data.head())
  
except pd.io.common.CParserError:
    print('Your data contained rows that could not be parsed.')
try:
    # Set warn_bad_lines to issue warnings about bad records
    data = pd.read_csv('vt_tax_data_2016_corrupt.csv', 
                       error_bad_lines=False, 
                       warn_bad_lines=True)
  
    # View first 5 records
    print(data.head())
  
except pd.io.common.CParserError:
    print('Your data contained rows that could not be parsed.')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07286821705426356, 'fraction_numerical': 0.01839793281653747, 'mean_word_length': 4.435955056179775, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '140605', 'license': 'MIT', 'n_tokens_mistral': 2890, 'n_tokens_neox': 2700, 'n_words': 1265, 'path': 'streamlined_data_ingestion_with_pandas/1_importing_data_from_flat_files.ipynb', 'repo_name': 'vhsenna/datacamp-courses', 'size': '9675'}