<jupyter_start><jupyter_text>Reading a csv filepandas is probably the most important library for Math 10. Exploring the dataHere are some first things you might try when importing a new dataset.<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>On my personal computer, the `cars.csv` file is located in a different folder from this notebook.  If it's in the same folder (like it is on Deepnote), you can just type `pd.read_csv("cars.csv")`.<jupyter_code>df = pd.read_csv("../data/cars.csv")<jupyter_output><empty_output><jupyter_text>Viewing the first 5 rows of the dataset.  You should think of each row as corresponding to one *instance* or one *data point*.<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>Or the first 10 rows.<jupyter_code>df.head(10)<jupyter_output><empty_output><jupyter_text>The number of rows and columns.<jupyter_code>df.shape<jupyter_output><empty_output><jupyter_text>The names of the columns.<jupyter_code>df.columns<jupyter_output><empty_output><jupyter_text>Some data about the numeric columns.<jupyter_code>df.describe()<jupyter_output><empty_output><jupyter_text>Some more information.  You can use this next information to determine which columns have missing values.<jupyter_code>df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 406 entries, 0 to 405
Data columns (total 9 columns):
 #   Column            Non-Null Count  Dtype  
---  ------            --------------  -----  
 0   Name              406 non-null    object 
 1   Miles_per_Gallon  398 non-null    float64
 2   Cylinders         406 non-null    int64  
 3   Displacement      406 non-null    float64
 4   Horsepower        400 non-null    float64
 5   Weight_in_lbs     406 non-null    int64  
 6   Acceleration      406 non-null    float64
 7   Year              406 non-null    object 
 8   Origin            406 non-null    object 
dtypes: float64(4), int64(2), object(3)
memory usage: 28.7+ KB<jupyter_text>The data types of the columns.<jupyter_code>df.dtypes<jupyter_output><empty_output><jupyter_text>You can see how the numeric columns are *correlated* with each other.  These correlation values range between -1 and 1, with 1 meaning the two columns are perfectly correlated.<jupyter_code>df.corr()<jupyter_output><empty_output><jupyter_text>IndexingThere are many different ways to select data within a pandas DataFrame.  The best way to remember them is to practice using them.<jupyter_code># Reminder of how df starts
df.head()<jupyter_output><empty_output><jupyter_text>The entry in the 2nd row, 3rd column (remember we start counting at 0).<jupyter_code>df.iloc[2,3]<jupyter_output><empty_output><jupyter_text>The entry in the row with label 2 and the column with label Displacement.  (Notice that the index and the label is the same for the rows; this is not uncommon.)<jupyter_code>df.loc[2,"Displacement"]<jupyter_output><empty_output><jupyter_text>You can get an entire row or column with the same syntax, using a colon `:` to represent "all rows or all columns".<jupyter_code># The row at index 2.
df.iloc[2,:]
# The column at index 3.
df.iloc[:,3]
# The column with label "Displacement".
df.loc[:,"Displacement"]<jupyter_output><empty_output><jupyter_text>There is an abbreviation for getting a certain column, using its label.<jupyter_code>df["Displacement"]<jupyter_output><empty_output><jupyter_text>The next abbreviation does not always work, but can be a further shortcut.  It is called "attribute" access.  The subtleties of attribute access won't be important for us in Math 10; you can read about those subtleties in the [pandas documentation](https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.htmlattribute-access).<jupyter_code>df.Displacement<jupyter_output><empty_output><jupyter_text>Missing valuesAn important concept (especially when working with real-world datasets) is the concept of missing data.  This particular dataset has missing values in the `Miles_per_Gallon` column and in the `Horsepower` column.  In this DataFrame, the missing data is denoted by the NumPy object `np.nan` which stands for "not a number".<jupyter_code>df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 406 entries, 0 to 405
Data columns (total 9 columns):
 #   Column            Non-Null Count  Dtype  
---  ------            --------------  -----  
 0   Name              406 non-null    object 
 1   Miles_per_Gallon  398 non-null    float64
 2   Cylinders         406 non-null    int64  
 3   Displacement      406 non-null    float64
 4   Horsepower        400 non-null    float64
 5   Weight_in_lbs     406 non-null    int64  
 6   Acceleration      406 non-null    float64
 7   Year              406 non-null    object 
 8   Origin            406 non-null    object 
dtypes: float64(4), int64(2), object(3)
memory usage: 28.7+ KB<jupyter_text>We create a Boolean DataFrame using the method `isna()`.  This DataFrame will be `True` where there are null values.<jupyter_code>df.isna()<jupyter_output><empty_output><jupyter_text>Let's find where the `np.nan` values are in the `Miles_per_Gallon` column.<jupyter_code>df[df["Miles_per_Gallon"].isna()]<jupyter_output><empty_output><jupyter_text>The same thing for the `Horsepower` column.<jupyter_code>df[df["Horsepower"].isna()]<jupyter_output><empty_output><jupyter_text>In Python, logical `or` is usually spelled out.<jupyter_code>True or True
True or False
False or False<jupyter_output><empty_output><jupyter_text>The equivalent of `or` in pandas is denoted with a vertical line `|`, which is sometimes called "pipe".<jupyter_code>df[df["Miles_per_Gallon"].isna() | df["Horsepower"].isna()]<jupyter_output><empty_output><jupyter_text>A fancier and more robust method is to use `any`.  In this example, `axis = 1` is saying, look one row at a time.  So `df.isna().any(axis=1)` is asking if there are any missing values in the entire row.<jupyter_code>df[df.isna().any(axis=1)]<jupyter_output><empty_output><jupyter_text>If we were to instead use `axis=0`, it would ask if there were any missing values in the entire column.<jupyter_code>df.isna().any(axis=0)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09724473257698542, 'fraction_numerical': 0.02593192868719611, 'mean_word_length': 4.384816753926701, 'pattern_counts': {'":': 0, '<': 97, '<?xml version=': 0, '>': 97, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '282101', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2123, 'n_tokens_neox': 1973, 'n_words': 620, 'path': '_build/jupyter_execute/Week3/intro_csv.ipynb', 'repo_name': 'ChristopherDavisUCI/UCI-Math-10-W22', 'size': '6170'}