<jupyter_start><jupyter_text>COMP 135 day01: Intro to Pandas and Matplotlib for Data Analysis and VizA Python-based version of the "Ch 2 lab" from James et al.'s "Introduction to Statistical Learning" textbookInspired by this original notebook: https://nbviewer.jupyter.org/github/emredjan/ISL-python/blob/master/labs/lab_02.3_introduction.ipynb What to doThe first half of this notebook gives several "demos" of how to use pandas and matplotlib.The second half of this notebook is a space for you to try out several "exercises" that would be useful when analyzing a dataset.If you get stuck, please ask a question on Piazza or search the web for help (Stack Overflow is a great resource!) Outline* [Line plots](plot)* [Subplots](subplots)* [Histograms](distplot)* [Loading CSV with NumPy's loadtxt](loadtxt)* [Loading CSV with Pandas' read_csv](read_csv)* [Scatter plots](jointplot)* [Pair plots](pairplot) Exercises* What is the *range* of horsepower values in the provided 'auto' dataset?* Can you plot the relationship between horsepower and miles per gallon (mpg)?* Within the loaded data frame, can you replace all missing values with the *population mean*? Takeaways* Pandas DataFrames are convienent for manipulating real datasets* * Makes reading from CSV files whose columns have multiple types a breeze* * Handles missing values a bit easier than just using raw NumPy functions* Matplotlib and seaborn offer some nice plot styles* * But take getting used to! Practice early and often!<jupyter_code>import numpy as np

# Import the pandas (data management library)
import pandas as pd
# import plotting libraries
import matplotlib
import matplotlib.pyplot as plt

%matplotlib inline
plt.style.use('seaborn') # pretty matplotlib plots

import seaborn as sns
sns.set('notebook', style='whitegrid', font_scale=1.25)<jupyter_output><empty_output><jupyter_text>Line plot<jupyter_code>plt.plot([3, 6, 9, 12], [4, 7, 6, 5], 'ks-');
plt.xlabel('hours studying');
plt.ylabel('grasp of material');
plt.legend(['line label']);<jupyter_output><empty_output><jupyter_text>SubplotsSubplots allows us to show multiple line plots side-by-sideEach subplot has its own "axis", which is represented as a object via Matplotlib.We can interact with that object via methods like 'plot' or 'set_title' to set its properties.<jupyter_code># Create multiple subplots in same figure
# We recommend using 'sharey=True' so y limits in all plots are the same. Makes comparing across plots easier!

fig_handle, axis_handles = plt.subplots(
    nrows=1, ncols=3, figsize=(15, 5),
    sharey=True, sharex=True);

ax0 = axis_handles[0]
ax0.plot([1, 2, 3], [1, 2, 3]);
ax0.set_title('Plot 0: Positive Slope');

ax1 = axis_handles[1]
ax1.plot([1, 2, 3], [2, 2, 2]);
ax1.set_title('Plot 1: Flat Slope');

ax2 = axis_handles[2]
ax2.plot([1, 2, 3], [3, 2, 1]);
ax2.set_title('Plot 2: Negative Slope');<jupyter_output><empty_output><jupyter_text>Histogram of an arrayExamine the empirical distribution of a sample of N examples<jupyter_code>N = 1000
x_N = np.random.normal(size=N) # 1000 draws from a "standard" Normal distribution
p = sns.distplot(x_N, bins=31, kde=False);
plt.xlabel('x value');
plt.ylabel('count');<jupyter_output><empty_output><jupyter_text>Joint plot of two arraysMake a scatterplot of points defined by two arrays.This plot is a great way to see if there is any *relationship* between two sets of observed measurements.<jupyter_code>N = 1000
x_N = np.random.normal(size=N)
y_N = np.random.normal(size=N)

# seaborn scatterplot
p = sns.jointplot(x_N, y_N, kind='scatter')
p.set_axis_labels(xlabel='x', ylabel='y');<jupyter_output><empty_output><jupyter_text>Loading dataset from CSV file with `np.loadtxt`<jupyter_code># Show the first few lines of the file!

with open('dataset_auto.csv', 'r') as f:
    for line in f.readlines()[:5]:
        print(line.strip())<jupyter_output>mpg,cylinders,displacement,horsepower,weight,acceleration,year,origin,name
18,8,307,130,3504,12,70,1,chevrolet chevelle malibu
15,8,350,165,3693,11.5,70,1,buick skylark 320
18,8,318,150,3436,11,70,1,plymouth satellite
16,8,304,150,3433,12,70,1,amc rebel sst<jupyter_text>So we have 9 total columns of data. The first 8 are all *numeric*. The last one is *text*.<jupyter_code># Load the file into numpy with `loadtxt`
# How is it represented??

auto_arr = np.loadtxt('dataset_auto.csv', delimiter=',', skiprows=1, dtype=str)
print("shape: " + str(auto_arr.shape))
print("dtype: " + str(auto_arr.dtype))<jupyter_output>shape: (397, 9)
dtype: <U36<jupyter_text>FYI, if you ever see numpy give a dtype of '<U...", its probably a text string of some kind.So, our data is currently in memory as ALL strings. That's not good. We want the numbers!<jupyter_code># Trying to make a numerical array out of this data will fail!
try:
    auto_num_arr = np.asarray(auto_arr, dtype=np.float64)
except Exception as e:
    print(str(type(e)) + ': ' + str(e))<jupyter_output><class 'ValueError'>: could not convert string to float: 'chevrolet chevelle malibu'<jupyter_text>Obviously, the last column is all strings, so we can't make this into numbers.What if we *ignore* the last column?<jupyter_code># Even ignoring the last column will fail!
try:
    auto_num_arr = np.asarray(auto_arr[:,:-1], dtype=np.float64)
except Exception as e:
    print(str(type(e)) + ': ' + str(e))<jupyter_output><class 'ValueError'>: could not convert string to float: '?'<jupyter_text>Tricky part: This CSV file was saved using '?' symbol to denote missing valuesWe need to replace missing value '?' strings with np.nan (numpy's built-in missing value type)<jupyter_code>auto_num_arr = auto_arr[:,:-1].copy()                       # copy all but last column into an array
auto_num_arr[auto_num_arr == '?'] = np.nan                  # replace any entry matching '?' with a NaN value
auto_num_arr = np.asarray(auto_num_arr, dtype=np.float64)   # now convert to float64 numeric type
np.mean(auto_num_arr, axis=0)
np.nanmean(auto_num_arr, axis=0) # ignore nan values when computing the mean
np.nanmin(auto_num_arr, axis=0)  # ignore nan values when computing the min<jupyter_output><empty_output><jupyter_text>Loading dataset from CSV file with `pd.read_csv`<jupyter_code># read csv data with pandas into dataframe, explicitly setting na_values.
# pandas read_xxx functions infer datatypes, headers, dates, etc. 
# without explicit declarations or required interventions from user

auto_df = pd.read_csv('dataset_auto.csv', na_values=['?']) # '?' symbol means missing value
print(auto_df)
auto_df.shape
# get column names of the dataframe
list(auto_df.columns)
# show the values of one column by name
auto_df['mpg']
# show the values of several column by name
auto_df[['mpg', 'weight']]
# dropping rows (axis=0) where there are not-a-number (NaN) values (inplace)
auto_df.dropna(axis=0, inplace=True)
auto_df.shape<jupyter_output><empty_output><jupyter_text>Scatterplot using a dataframe<jupyter_code># seaborn scatterplot
pl = sns.jointplot(x='horsepower', y='mpg', data=auto_df);<jupyter_output><empty_output><jupyter_text>Factor plots using a dataframe<jupyter_code># changing data type of a column into category
auto_df['cylinders'] = auto_df['cylinders'].astype('category')
# seaborn boxplot implementation
sns.boxplot(x='cylinders', y='mpg', data=auto_df);<jupyter_output><empty_output><jupyter_text>Histograms<jupyter_code># seaborn enhanced histogram with density plot
sns.distplot(auto_df['mpg'], bins=31, kde=True);
plt.xlim([0, 50]);
# same as above, but WITHOUT the density line plot
sns.distplot(auto_df['mpg'], bins=31, kde=False);
plt.xlim([0, 50]);<jupyter_output><empty_output><jupyter_text>Pair plot<jupyter_code># seaborn pairplot for selected variables, colored by another
sns.pairplot(auto_df, vars=['mpg', 'displacement', 'horsepower', 'weight', 'acceleration'], hue='cylinders');
# summary statistics for a single column
# wrapped as dataframe for pretty table display in jupyter
pd.DataFrame(auto_df['mpg'].describe())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11433212545295514, 'fraction_numerical': 0.022991378233162565, 'mean_word_length': 6.140053523639607, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 59, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '246404', 'license': 'MIT', 'n_tokens_mistral': 2830, 'n_tokens_neox': 2635, 'n_words': 859, 'path': 'labs/day01-IntroToPandasAndMatplotlib.ipynb', 'repo_name': 'milesizydorczak12/comp135-20f-assignments', 'size': '8003'}