<jupyter_start><jupyter_text>CS109A Introduction to Data Science  Lab 5:  Exploratory Data Analysis, `seaborn`, more Plotting**Harvard University****Fall 2019****Instructors:** <NAME>, <NAME>, and <NAME>**Material Preparation:** <NAME>.---<jupyter_code>#RUN THIS CELL 
import requests
from IPython.core.display import HTML
styles = requests.get("https://raw.githubusercontent.com/Harvard-IACS/2018-CS109A/master/content/styles/cs109.css").text
HTML(styles)
# import the necessary libraries
%matplotlib inline
import numpy as np
import scipy as sp
import matplotlib as mpl
import matplotlib.cm as cm
import matplotlib.pyplot as plt
import pandas as pd
import time
pd.set_option('display.width', 500)
pd.set_option('display.max_columns', 200)
pd.set_option('display.notebook_repr_html', True)
import seaborn as sns

from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

import warnings
warnings.filterwarnings('ignore')
%config InlineBackend.figure_format ='retina'
%%javascript
IPython.OutputArea.auto_scroll_threshold = 9999;<jupyter_output><empty_output><jupyter_text>Learning GoalsBy the end of this lab, you should be able to:- know how to implement the different types of plots such as histograms, boxplots, etc, that were mentioned in class.- have `seaborn` as well as `matplotlib` in your plotting toolbox.**This lab corresponds to lecture 6 up to 9 and maps to homework 3.**  Table of Contents* [1 - Visualization Inspiration](inspire)* [2 - Intro to `seaborn` plotting](seaborn)* [3 - Different types of plots (histograms, boxplots, etc)](plots)* [4 - Excercise: EDA on Medical Data](exercise)  1 - Visualization Inspiration![title](../images/WGOITG.jpg)[source: nytimes.org](https://www.nytimes.com/2018/08/27/learning/whats-going-on-in-this-graph-is-now-weekly-stem-teachers-explain-why-its-a-powerful-activity.html)Notice that in “Summers Are Getting Hotter,” above, the histogram has intervals for global summer temperatures on the x-axis, designated from extremely cold to extremely hot, and their frequency on the y-axis. That was an **infographic** intended for the general public. In contrast, take a look at the plots below of the same data published at a **scientific journal**. They look quite different, don't they?![title](../images/F4.jpeg)*<NAME>, <NAME>, and <NAME>*, Perception of climate change. [PNAS](https://www.pnas.org/content/109/37/E2415.fullabstract-1)  2 - Implementing Various Types of Plots using `matplotlib`  and `seaborn`.Before you start coding your visualization, you need to decide what **type** of vizualization to use. A box plot, a histogram, a scatter plot, or something else? That will depend on the purpose of the plot (is it for performing an inspection on your data (EDA, or for showing your results/conclusions to people) and the number variables that you want to plot.You have a lot of tools for plotting in Python. The basic one, of course, is `matplotlib` and there are other libraries that are built on top of it, such as `seaborn`, `bokeh`, or `altair`. In this class we will continue using `matplotlib` and also look into [`seaborn`](https://seaborn.pydata.org). Those two libraries are the ones you should be using for homework. Introduction to `seaborn``Seaborn` is a Python data visualization library based on matplotlib. It provides a high-level interface for drawing attractive and informative statistical graphics. The library provides a database of useful datasets for educational purposes that can be loaded by typing:```pythonseaborn.load_dataset(name, cache=True, data_home=None, **kws)```For information on what these datasets are : https://github.com/mwaskom/seaborn-data The plotting functions in `seaborn` can be decided in two categories- **'axes-level'** functions, such as `regplot`, `boxplot`, `kdeplot`, `scatterplot`, `distplot` which can connect with the `matplotlib` Axes object and its parameters. You can use that object as you would in `matplotlib`:```pythonf, (ax1, ax2) = plt.subplots(2)sns.regplot(x, y, ax=ax1)sns.kdeplot(x, ax=ax2)ax1 = sns.distplot(x, kde=False, bins=20)```- **'figure-level'** functions, such as `lmplot`, `factorplot`, `jointplot`, `relplot`. In this case, `seaborn` organizes the resulting plot which may include several Axes in a meaningful way. That means that the functions need to have total control over the figure, so it isn't possible to plot, say, an `lmplot` onto one that already exists. Calling the function always initializes a figure and sets it up for the specific plot it's drawing. These functions return an object of the type `FacetGrid` with its own methods for operating on the resulting plot.To set the parameters for figure-level functions:```pythonsns.set_context("notebook", font_scale=1, rc={"lines.linewidth": 2.5})```  The Titanic datasetThe `titanic.csv` file contains data for 887 passengers on the Titanic. Each row represents one person. The columns describe different attributes about the person including whether they survived, their age, their on-board class, their sex, and the fare they paid.<jupyter_code>titanic = sns.load_dataset('titanic');
titanic.info();
titanic.columns<jupyter_output><empty_output><jupyter_text>Exercise:  Drop the following features:    'embarked', 'who', 'adult_male', 'embark_town', 'alive', 'alone'<jupyter_code># your code here<jupyter_output><empty_output><jupyter_text>Exercise:  Find for how many passengeres we do not have their deck information.<jupyter_code># your code here<jupyter_output><empty_output><jupyter_text>Histograms Plotting one variable's distribution (categorical and continous)The most convenient way to take a quick look at a univariate distribution in `seaborn` is the `distplot()` function. By default, this will draw a histogram and fit a kernel density estimate (KDE).A histogram displays a quantitative (numerical) distribution by showing the number (or percentage) of the data values that fall in specified intervals. The intervals are on the x-axis and the number of values falling in each interval, shown as either a number or percentage, are represented by bars drawn above the corresponding intervals.<jupyter_code># What was the age distribution among passengers in the Titanic?
import seaborn as sns
sns.set(color_codes=True)

f, ax = plt.subplots(1,1, figsize=(8, 3));
ax = sns.distplot(titanic.age, kde=False, bins=20)

# bug
#ax = sns.distplot(titanic.age, kde=False, bins=20).set(xlim=(0, 90));

ax.set(xlim=(0, 90));
ax.set_ylabel('counts');
f, ax = plt.subplots(1,1, figsize=(8, 3))
ax.hist(titanic.age, bins=20);
ax.set_xlim(0,90);<jupyter_output><empty_output><jupyter_text>Exercise (pandas trick): Count all the infants on board (age less than 3) and all the children ages 5-10.<jupyter_code># your code here
infants = len(titanic[(titanic.age < 3)]) 
children = len(titanic[(titanic.age >= 3) & (titanic.age < 10)]) 
print(f'There were {infants} infants and {children} children on board the Titanic')<jupyter_output><empty_output><jupyter_text>**Pandas trick:** We want to creat virtual "bins" for readability and replace ranges of values with categories. We will do this in an ad hoc way, **it can be done better**. For example in the previous plot we could set:    - `(age<3) = 'infants'`, - `(3<age<18)='children'`, - `(18See matplotlib [colors](https://matplotlib.org/3.1.0/tutorials/colors/colormaps.html) here.<jupyter_code># set the colors
cmap = plt.get_cmap('Pastel1')
young = cmap(0.5)
middle = cmap(0.2)
older = cmap(0.8)

# get the object we will change - patches is an array with len: num of bins
fig, ax = plt.subplots()
y_values, bins, patches = ax.hist(titanic.age, 10)

[patches[i].set_facecolor(young) for i in range(0,1)] # bin 0
[patches[i].set_facecolor(middle) for i in range(1,3)] # bins 1 and 2
[patches[i].set_facecolor(older) for i in range(3,10)] # 7 remaining bins 

ax.grid(True)
fig.show()<jupyter_output><empty_output><jupyter_text>Kernel Density EstimationThe kernel density estimate can be a useful tool for plotting the shape of a distribution. The **bandwidth (bw)** parameter of the KDE controls how tightly the estimation is fit to the data, much like the bin size in a histogram. It corresponds to the width of the kernels we plotted above. The default behavior tries to guess a good value using a common reference rule, but it may be helpful to try larger or smaller values.<jupyter_code>sns.kdeplot(titanic.age, bw=0.6, label="bw: 0.6", shade=True, color="r");
sns.kdeplot(titanic.age, bw=2, label="bw: 2", shade=True);<jupyter_output><empty_output><jupyter_text>Exercise:  Plot the distribution of fare paid by passengers<jupyter_code># your code here
sns.kdeplot(titanic.fare, bw=0.5, label="bw: 0.5", shade=True);<jupyter_output><empty_output><jupyter_text>You can mix elements of `matplotlib` such as Axes with `seaborn` elements for a best use of both worlds.<jupyter_code>import seaborn as sns
sns.set(color_codes=True)

x1 = np.random.normal(size=100)
x2 = np.random.normal(size=100)

fig, ax = plt.subplots(1,2, figsize=(15,5))

# seaborn goes in first subplot
sns.set(font_scale=0.5)
sns.distplot(x1, kde=False, bins=15, ax=ax[0]);
sns.distplot(x2, kde=False, bins=15, ax=ax[0]);
ax[0].set_title('seaborn Graph Here', fontsize=14)
ax[0].set_xlabel(r'$x$', fontsize=14)
ax[0].set_ylabel(r'$count$', fontsize=14)

# matplotlib goes in second subplot
ax[1].hist(x1, alpha=0.2, bins=15, label=r'$x1$');
ax[1].hist(x2, alpha=0.5, bins=15, label=r'$x2$');
ax[1].set_xlabel(r'$x$', fontsize=14)
ax[1].set_ylabel(r'$count$', fontsize=14)
ax[1].set_title('matplotlib Graph Here', fontsize=14)
ax[1].legend(loc='best', fontsize=14);<jupyter_output><empty_output><jupyter_text>Introduding the heart disease dataset. More on this in the in-class exercise at the end of the notebook.<jupyter_code>columns = ["age", "sex", "cp", "restbp", "chol", "fbs", "restecg", 
           "thalach", "exang", "oldpeak", "slope", "ca", "thal", "num"]
heart_df = pd.read_csv('../data/heart_disease.csv', header=None, names=columns)

heart_df.head()<jupyter_output><empty_output><jupyter_text>Boxplots One variable.<jupyter_code># seaborn
ax = sns.boxplot(x='age', data=titanic)
#ax = sns.boxplot(x=titanic['age']) # another way to write this
ax.set_ylabel(None);
ax.set_xlabel('age', fontsize=14);
ax.set_title('Distribution of age in the Titanic', fontsize=14);<jupyter_output><empty_output><jupyter_text>Two variables   Exercise:  Did more young people or older ones get first class tickets on the Titanic?<jupyter_code># your code here
# two variables seaborn

# two variable boxplot in pandas<jupyter_output><empty_output><jupyter_text>Scatterplots Plotting the distribution of two variablesAlso called a bivariate distribution where each observation is shown with  a point with x and y values. You can draw a scatterplot with the `matplotlib plt.scatter` function, or the `seaborn jointplot()` function:<jupyter_code>f, ax = plt.subplots(1,1, figsize=(10, 5))
sns.scatterplot(x="fare", y="age", data=titanic, ax=ax); 
sns.jointplot("fare", "age", data=titanic, s=40, edgecolor="w", linewidth=1)<jupyter_output><empty_output><jupyter_text>You may control the `seaborn` [Figure aesthetics](https://seaborn.pydata.org/tutorial/aesthetics.html).<jupyter_code># matplotlib
fig, ax = plt.subplots(1,1, figsize=(10,6))
ax.scatter(heart_df['age'], heart_df['restbp'], alpha=0.8);
ax.set_xlabel(r'$Age (yrs)$', fontsize=15);
ax.set_ylabel(r'Resting Blood Pressure (mmHg)', fontsize=15);
ax.set_title('Age vs. Resting Blood Pressure', fontsize=14)
plt.show();<jupyter_output><empty_output><jupyter_text>Plotting the distribution of three variables<jupyter_code>f, ax = plt.subplots(1,1, figsize=(10, 5))
sns.scatterplot(x="fare", y="age", hue="survived", data=titanic, ax=ax);<jupyter_output><empty_output><jupyter_text>Plotting the distribution of four variables (going too far?)   Exercise:  Plot the distribution of fare paid by passengers according to age, survival and sex.     Use `size=` for the fourth variable<jupyter_code># your code here<jupyter_output><empty_output><jupyter_text>Pairplots<jupyter_code>titanic.columns
to_plot = ['age', 'fare', 'survived', 'deck']
df_to_plot = titanic.loc[:,to_plot]
sns.pairplot(df_to_plot);
from pandas.plotting import scatter_matrix
scatter_matrix(df_to_plot, alpha=0.8, figsize=(10, 10), diagonal='kde');<jupyter_output><empty_output><jupyter_text>Plotting Categorical Variables<jupyter_code>titanic = sns.load_dataset('titanic')
f, ax = plt.subplots(figsize=(7, 3));
ax = sns.countplot(y="deck", data=titanic, color="c");
ax.set_title('Titanic');
ax = sns.countplot(x="class", data=titanic)
ax.set_title('Titanic');
fig, ax = plt.subplots(figsize=(10,6)) # Create figure object
sns.set_context("notebook", font_scale=1, rc={"lines.linewidth": 2.5})
ax = sns.countplot(x="deck", data=titanic)
sns.set(style="ticks", palette="muted")
sns.relplot(x="age", y="deck", col="class", data=titanic);
sns.set_context("notebook", font_scale=1.5, rc={"lines.linewidth": 2.5})
sns.pairplot(data=titanic, hue="deck");<jupyter_output><empty_output><jupyter_text>Introduction to `pandas` plotting.There is plotting functionality built in `pandas`. Look for it in the `pandas` "encyclopedia", a mere 2883-page pdf from the creator <NAME>: [pandas documentation (pdf)](https://pandas.pydata.org/pandas-docs/stable/pandas.pdf) **Example:** The `value_counts()` Series method and top-level function computes a histogram of a 1D array of values. It canalso be used as a function on regular arrays.Reminder: DataFrame: “index” (axis=0, default), “columns” (axis=1)   Line Graph Good for time dependency or when a variable evolves<jupyter_code>df = pd.DataFrame(np.random.randn(1000, 4), columns=['A', 'B', 'C', 'D'])
df.head()
# cumulative sum adds column values as it goes
df = df.cumsum()
df.head()
plt.figure();
df.plot();
plt.legend(loc='best');
ts = pd.Series(np.random.randn(1000),
               index=pd.date_range('1/1/2000', periods=1000))
df = pd.DataFrame(np.random.randn(1000, 4), 
                  index=ts.index, columns=list('ABCD'))

df = df.cumsum()
plt.figure();
df.plot();<jupyter_output><empty_output><jupyter_text>Plotting methods allow for a handful of plot styles other than the default line plot. These methods can be provided as the kind keyword argument to plot(), and include:- ‘bar’ or ‘barh’ for bar plots- ‘hist’ for histogram- ‘box’ for boxplot- ‘kde’ or ‘density’ for density plots- ‘area’ for area plots- ‘scatter’ for scatter plots- ‘hexbin’ for hexagonal bin plots- ‘pie’ for pie plotsIn addition to these kind s, there are the `DataFrame.hist()`, and `DataFrame.boxplot()` methods, which use a separate interface. `scatter_matrix` in pandas.plotting takes a Series or DataFrame as an argument.  Bar Plots<jupyter_code>plt.figure();
df.iloc[0].plot(kind='bar');
df2 = pd.DataFrame(np.random.rand(10, 4), columns=['a', 'b', 'c', 'd'])
df2
df2.plot.bar();
# horizontal bar plot
df2.plot.barh(stacked=False);<jupyter_output><empty_output><jupyter_text>Histograms<jupyter_code>df4 = pd.DataFrame({'a': np.random.randn(1000) + 1, 'b': np.random.randn(1000), 
                    'c': np.random.randn(1000) - 1}, columns=['a', 'b', 'c'])

plt.figure();
df4.plot.hist(alpha=0.5, stacked=False, bins=60);<jupyter_output><empty_output><jupyter_text>Boxplots<jupyter_code>color = {'boxes': 'DarkGreen', 'whiskers': 'DarkOrange',
         'medians': 'DarkBlue', 'caps': 'Gray'}

df = pd.DataFrame(np.random.rand(10, 5), columns=['A', 'B', 'C', 'D', 'E'])
df.plot.box(color=color );<jupyter_output><empty_output><jupyter_text>Area plotsYou can create area plots with `Series.plot.area()` and `DataFrame.plot.area()`. Area plots are stackedby default. To produce stacked area plot, each column must be either all positive or all negative values.<jupyter_code>df = pd.DataFrame(np.random.rand(10, 4), columns=['a', 'b', 'c', 'd'])
df.plot.area(stacked=True);
df.plot.area(stacked=False);<jupyter_output><empty_output><jupyter_text>ScatterplotScatter plot can be drawn by using the DataFrame.plot.scatter() method. Scatter plot requires numericcolumns for the x and y axes. These can be specified by the x and y keywords.<jupyter_code>ax = df.plot.scatter(x='a', y='b', color='DarkBlue', label='Group 1');
df.plot.scatter(x='c', y='d', color='DarkGreen', label='Group 2', ax=ax);<jupyter_output><empty_output><jupyter_text>`pandas` TricksThe copy() method on pandas objects copies the underlying data (though not the axis indexes, since they are immutable)and returns a new object. Note that it is seldom necessary to copy objects. For example, there are only ahandful of ways to alter a DataFrame in-place:- Inserting, deleting, or modifying a column.- Assigning to the index or columns attributes.- For homogeneous data, directly modifying the values via the values attribute or advanced indexing.To be clear, no pandas method has the side effect of modifying your data; almost every method returns a new object,leaving the original object untouched. If the data is modified, it is because you did so explicitly  4 - Group Exercise: 1/2 hour in the Life of a Cardiologist Try each exercise on your own and then discuss with your peers sitting at your table. Feel free to engage the TFs and instructors as well. Visualize and explore the data. Use `.describe()` to look at your data and also examine if you have any missing values.  What is the actual number of feature variables after converting categorical variables to dummy ones?**List of available variables (includes target variable `num`):**- **age**: continuous- **sex**: categorical, 2 values {0: female, 1: male}- **cp** (chest pain type): categorical, 4 values    {1: typical angina, 2: atypical angina, 3: non-angina, 4: asymptomatic angina}- **restbp** (resting blood pressure on admission to hospital): continuous (mmHg)- **chol (serum cholesterol level)**: continuous (mg/dl)- **fbs** (fasting blood sugar): categorical, 2 values {0:  120 mg/dl}- **restecg** (resting electrocardiography): categorical, 3 values    {0: normal, 1: ST-T wave abnormality, 2: left ventricular hypertrophy}- **thalach** (maximum heart rate achieved): continuous- **exang** (exercise induced angina): categorical, 2 values {0: no, 1: yes}- **oldpeak** (ST depression induced by exercise relative to rest): continuous- **slope** (slope of peak exercise ST segment): categorical, 3 values     {1: upsloping, 2: flat, 3: downsloping}- **ca** (number of major vessels colored by fluoroscopy): discrete (0,1,2,3)- **thal**: categorical, 3 values {3: normal, 6: fixed defect, 7: reversible defect}- **num** (diagnosis of heart disease): categorical, 5 values     {0: less than 50% narrowing in any major vessel,     1-4: more than 50% narrowing in 1-4 vessels}<jupyter_code># load the dataset
heart_df = pd.read_csv('../data/heart_disease.csv', header=None, names=columns)
heart_df.head()<jupyter_output><empty_output><jupyter_text>Answer the following question using plots01. At what ages do people seek cardiological exams? 02. Do men seek help more than women? 03. Examine the variables. How do they relate to one another?04. (Variation on 02): What % of men and women seek cardio exams? 05. Does resting blood pressure increase with age? **Pandas trick: `.replace`** The response variable (num) is categorical with 5 values, but we don't have enough data to predict all the categories.  Therefore we'll replace `num` with `hd` (heart disease): **categorical, 2 values {0: no, 1: yes}**. Use the code below (take a minute to understand how it works, it's very useful!):<jupyter_code># Replace response variable values with a binary response (1: heart disease(hd) or 0: not)
heart_df['num'].replace(to_replace=[1,2,3,4],value=1,inplace=True)

# Rename column for clarity
heart_df = heart_df.rename(columns = {'num':'hd'})
heart_df.head()
# look at the features
heart_df.info();
heart_df.describe()<jupyter_output><empty_output><jupyter_text>At this point you should split in train and test set and work only on the **train!!**. For simplicity we will not do this in the solutions.<jupyter_code># your code here
# 01. what ages do people seek cardiological exams? 

# %load solutions/q01.py
# your code here
# 02. do men seek help more than women?

# %load solutions/q02.py
# your code here
# 03. Examine the variables. How do they relate to one another?

# %load solutions/q03.py
# your code here
# 04. What percentage of men and women seek cardio exams? 

# %load solutions/q04.py
# your code here
# 05. Does resting blood pressure increase with age?

# %load solutions/q05.py<jupyter_output><empty_output><jupyter_text>Bonus: Find the hidden patternRead the following file into a `pandas` Dataframe: '../data/mystery.csv' and plot it.  How does it look? You should see a beautiful pattern. If not, think of ways to fix the issue.<jupyter_code>mystery = pd.read_csv('../data/mystery.csv',  sep=' ', header=None) 
mystery.head()
# your code here
# %load solutions/mystery.py<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12526751319731774, 'fraction_numerical': 0.019070718599895373, 'mean_word_length': 5.811791383219955, 'pattern_counts': {'":': 3, '<': 130, '<?xml version=': 0, '>': 126, 'https://': 8, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '29', 'dir': 'jupyter-structured-clean-dedup', 'id': '368233', 'license': 'MIT', 'n_tokens_mistral': 7159, 'n_tokens_neox': 6706, 'n_words': 2329, 'path': 'docs/labs/lab05/notebook/cs109a_lab5_EDA_plotting.ipynb', 'repo_name': 'biankaursul/2019-CS109A', 'size': '21077'}