<jupyter_start><jupyter_text>====================================================== MA477 - Theory and Applications of Data Science   Lesson 5: Vizualising Data with Seaborn   Dr. <NAME> United States Military Academy, West Point AY20-2====================================================== Lecture Outline    Quick Overview and Installation    Distribution Plots    Categorical Plots    Matrix Plots    Regression Plots  Overview and InstallationSeaborn is a Python visualization library built on top of Matplotlib and is especially designed for statistical plotting. In addition to having a wide range of beautiful styles and graphs it also works really well with Panda DataFrames. If you don't already have Seaborn installed in your computer, then you may install it either from your command line using the `pip` install command or from the anaconda terminal using the `conda` command: ```pythonpip install seabornconda install seaborn```For a more detailed and complete tutorial of Matplotlib you may visit the official website by clicking on the: Seaborn WebsiteLet's begin by importing seaborn, pandas, and numpy.<jupyter_code>import seaborn as sns
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Data Set`Seaborn` comes with a few pre-loaded data sets, so for the sake of illustration, we will use one of these sets.<jupyter_code>diamonds=sns.load_dataset('diamonds', cache=True)
diamonds.shape
[np.random.randint(50000) for i in range(10)]
[0 for i in range(10)]
diamonds=diamonds.iloc[[np.random.randint(50000) for i in range(10000)]]
diamonds.head()<jupyter_output><empty_output><jupyter_text>Description of DatasetThis classic dataset contains the prices and other attributes of almost 54,000 diamonds. It's a great dataset for beginners learning to work with data analysis and visualization.Contentprice price in US dollars (\$326--\$18,823)carat weight of the diamond (0.2--5.01)cut quality of the cut (Fair, Good, Very Good, Premium, Ideal)color diamond colour, from J (worst) to D (best)clarity a measurement of how clear the diamond is (I1 (worst), SI2, SI1, VS2, VS1, VVS2, VVS1, IF (best))x length in mm (0--10.74)y width in mm (0--58.9)z depth in mm (0--31.8)depth total depth percentage = $z / mean(x, y) = 2 * z / (x + y)$ (43--79)Source: Kaggle If you want to check all the different pre-loaded datasets in `seaborn`, we may do so as follows:<jupyter_code>sns.get_dataset_names()<jupyter_output>C:\Users\Valmir.Bucaj\AppData\Local\Programs\Python\Python37\lib\site-packages\seaborn\utils.py:376: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("html.parser"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.

The code that caused this warning is on line 376 of the file C:\Users\Valmir.Bucaj\AppData\Local\Programs\Python\Python37\lib\site-packages\seaborn\utils.py. To get rid of this warning, pass the additional argument 'features="html.parser"' to the BeautifulSoup constructor.

  gh_list = BeautifulSoup(http)<jupyter_text>Distribution Plots Say we wanted to know how the `price` is distributed:<jupyter_code>sns.set_style('whitegrid')
plt.figure(figsize=(8,6))
plt.xticks(fontsize=12)
plt.yticks(fontsize=12)
plt.xlabel('price',fontsize=14)
sns.distplot(diamonds['price'],bins=30,kde=False)
plt.show()<jupyter_output><empty_output><jupyter_text>Observe above that we get a `smoothing` of the distribution. This is called the Kernel Density Estimate (KDE) and is an empirical estimate of the underlying distribution of our data. We may turn this off, if we want to, by setting `kde=False`, in which case we just have a histogram. You can experiment with all the different parmeters such as `bins, color` etc.What are some quick observations regarding the prices of the diamonds?Comparing Distributions of Two Features  In case we are interested in comparing (visually) the distributions of two different features, say for example we are interested in knowing how the `carat` distribution compares to the `price`, then we may do so by using the `.jointplot()` method:<jupyter_code>sns.set_style('whitegrid')
sns.jointplot(x='carat',y='price',data=diamonds,kind='kde')

plt.show()<jupyter_output><empty_output><jupyter_text>PracticeGo ahead, and experiment with the differnt types of parameters, specifically try setting `kind=` to `hex`, `kde` and `reg` and see what you get. Question: What are some quick observations you can draw by looking at this scatterplot? In case we wanted to compare distributions for every single combination of the (numerical)features in our data, instead of using the `.jointplot()` repeatedly, we can do it all at once via the `.pairplot()` method:<jupyter_code>sns.pairplot(data=diamonds,hue='cut',palette='coolwarm')<jupyter_output><empty_output><jupyter_text>In the pairplot we can pass on a categorical column for the `hue` parameter. For example, if we pass on the `color` or `cut` column for `hue`, then what's going to happen is that the scatter plots will be colored based off the color or cut of the diamond.  Practice: Experiment with the different parameters.  Categorical PlotsWhat if for example we wanted to visualize a certain statistic of a numerical feature, such as `price, carat, depth ` etc. based off a categorical variable such as `color, cut ` etc.? In what follows we will talk about a few of the ways we can do this. BarplotA barplot takes in a categorical variable for x and a numerical variable for y, and it computes and visualizes different statistics(e.g. mean, standard deviation etc.) of the numerical feature for each of the categories. This should remind you of the `.groupby()` method we learned when we talked about Pandas. Depending on what statistic we want to compute, we can determine this by setting `estimator=` to `np.mean, np.std, np.median` etc.<jupyter_code>diamonds.head()
plt.figure(figsize=(8,6))
sns.barplot(x='cut',y='price',data=diamonds,estimator=np.std)
plt.show()<jupyter_output><empty_output><jupyter_text>PracticeTake a few moments and experiment with the different categorical and numerical features and different aggregate functions. Try to see if you observe anything interesting! CountplotSometimes we may simply be interested in knowing home many members of each category we have in our data. For this purpose we may use a `countplot`. It is essentially a `barplot` with the `estimator` set to be a count function. It also has a `hue` parameter, where you may specifically see how many of the elements in the selected `hue` are in each of the categories you are counting.<jupyter_code>plt.figure(figsize=(8,6))
sns.countplot(x='color',data=diamonds,hue='clarity')
plt.show()<jupyter_output><empty_output><jupyter_text>BoxplotsWe already discussed Boxplots when we talked about Matplotlib. We just want to briefly mention here that we can built them via seaborn as well. As we have already seen, Boxplots are great if we want to gain a good understanding of the distributions of different variables, as well as the distribution of a certain numerical value for the different categories. For example, if we were interested in knowing how the `price` of the diamonds is distributed among the diamonds of different `cut, color` etc. then a boxplot is a great way to gain a visual representation.It also has a `hue` parameter, which may be useful if we want to gain a better understanding of the distribution of the values of another category within each of the first selected category. Let's discuss and illustrate all of these features below:<jupyter_code>plt.figure(figsize=(12,6))
sns.boxplot(x='cut',y='price',data=diamonds,hue='color')<jupyter_output><empty_output><jupyter_text>Practice: Take a few moments and experiment with all the different parameters and features and see if you notice anything interesting in the data. ViolinplotsViolinplots are very similar to boxplots, in the sense that it also shows the distributions of the categorical data but in a slightly different way: it shows the kernel density estimations for each of the categories. Its innerworkings and parameters,however, are very similar as for the boxplot, including the `hue` parameter.<jupyter_code>plt.figure(figsize=(12,6))

sns.violinplot(x='cut',y='price',data=diamonds)
plt.show()<jupyter_output><empty_output><jupyter_text>Violinplots are especially aesthetacilly pleasing if we have a binary categorical data to set for `hue`.    ExerciseIn this exercise you are to replace all the colors `E, G, I, J, F` with the color `D`. So, at the end of the day you should end up with only colors `D` and `H`. Once you do this, go ahead and set `hue` to `color`.<jupyter_code>#Enter Code here 

diamonds.head()<jupyter_output><empty_output><jupyter_text><jupyter_code>plt.figure(figsize=(12,6))
sns.violinplot(x='cut',y='carat',data=diamonds,hue='color',split=True)<jupyter_output><empty_output><jupyter_text>Matrix Plots - Heatmaps & ClustermapsOne situation where we will often use the Heatmaps is to display the correlation between the different (numerical) features in our data. Recall that the  correlation coefficient $\rho_{X,Y}$ between two random variables $X$ and $Y$ is given by:$$\rho_{X,Y}=\frac{Cov(X,Y)}{\sigma_X\sigma_Y},$$where $Cov(X,Y)=E\left[XY\right]-E\left[X\right]E\left[Y\right]$ and $\sigma_X,\sigma_Y$ are the respective standard deviations.Recall from Linear Algebra, that if our data is normalized to have mean zero and standard deviation one, then the correlation matrix for our data is is simply given by $$Corr(A)=\frac{1}{n-1}A^TA$$where by $A$ we have denoted our datamatrix (for example the dataframe `diamonds`, after normalization, in our case)For example, if we were interested in understanding what the correlation between all the different features in the `diamonds` dataset, we can represent that nicely via a heatmap:<jupyter_code>diamonds.corr()
plt.figure(figsize=(12,8))
sns.heatmap(diamonds.corr(),cmap='coolwarm',annot=True,linecolor='white',lw='1')
plt.xticks(fontsize=13)
plt.yticks(fontsize=13)
plt.show()<jupyter_output><empty_output><jupyter_text>ClustermapsClustermaps are similar in flavor to Heatmaps, however, instead of plotting correlations between the variables, under the hood firt Hiearchical Clustering is run (we will learn about this when we discuss the unsupervised learning methods later in the semester, so don't worry too much about understanding the mechanics behind it for now) and then a variables that are more similar to one another are placed closer to each other.<jupyter_code>plt.figure(figsize=(12,8))

sns.clustermap(diamonds.corr(),cmap='coolwarm',standard_scale=1)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09080831408775981, 'fraction_numerical': 0.010808314087759814, 'mean_word_length': 5.895541401273885, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '70600', 'license': 'MIT', 'n_tokens_mistral': 3220, 'n_tokens_neox': 3013, 'n_words': 1346, 'path': 'MA477 - Theory and Applications of Data Science/Lessons/Lesson 5 - Seaborn/Lesson 5 - Seaborn.ipynb', 'repo_name': 'jkstarling/MA477-copy', 'size': '10831'}