<jupyter_start><jupyter_text>Python Tutorial: Intro to the Matplotlib libraryThursday 2020-02-04 17:00 to 18:00, UTC-05:00Instructor: Dr <NAME>, Ph.D.This tutorial aims at introducing participant to basic functionalities of the Python's library **Matplotlib**. The tutorial is divided in five parts: basic plots, plot attributes, subplots, and plotting the *iris dataset*.Contents```1. General Setup2. Basic Plots    2.1. Scatter plot    2.2. Histogram3. Plot Attributes    3.1. Axis labels and plot title    3.2. Plot style and colors    3.3. Adding horizontal and vertical lines    3.4. Labels and 4. Figure and Subplots5. Plotting the Iris Dataset (example)```  1. General SetupWe are using Python 3.<jupyter_code>import sys
print("Python version :", sys.version)<jupyter_output>Python version : 3.7.4 (default, Aug 13 2019, 15:17:50) 
[Clang 4.0.1 (tags/RELEASE_401/final)]<jupyter_text>Import the **Matplotlib** library and check the version.<jupyter_code>import matplotlib as mpl
import matplotlib.pyplot as plt
print ("Matplotlib version :", mpl.__version__)<jupyter_output>Matplotlib version : 3.1.1<jupyter_text>To test various plots and plot attributes, we will be using synthetic datasets. In order to generate them, we need to import some librairies.<jupyter_code>import numpy as np
import random<jupyter_output><empty_output><jupyter_text>2. Basic Plots  Line PlotRessource: https://matplotlib.org/3.2.1/api/_as_gen/matplotlib.pyplot.plot.htmlLine plots are useful, when we want to visualize $y = f(x)$ functions.Let's use $\sin$ and $\cos$ functions in our examples.<jupyter_code>## Numpy arange (start, end, jump)
x_line = np.arange(0, 15, 0.1)

y_sin = np.sin(x_line)
y_cos = np.cos(x_line)<jupyter_output><empty_output><jupyter_text>We can easily draw lines using the ```plot``` function: ```plt.plot(x, y, args**)```. To display a plot without saving it to a file, we can use the ```show``` function: ```plt.show()```.<jupyter_code>## Plot the sin function
plt.plot(x_line, y_sin)
plt.show()<jupyter_output><empty_output><jupyter_text>Now, let's generate two new datasets.<jupyter_code>y_sin_shift = [i+2 for i in y_sin]
y_cos_shift = [i+2 for i in y_cos]<jupyter_output><empty_output><jupyter_text>When plotting line graphs, we can specify various **line types** with the argument ```ls```.The main **line types** are:1. Solid ______2. Dashed -----3. Dashdot -.-.-.-4. Dot ......The **line width** can also be modified using the ```lw``` argument.<jupyter_code>## Plot the sin function with solid linestyle (ls)
plt.plot(x_line, y_sin, ls='-', lw=1)

## Plot the cos function with dashed linestyle 
plt.plot(x_line, y_cos, ls='--')

## Plot the cos function with dashdot linestyle 
plt.plot(x_line, y_sin_shift, ls='-.', lw=3)

## Plot the cos function with dot linestyle 
plt.plot(x_line, y_cos_shift, ls=':')
plt.show()<jupyter_output><empty_output><jupyter_text>We can also define a specific ```marker``` to illustrate are linked datapoints.<jupyter_code>x = range(10)
y = np.random.normal(0, 10, 10)

## Plot the x-y relationship and identify the data points with '*'
plt.plot(x, y, marker='*')
plt.show()<jupyter_output><empty_output><jupyter_text>2.1. Scatter plotRessource: https://matplotlib.org/3.2.1/api/_as_gen/matplotlib.pyplot.plot.htmlThere are two ways to draw scatter plots:1. using the ```plot``` function (faster when markers are identical in size and color) -> ```plt.plot(x, y, 'o', args**)```2. using the ```scatter```function -> ```plt.scatter(x, y, args**)``` We will be using three randomly generated gaussian datasets.<jupyter_code>N = 150

x_scatter = [np.random.normal(0, 2, N), np.random.normal(5, 2, N), np.random.normal(10, 2, N)]
y_scatter = [np.random.normal(4, 5, N), np.random.normal(0, 2, N), np.random.normal(8, 2, N)]<jupyter_output><empty_output><jupyter_text>Let's first try with the ```plot```function...Remember what is the format of our $x$ and $y$ data!<jupyter_code>## Plot the three datasets as scatter plots
plt.plot(x_scatter, y_scatter, 'o', alpha=0.7)
plt.show()<jupyter_output><empty_output><jupyter_text>Now, let's try with the ```scatter```function.<jupyter_code>## Plot the three datasets as scatter plots
plt.scatter(x_scatter, y_scatter)
plt.show()<jupyter_output><empty_output><jupyter_text>Fundamentally, ```scatter``` works with 1-D arrays; $x$ and $y$ may be input as 2-D arrays, but they will be flattened. This is why we are losing our three distinct datasets. To draw the various datasets as distinct instances within the same plot, we could use a ```for```loop. Each dataset will be assigned a default color. We can use the ```alpha``` parameter to play with our marker transparency, which is useful when plotting a lot of overlapping points.<jupyter_code>## Plot the three datasets as distinct scatter plots
for i in range(len(x_scatter)) :
    plt.scatter(x_scatter[i], y_scatter[i], alpha=0.7)
    
plt.show()<jupyter_output><empty_output><jupyter_text>The default marker is ```o``` but there exists many more options.<jupyter_code>filled_markers = ['o', 'v', '^', '<', '>', '8', 's', 'p', '*', 'h', 'H', 'D', 'd', 'P', 'X']<jupyter_output><empty_output><jupyter_text>To change the marker style of a scatter plot, we must specify the ```marker``` argument. We can also specify the marker size with the ```s``` argument.<jupyter_code>x_marker = range(len(filled_markers))
y_marker = x_marker

## Plot the marker datapoints with different marker 
for i in x_marker :
    plt.scatter(x_marker[i], y_marker[i], s=50, marker=filled_markers[i])<jupyter_output><empty_output><jupyter_text>2.2. HistogramRessource: https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.hist.htmlWe can plot the histogram of $x$ using the ```hist```function. Let's generate a dataset of 5 randomly generated gaussian distributions.<jupyter_code>M = 250

mu = [0, 4, 6, 10, 15]
sigma = [4, 2, 1, 5, 3]

norm_dist = [np.random.normal(mu[i], sigma[i], M) for i in range(len(mu))]<jupyter_output><empty_output><jupyter_text>Multiple data can be provided via x as a list of datasets of potentially different lengths, or as a 2-D ndarray in which each column is a dataset.The default ```histtype``` is set to ```bar```.<jupyter_code>## Plot the histograms of the various distributions.
plt.hist(norm_dist)
plt.show()<jupyter_output><empty_output><jupyter_text>In our case, we would like to see the distributions, and the ```bar``` representation might not be appropriate. We can try to plot individual histograms using a ```for```loop.<jupyter_code>## Plot the histograms of the various distribution
for i in range(len(norm_dist)) :
    plt.hist(norm_dist[i], alpha=0.7)
    
plt.show()<jupyter_output><empty_output><jupyter_text>The results is more of what we expect, but it is not quite there yet: we want the various histograms to share the same bins. To do so, we can define the ```histtype``` to be either ```barstacked``` or ```stepfilled```.We can also specify that we want the density distribution, rather than the number of occurrences, by setting the ```density``` argument to ```True``` (by default, it is set to ```False```)<jupyter_code>## Plot the histograms of the distributions using barstacked or stepfilled.
plt.hist(norm_dist, histtype='stepfilled', alpha=0.7)
plt.show()<jupyter_output><empty_output><jupyter_text>It can also be useful to define the number of bins, using the ```bins``` argument.<jupyter_code>## Change the number of bins to 50.
plt.hist(norm_dist, bins=50, histtype='stepfilled', density=True,  alpha=0.7)
plt.show()<jupyter_output><empty_output><jupyter_text>3. Plot attributes  3.1. Axis labels and plot title We can add a title and axis labels to our plot by using the ```title```, ```xlabel``` and ```ylabel``` functions. They all take as input a string and optional arguments (*e.g.* font size and color).We ca integrate LaTEX notation with ```$...$``` and some variable values with ```%d```, ```%s``` and ```%f```.* ```'$K=3$, $n_{k}=150$' ``` -> $K=3$, $n_{k}=150$* ```'There are %d points' % (len(x_scatter[0]))```-> There are 150 points<jupyter_code>for i in range(len(x_scatter)) :
    plt.scatter(x_scatter[i], y_scatter[i], marker=filled_markers[i*3], alpha=0.7)

## Add a title to our plot
plt.title('Scatter Plot ($K=3$, $n_{k}=$%s)' % (len(x_scatter[0])))

## Add axis labels to our plot
plt.xlabel('First Component')
plt.ylabel('Second Component')

plt.show()<jupyter_output><empty_output><jupyter_text>3.2. Plot style and colors It is possible to use predefined plot styles. Matplotlib has a lot of options to choose from!<jupyter_code>print (plt.style.available)<jupyter_output>['seaborn-dark', 'seaborn-darkgrid', 'seaborn-ticks', 'fivethirtyeight', 'seaborn-whitegrid', 'classic', '_classic_test', 'fast', 'seaborn-talk', 'seaborn-dark-palette', 'seaborn-bright', 'seaborn-pastel', 'grayscale', 'seaborn-notebook', 'ggplot', 'seaborn-colorblind', 'seaborn-muted', 'seaborn', 'Solarize_Light2', 'seaborn-paper', 'bmh', 'tableau-colorblind10', 'seaborn-white', 'dark_background', 'seaborn-poster', 'seaborn-deep']<jupyter_text>To select a style, we use the ```style.use``` function. We can also define other attributes such as font size by using the ```rc``` functions. This style and these attributes will be applied to all new plots.Ressource: https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.rc.html<jupyter_code>plt.style.use('seaborn-whitegrid')
plt.rc('font', size=15)<jupyter_output><empty_output><jupyter_text>Let's see what our scatter plot looks like with this new style!<jupyter_code>for i in range(len(x_scatter)) :
    plt.scatter(x_scatter[i], y_scatter[i], marker=filled_markers[i*3], alpha=0.7)
    
plt.title('Scatter Plot ($K=3$, $n_{k}=$%s)' % (len(x_scatter[0])))
plt.xlabel('First Component')
plt.ylabel('Second Component')

plt.show()<jupyter_output><empty_output><jupyter_text>Commands which take color arguments can use several formats to specify the colors.Each style has its own cycle of color: these are the colors used by default when we do not define a specific color. If we plot more elements than there are colors in the cycle, we will have color duplicates in our plot.We can access the various colors of the cycle with ```C``` followed by a digit: ```C0``` is the first color of our cycle. Ressources: https://matplotlib.org/2.0.2/api/colors_api.html<jupyter_code>for i in range(20) :
    #plt.scatter(i, i, s=50)
    plt.scatter(i, i, s=50, color='C%d' % (i))
    
plt.show()<jupyter_output><empty_output><jupyter_text>We can also use the basic built-in colors and refer to them either with a single letter or their complete name: * ```b```: blue* ```g```: green* ```r```: red* ```c```: cyan* ```m```: magenta* ```y```: yellow* ```k```: black* ```w```: white<jupyter_code>cl = ['b', 'g', 'r', 'c', 'm', 'y', 'k', 'w']

for i in range(len(cl)) :
    plt.scatter(i, i, s=50, color=cl[i])
    
plt.show()<jupyter_output><empty_output><jupyter_text>We can also explicitely define the color from the available color list, or with color codes.Ressources: https://matplotlib.org/2.0.0/_images/named_colors.png<jupyter_code>cl = ['paleturquoise',  'mediumorchid', 'goldenrod', 'antiquewhite', '#ff800d']

for i in range(len(cl)) :
    plt.scatter(i, i, s=50, color=cl[i])
    
plt.show()<jupyter_output><empty_output><jupyter_text>3.3. Adding horizontal and vertical linesHorizontal and vertical lines can be very informative! They can illustrate mean and median, the bounds of a confidence interval, a threshold, a reference value, etc.They are very simple to plot :D * ```plt.axhline(y, args**)```* ```plt.axvline(x, args**)```The arguments are similar to the ones used when drawing a line plot: ```ls```, ```lw```, ```color```, etc.Ressource (horizontal line): https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.axhline.htmlRessource (vertical line): https://matplotlib.org/api/_as_gen/matplotlib.pyplot.axvline.html<jupyter_code>for i in range(len(x_scatter)) :
    plt.scatter(x_scatter[i], y_scatter[i], marker=filled_markers[i*3], alpha=0.7, color='C%d' % (i))
    
    ## Plot the x-mean
    plt.axvline(x=np.mean(x_scatter[i]), color='C%d' % (i), ls='-')
    
    ## Plot the y-mean
    plt.axhline(y=np.mean(y_scatter[i]), color='C%d' % (i), ls='--')
    
    
plt.title('Scatter Plot ($K=3$, $n_{k}=$%s)' % (len(x_scatter[0])))
plt.xlabel('First Component')
plt.ylabel('Second Component')

plt.show()<jupyter_output><empty_output><jupyter_text>Here's an example when plotting histograms<jupyter_code>plt.hist(norm_dist, bins=50, histtype='stepfilled', density=True, alpha=0.7)

for i in range(len(norm_dist)) :
    plt.axvline(np.mean(norm_dist[i]), color='C%d' % (i), alpha=1)
    
plt.title('Normal Distributions\n($N=$%d, $n_{k}=$%d)' % (len(norm_dist), len(norm_dist[0])))
plt.xlabel('Value')
plt.ylabel('Density')
    
plt.show()
x = [[1], [2], [3], [4], [5]]
cat = ['a', 'b', 'c', 'd', 'e']
mean_x = [1, 2, 3, 4, 5]

plt.bar(cat, mean_x, color=['k', 'g', 'y', 'r', 'b'])
plt.show()<jupyter_output><empty_output><jupyter_text>Labels and LegendRessource: https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.legend.html<jupyter_code>## Define the labels
labels = ['Distribution %d' % (i) for i in range(len(norm_dist))]

## Add the labels
plt.hist(norm_dist, bins=50, histtype='stepfilled', density=True, alpha=0.7, label=labels)

for i in range(len(norm_dist)) :
    plt.axvline(np.mean(norm_dist[i]), color='C%d' % (i))
    
plt.title('Normal Distributions\n($N=$%d, $n_{k}=$%d)' % (len(norm_dist), len(norm_dist[0])))
plt.xlabel('Value')
plt.ylabel('Density')

## Show the legend and place it outside
plt.legend(bbox_to_anchor=(1.,1.), ncol=2, frameon=True, facecolor='white')

plt.show()<jupyter_output><empty_output><jupyter_text>4. Figure and SubplotsRessource: https://matplotlib.org/3.2.1/api/_as_gen/matplotlib.pyplot.subplots.html```plt.subplots(nrows, ncols, figsize, sharex, sharey)```<jupyter_code>N = len(x_scatter)
fig, ax = plt.subplots(len(x_scatter), len(y_scatter), figsize=(12, 9), sharex='row', sharey='row')

### First row: scatter plot of FC vs. SC
r = 0

for i in range(N) :
    c = i
    ax[r][c].scatter(x_scatter[c], y_scatter[c], color='C%d' % (c))
    ax[r][c].set_xlabel('First Component')
    ax[r][c].set_ylabel('Second Component')
    ax[r][c].set_title('Distribution %d' % (c))
    
### Second row: plot distribution of first component w/ mean
r = 1

for c in range(N) :    
    ax[r][c].hist(x_scatter[c], histtype='stepfilled', color='C%d' % (c))
    ax[r][c].axvline(np.mean(x_scatter[c]), ls='-', color='k', label="%.2f" % (np.mean(x_scatter[c])))
    ax[r][c].legend(frameon=True)

    
### Third row: plot distribution of second component w/ mean
r = 2

for c in range(N) :    
    ax[r][c].hist(y_scatter[c], histtype='stepfilled', color='C%d' % (c))
    ax[r][c].axvline(np.mean(y_scatter[c]), ls='-', color='k', label="%.2f" % (np.mean(y_scatter[c])))
    ax[r][c].legend(loc='upper right', frameon=True)

plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>Here's an example if we plot the datasets column-wise rather than row-wise. This visualization can be very helpful when comparing various datasets.<jupyter_code>N = len(x_scatter)
fig, ax = plt.subplots(len(x_scatter), len(y_scatter), figsize=(12, 9), sharex='col', sharey='col')

### First col: scatter plot of FC vs. SC
c = 0

for r in range(N) :
    ax[r][c].scatter(x_scatter[r], y_scatter[r], color='C%d' % (r))
    ax[r][c].set_xlabel('First Component')
    ax[r][c].set_ylabel('Second Component')
    ax[r][c].set_title('Dataset %s' % r)
    
### Second col: plot distribution of first component w/ mean
c = 1

for r in range(N) :
    ax[r][c].hist(x_scatter[r], bins=20, histtype='stepfilled', density=True, color='C%d' % (r))
    ax[r][c].axvline(np.mean(x_scatter[r]), ls='-', color='k')
    ax[r][c].set_xlabel('First Component')
    ax[r][c].set_ylabel('Density')
    
### Third col: plot distribution of second component w/ mean
c = 2

for r in range(N) :
    ax[r][c].hist(y_scatter[r], bins=20, histtype='stepfilled', density=True, color='C%d' % (r))
    ax[r][c].axvline(np.mean(y_scatter[r]), ls='-', color='k')
    ax[r][c].set_xlabel('Second Component')
    ax[r][c].set_ylabel('Density')

plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>5. Plotting the Iris Dataset (example)<jupyter_code>## Import scikit learn library
from sklearn import datasets
from sklearn.decomposition import PCA
## Import the iris dataset
iris = datasets.load_iris()
names = iris.target_names
feature = iris.feature_names
target = iris.target
X_reduced = PCA(n_components=len(feature)).fit_transform(iris.data)
fig, ax = plt.subplots(1, 6, figsize=(24, 4))

for i in range(len(target)) :
    tg = target[i]
    tmp = X_reduced[i]
    
    ax[0].scatter(tmp[0], tmp[1], color='C%d' % (tg))
    ax[1].scatter(tmp[0], tmp[2], color='C%d' % (tg))
    ax[2].scatter(tmp[0], tmp[3], color='C%d' % (tg))
    ax[3].scatter(tmp[1], tmp[2], color='C%d' % (tg))
    ax[4].scatter(tmp[1], tmp[3], color='C%d' % (tg))
    ax[5].scatter(tmp[2], tmp[3], color='C%d' % (tg))
    
ax[0].set_xlabel('PC1')
ax[0].set_ylabel('PC2')

ax[1].set_xlabel('PC1')
ax[1].set_ylabel('PC3')

ax[2].set_xlabel('PC1')
ax[2].set_ylabel('PC4')

ax[3].set_xlabel('PC2')
ax[3].set_ylabel('PC3')

ax[4].set_xlabel('PC2')
ax[4].set_ylabel('PC4')

ax[5].set_xlabel('PC3')
ax[5].set_ylabel('PC4')

#ax[0].legend()

plt.tight_layout()
plt.show()
fig, ax = plt.subplots(1, 6, figsize=(24, 4))

for i in range(len(target)) :
    tg = target[i]
    tmp = iris.data[i]
    
    ax[0].scatter(tmp[0], tmp[1], color='C%d' % (tg))
    ax[1].scatter(tmp[0], tmp[2], color='C%d' % (tg))
    ax[2].scatter(tmp[0], tmp[3], color='C%d' % (tg))
    ax[3].scatter(tmp[1], tmp[2], color='C%d' % (tg))
    ax[4].scatter(tmp[1], tmp[3], color='C%d' % (tg))
    ax[5].scatter(tmp[2], tmp[3], color='C%d' % (tg))


plt.tight_layout()
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17295790354056315, 'fraction_numerical': 0.018790075271814886, 'mean_word_length': 5.465753424657534, 'pattern_counts': {'":': 0, '<': 128, '<?xml version=': 0, '>': 132, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '32', 'dir': 'jupyter-structured-clean-dedup', 'id': '163692', 'license': 'MIT', 'n_tokens_mistral': 6917, 'n_tokens_neox': 6625, 'n_words': 1757, 'path': 'Tutorials/Python_Tutorial/Workshop_data_visualization_with_Python_and_Matplotlib/Python_Matplotlib_tutorial.ipynb', 'repo_name': 'sebhtml/data-trek-2020', 'size': '17947'}