<jupyter_start><jupyter_text>Introduction  Context **Prerequisite:** We assume in this workshop that basic notions of Python are already mastered (data types and variables, data operations, dataframes, conditional statements, etc.). **Content:** Understand data visualization fundamentals and learn how to leverage it via Python's main libraries (MatplotLib, Seaborn, Bokeh, etc.) **Length:** 3h  Colab Hosted Jupyter Notebook 101 Before starting the workshop, take a few minutes to play with a new Notebook. Mainly, *  Hosted and shared. *  Librairies & Installation.*  Environment (CPU, GPU, TPU).Now let's look at *  Basic operations in Python.*  Markup and Structure of an Notebook. Basics of Data Visualization with MatplotLib Summary – Why data visualization? **It is said that an image is worth a thousand words.** Actually, an image can say it all if it’s properly shaped and well presented. As such, it’s a powerful tool to tell a story, explain a phenomenon or raise a case and defend a plan. **Here’s a great example:** you want to hire more people and want your boss to understand why quickly?(**source:** "Storytelling with Data" by <NAME> Knafli)  **Note:** It doesn’t always need to be a complex display; But to get there, you need to master a few visualization concepts & tools. The purpose of this workshop is to get you there by exploring a few topics. **But first let’s start by the basics, MatPlotLib.**  Introduction to MatplotLib [MatplotLib](https://matplotlib.org/) is a free open source library that enable fast and powerful 2D data visualization. It’s is often the way to go to start exploring data in Python. Other libraries build on MatplotLib to offer off-theshelve advanced visualisation tools (e.g., [seaborn](https://seaborn.github.io/), [holoviews](http://holoviews.org/), [ggplot](http://ggplot.yhathq.com/), [basemap](http://matplotlib.org/basemap) and [cartopy](http://scitools.org.uk/cartopy/docs/latest)).  PyPlot **Pyplot is MatplotLib’s main visualization module.** It was designed to mimic Matlab while being used in Python for free.  **Note:** MatPlotLib has also a nother module, pylab ; “pylab combines pyplot with numpy into a single namespace. This is convenient for interactive work, but for programming it is recommended that the namespaces be kept separate” (source) **In this workshop, we’ll focus on Pyplot as it is the visualization module.** First Steps with PyPlot  **GOAL:** Before diving into MatPlotLib's technicalities, explore a simple example of data visualization: generate data, prepare data to be displayed, and annotate the plot. **Note: ** this first example is available on MatplotLib's website. We modified its code a bit in order to focus only on MatplotLib & Pandas (instead of Numpy). **We assume that you know how DataFrames work.**<jupyter_code>#Import Libraries
import matplotlib.pyplot as plt
import pandas as pd<jupyter_output><empty_output><jupyter_text>Usually you would import your data from a csv or a database. But for the sake of the exercise, we'll create a dataframe from scratch. Click on the execute button (left of "Data Preparation").If you want to explore the code behind it, double click on it.<jupyter_code>#@title Data Preparation (click on the execute button)
#Data Preparation - usually you have already a dataframe ready to be explored. But for the sake of this workshop we'll start by setting up one from scratch. 
df = pd.DataFrame({
        'menMeans':[20, 35, 30, 35, 27],
        'womenMeans':[25, 32, 34, 20, 25],
        'menStd':[2, 3, 4, 1, 2],
        'womenStd':[3, 5, 2, 3, 3]})
#Explore the DataFrame
df.head()
#We focus on plotting a bar chart

l_col = len(df.values) # the x locations for the groups
ind = list(range(l_col))
width = 0.5       # 1 means that all bars are connected. Less than 1 creates a space between bars. 

# Plotting functionality based on the available data in the DataFrame.
p1 = plt.bar(ind, df['menMeans'], width, yerr=df['menStd'])
p2 = plt.bar(ind, df['womenMeans'], width,
             bottom=df['menMeans'], yerr=df['womenStd'])

# Axes definition
plt.ylabel('Scores')
plt.title('Scores by group and gender')
plt.xticks(ind, ('G1', 'G2', 'G3', 'G4', 'G5'))
plt.yticks(list(range(0, 81, 10)))
plt.legend((p1[0], p2[0]), ('Men', 'Women'))

plt.show()<jupyter_output><empty_output><jupyter_text>Matplotlib's Anatomy Let's dive in and explore how a PyPlot display is decomposed. You'll have to manipulate its components in order to get the expected result.  PyPlot Display Objects **MatPlotLib's key aspects:** **Figure:** Every display in Matplotlib is contained in a single box or object, refered to as **Figure**. It encapsulates everything. **Axes:** Within a Figure, you can have a single or several "Axes". each "Axes" object represents a two dimensional plot, with:*   **Title:** Name of the Axes ; it usually summarizes what we expect to see in the plot.*   **xAxis & yAxis:** Orthogonal axes that support the plotted data in 2D.*   **xLabel & yLabel:** Name of xAxis and yAxis ; usually it help reading the plot and understanding the displayed data.*   **xTicks & yTicks:** there are two types, minor and major ticks. They decompose xAxis and yAxis into smaller parts (easier to read)*   **"lines":** This is the data to display ![MatplotLib's Anatomy](https://matplotlib.org/_images/sphx_glr_anatomy_001.png)  Can you draw it yourself?  Build the basic frame, axes and lines.<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
data = np.linspace(0.5, 3.5, 100)

#Transform Data as f1=3+cos(x) and f2=1+cos(1+3x/4)/2
Y1 = 3+np.cos(data)
Y2 = 1+np.cos(1+data/0.75)/2

#Create a figure object
fig = 

#Add Axes via the subplot function.
ax = 
#ax = fig.subplots(1,1)

#Define xaxis and yaxis limites 

#Plot lines!! Include Label information as well.

plt.show()<jupyter_output><empty_output><jupyter_text>Add Title, Labels & Legend<jupyter_code># title, axes labels, legend

# add a grid

fig
#For the rest of the exercice, we'll include random dots on the plot. Then you'll have to add Tickers and the rest of the missing information on the figure.
np.random.seed(19680801)
Y3 = np.random.uniform(Y1, Y2, len(data))
#Add Y3 to the graph, you can manipulate the color and shape of the dots using the parameters marker, markerfacecolor & markeredgecolor.

fig<jupyter_output><empty_output><jupyter_text>Ticker functions Tickers are set automatically based on the data range except ; but you also define them yourself when needed. In this subsection we'll explore how we can manipulate them via [matplotlib.ticker](https://matplotlib.org/3.1.1/api/ticker_api.html) library and its subfunctions AutoMinorLocator, MultipleLocator & FuncFormatter. "This module contains classes to support completely configurable tick locating and formatting."*   **MultipleLocator:** Defines the scale of the major ticks, e.g., "ax.xaxis.set_major_locator(MultipleLocator(x))" where x could be 0.5 for instance, would give you a step size of 0.5 on the xaxis.*   **AutoMinorLocator:** Defines the number of intervals between two major ticks, e.g., "ax.xaxis.set_minor_locator(AutoMinorLocator(2))" would divide the interval between two major ticks in 2 (so a single minor tick between them is added). *   **ScalarFormatter:** "Default formatter for scalars: autopick the format string.", e.g., just instanciated as follows "ax.xaxis.set_minor_formatter(ScalarFormatter())", and it will manage the rest.<jupyter_code>from matplotlib.ticker import AutoMinorLocator, MultipleLocator, ScalarFormatter
#Setup Major ticks on both xAxis and yAxis with a step size of 1 and divide intervals between major ticks in 4 sub intervals
ax.xaxis...
ax.yaxis...
#Bonus shape the ticker width & label
#Note: you can define a tick's width and length via the function tick_params, e.g., ax.tick_params(which='major', param=1.0) param: length, width, labelsize, labelcolor. which coulb be "minor" as well.

fig<jupyter_output><empty_output><jupyter_text>Now, last but not least, setting up minor tick's values.<jupyter_code># setup minor ticks' values automatically. 

fig
# Learn to declutter: remove top and right spines<jupyter_output><empty_output><jupyter_text>Bonus: finalizing the figure with text labels and annotating circles<jupyter_code>##For the purpose of the exercise, we define these two functions, Circle & Text that you can use.
#Circle: draws a cercle with a fixed radius. x & y represent it's position relatively to xAxis and yAxis.
def circle(x, y, radius=0.15):
    from matplotlib.patches import Circle
    from matplotlib.patheffects import withStroke
    circle = Circle((x, y), radius, clip_on=False, zorder=10, linewidth=1,
                    edgecolor='black', facecolor=(0, 0, 0, .0125),
                    path_effects=[withStroke(linewidth=5, foreground='w')])
    ax.add_artist(circle)

#Text: displays a text field. x & y represent it's position relatively to xAxis and yAxis.
def text(x, y, text):
    ax.text(x, y, text, backgroundcolor="white",
            ha='center', va='top', weight='bold', color='blue')

#@title Generate circles & Text Labels for Ticks
# Minor tick label
circle(0.50, -0.10)
text(0.50, -0.32, "Minor tick label")

# Major tick

# Minor tick

# Major tick label

# X Label

# Y Label

# Title

# Blue plot

# Red plot

# Scatter plot

# Grid

# Legend

# Axes

# Figure

fig<jupyter_output><empty_output><jupyter_text>Matplotlib's charts & when to use them? In this section, we'll guide you to explore MatPlotLib's available charts. For documentation, you can rely on [Matplotlib's website](https://matplotlib.org/gallery/index.html).*   Histogram*   Pie Chart*   Box PlotFor the sake of the exercice, we'll rely on this dataset: Paris's real estate data. It's publicly available data that you find [here](https://www.data.gouv.fr/fr/datasets/5cc1b94a634f4165e96436c1/). We took the full dataset, cleaned it a bit and reduced it to sold appartements in Paris. The original file is available [here](https://drive.google.com/open?id=1BJdsMw8i8Ad0gT0LyrhAZYEL0JhN8c0m) (about 4MB). You can download it then load it on colab (under the folder "content"), or you can just use [this raw git url](https://raw.githubusercontent.com/annemariet/tutorials/master/data/RealEstate_Paris_cleaned_17_12_2019.csv) to load it directly with `pandas`.  Load Data<jupyter_code>#To make this section self sufficient, we'll re-import needed libraries. 
import pandas as pd
import matplotlib.pyplot as plt
#Load data and display columns and shape
path_data = "https://raw.githubusercontent.com/annemariet/tutorials/master/data/RealEstate_Paris_cleaned_17_12_2019.csv" 
df_housing = pd.read_csv(path_data, low_memory=False, index_col=0)
print('Data Columns: {}'.format(df_housing.columns))
print('Data shape: {}'.format(df_housing.shape))
# Check the number of zipcodes ("code_postal" or "code_commune") and the number of houses per district<jupyter_output><empty_output><jupyter_text>Histogram Histograms aggregate data in ordered bins:*   xAxis representing the ordered values of bins*   yAxis: reprsenting the aggreagted data as a count of occurences (or frequency).Histograms are used for quantitative variable (not categorical ones, otherwise we would use bar charts instead). It is very useful When looking a single quantitative variable to explore how values are distributed. It often gives a strong indication on it's variations and predictability. When a variable is projected on subdimensions e.g., variable: salary, projection on male/female. You can plot and compare both histogram, which gives an indication on how this variable (salary) evolves within the different categories of the population(male/female).In the following exercise we will look at the distribution of housing prices, and compare it accross Paris area districts. There are several functions available : use whichever you find more convenient.- `plt.hist`- `ax.hist`- `pd.DataFrame.plot` with `kind="hist"`- ...<jupyter_code>#Exercice1: Use hist() function to display a histogram in Pyplot.
#@todo: display a histogram representing prices in Paris's 13th arrondissement. 
#Don't forget to label the axes and to provide a title!

#Exercice2: display several histograms in the same figure
#@todo: compare 18th and 16th arrondissements. 
#Don't forget to label the axes and to provide a title and a legend!
#Bonus: put a logarithmic scale on the y-axis<jupyter_output><empty_output><jupyter_text>PieChart **Pie chart** is a simple circular representation of either quantitative (aggregated) or categorical variables. the full pie represents 100% of the visualized data and each slice represent a percentage of the occurences. Very useful to highlight certain slices and offers a constrasts between the dominant categories and the others.<jupyter_code>data = df_housing['code_commune'].value_counts()

data.index
data.index[0]
data.values[0]
#Exercice3: Use pie() function to display a Pie chart in Pyplot.
#@todo: Display a piechart representing all #sales in Paris, where each category is the arrondissement ("code_commune").  
#Don't forget to label the axes and to provide a title and a legend!
#Bonus: how can you reduce the ugliness of this?<jupyter_output><empty_output><jupyter_text>BoxPlot "The box-and-whisker plot is an exploratory graphic, created by <NAME>, used to show the distribution of a dataset (at a glance). Think of the type of data you might use a histogram with, and the box-and-whisker (or box plot, for short) could probably be useful." [source](https://flowingdata.com/2008/02/15/how-to-read-and-use-a-box-and-whisker-plot/)There are several functions available : use whichever you find more convenient.- `plt.boxplot`- `ax.boxplot`- `pd.DataFrame.boxplot`- ... ![boxplot](https://i2.wp.com/flowingdata.com/wp-content/uploads/2008/02/box-plot-explained.gif?w=1090)<jupyter_code>#Exercice4: Use boxplot() function to display boxplots in Pyplot.
#@todo: Represents all prices per district in a single figure with boxplot().  
#Don't forget to label the axes and to provide a title and a legend!<jupyter_output><empty_output><jupyter_text>Leverage MatplotLib On Real Data - Answering Open Questions  Load Data  Let's explore Paris's real estate data. It's publicly available data that you find [here](https://www.data.gouv.fr/fr/datasets/5cc1b94a634f4165e96436c1/). We took the full dataset, cleaned it a bit and reduced it to sold appartements in paris. The file is available [here](https://drive.google.com/open?id=1BJdsMw8i8Ad0gT0LyrhAZYEL0JhN8c0m) (about 4MB). **It is the same file as in the previous exercise.** You can download it then load it on colab (under the folder "content"), or you can just use [this raw git url](https://raw.githubusercontent.com/annemariet/tutorials/master/data/RealEstate_Paris_cleaned_17_12_2019.csv) to load it directly with `pandas`.<jupyter_code>#To make this section self sufficient, we'll re-import needed libraries. 
import pandas as pd
import matplotlib.pyplot as plt
#Load data and display columns and shape<jupyter_output><empty_output><jupyter_text>Basic insights Exploring data means answering questions, so let's ask a few question that you'll have to answer via visualization. PS: if you need a few minutes to look at the data, understand columns, explore missing points, etc. Go ahead :)Here are a few "open" questions for you to further explore the data.  **Question1:** What can you tell us about the number of sold items per month and per arrondissement?<jupyter_code>#@todo<jupyter_output><empty_output><jupyter_text>**Question2:** What about prices? which regions are the most/least expensive in Paris (in average)?<jupyter_code>#@todo<jupyter_output><empty_output><jupyter_text>**Question3:** What can we say about the relationship between price & surface?<jupyter_code>#@todo<jupyter_output><empty_output><jupyter_text>**Question4:** Let's assume we build a prediction model to predict the price, how precise can it be?<jupyter_code>#@todo<jupyter_output><empty_output><jupyter_text>Complex view: master Matplotlib's multiple Axes in a Figure  HowTo & Example Putting data into perspective can be simpler with a single Figure divided into several subplots. Matplotlib enables you to do just that. **How?** simply by relying on the pyplot function subplot2grid(). See example below.<jupyter_code>#Example: how to setup multiple axes. 
ex_fig = plt.figure(figsize=(16, 8))
gridsize = (4, 4)
ax1 = plt.subplot2grid(gridsize, (0, 0), colspan=2, rowspan=4)
ax2 = plt.subplot2grid(gridsize, (0, 2), colspan=2, rowspan=2)
ax3 = plt.subplot2grid(gridsize, (2, 2), colspan=1, rowspan=2)
ax4 = plt.subplot2grid(gridsize, (2, 3), colspan=1, rowspan=2)
plt.show()<jupyter_output><empty_output><jupyter_text>Practices multiple Axes<jupyter_code>#@todo: Answer Exercices 2, 3 & 4 in a single figure.<jupyter_output><empty_output><jupyter_text>Built in Visualization with Seaborn For this part of the workshop, we'll explore the same dataset, but rely on another library built on top of Matplotlib called `seaborn`.<jupyter_code>#@todo: import Seaborn library
#@todo: answer question 1 to 4 relying on Seaborn
#@todo: Explore correlations between prices and other features of the dataset.
#@todo: Find and use Seaborn's function that enables you to plot prices, based on the surface, while computing and displaying its linear regression.
#Bonus: build a basic xgboost model to predict prices and visualize:
#1. The curve describing the real prices vs predicted prices
#2. show the centered distribution of prices
#3. what proportion of predicted prices is within 20% of the real price.<jupyter_output><empty_output><jupyter_text>Interactive Visualization with Bokeh  Introduction to Bokeh We'll continue exploring our dataset, but with an interactive library this time: Bokeh. Its official documentation/training can be found [here](https://hub.gke.mybinder.org/user/bokeh-bokeh-notebooks-qikuq6ov/notebooks/tutorial/00%20-%20Introduction%20and%20Setup.ipynb). We'll focus on a few elements to get you acquainted and operational ; you can explore the rest on your own as needed. **Why Bokeh?** Unlike Matplotlib or Seaborn, Bokeh renders its display in HTML and JavaScript. Its purpose is to offer an interactive visualisation tool that can be easily embedded in web applications.  **Basic plotting template?** assuming you already prepared your data, rendering it on Bokeh requires a few steps:1.   Importing Bokeh librairies2.   Choosing rendering type (e.g., html or inline in a notebook)3.   Prepare a plotting canvas (aka Figure)4.   Define plotting schema and layout5.   Invoke show() that will finally display the data based on the previous setps.<jupyter_code># Import Data manipulation Libraries
import pandas as pd


#1 Import Bokeh Libraries
from bokeh.plotting import output_file, output_notebook, show, figure
#2 Rendering File & Enabling Rendering on Jupyter Notebooks
output_file('myDataViz.html')  # Static HTML or use the second line to enable 
output_notebook()  # Rendering in a Jupyter Notebook

#3 Prepare the canvas in which the data will be displayed: figure
fig = figure()  

#4 Define plotting schema and layout (will be explored in the next subsections)

#5 Preview and save 
show(fig)  # See what I made, and save if I like it

## You should observe an empty display since we didn't plug the data (#4)<jupyter_output><empty_output><jupyter_text>First plot based on real data<jupyter_code>#Price/Surface relationship (relying on the same real estate data used before)
#Reminder: we'll be working with the loaded data in a dataframe
df_housing.columns
#Focus on a single arrondissement first (e.g., 75)
df_housing_arr=df_housing[df_housing['code_commune']==75113]
df_housing_arr.shape
# Import Data manipulation Libraries
import pandas as pd

#1 Import Bokeh Libraries
from bokeh.plotting import output_file, output_notebook, show, figure

#2 Rendering File & Enabling Rendering on Jupyter Notebooks
output_file('Real_Estate_Price_Surface_Relationship.html')  # Static HTML or use the second line to enable 
output_notebook()  # Rendering in a Jupyter Notebook

#3 Prepare the canvas in which the data will be displayed: figure
fig = figure(title="Price_Surface_Relationship", x_axis_label='Surface', y_axis_label='Price')  

#4 Define plotting schema and layout (will be explored in the next subsections)
fig.circle(x=, y=, line_width=2)


#5 Preview and save 
show(fig)  # See what I made, and save if I like it<jupyter_output><empty_output><jupyter_text>Using ColumnDataSource We'll update step 4 to rely on Bokeh's ColumDataSource, key elements that we'll rely on to add interactions in the next subsections. For now we'll plot the same data.<jupyter_code># Import Data manipulation Libraries
import pandas as pd

#1 Import Bokeh Libraries
from bokeh.plotting import output_file, output_notebook, show, figure
from bokeh.models import ColumnDataSource

#2 Rendering File & Enabling Rendering on Jupyter Notebooks
output_file('Real_Estate_Price_Surface_Relationship.html')  # Static HTML or use the second line to enable 
output_notebook()  # Rendering in a Jupyter Notebook

#3 Prepare the canvas in which the data will be displayed: figure
fig = figure(title="Price_Surface_Relationship", x_axis_label='Surface', y_axis_label='Price')

#4 Define plotting schema and layout (will be explored in the next subsections)
src = ColumnDataSource(todo)
fig.circle(source=src,x=todo,y=todo, line_width=2)


#5 Preview and save 
show(fig)  # See what I made, and save if I like it<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08740300171925096, 'fraction_numerical': 0.015473258677570744, 'mean_word_length': 5.198732718894009, 'pattern_counts': {'":': 1, '<': 91, '<?xml version=': 0, '>': 91, 'https://': 15, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '263171', 'license': 'Apache-2.0', 'n_tokens_mistral': 6731, 'n_tokens_neox': 6335, 'n_words': 2681, 'path': '02_Data_Visualization_With_Python.ipynb', 'repo_name': 'annemariet/tutorials', 'size': '21537'}