<jupyter_start><jupyter_text>![header](https://dbdmg.polito.it/wordpress/wp-content/uploads/2021/05/data_viz_header.jpg)  IntroductionIn this laboratory, you will get acquainted to the basic visualization techniques for the Exploratory Data Analysis (EDA). You will deal with charts to- analyze a data distribution, like **histograms**, **boxplots** and **violin plots**- explore relationships and (linear) correlations in your data, like **scatterplots** or **heatmaps**- visualize the trend and some salient information of a **time series**.Choosing the right type of chart for the type of data at hand is a never ending job. Some [communities](https://www.reddit.com/r/dataisbeautiful/) took it seriously since a while. You will soon find yourself eager to test many approaches: this laboratory will help you getting started.   StructureEach main section of the notebook introduces one of the chart categories listed above. Within the sections, you will find a brief description of type of plot you are required to build, along with a short snippet of code that produces the chart on syntetic data.The whole notebook is written in **Python**. The code level is introductory: if your are a Python master, bear with me. On the other hand, not all the functions are described, but their functioning can be easily inferred either by the name or the output.   For your convenience, some parts of the notebook are pre-compiled, we filled the boilerplate code for you. Although the notebook is sequential, each section is self-contained: feel free to skip back and forth you are more interested in specific parts.  --- So, to recap. Let's say your client provides you and your team a large set of raw, unprocessed data and you are the one in charge to do a first EDA pass. And you happen to know Python.![Data Viz](https://venngage-wordpress.s3.amazonaws.com/uploads/2020/06/image17.png)  Before we start: Python & Co.Python is the *standard de-facto* ecosystem for data scientists and practitionaires. If I had to summarized its advantages over other languages like R, or even some GUI-enabled tools, I would say:- a large, well-documented standard library, and a *huge* number of third-part libraries for virtually *everything* you need- a large, supportive community with a lot of resources- modern Python keeps the syntax at a minimum. To run data science and visualization experiments, the python code almost reduces to *plain english*- a series of online tools that let you experiment with the language: you only need a Google Account to create [Colab](https://colab.research.google.com/notebook) Notebooks, and the same applies for [Kaggle](https://www.kaggle.com/), [Deepnote](https://deepnote.com/), and many more.In this laboratory, we will use three popular libraries: [NumPy](https://numpy.org/devdocs/index.html), [pandas](https://pandas.pydata.org/docs/) and [seaborn](https://seaborn.pydata.org/). Let's run the cell below to check and install all the dependecies for these libraries (do not worry about warnings).<jupyter_code>!pip install --quiet --user numpy
!pip install --quiet --user pandas
!pip install --quiet --user seaborn<jupyter_output><empty_output><jupyter_text>ImportsWe can import them now (along with other useful standard libraries) and provide an alias with:<jupyter_code>import numpy as np
import pandas as pd
import seaborn as sns

# set the style for all the charts in the notebook
sns.set_theme("notebook")

from os.path import join<jupyter_output><empty_output><jupyter_text>Data ManipulationNumPy and Pandas let you easily handle numerical arrays and tabular data respectively.Simply put, NumPy (or numpy in the following) is your best companion for the Geometry class, while Pandas provides you a table-like access to your data (with index and column names, and so on). "Tables" in pandas are known as DataFrames.They have been thoroughly covered in the course Data Science Lab: process and methods at Politecnico di Torino. If you are interest, feel free to visit the [course website](https://dbdmg.polito.it/wordpress/teaching/data-science-lab-process-and-methods-2020-2021/) for slides and laboratories.To give a practical example, let's see how we can create a simple array with all the integers between 0 (included) and 100 (excluded) in numpy:<jupyter_code>np.arange(100)<jupyter_output><empty_output><jupyter_text>or a 3x3 matrix with random floating point numbers from 0 and 1:<jupyter_code>np.random.random((3,3))<jupyter_output><empty_output><jupyter_text>For pandas instead, let's create a toy table-like dataset:<jupyter_code>toy_df = pd.DataFrame(
    {
        "Age": [22, 43, 19, 23, 55],
        "Gender": ["Male", "Female", "Female", "Male", "Female"],
        "Origin": ["Italy", "France", "France", "USA", "USA"]
    }
)
toy_df<jupyter_output><empty_output><jupyter_text>Data VisualizationThe most used library for data visualization in python is [matplotlib](https://matplotlib.org/). However, in this laboratory, you will use **seaborn**.Seaborn is a matplotlib wrapper which provides simpler-to-use, high-level APIs to produce charts with better quality and lower effort. It integrates pandas off-the-shelf and provides [an extended list](https://seaborn.pydata.org/api.html) of chart types already implemented. You better take a look at it: most of the cases the chart you are trying to achieve manually is already there.If you want to do something low-level, like changing the size for all the future figures in the notebook, you have to use matplolib. Let us do it for you.<jupyter_code>import matplotlib.pyplot as plt
plt.rcParams["figure.figsize"] = (8, 4)<jupyter_output><empty_output><jupyter_text>DatasetsYou will deal with two different types of data.First, we will focus on tabular data, with the [Stroke Prediction Dataset](https://www.kaggle.com/fedesoriano/stroke-prediction-dataset).Next, we will do some finance. We collected the ticker prices of eight publicly traded companies, namely Amazon (AMZN), AAPL (AAPL), Alphabet Inc (GOOG), Microsoft (MSFT), Johnson & Johnson (JNJ), Pfizer (PFE), Sanofi (SNY), and AstraZeneca (AZN). Not by coincidence, four of them belong to the tech sector, and the rest to healthcare.Run the cell below to download and extract the datasets.<jupyter_code>!wget -q https://dbdmg.polito.it/wordpress/wp-content/uploads/2021/05/datasets_Data_Theory_Python.zip -O datasets.zip
!unzip -qu datasets.zip
!rm datasets.zip<jupyter_output><empty_output><jupyter_text>Exercise 1. Tabular Data: Stroke Prediction DatasetThe dataset collects information on patients that are likely to get stroke. Each row of the dataset contains relevant information about a patient, like gender, age, various diseases, and smoking status.(From Kaggle) The attributes are:- **id**: unique identifier- **gender**: "Male", "Female" or "Other"- **age**: age of the patient- **hypertension**: 0 if the patient doesn't have hypertension, 1 if the patient has hypertension- **heart_disease**: 0 if the patient doesn't have any heart diseases, 1 if the patient has a heart disease- **ever_married**: "No" or "Yes"- **work_type**: "children", "Govt_jov", "Never_worked", "Private" or "Self-employed"- **Residence_type**: "Rural" or "Urban"- **avg_glucose_level**: average glucose level in blood- **bmi**: body mass index- **smoking_status**: "formerly smoked", "never smoked", "smokes" or "Unknown"*- **stroke**: 1 if the patient had a stroke or 0 if not*Note: "Unknown" in smoking_status means that the information is unavailable for this patientReading and parsing the dataset with pandas is as simple as:<jupyter_code>stroke_df = pd.read_csv(join("datasets", "healthcare-dataset-stroke-data.csv"))
stroke_df.head()<jupyter_output><empty_output><jupyter_text>Explore Data DistributionLet's now build some of the most common charts to analyze data distribution: **histograms**, **box plots** and **violin plots** with seaborn. More advanced configurations can be achieved with **displot** (generic distribution plot), or **catplot** for "category plot".Before addressing the real dataset, let's create a few syntetic data points to test the searborn APIs.<jupyter_code>synt = np.random.random(1000)
synt.shape<jupyter_output><empty_output><jupyter_text>We can then plug them into one of the various *histplot*, *boxplot* or *violinplot* already available in seaborn. *Note that boxplots and violinplots cannot directly be applied to non-numerical variables as they show distributions. Histplot, instead, can display simple counts for each value of a categorical variable.*<jupyter_code>sns.histplot(y=synt)<jupyter_output><empty_output><jupyter_text>From the output, we can see that the method np.random.random samples points from a uniform distribution between 0 and 1.As previously mentioned, seaborn comes integrated with pandas. You can call any of its plotting methods specifying a dataframe to the `data` argument and a column name for the other parameters, like `x` in the previous cell.<jupyter_code>sns.boxplot(x="Age", data=toy_df)<jupyter_output><empty_output><jupyter_text>Across almost all its functions, seaborn let's you easily separate the plot based on a variable of interest. You can specify it with the parameter `hue`:<jupyter_code>sns.histplot(x="Age", hue="Gender", data=toy_df)<jupyter_output><empty_output><jupyter_text>---Let's now analyze the real stroke dataset. Specifically, try to answer the following questions:- Q1: what is the age distribution of patients?- Q2: are there more male or female smokers?- Q3: how is the age distributed for people who got stroke and those who did not?<jupyter_code>#### TODO: YOUR CODE HERE (~10 minutes)<jupyter_output><empty_output><jupyter_text>Explore Data CorrelationLet's now focus on exploring data correlation in our dataset. We will mainly use scatterplot and, again, **displot**. The latter is a versatile function that accepts either a single or two inputs.Again, let's start with a few syntetic data points.<jupyter_code>synt = np.random.normal(size=(1000, 2))
synt.shape
sns.displot(x=synt[:, 0], y=synt[:, 1], kind="hist", rug=True)<jupyter_output><empty_output><jupyter_text>Here we changed the generator: np.random.normal, as per its name, draws data point from a normal distribution. The displot plots the first 1000 points (`synt[:, 0]`) on the horizontal axis and the second 1000 points (`synt[:, 1]`) on the vertical axis. The `rug=True` its a nice addition: it adds small segments corresponding to data points on a single dimension (mathematicians will tell you that those are the two marginal distributions).Similarly, the **scatterplot** results in:<jupyter_code>sns.scatterplot(x=synt[:, 0], y=synt[:, 1], alpha=0.5) # alpha: "opacity"<jupyter_output><empty_output><jupyter_text>---Let's now analyze the real stroke dataset. Specifically, try to answer the following questions:- Q1: what is the correlation between the body mass index and average glucose level?- Q2: what is the correlation between the body mass index and average glucose level, separately by stroke yes/no?- Q3: what it the correlation between the body mass index, the average glucose level, and the age?- Q4: what it the correlation between the body mass index, the average glucose level, the age, and the probability of get stroke?*Note: to answer Q2 and Q3 dive into the documentation of scatterplot/displot and search for color and/or size*<jupyter_code>#### TODO: YOUR CODE HERE (~10 minutes)<jupyter_output><empty_output><jupyter_text>The easiest waySeaborn provides you a single method that generates multiple type of correlation plots in a single figure. Let's try **pairplot**.<jupyter_code>sns.pairplot(data=stroke_df, vars=["age", "bmi", "avg_glucose_level"], hue="stroke")<jupyter_output><empty_output><jupyter_text>Exercise 2. Financial Time SeriesWe switch now to a completely different domain: financial time series. We have the candlestick values (Open, High, Low, Close, Adjusted Close, and Volume) of eight public companies. The stock data is separated in different files. Each file contains the OHLCV values in tabular format: each row corresponds to a trading day. Our data is two-years long, from May 1, 2019 to May 1, 2021.The time series will give use the opportunity two use two more types of chart: **lineplot** and **heatmap**.First, we load them into a dictionary with company names as keys and pandas dataframes as values.Then, for each ticker, we also add a new column representing the *percentage_change* between the current day's close and the previous one.$$ pct\_change_t = \frac{(close_t - close_{t-1})}{close_{t-1}} $$<jupyter_code>tech = ["AAPL", "AMZN", "GOOG", "MSFT"] 
healthcare = ["JNJ", "PFE", "SNY", "AZN"]
tickers = tech + healthcare

ticker_dfs = {
    tick: pd.read_csv(join("datasets", f"{tick}.csv"), parse_dates=True, index_col="Date")
    for tick in tickers
}

# add pct_change
for _, tick_df in ticker_dfs.items():
    tick_df["pct_change"] = tick_df["Close"].pct_change()

ticker_dfs["AAPL"].head()<jupyter_output><empty_output><jupyter_text>Explore trend and search co-trending stocksThe syntax of seaborn methods is the same also for the next charts. If we want to analyze a single series, we can simply pass it to the parameter `data`, seaborn will take care of the rest (axis scaling, choose the right labels for days, etc.).<jupyter_code>sns.lineplot(data=ticker_dfs["AAPL"]["Close"], label="AAPL")<jupyter_output><empty_output><jupyter_text>In all the previous examples we run a single plot per cell. Try now to execute several lineplot in the same cell, specifically plot:- Q1: the Close series for all the healthcare stocks- Q2: the pct_change series for all the tech stocks<jupyter_code># TODO: YOUR CODE HERE (~10 mins)<jupyter_output><empty_output><jupyter_text>The last chart is pretty much garbage. Let's **search for co-trending behaviors** in another way. Let's go through the following steps:1. collect in a new dataframe only the pct_change series2. compute the [Pearson correlation](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient) for each pair of series. A strong correlation of any two series signals that the two stocks were moving very similarly3. plot a heatmap chart to easily spot correlations<jupyter_code>pct_change_df = pd.DataFrame({tick: ticker_dfs[tick]["pct_change"] for tick in tickers})
pct_change_df.head()
sns.heatmap(pct_change_df.corr())<jupyter_output><empty_output><jupyter_text>We can still improved the chart a bit. Since we know that the Pearson correlation goes from -1 (negatively correlated - if one increase, the other one decreases) to 1 (positively correlated - if one increases, the other one does the same), **we can set the color scale to match the range \[-1, 1\]. Search the heatmap documentation and add this change**.<jupyter_code># TODO: YOUR CODE HERE (~5 mins)<jupyter_output><empty_output><jupyter_text>Conclusion (and what we have left aside)In this laboratory we have introduced many of the basic types of charts offered by the seaborn library. The API is vaste though, and we are far from having completely explored it all. Furthermore, note that custom, low-level charts require the a solid knowledge of the inner matplotlib functioning, but that went beyond the scope of the laboratory. On the other hand, we have left aside other charting strategies, such as **part-whole** visualizations, or plots that help visualizing **rankings**.Finally, other charts complete the landscape. **Maps** are a completely different matter, and require specific, more advanced libraries. The same applies to **wordclouds** (and in general textual representations).  Bonus. InteractivityWe want to give you a final pointer to the topic of interactive charting. One of the most used open source charting library for interactivity is [plotly](https://plotly.com/graphing-libraries/). The library is available for Python, R, and Javascript.The class names are different, the name of the functions are different but the syntax of the [plotly-express](https://plotly.com/python/plotly-express/) interface is similar to seaborn.The following cells install plotly for python and produce the lineplot and the correlation heatmap drawn with seaborn before, but with interactivity enabled.  DISCLAIMERAt the time of writing, Plotly is not fully supported by Jupyter Lab. In order to run and see the results of the last cell, you have to switch to the classical Jupyter Notebook interface. You can do it by simply replacing *lab* with *tree* in the current URL string.<jupyter_code>!pip install --quiet --user plotly
import plotly.express as px
fig = px.line(ticker_dfs["AAPL"]["Close"], title="AAPL: Close Price")
fig.show()
fig = px.imshow(pct_change_df.corr(), title="Close Price Correlation")
fig.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08832826385161328, 'fraction_numerical': 0.007514761137949544, 'mean_word_length': 5.496706702828361, 'pattern_counts': {'":': 3, '<': 93, '<?xml version=': 0, '>': 93, 'https://': 17, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '85217', 'license': 'MIT', 'n_tokens_mistral': 4879, 'n_tokens_neox': 4592, 'n_words': 2123, 'path': 'main.ipynb', 'repo_name': 'g8a9/intro-data-visualization', 'size': '16767'}