<jupyter_start><jupyter_text>Lecture Notebook: Information VisualizationThis notebook covers several things:1. The basics of plotting Pandas dataframes using matplotlib.2. Some rules of thumb about bar vs line charts, axes, normalization, and whether to interpolate.3. Basics of ggplot on Python4. Seaborn and visualization of statistical data<jupyter_code>import matplotlib.pyplot as plt
import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>Loading familiar data into Pandas We'll use the CEOs dataset from Wikipedia as an example to compare two different sub-populations: those CEOs who are actually **founders**, and those who are simply "**regular CEOs**".<jupyter_code># Read the Wikipedia HTML table containing information about CEOs!

company_ceos_df = pd.read_html('https://en.wikipedia.org/wiki/List_of_chief_executive_officers#List_of_CEOs')[0]

# Clean the references out of the Since field and the Title field...
company_ceos_df['Since'] = company_ceos_df['Since'].apply(lambda x: int(x.split('[')[0].strip()) if '[' in x else int(x))
company_ceos_df['Title'] = company_ceos_df['Title'].apply(lambda x: x.split('[')[0].strip() if '[' in x else x)

# Show the output
company_ceos_df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 185 entries, 0 to 184
Data columns (total 6 columns):
Company      185 non-null object
Executive    185 non-null object
Title        185 non-null object
Since        185 non-null int64
Notes        184 non-null object
Updated      184 non-null object
dtypes: int64(1), object(5)
memory usage: 8.8+ KB<jupyter_text>Now that we have the data, let's split into two dataframes.<jupyter_code># Split the founders
founders_df = company_ceos_df[company_ceos_df['Title'].apply(lambda s: True if 'founder' in s.lower() else False)]

# This is a set difference: we keep only items that are duplicated
regular_ceos_df = pd.concat([company_ceos_df, founders_df]).drop_duplicates(keep=False)
# For inspection: who are non-founders?
regular_ceos_df.info()
# For inspection: who are non-founders?

founders_df<jupyter_output><empty_output><jupyter_text>Plotting our first graphOK, so we'll do our first plot.  We want to see company vs CEO start year, for CEOs who are also founders.  This is a *bar chart* since companies are categorical rather than continuous-valued.<jupyter_code>founders_df.plot(kind='bar', x='Company', y='Since', color='gray')<jupyter_output><empty_output><jupyter_text>This looks pretty ridiculous, because the assumption is that dates start at 0, and that we are measuring dates!  Could we change the graphed value to that conceptually makes more sense, e.g., maybe we should look at **how long** people have been CEOs?<jupyter_code>import datetime
now = datetime.datetime.now()

founders_df['Years'] = founders_df['Since'].apply(lambda x: now.year - x)

fig = founders_df.plot(kind='bar', x='Company', y='Years')


# Based on "domain expertise", we will assume no one should be CEO for more 
# than ~70 years -- if they started at 20, they would be 90...
fig.set_ylim([0, 70])<jupyter_output>/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:4: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  after removing the cwd from sys.path.<jupyter_text>Plotting for comparisonLet's look at how many folks founded companies in each year, comparing founding CEOs vs "regular" CEOs...Here, year can be considered a continuous-valued parameter (although note that we are actually quantizing it to integer values, so fractional years aren't really useful here).<jupyter_code># gca stands for 'get current axis'
ax = plt.gca()

# This will create counts for how many founders started in each year
founders_by_year = founders_df.groupby(['Since']).count()

founder = founders_by_year.plot(kind='line',y='Company',ax=ax, label='Founding CEOs')
regular_ceos_by_year = regular_ceos_df.groupby(['Since']).count()

regular = regular_ceos_by_year.plot(kind='line',y='Company', color='red', ax=ax, \
                                    label='Other CEOs')<jupyter_output><empty_output><jupyter_text>If we look closely at this graph, we'll note there seems to be one founding CEO every year. Could that be?  Maybe we should look more closely!!!We'll re-plot, putting a marker at each point.  And perhaps we can even remove the line from the "founding CEO" plot, just looking at the markers...<jupyter_code># gca stands for 'get current axis' and if we share the x-axis,
# we will be able to plot multiple items against it
ax = plt.gca()

# This will create counts for how many founders started in each year
founders_by_year = founders_df.groupby(['Since']).count()

founder = founders_by_year.plot(kind='line',y='Company',ax=ax, label='Founding CEOs', 
                                marker='x', linewidth=0)
regular_ceos_by_year = regular_ceos_df.groupby(['Since']).count()

regular = regular_ceos_by_year.plot(kind='line',y='Company', color='red', ax=ax, 
                                    label='Other CEOs', marker='+')

plt.xlabel('Year CEO started', fontsize = 16)<jupyter_output><empty_output><jupyter_text>Much clearer -- in fact the blue x's show that founding CEOs are not that common!  Plotting and Thinking about ScaleLet's try another plot, here comparing three items...<jupyter_code>graph_df = pd.DataFrame([{'scale': 100, 'value': 800}, {'scale': 200, 'value': 1200}, {'scale': 500, 'value': 2400}])

graph_df.plot(kind='bar', x='scale', y='value', label='Execution time')<jupyter_output><empty_output><jupyter_text>This plot is perfectly fine, but note that the x-axis actually contains **numeric** items, which might be continuous-valued.  Moreover, there is neither a log-scale nor a linear-scale progression along the axis -- so while our eyes see something that looks non-linear, in fact we can plot this as a line chart and see what's really happening...<jupyter_code>graph_df.plot(kind='line', x='scale', y='value', label='Execution time', marker='o')<jupyter_output><empty_output><jupyter_text>Plotting and NormalizationNow let's look at data and scaling, where perhaps we are looking at phenomena that are quite different.  A common situation is to measure the running time of three computations, using some baseline computation and comparing it with some alternate computation.  We can plot this using bar charts as we see below.<jupyter_code># Suppose we are counting, for three computations, two different components, the
# baseline computation and the alternative.

# These are the "baseline" numbers for some computation
baseline_df = pd.DataFrame([{'comp': 1, 'value': 800}, {'comp': 2, 'value': 5}, {'comp': 3, 'value': 2400}])
# These are alternative computations
alternative_df = pd.DataFrame([{'comp': 1, 'value': 720}, {'comp': 2, 'value': 3}, {'comp': 3, 'value': 2100}])

# We want to plot side-by-side
combined_df = baseline_df.rename(columns={'value': 'baseline'})
combined_df['alternative'] = alternative_df['value']

fig = combined_df.plot.bar(x='comp')<jupyter_output><empty_output><jupyter_text>Wow, we can't see computation 2 at all!  Given that each plot is very different, we may want to normalize each...<jupyter_code>rescaled_df = combined_df.copy()
rescaled_df['alternative'] = combined_df.apply(lambda r: r['alternative'] / r['baseline'], axis=1)
rescaled_df['baseline'] = combined_df.apply(lambda r: 1.0, axis=1)

fig = rescaled_df.plot(kind='bar', x='comp')
fig.set_title('Normalized Performance')<jupyter_output><empty_output><jupyter_text>Note that an "honest" presentation of the data will emphasize that these are normalized, and that the relative running times are quite different.  In fact, sometimes people will put a caption above each bar showing the actual timings.  ggplot and the Grammar of GraphicsThis part of the notebook shows some examples of using ggplot (as ported from its original R implementation) in Python.<jupyter_code>!pip3 install ggplot
# NOTE: this is (on 10-29-2019) a known incompatibility between Pandas and ggplot
# https://github.com/yhat/ggpy/issues/662

# So the cell does not work as it should at the moment!

#from ggplot import *

#ggplot(rescaled_df, aes(x='comp', y='alternative')) + geom_line() + stat_smooth()<jupyter_output><empty_output><jupyter_text>Visualization of Statistical Data with Seaborn<jupyter_code>!pip install seaborn

import seaborn as sb
# Some simple data, random points around a line
points = 150
slope = 0.3

x = np.array(range(points))
# We'll plot these
y = np.random.randn(points) * 5 + x * slope
# Choose a random integer, set z to True if it's positive, else set z to False
z = map(lambda x: x >= 0, np.random.randn(points))

sample_df = pd.DataFrame({'x': x, 'y': y, 'z': z})

sample_df
# Do a scatter plot, with height 4, shading the points based on whether z is True
sb.lmplot('x', 'y', data=sample_df, height=4, aspect=1.5, fit_reg=False, hue="z")

# Do a scatter plot, with height 4, shading the points based on whether z is True
sb.lmplot('x', 'y', data=sample_df, height=4, aspect=1.5, fit_reg=True, hue="z")
# Sample dataset with people + tips
tips_dataset = sb.load_dataset('tips')

tips_dataset
# Sample dataset with people + tips
tips_dataset = sb.load_dataset('tips')

# The question: how do people tip on different days of the week?
tips_dataset['tip_pct'] = tips_dataset.apply(lambda r: r['tip'] / r['total_bill'], axis=1)

# We will create a different graph for each value of 'time' (lunch vs dinner)
g = sb.FacetGrid(tips_dataset, col='time', hue='day',
                 height=4, aspect=1)

# Within each graph, plot total bill vs tip
g.map(plt.scatter, 'total_bill', 'tip_pct')
g.add_legend()
# We will create a different graph for each value of 'time' (lunch vs dinner)
g = sb.FacetGrid(tips_dataset, col='time', hue='day',
                 height=4, aspect=1)

# Within each graph, plot total bill vs tip
g.map(plt.scatter, 'total_bill', 'size')
g.add_legend()
# Let's look at how different factors are influenced by the size
# of the party
sb.pairplot(data=tips_dataset,kind='scatter', hue='size')
# Create histogram bins of size 5
bins = np.arange(tips_dataset.total_bill.min(), tips_dataset.total_bill.max(), 5)

# Cut the bins, and group by size
by_bill_binned = tips_dataset.groupby([pd.cut(tips_dataset.total_bill, bins, precision=0), 
                                       'size']).size().unstack().fillna(0)

by_bill_binned
sb.set(font_scale=1.0)

sb.heatmap(by_bill_binned[by_bill_binned.sum(axis=1) > 3])
sb.boxplot(tips_dataset.time, tips_dataset.total_bill)
tips_dataset.total_bill.sort_values()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11344304733727811, 'fraction_numerical': 0.013313609467455622, 'mean_word_length': 5.184676958261864, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 54, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '475583', 'license': 'Apache-2.0', 'n_tokens_mistral': 3562, 'n_tokens_neox': 3356, 'n_words': 1171, 'path': 'opends4all-resources/opends4all-exploratory-data-analysis/VISUAL-ANALYTICS-visualization.ipynb', 'repo_name': 'ophdesdi/OpenDS4All', 'size': '10816'}