<jupyter_start><jupyter_text>Module 6: Data Visualization After cleaning your data, let's get to know the data a bit better with a data analyst's best friend -- graphs.Data visualization is a very important skill to have while analyzing your data and while presenting your findings to others, since it displays the patterns in the data in a clear format that's easier for others to understand. Often times, a good graph is better than whole paragraphs of explanation. For Data visualization, let's take a look at a rather cute topic: penguins.This dataset can be found [here](https://www.kaggle.com/parulpandey/palmer-archipelago-antarctica-penguin-data)<jupyter_code>import matplotlib.pyplot as plt
#import seaborn as sns
import pandas as pd
import numpy as np
from datascience import *

## reference back to data8 curriculum

penguins = pd.read_csv("penguins_size.csv")
penguins = penguins.dropna()

##don't worry about this line, it just reads in a dataframe in pandas and converts it to a table object in datascience
penguins_tbl = Table.from_df(penguins)
penguins.head()<jupyter_output><empty_output><jupyter_text>Plots You've Used In Data8  Let's fist take a look at some plots we are already familiar with from data 8, notably barplots, histograms, scatter plots, and line plots. If you need to touch up on these concepts, here's the graph section of the [textbook](https://www.inferentialthinking.com/chapters/07/Visualization.html)  Bar plotsBarplots are best used when you are trying to compare numerical properties of different categories. There are five distinct penguin species in this dataset, and barplots can make comparing the mean body mass of the three species fairly easy.In data8, the syntax to generate bar plots is  `tbl.barh("category_col", "numeric_col")` as shown below<jupyter_code>species_body_mass_tbl = penguins_tbl.group("species", np.mean)

species_body_mass_tbl.barh("species", "body_mass_g mean")<jupyter_output><empty_output><jupyter_text>While using the `matplotlib` package to graph, however, we are making a plot object with the first `plt.bar(x, y)` command,then the two other `plt.bar()` commands are adding more bars to the same graph object. The same hold true for other graphing commands in matplotlib such as `plt.hist()`, `plt.scatter()`, `plt.plot()`, etc.<jupyter_code>species_body_mass = penguins.groupby("species").agg({"body_mass_g": np.mean})

#try commenting out some of the code to see the plot object adding/deleting bars in action!
plt.bar(0, species_body_mass.loc["Adelie"]["body_mass_g"], label = "Adelie")
plt.bar(1, species_body_mass.loc["Chinstrap"]["body_mass_g"], label = "Chinstrap")
plt.bar(2, species_body_mass.loc["Gentoo"]["body_mass_g"], label = "Gentoo")

plt.legend();<jupyter_output><empty_output><jupyter_text>From the plot above, we can see that the species Gentoo on Biscoe island is significantly heavier in weight than all other penguin species in this dataset.   HistogramsAs discussed in data8, histograms are best used to see the distribution of one single numeric variable. From the bar plot above, we already know that the Gentoo penguins have the highest mean body mass, but can this be caused by a few abnormally heavy Gentoo penguins? Let's find out using a histogram.<jupyter_code>gentoo = penguins[penguins.species == "Gentoo"]

plt.hist(gentoo["body_mass_g"]);<jupyter_output><empty_output><jupyter_text>you might have noticed that the y-axis of the histogram above is not the same y axis you got introduced to in data8, this is because the dafult histogram-drawing method in matplotlib just sets the y-axis to `count`-the number of datapoints that falls within that bin.To change the y-axis to the normal %per unit, we need to set `density=True` in the function call.If you are interested in learning more about the optional arguments for the matplotlib histogram function, you can visit their function documentation page [here](https://matplotlib.org/3.3.1/api/_as_gen/matplotlib.pyplot.hist.html)<jupyter_code>plt.hist(gentoo["body_mass_g"], density=True);<jupyter_output><empty_output><jupyter_text>the histogram displayed a rather normal looking curve with no apparent skewness, guess the Gentoo penguins are just heavier in general after all.  Scatter plotsscatter plots are most useful when we are trying to determine the relationship between two numeric variables. For example, would a heavier body mass in penguins imply a longer flipper?Let's draw a scatter plot of body mass vs flipper length for species Gentoo and Chinstrap to compare.<jupyter_code>chinstrap = penguins[penguins.species == "Chinstrap"]

plt.scatter(chinstrap["flipper_length_mm"], chinstrap["body_mass_g"], label = "Chinstrap")
plt.scatter(gentoo["flipper_length_mm"], gentoo["body_mass_g"], label = "Gentoo")
plt.legend();<jupyter_output><empty_output><jupyter_text>From the scatter plot above, we can safely conclude that for species Chinstrap and Gentoo, their flipper length is positively associated wih body mass.   Line plotsLine plots are often used for displaying a trend over time. This functionality makes them the top graph of choice when looking at stock market or any other time series data. The penguin dataset, however, does not include time as a variable, and line plots are not suitable for visualizing this dataset.   2. Customizing your PlotsTitles, Axes, Plotsize, Legends, Axes ticks, Font size There are a lot of ways you can customize your plot in `matplotlib`. Some of the most commonly ones are shown below. Any good plot must have a title for the plot, as well as axis lables. These are necessary for a user to fully understand what is being displayed. * To add a title to a plot, we can use `plt.title("title")`.* For the x axis label, we can use `plt.xlabel("label")`.* For the y axis label, we can use `plt.ylabel("label")`.<jupyter_code>plt.scatter(gentoo["flipper_length_mm"], gentoo["body_mass_g"])
plt.title("Flipper Length vs Body Mass for Gentoo penguins")
plt.xlabel("Flipper length in mm")
plt.ylabel("Body mass in g");<jupyter_output><empty_output><jupyter_text>For all three commands shown above, we can use optional parameters `fontsize` to change the size of the labels.<jupyter_code>plt.scatter(gentoo["flipper_length_mm"], gentoo["body_mass_g"])
plt.title("Flipper Length vs Body Mass for Gentoo penguins", fontsize= 16)
plt.xlabel("Flipper length in mm", fontsize= 14 )
plt.ylabel("Body mass in g", fontsize= 14);<jupyter_output><empty_output><jupyter_text>Another important part of a plot is the legend. For this, let recreate the scatter plot from before.Notice that in creating this, we have used the parameter `label` in the `plot.scatter()` command. The label mentioned is the then used as the text in the legend made using `plt.legend`.<jupyter_code>plt.scatter(chinstrap["flipper_length_mm"], chinstrap["body_mass_g"], label = "Chinstrap")
plt.scatter(gentoo["flipper_length_mm"], gentoo["body_mass_g"], label = "Gentoo")
plt.legend();<jupyter_output><empty_output><jupyter_text>Another way to ge the same plot is to pass in the labels as a list in `plt.legend`. Here we have to be careful with the order of the labels. The first label in the list is automatically assigned to the first dataset and so on.<jupyter_code>plt.scatter(chinstrap["flipper_length_mm"], chinstrap["body_mass_g"])
plt.scatter(gentoo["flipper_length_mm"], gentoo["body_mass_g"])
plt.legend(["Chinstrap","Gentoo"]);<jupyter_output><empty_output><jupyter_text>Notice that by default matplotlib will itself chose where 'best' to place the legend in the plot. But we can use the optional parameter `loc` to set the location ourselves. Here we will palce it in the 'lower right' corner.All the options for legend placement can be found [here](https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.legend.html).<jupyter_code>plt.scatter(chinstrap["flipper_length_mm"], chinstrap["body_mass_g"], label = "Chinstrap")
plt.scatter(gentoo["flipper_length_mm"], gentoo["body_mass_g"], label = "Gentoo")
plt.legend(loc= 'lower right');<jupyter_output><empty_output><jupyter_text>To change the size of the text in the legend, we can use the parameter `fontsize` in the `plot.legend()` command like this.<jupyter_code># fontsize has to be in plt.legend(), not where you mention labels
plt.scatter(chinstrap["flipper_length_mm"], chinstrap["body_mass_g"], label = "Chinstrap")
plt.scatter(gentoo["flipper_length_mm"], gentoo["body_mass_g"], label = "Gentoo")
plt.legend(loc= 'lower right', fontsize = 14);<jupyter_output><empty_output><jupyter_text>Another important elements is the axis ticks. In the following plot, the x-axis ticks are in intervals of 5.<jupyter_code>plt.scatter(gentoo["flipper_length_mm"], gentoo["body_mass_g"])
plt.title("Flipper Length vs Body Mass for Gentoo penguins")
plt.xlabel("Flipper length in mm")
plt.ylabel("Body mass in g");<jupyter_output><empty_output><jupyter_text>By default, python will choose the range for x-axis and y-axis. But we can set the range ouselves as well using `plt.xlim([lower_limit, higher_limit])` for the x-axis and `plt.ylim([lower_limit, higher_limit])` for the y-axis.<jupyter_code>plt.scatter(gentoo["flipper_length_mm"], gentoo["body_mass_g"])
plt.title("Flipper Length vs Body Mass for Gentoo penguins")
plt.xlabel("Flipper length in mm")
plt.ylabel("Body mass in g")
plt.xlim([200, 240])
plt.ylim([3500, 7000])<jupyter_output><empty_output><jupyter_text>We have the option to change the freqency of the x-axis ticks using `plt.xticks(array_with_ticks)`<jupyter_code>import numpy as np
# Here we are using intervals of 3. 
# Therefore, we are genating an array that starts at 200 and increments by 3.
plt.scatter(gentoo["flipper_length_mm"], gentoo["body_mass_g"])
plt.title("Flipper Length vs Body Mass for Gentoo penguins")
plt.xlabel("Flipper length in mm")
plt.ylabel("Body mass in g")
plt.xticks(np.arange(200, 239, 3));<jupyter_output><empty_output><jupyter_text>Similarly, for y-axis ticks, we can use `plt.yticks(array_with_ticks)`.<jupyter_code># Here we are using intervals of 500. 
# Therefore, we are genating an array that starts at 4000 and increments by 500.
plt.scatter(gentoo["flipper_length_mm"], gentoo["body_mass_g"])
plt.title("Flipper Length vs Body Mass for Gentoo penguins")
plt.xlabel("Flipper length in mm")
plt.ylabel("Body mass in g")
plt.xticks(np.arange(200, 239, 3))
plt.yticks(np.arange(4000, 7000, 500));<jupyter_output><empty_output><jupyter_text>SubplotsAnother good functionality that `matplotlib` plots offer is subplots. Rememeber from the first section that each plot is actually a graph object, and we are just adding points/bars to the same graph object each time.With subplots, however, we are creating multiple graph objects with one command, and adding plots to each of the graphs seperately.To generate subplots, use the command `plt.subplots(row_number, column_number)` as shown below, here the `figsize=(width, length)` is just regulating how big the entire graph figure should be.<jupyter_code>fig, ax = plt.subplots(1, 3, figsize=(15, 5))<jupyter_output><empty_output><jupyter_text>To add plots to a specific subplot, we need to specify the row and column indices of the graph we want to plot on using `ax[row_inex, col_index]`, if there is just one row, the syntax changes to `ax[col_index]`. Similarly, if there is just one column, the syntax changes to `ax[row_index]`.For example, if we want a scatter plot of Flipper Length vs Body Mass for Gentoo penguins on the first subplot(the left most subplot),we need to specify `ax[0].scatter()`, where `ax[0]` signifies we want the plot to be on the first subplot.<jupyter_code>fig, ax = plt.subplots(1, 3, figsize=(15, 5))
adelie = penguins[penguins.species == "Adelie"]
ax[0].scatter(adelie["flipper_length_mm"], adelie["body_mass_g"], label = "Adelie")

ax[0].legend()<jupyter_output><empty_output><jupyter_text>Let's add scatter plots for the rest of the subplots. The syntax for setting title and axes range is slightly different for subplots, but it's nothing that a google search won't find. Don't be afraid to use search engines when doing your own project, if you are stuck on a piece of code, chances are many other people had the same problem. Besides slight syntax changes on some of the commands, each of the `ax[row, column]` objects are just regular graph objects returned by `plt.plot()` and the alike.<jupyter_code>fig, ax = plt.subplots(1, 3, figsize=(15, 5))
adelie = penguins[penguins.species == "Adelie"]
ax[0].scatter(adelie["flipper_length_mm"], adelie["body_mass_g"], label = "Adelie")
ax[1].scatter(gentoo["flipper_length_mm"], gentoo["body_mass_g"], label = "Gentoo")
ax[2].scatter(chinstrap["flipper_length_mm"], chinstrap["body_mass_g"], label = "Chinstrap")

ax[0].set_title("Adelie")
ax[1].set_title("Gentoo")
ax[2].set_title("Chinstrap")


ax[0].set_xlim(170, 240)
ax[0].set_ylim(2500, 6500)
ax[1].set_xlim(170, 240)
ax[1].set_ylim(2500, 6500)
ax[2].set_xlim(170, 240)
ax[2].set_ylim(2500, 6500)<jupyter_output><empty_output><jupyter_text>3. Good and Bad Data Visualization Practices Lab visualizations are great for understanding our data in a grphical fashion but as data scientists we must becognizant of when our visualizations start veering from infromative to misleading. Misleading graphs and bad visualizationscan range from comical to downright damaging. A comical example of bad graphs is given by XKCD:![image](https://imgs.xkcd.com/comics/y_axis_2x.png) Though it seems silly to zoom into a line and exaggerate the feautures, we as data scientists may fall into this trap.To connect back to what we learned in Data 8, let's say thatwe create this visualization:<jupyter_code>plt.scatter(np.random.rand(100,1), np.random.rand(100,1))
plt.title("Random Points")
plt.xlabel("Random x Points")
plt.ylabel("Random y points")<jupyter_output><empty_output><jupyter_text>It's obvious that the graph just shows random data. However, if we screw around with the visualization lines:<jupyter_code>plt.scatter(np.random.rand(100,1), np.random.rand(100,1))
plt.title("Random Points")
plt.xlabel("Random x Points")
plt.ylabel("Random y points") 
plt.yticks(np.arange(-50, 20, 3));<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09870972290770641, 'fraction_numerical': 0.012620743143199605, 'mean_word_length': 6.2109811896288765, 'pattern_counts': {'":': 1, '<': 85, '<?xml version=': 0, '>': 85, 'https://': 5, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '198177', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 4664, 'n_tokens_neox': 4380, 'n_words': 1641, 'path': 'module_6/.ipynb_checkpoints/module6-checkpoint.ipynb', 'repo_name': 'ds-modules/DATA-198-SP21', 'size': '14183'}