<jupyter_start><jupyter_text>Meeting in-person? Have you signed in?         Visualizing Your Data and Advanced Visualization         by:          None        (@cokeacolaking)             None        (@calvinyong)     on 2020-02-13<jupyter_code>import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns<jupyter_output><empty_output><jupyter_text>Loading in and Reading Data<jupyter_code>tips = pd.read_csv('tips.csv')  # import dataset
tips  # look at the data<jupyter_output><empty_output><jupyter_text>By default, `tips.head()` will give us the first 5 rows. A number will give us a certain number of rows.`tips.head(7)` or `tips.head(3)`<jupyter_code>tips.head()
tips.tail()<jupyter_output><empty_output><jupyter_text>To see a summary of the data with statistical numbers.<jupyter_code>tips.describe()<jupyter_output><empty_output><jupyter_text>You may have noticed we didn't see the other categories. It's not possible to get the mean or standard deviation of those columns. The next command lets us see the other columns.<jupyter_code>tips.describe(include=['O'])<jupyter_output><empty_output><jupyter_text>Missing Values `tips.isnull()` gets us a table of booleans. If the data is missing in that position, it will return True.<jupyter_code>tips.isnull()<jupyter_output><empty_output><jupyter_text>By using  `tips.isnull().any(axis=0)` we compare each column and if any values are missing in the column it returns True for the column.<jupyter_code>tips.isnull().any(axis=0)<jupyter_output><empty_output><jupyter_text>If we change `axis` to 1, it will return if any row has a missing value.<jupyter_code>tips.isnull().any(axis=1).head(13)<jupyter_output><empty_output><jupyter_text>Now we know how to see if there is a missing value in the data, but how much of our data are we actually missing?<jupyter_code>tips.isnull().sum(axis=0)<jupyter_output><empty_output><jupyter_text>If you want to see the total number of missing values, take the sum of the above line<jupyter_code>tips.isnull().sum(axis=0).sum()
tips.shape
# the size of the dataset (rows, cols)<jupyter_output><empty_output><jupyter_text>In many datasets, the data is larger than only 244, so we want to look at the impact the missing values have on our dataset.3 missing values out of 244 would be more significant than 3 missing values out of thousands. To see the impact, we want to divide each sum by 244 to see the percent of missing data per category.<jupyter_code>tips.isnull().sum(axis=0) / tips.shape[0]<jupyter_output><empty_output><jupyter_text>Plotting Continuous DataData comes in different types. With our `tips` dataset, we have three variables which are numbers (`total_bill`, `tip`, `size`). Although `size` is a number, we are going to treat it as a categorical value.  ScatterplotsThese graphs are useful for finding correlations or patterns in number values. In our graphs, we will assign `total_bill` to the x-axis and `tip` to the y-axis.<jupyter_code>sns.relplot(x="total_bill", y="tip", data=tips)<jupyter_output><empty_output><jupyter_text>`col=""` splits the graphs<jupyter_code>sns.relplot(x="total_bill", y="tip", col="time", data=tips)<jupyter_output><empty_output><jupyter_text>`hue=""` shades different points<jupyter_code>sns.relplot(x='total_bill', y='tip', hue='sex', data=tips)<jupyter_output><empty_output><jupyter_text>`style=""` changes the dots<jupyter_code>sns.relplot(x='total_bill', y='tip', hue='smoker', style='smoker', data=tips)<jupyter_output><empty_output><jupyter_text>`size=""` changes the size<jupyter_code>sns.relplot(x='total_bill', y='tip', size='size', data=tips)<jupyter_output><empty_output><jupyter_text>All together now!<jupyter_code>sns.relplot(x='total_bill', y='tip', col='time',
            hue='smoker', style='sex', size='size', data=tips)<jupyter_output><empty_output><jupyter_text>HistogramsViewing the univariate distribution<jupyter_code>sns.distplot(tips.total_bill, kde=False)<jupyter_output><empty_output><jupyter_text>Now we add in the kernel density estimate, `kde`, which is basically a smoothed curve over the data. This helps us plot the shape of the distribution.<jupyter_code>sns.distplot(tips.total_bill, kde=True)<jupyter_output><empty_output><jupyter_text>The number of `bins` divides the data into `n` equal parts, but the default `bins` number is 14.<jupyter_code>sns.distplot(tips.total_bill, bins=20, kde=True)<jupyter_output><empty_output><jupyter_text>Lastly, we have `rug`. It places a dash where each datapoint lies.<jupyter_code>sns.distplot(tips.total_bill, bins=20, kde=True, rug=True)<jupyter_output><empty_output><jupyter_text>In certain scenarios, `rug` helps us to see data that we might've forgotten about. In `tips.tip`, it is hard to tell where the points lie above 7.<jupyter_code>sns.distplot(tips['tip'], bins=12, kde=True, rug=True)<jupyter_output><empty_output><jupyter_text>Joint GraphsThe `jointplot` is a combination of a scatterplot and histogram.<jupyter_code>sns.jointplot(x="total_bill", y="tip", data=tips)
sns.pairplot(tips)<jupyter_output><empty_output><jupyter_text>Plotting Categorical Data  Categorical PlotsUsing `sns.catplot()` requires you to change the `kind` variable. Each kind shows a different way to visualize the data.<jupyter_code>sns.catplot(x="time", y="total_bill", data=tips)<jupyter_output><empty_output><jupyter_text>In the `catplot` we can use `hue` like before<jupyter_code>sns.catplot(x="time", y="total_bill", hue='smoker', data=tips)<jupyter_output><empty_output><jupyter_text>What do we do if we want to split the data more than just time? `dodge`!<jupyter_code>sns.catplot(x="time", y="total_bill", hue='smoker', dodge=True, data=tips)<jupyter_output><empty_output><jupyter_text>There are different kinds of `catplots`. A few of them are `swarm`, `box`, `boxen`, `bar`.<jupyter_code>sns.catplot(x="time", y="total_bill", hue='sex',
            dodge=True, kind='swarm', data=tips)<jupyter_output><empty_output><jupyter_text>If you want to use the `x` as a `hue`, make sure you set `dodge` to False<jupyter_code>sns.catplot(x="time", y="total_bill", hue='time',
            dodge=False, kind='bar', data=tips)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12113860585476306, 'fraction_numerical': 0.006631085233705321, 'mean_word_length': 7.126149802890933, 'pattern_counts': {'":': 0, '<': 113, '<?xml version=': 0, '>': 113, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '28', 'dir': 'jupyter-structured-clean-dedup', 'id': '228004', 'license': 'MIT', 'n_tokens_mistral': 2229, 'n_tokens_neox': 2112, 'n_words': 572, 'path': 'sp20/02-13-visualization/visualization.solution.ipynb', 'repo_name': 'AHKerrigan/supplementary', 'size': '6183'}