<jupyter_start><jupyter_text>Week 2 lecture notesFrom Coursera course, Understanding and Visualizing Data.  <NAME>  University of Michigan Categorical data: Tables, Bar Charts and Pie ChartsCategorial data classfies individuals or items into different groups.The most common way to summarize the data is to group by category and assemble a frequency table of counts, percentages. The most common way to visualize is with a bar chart. Sometimes you arrange in descending order. Don't recommend pie charts: Labels can overlap, it's hard to compare slice sizes.  Quantitative data: HistogramGreat first look at the data. Allows you to see the distribution of data in a compact way. Each rectangle is called a bin. Python will automatically set bin edges.4 aspects:* Shape: Overall appearance. Can be symmetric, bell-shaped, left skewed, right skewed, etc. Unimodal = one peak. "Right skewed" means it leans to the left and has a tail to the right. * Center: Mean or median position* Spread: How far our data spreads. Range, interquartile range (IQR), standard deviation, variance* Outliers: Data points that fall far from the bulk of the dataExample summary of the features on a distribution- cover each of those four aspects. "The distribution of adult male heights is roughly bell shaped with a center of about 68 inches, a range of 13 inches (62 to 75) and no apparent outliers." Quantitative data: Numerical summaries5 number summary:* Min (0%)* 1st quartile (25%)* Median (50%)* 3rd quartile (75%)* Max (100%)Others* Mean* IQR Inter quartile range (20-75%), useful when the data is left or right skewed.* Standard deviation: Average size that points fall away from the mean* Sample size Standard score (Empirical Rule)For a normal distribution:* 68% of data fall within 1-sigma of the mean* 95% of data fall within 2-sigma of the mean* 99.7% of data fall within 3-sigma of the meanThis allows us to get an idea of how unusual a data point is. Calculate a standard score:$ z = (x-\mu)/\sigma$A negative value means $x$ is below the mean.  Quantitative data: box plotsGives a visual representation of the 5-number summary. ![Boxplot](fig/boxplot.PNG) Often called a box and whisker plot. The 1st and 3rd quartiles define the box. The median is the line through the box. The min and max are the whiskers.Boxplots have a separate function for determining outliers and they are plotted as individual data plots outside the mix and max. Limitation: Hides some shape, which histogram displays better.  Tables, histograms, and boxplots in Python<jupyter_code>import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Load a tips dataset, which is a default dataset in seaborn<jupyter_code>tips_data = sns.load_dataset("tips")<jupyter_output><empty_output><jupyter_text>View it in table form, just the first five lines.<jupyter_code>tips_data.head()<jupyter_output><empty_output><jupyter_text>Describe the table to me. Default behavior is only to describe quantitative metrics in numerical columns. I can use the `include` parameter to explicitly add additional columns<jupyter_code>tips_data.dtypes
tips_data.describe()
tips_data.describe(include=['float','int64','category'])<jupyter_output><empty_output><jupyter_text>Create a histogramLook at the total bill.<jupyter_code># Plot a histogram of the total bill amount
sns.distplot(tips_data['total_bill'], kde=False).set_title("Histogram of total bill")
plt.show()
# Plot a histogram of the tips
# This time leave kde on... the smoothing curve
sns.distplot(tips_data['tip']).set_title("Histogram of tips")
plt.show()
# Plot both together.
sns.distplot(tips_data['total_bill'], kde=False)
sns.distplot(tips_data['tip'], kde=False).set_title("Histogram of total bill")
plt.show()<jupyter_output><empty_output><jupyter_text>Create a boxplot<jupyter_code>sns.boxplot(tips_data["total_bill"]).set_title("Box plot of total bill")
plt.show()<jupyter_output><empty_output><jupyter_text>By default, if we want to plot them both together, it will pile separate plots on top of each other. Let's see:<jupyter_code>sns.boxplot(tips_data["total_bill"])
sns.boxplot(tips_data["tip"]).set_title("Box plot of total bill")
plt.show()<jupyter_output><empty_output><jupyter_text>We don't want these on the same axis. How do we plot them separately? Plot by groupsSeparate by smokers and not smokers. All we have to do is provide the category by which we are splitting the data as the y-axis.<jupyter_code>tips_data.head()
sns.boxplot(x = tips_data["tip"], y=tips_data["smoker"])
plt.show()<jupyter_output><empty_output><jupyter_text>The quartile ranges look similar for these, but are the shapes of the data the same? Look at the histograms separately with a function `sns.FacetGrid`.This automatically matches the axes range so they are directly comparable.<jupyter_code>g = sns.FacetGrid(tips_data, row = "smoker")
g = g.map(plt.hist, "tip")<jupyter_output><empty_output><jupyter_text>Look as grouped by time of day.<jupyter_code>sns.boxplot(x = 'tip',
            y = 'time',
            data = tips_data)
plt.show()<jupyter_output><empty_output><jupyter_text>Dinner generates higher tips than lunch. Look at a histogram.<jupyter_code>g = sns.FacetGrid(tips_data, row = "time")
g = g.map(plt.hist, "tip")<jupyter_output><empty_output><jupyter_text>Observation: There are a lot more diners at dinner than lunch, and tips are higher then, so if I were a server I would want to work during dinner.  Invert and look vs. day.<jupyter_code>sns.boxplot(x="day",
            y="tip",
            data = tips_data)
plt.show()<jupyter_output><empty_output><jupyter_text>How would you decide what day to work? A few observations:* Sunday has the largest tip on average, Thursday has the worst tip on average* Saturday has the most high outliers* Friday has the smallest rangeNow look at the shapes.<jupyter_code>g = sns.FacetGrid(tips_data, "day")
g = g.map(plt.hist, "tip")<jupyter_output><empty_output><jupyter_text>Now compare smokers and non-smokers within each day. You can do this by adding as a `hue`.<jupyter_code>sns.boxplot(x=tips_data["day"], y = tips_data["tip"], hue=tips_data["smoker"])
plt.show()<jupyter_output><empty_output><jupyter_text>Another way to input all of this is just with the column names, and then provide the name of the dataframe as the `data` input parameter. This simplifies things.<jupyter_code>sns.boxplot(x=  "day", 
            y = "tip", 
            hue= "smoker",
            data = tips_data)
plt.savefig('tips_data.png')
plt.show()<jupyter_output><empty_output><jupyter_text>It looks like we can break out the FacetGrid into eight plots to see the distributions for these eight boxes.<jupyter_code>g = sns.FacetGrid(tips_data, row="day", col="smoker")
g = g.map(plt.hist, "tip")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09008349201699135, 'fraction_numerical': 0.006152043357257947, 'mean_word_length': 5.502857142857143, 'pattern_counts': {'":': 0, '<': 66, '<?xml version=': 0, '>': 66, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '55235', 'license': 'MIT', 'n_tokens_mistral': 2169, 'n_tokens_neox': 2037, 'n_words': 814, 'path': 'week_2_notes.ipynb', 'repo_name': 'pfschus/coursera_stats_1_understanding_visualizing_data', 'size': '6838'}