<jupyter_start><jupyter_text>3. Grouping and summarizingSo far you've been answering questions about individual country-year pairs, but you may be interested in aggregations of the data, such as the average life expectancy of all countries within each year. Here you'll learn to use the group by and summarize verbs, which collapse large datasets into manageable summaries.<jupyter_code>library(gapminder)
library(dplyr)<jupyter_output>Warning message:
"package 'dplyr' was built under R version 3.6.3"
Attaching package: 'dplyr'

The following objects are masked from 'package:stats':

    filter, lag

The following objects are masked from 'package:base':

    intersect, setdiff, setequal, union<jupyter_text>The summarize verbyou'll return to the topic of data transformation with `dplyr` to learn more ways to explore your data. Data transformation and visualizationAnalyses will usually involve a cycle between these steps of data transformation and visualization, as well as additional components of the data science workflow, like modeling. Once you've learned these new verbs, you'll be able to create a much larger variety of informative visualizations with `ggplot2`.  Extracting dataUse filter verb to pull out individual observations, such as statistics for the United States in 2007.<jupyter_code>gapminder_US_2007 <- gapminder %>%
    filter(country == "United States", year == 2007)
gapminder_US_2007<jupyter_output><empty_output><jupyter_text>The summarize verbFor example, suppose you want to know the average life expectancy across all countries and years in the dataset. You would do this with the `summarize` verb. Take your gapminder data, pipe it into summarize, and specify that you're creating a summary column called `meanLifeExp`.<jupyter_code>gapminder %>%
    summarize(meanLifeExp = mean(lifeExp))<jupyter_output><empty_output><jupyter_text>This is calling the function mean on the variable lifeExp. The mean function takes the average of a set of values, and R comes with many built-in functions like this. Notice that summarize collapses the entire table down into one row. In the output, we see the answer to our question: the mean life expectancy was about 59.47 years.If you think about it, it doesn't really make sense to summarize across all countries and across all years. It may make more sense to ask questions about averages in a particular year, such as 2007.<jupyter_code>gapminder %>%
    filter(year == 2007) %>%
    summarize(meanLifeExp = mean(lifeExp))<jupyter_output><empty_output><jupyter_text>Summarizing into multiple columnsFor example, suppose that along with finding the average life expectancy in 2007, you want to find the total population in that year.<jupyter_code>gapminder %>%
    filter(year == 2007) %>%
    summarize(meanLifeExp = mean(lifeExp),
             totalPop = sum(pop))<jupyter_output><empty_output><jupyter_text>`mean` and `sum` are just two of the built-in functions you could use to summarize a variable within a dataset. Another example is `median`: the `median` represents the point in a set of numbers where half the numbers are above that point and half of the numbers are below. Two others are `min`, for minimum, and `max`, for maximum.   Exercise<jupyter_code># Summarizing the median life expectancy
# Summarize to find the median life expectancy
gapminder %>%
    summarize(medianLifeExp = median(lifeExp))
# Summarizing the median life expectancy in 1957
# Filter for 1957 then summarize the median life expectancy
gapminder %>%
    filter(year == 1957) %>%
    summarize(medianLifeExp = median(lifeExp))
# Summarizing multiple variables in 1957
# Filter for 1957 then summarize the median life expectancy and the maximum GDP per capita
gapminder %>%
    filter(year == 1957) %>%
    summarize(medianLifeExp = median(lifeExp), 
              maxGdpPercap = max(gdpPercap))<jupyter_output><empty_output><jupyter_text>--- The group_by verbWhat if we weren't interested just in the average for the year 2007, but for each of the years in the dataset?We can use the `group_by` verb, which tells `dplyr` to summarize within groups instead of summarizing the entire dataset. Summarizing by year<jupyter_code>gapminder %>%
    group_by(year) %>%
    summarize(meanLifeExp = mean(lifeExp), 
              totalPop = sum(pop))<jupyter_output><empty_output><jupyter_text>Summarizing by continentSuppose you're interested in the average life expectancy and the total population in 2007 within each continent. You can find this by first filtering for the year 2007, grouping by continent (instead of year), and then performing your summary.<jupyter_code>gapminder %>%
    filter(year == 2007) %>%
    group_by(continent) %>%
    summarize(meanLifeExp = mean(lifeExp), 
              totalPop = sum(pop))<jupyter_output><empty_output><jupyter_text>Summarizing by continent and yearTo do so, you can summarize by both year and continent, by adding year comma continent within the group by.<jupyter_code>gapminder %>%
    group_by(year, continent) %>%
    summarize(totalPop = sum(pop), 
              meanLifeExp = mean(lifeExp))<jupyter_output>`summarise()` has grouped output by 'year'. You can override using the `.groups` argument.<jupyter_text>Now the output has one row for each combination of a year and continent. For example, we see the total population and average life expectancy in 1952 for Africa, the Americas, Asia, Europe, and Oceania, followed by each of the continent-level summaries for 1957.  Exercise<jupyter_code># Summarizing by year
# Find median life expectancy and maximum GDP per capita in each year
gapminder %>%
    group_by(year) %>%
    summarize(medianLifeExp = median(lifeExp), 
              maxGdpPercap = max(gdpPercap))<jupyter_output><empty_output><jupyter_text>*Notice that median life expectancy across countries is generally going up over time, but maximum GDP per capita is not.*<jupyter_code># Summarizing by continent
# Find median life expectancy and maximum GDP per capita in each continent in 1957
gapminder %>%
    filter(year == 1957) %>%
    group_by(continent) %>%
    summarize(medianLifeExp = median(lifeExp), 
              maxGdpPercap = max(gdpPercap))
# Find median life expectancy and maximum GDP per capita in each continent in 2007
gapminder %>%
    filter(year == 2007) %>%
    group_by(continent) %>%
    summarize(medianLifeExp = median(lifeExp), 
              maxGdpPercap = max(gdpPercap))
# Summarizing by continent and year
# Find median life expectancy and maximum GDP per capita in each continent/year combination
gapminder %>%
    group_by(continent, year) %>%
    summarize(medianLifeExp = median(lifeExp), maxGdpPercap = max(gdpPercap))<jupyter_output>`summarise()` has grouped output by 'continent'. You can override using the `.groups` argument.<jupyter_text>--- Visualizing summarized dataNow you'll learn how to turn those summaries into informative visualizations, by returning to the ggplot2 package from Chapter 2. Summarizing by yearNow instead of viewing the summarized data as a table, let's save it as an object called `by_year`, so you can visualize the data using `ggplot2`.<jupyter_code>by_year <- gapminder %>%
    group_by(year) %>%
    summarize(totalPop = sum(pop),
             meanLifeExp = mean(lifeExp))

by_year<jupyter_output><empty_output><jupyter_text>Visualizing population over timeYou would construct the graph with the three steps of `ggplot2`: the data, which is `by_year`. The aesthetics, which puts year on the x-axis and total population on the y-axis. And the type of graph, which in this case is a scatter plot, represented by `geom_point`. Notice that the steps are the same as when you were graphing countries in a scatter plot, even though it's a new dataset.<jupyter_code>library(ggplot2)
ggplot(by_year, aes(x = year, y = totalPop)) +
    geom_point()<jupyter_output><empty_output><jupyter_text>The resulting graph of population by year shows the change in the total population, which is going up over time. `ggplot2` puts the y-axis is in scientific notation, since showing it with nine zeros would be hard to read. The global starts a little under 3 times 10 to the 9th power- that's three billion- and goes up to more than 6 billion. You might notice that the graph is a little misleading because it doesn't include zero: you don't have a sense of how much the population grew relative to where it was when it started. This is a good time to introduce another graphing option. Starting y-axis at zeroBy adding `expand_limits(y = 0)` to the end of the ggplot call, you can specify that you want the y-axis to start at zero.<jupyter_code>ggplot(by_year, aes(x = year, y = totalPop)) +
    geom_point() +
    expand_limits(y = 0)<jupyter_output><empty_output><jupyter_text>Now the graph makes it clearer that the population is almost tripling during this time. You could have created other graphs of summarized data, such as a graph of the average life expectancy over time, by changing the y aesthetic. Summarizing by year and continent & Visualizing population by year and continent<jupyter_code>by_year_continent <- gapminder %>%
    group_by(year, continent) %>%
    summarize(totalPop = sum(pop), 
              meanLifeExp = mean(lifeExp))<jupyter_output>`summarise()` has grouped output by 'year'. You can override using the `.groups` argument.<jupyter_text>By setting color equals continent, you can show five separate trends on the same graph.<jupyter_code>ggplot(by_year_continent, aes(x = year, y = totalPop, color = continent)) +
    geom_point() +
    expand_limits(y = 0)<jupyter_output><empty_output><jupyter_text>This lets us see that Asia was always the most populated continent and has been growing the most rapidly, that Europe has a slower rate of growth, and that Africa has grown to surpass both Europe and the Americas in terms of population.  Exercise<jupyter_code>by_year_2 <- gapminder %>%
  group_by(year) %>%
  summarize(medianLifeExp = median(lifeExp), 
            maxGdpPercap = max(gdpPercap))
# Visualizing median life expectancy over time
# Create a scatter plot showing the change in medianLifeExp over time
ggplot(by_year_2, aes(x = year, y = medianLifeExp)) +
  geom_point() +
  expand_limits(y = 0)<jupyter_output><empty_output><jupyter_text>*It looks like median life expectancy across countries is increasing over time.*<jupyter_code># Visualizing median GDP per capita per continent over time
# Summarize medianGdpPercap within each continent within each year: by_year_continent
by_year_continent_2 <- gapminder %>%
    group_by(continent, year) %>%
    summarize(medianGdpPercap = median(gdpPercap))

# Plot the change in medianGdpPercap in each continent over time
ggplot(by_year_continent_2, aes(x = year, y = medianGdpPercap, color = continent)) +
    geom_point() +
    expand_limits(y = 0)
# Comparing median life expectancy and median GDP per continent in 2007
# Summarize the median GDP and median life expectancy per continent in 2007
by_continent_2007 <- gapminder %>%
  filter(year == 2007) %>%
  group_by(continent) %>%
  summarize(medianGdpPercap = median(gdpPercap),
            medianLifeExp = median(lifeExp))

# Use a scatter plot to compare the median GDP and median life expectancy
ggplot(by_continent_2007, aes(x = medianGdpPercap, y = medianLifeExp, color = continent)) +
  geom_point()<jupyter_output><empty_output><jupyter_text>------ 4. Types of visualizationsIn this chapter, you'll learn how to create line plots, bar plots, histograms, and boxplots. You'll see how each plot requires different methods of data manipulation and preparation, and you’ll understand how each of these plot types plays a different role in data analysis. Line plots Types of plotsIn this chapter, you'll learn to make four more types of graphs. You'll make line plots, which are useful for showing change over time. You'll create bar plots, which are good at comparing statistics for each of several categories. You'll learn to make histograms, which describe the distribution of a one-dimensional numeric variable. And you'll make box plots, which compare the distribution of a numeric variable among several categories. Scatter vs line plot`geom_point()` vs `geom_line()`<jupyter_code>ggplot(by_year_continent, aes(x = year, y = meanLifeExp, color = continent)) +
    geom_line()<jupyter_output><empty_output><jupyter_text>Exercise<jupyter_code># Visualizing median GDP per capita over time
# Summarize the median gdpPercap by year, then save it as by_year
by_year <- gapminder %>%
    group_by(year) %>%
    summarize(medianGdpPercap = median(gdpPercap))

# Create a line plot showing the change in medianGdpPercap over time
ggplot(by_year, aes(x= year, y = medianGdpPercap)) +
    geom_line() +
    expand_limits(y = 0)<jupyter_output><empty_output><jupyter_text>*Looks like median GDP per capita across countries has gone up over time.*<jupyter_code># Visualizing median GDP per capita by continent over time
# Summarize the median gdpPercap by year & continent, save as by_year_continent
by_year_continent <- gapminder %>%
    group_by(year, continent) %>%
    summarize(medianGdpPercap = median(gdpPercap))

# Create a line plot showing the change in medianGdpPercap by continent over time
ggplot(by_year_continent, aes(x = year, y = medianGdpPercap, color = continent)) + 
    geom_line() +
    expand_limits(y = 0)<jupyter_output>`summarise()` has grouped output by 'year'. You can override using the `.groups` argument.<jupyter_text>--- Bar plotsBar plots are useful for comparing values across discrete categories, such as continents. Summarizing by continentFor example, this code finds the average life expectancy within each continent in the year 2007. That creates a table that looks like this, with one observation for each continent. Instead of just printing the table, you might want to represent the summary visually.<jupyter_code>by_continent <- gapminder %>%
    filter(year == 2007) %>%
    group_by(continent) %>%
    summarize(meanLifeExp = mean(lifeExp))

by_continent<jupyter_output><empty_output><jupyter_text>Bar plotThis kind of plot represents the data using one bar for each continent, with the height of the bar representing the mean life expectancy.<jupyter_code>ggplot(by_continent, aes(x = continent, y = meanLifeExp)) +
    geom_col()<jupyter_output><empty_output><jupyter_text>This makes it easy to see which continents have higher life expectancy than others. Notice that unlike scatter plots or line plots, bar plots always start at zero.  Exercise<jupyter_code># Visualizing median GDP per capita by continent

# Summarize the median gdpPercap by continent in 1952
by_continent_1952 <- gapminder %>%
    filter(year == 1952) %>%
    group_by(continent) %>%
    summarize(medianGdpPercap = median(gdpPercap))

# Create a bar plot showing medianGdp by continent
ggplot(by_continent_1952, aes(x = continent, y = medianGdpPercap)) + 
    geom_col()
# Visualizing GDP per capita by country in Oceania
# Filter for observations in the Oceania continent in 1952
oceania_1952 <- gapminder %>%
    filter(continent == "Oceania", year == 1952)

# Create a bar plot of gdpPercap by country
ggplot(oceania_1952, aes(x = country, y = gdpPercap)) +
    geom_col()<jupyter_output><empty_output><jupyter_text>*Looks like the GDP per capita of these two countries was similar in 1952.* --- HistogramsA histogram shows a distribution. In this case, it's the distribution of life expectancy across countries in the year 2007.<jupyter_code>expectancy_2007 <- gapminder %>%
    filter(year == 2007)

ggplot(expectancy_2007, aes(x = lifeExp)) +
    geom_histogram()<jupyter_output>`stat_bin()` using `bins = 30`. Pick better value with `binwidth`.<jupyter_text>Every bar represents a bin of life expectancies, and the height represents how many countries fall into that bin. This lets you get a sense of the distribution based on the histogram's shape. We can see that most countries have a life expectancy between 70 to 80 years, but that another set of countries have life expectancies between 40 and 65.It has only one aesthetic: the x-axis, the variable whose distribution you are examining. The width of each bin in the histogram is chosen automatically, and it has a large effect on how the histogram communicates the distribution.  Adjusting bin widthYou may need to customize that width. It is possible to do so with the binwidth option, which is set inside the parentheses of the geom underscore histogram layer. Setting binwidth equals 5 means that each of the bars in the histograms represents a width of five years.<jupyter_code>ggplot(expectancy_2007, aes(x = lifeExp)) +
    geom_histogram(binwidth = 5)<jupyter_output><empty_output><jupyter_text>Setting a wide binwidth like this makes the histogram a bit blockier, which focuses on the general shape more than the small details.  In some cases, you may need to put the x-axis of a histogram on a log scale for it to be understandable.   Exercise<jupyter_code># Visualizing population

gapminder_1952 <- gapminder %>%
  filter(year == 1952) %>%
  mutate(pop_by_mil = pop / 1000000)

# Create a histogram of population (pop_by_mil)
ggplot(gapminder_1952, aes(x = pop_by_mil)) +
  geom_histogram(bins = 50)<jupyter_output><empty_output><jupyter_text>*Notice that most of the distribution is in the smallest (leftmost) bins.*<jupyter_code># Visualizing population with x-axis on a log scale

gapminder_1952 <- gapminder %>%
  filter(year == 1952)

# Create a histogram of population (pop), with x on a log scale
ggplot(gapminder_1952, aes(x = pop)) +
  geom_histogram() +
  scale_x_log10()<jupyter_output>`stat_bin()` using `bins = 30`. Pick better value with `binwidth`.<jupyter_text>*Notice that on a log scale, the distribution of country populations is approximately symmetrical.* --- BoxplotsRecall that we used a histogram when we wanted to examine the distribution of one variable, such as life expectancy, across all countries. Notice that a histogram combines all the life expectancies across all continents, without distinguishing them. But what if your goal is to compare the distribution of life expectancies among continents? Box plotsThis is a box plot, which shows the distribution of life expectancies within each continent, so that you can compare them.<jupyter_code>ggplot(expectancy_2007, aes(x = continent, y = lifeExp)) +
    geom_boxplot()<jupyter_output><empty_output><jupyter_text>A box plot takes a bit of practice to interpret, so here's what each of the components means. - The black line in the middle of each white box is the **median** of that continent's distribution. - The top and bottom of each box represent the 75th percentile and the 25th percentile of that group, meaning half of the distribution lies within that box. - The lines going up and down from the box, called "whiskers", cover additional countries. - The two dots below the whiskers for Asia and the Americas represent outliers: countries with unusually low life expectancy relative to the rest of the distribution. So there's a lot that this plot tells us about differences in life expectancy across continents. We can see that the median life expectancy of Europe is one of the highest, and that the two countries in Oceana (Australia and New Zealand) both have very high values. We can also see that the distribution for Africa is unusually low, with about half of its countries having a life expectancy between 50 and 60 years. Histogram vs box plotA boxplot helps give more context to the shape of the earlier histogram, where there were two bumps: one for countries between 65 and 80 representing most of Europe, Asia, and the Americas, and another, lower distribution.  Exercise<jupyter_code># Comparing GDP per capita across continents

gapminder_1952 <- gapminder %>%
  filter(year == 1952)

# Create a boxplot comparing gdpPercap among continents
ggplot(gapminder_1952, aes(x = continent, y = gdpPercap)) +
  geom_boxplot() +
  scale_y_log10()
# Adding a title to your graph

gapminder_1952 <- gapminder %>%
  filter(year == 1952)

# Add a title to this graph: "Comparing GDP per capita across continents"
ggplot(gapminder_1952, aes(x = continent, y = gdpPercap)) +
  geom_boxplot() +
  scale_y_log10() + 
  ggtitle("Comparing GDP per capita across continents")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07069712485230406, 'fraction_numerical': 0.015015754233950374, 'mean_word_length': 5.016883886255924, 'pattern_counts': {'":': 0, '<': 130, '<?xml version=': 0, '>': 170, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '30', 'dir': 'jupyter-structured-clean-dedup', 'id': '362407', 'license': 'FSFAP', 'n_tokens_mistral': 6186, 'n_tokens_neox': 5561, 'n_words': 2454, 'path': 'Data Analyst with R/03_Introduction_to_the_Tidyverse/03_2_Grouping_and_summarizing&Types_of_visualizations.ipynb', 'repo_name': 'brendensong/DataCamp', 'size': '20312'}