<jupyter_start><jupyter_text>Exploring The Historical Voting of USGA> Using data manipulation and visualization skills to explore the historical voting of the United Nations General Assembly.- author: <NAME>- toc: true- comments: true- categories: [eda, data-analysis, r]- image:images/eda-usga-shield.png  OverviewWe'll explore the historical voting of the United Nations General Assembly, including analyzing differences in voting between countries, across time, and among international issues. We'll be using dplyr and ggplot2 packages, broom package for tidying model output, and experience the kind of start-to-finish exploratory analysis common in data science.  Libraries<jupyter_code>library(dplyr)
library(countrycode)
library(ggplot2)
library(tidyr)
library(broom)
library(purrr)<jupyter_output>Warning message:
"package 'dplyr' was built under R version 3.6.3"
Attaching package: 'dplyr'

The following objects are masked from 'package:stats':

    filter, lag

The following objects are masked from 'package:base':

    intersect, setdiff, setequal, union

Warning message:
"package 'tidyr' was built under R version 3.6.3"Warning message:
"package 'broom' was built under R version 3.6.3"Warning message:
"package 'purrr' was built under R version 3.6.3"<jupyter_text>Data cleaning and summarizing with dplyrThe best way to use data wrangling skills is to apply them to a specific case study. Here we'll explore how to clean and filter the United Nations voting dataset using the dplyr package, and how to summarize it into smaller, interpretable units.  The United Nations Voting Dataset<jupyter_code>voting <- readRDS(gzcon(url("https://assets.datacamp.com/production/repositories/420/datasets/ddfa750d993c73026f621376f3c187f276bf0e2a/votes.rds")))
head(voting)<jupyter_output><empty_output><jupyter_text>- each row is a country vote pair- `rcid` roll call id- `session` session (year)- `ccode` country code- `votes`  - 1 = Yes  - 2 = Abstain  - 3 = No  - 8 = Not present  - 9 = Not a member<jupyter_code>unique(voting$vote)
table(voting$vote)<jupyter_output><empty_output><jupyter_text>>  dplyr verbs- `filter` - keeps observations based on a condition- `mutate` - adds an additional variable<jupyter_code># Filter for votes that are "yes", "abstain", or "no"
votes_processed <- voting %>%
  filter(vote<=3) %>%
    # Add another %>% step to add a year column
    mutate(year=session+1945)
head(votes_processed)<jupyter_output><empty_output><jupyter_text>Adding a country columnThe country codes in the `ccode` column are what's called [Correlates of War codes](http://cow.dss.ucdavis.edu/data-sets/cow-country-codes). This isn't ideal for an analysis, since we'd like to work with recognizable country names.<jupyter_code>votes_processed<- votes_processed %>%
    mutate(year = session + 1945, country=countrycode(ccode, "cown", "country.name"))
head(votes_processed)<jupyter_output>Warning message in countrycode(ccode, "cown", "country.name"):
"Some values were not matched unambiguously: 260
"<jupyter_text>Grouping and summarizing>  Using “% of Yes votes” as a summary- `summarize()` turns many rows into one<jupyter_code>votes_processed %>%
    summarise(total=n())
votes_processed %>%
    summarise(total=n(), percent_yes=mean(vote==1))<jupyter_output><empty_output><jupyter_text>> - `mean(vote == 1)` is a way of calculating “percent of vote equal to 1”- `group_by()` before `summarize()` turns groups into one row each<jupyter_code>head(votes_processed %>%
    group_by(year) %>%
        summarise(total=n(), percent_year=mean(vote==1)), 
     10)
# Summarize by country: by_country
head(by_country<- votes_processed %>%
    group_by(country) %>%
        summarize(total = n(), percent_yes = mean(vote == 1)),
     10)<jupyter_output>`summarise()` ungrouping output (override with `.groups` argument)<jupyter_text>Sorting and filtering summarized data- `arrange` - sorts a table based on a variable<jupyter_code>head(by_country %>%
    arrange(percent_yes), 
    10)
head(by_country %>% 
     arrange(desc(percent_yes)),
     10)<jupyter_output><empty_output><jupyter_text>Filtering summarized output Zanzibar, had only 2 votes in the entire dataset. We certainly can't make any substantial conclusions based on that data!<jupyter_code>head(by_country %>%
     arrange(percent_yes) %>%
         filter(total>=100), 
     10)<jupyter_output><empty_output><jupyter_text>Data visualization with ggplot2Once we've cleaned and summarized data, we'll visualize them to understand trends and extract insights. Here we'll use the ggplot2 package to explore trends in United Nations voting within each country over time.  Visualization with ggplot2>  Visualizing by-year data<jupyter_code>head(by_year <- votes_processed %>%
    group_by(year) %>%
        summarise(total = n(), percent_yes = mean(vote==1)), 
     10)
by_year %>%
    ggplot(aes(x=year, y=percent_yes)) + 
    geom_line()
# Change to scatter plot and add smoothing curve
ggplot(by_year, aes(year, percent_yes)) +
  geom_point() +
  geom_smooth()<jupyter_output>`geom_smooth()` using method = 'loess' and formula 'y ~ x'<jupyter_text>Visualizing by country<jupyter_code>head(by_year_country <- votes_processed %>%
     group_by(year, country) %>%
         summarise(total=n(), percent_yes=mean(vote==1)), 
     10)<jupyter_output>`summarise()` regrouping output by 'year' (override with `.groups` argument)<jupyter_text>>  Filtering for one country<jupyter_code>by_year_country %>%
    filter(country=="United States") %>%
        ggplot(aes(x=year, y=percent_yes)) +
        geom_line()
by_year_country %>%
    filter(country=="United States") %>%
        ggplot(aes(x=year, y=percent_yes)) +
        geom_point() +
        geom_smooth()<jupyter_output>`geom_smooth()` using method = 'loess' and formula 'y ~ x'<jupyter_text>>  Filtering for multiple countries<jupyter_code>head(us_france <- by_year_country %>%
    filter(country %in% c("United States", "France")), 10)
us_france %>%
    ggplot(aes(x=year, y=percent_yes, color=country)) +
    geom_line()
us_france %>%
    ggplot(aes(x=year, y=percent_yes, color=country)) +
    geom_point() +
    geom_smooth()<jupyter_output>`geom_smooth()` using method = 'loess' and formula 'y ~ x'<jupyter_text>Plotting just the UK over timeNow that we have the percentage of time that each country voted "yes" within each year, we can plot the trend for a particular country. In this case, we'll look at the trend for just the United Kingdom<jupyter_code>by_year_country %>%
    filter(country=="United Kingdom") %>%
        ggplot(aes(x=year, y=percent_yes)) +
        geom_line()
by_year_country %>%
    filter(country=="United Kingdom") %>%
        ggplot(aes(x=year, y=percent_yes)) +
        geom_point() +
        geom_smooth()<jupyter_output>`geom_smooth()` using method = 'loess' and formula 'y ~ x'<jupyter_text>Plotting multiple countriesPlotting just one country at a time is interesting, but we really want to compare trends between countries. For example, suppose we want to compare voting trends for the United States, the UK, France, and India.<jupyter_code>by_year_country %>%
    filter(country %in% c("United States", "United Kingdom", "France", "India")) %>%
        ggplot(aes(x=year, y=percent_yes, color=country)) +
        geom_line()
by_year_country %>%
    filter(country %in% c("United States", "United Kingdom", "France", "India")) %>%
        ggplot(aes(x=year, y=percent_yes, color=country)) +
        geom_point() +
        geom_smooth(se=F)<jupyter_output>`geom_smooth()` using method = 'loess' and formula 'y ~ x'<jupyter_text>Faceting by country<jupyter_code>by_year_country %>%
    filter(country %in% c("United States", "United Kingdom", "France", "India")) %>%
        ggplot(aes(x=year, y=percent_yes, color=country)) +
        geom_line() +
        facet_wrap(~country)
by_year_country %>%
    filter(country %in% c("United States", "United Kingdom", "France", "India")) %>%
        ggplot(aes(x=year, y=percent_yes, color=country)) +
        geom_point() +
        geom_smooth() +
        facet_wrap(~country)<jupyter_output>`geom_smooth()` using method = 'loess' and formula 'y ~ x'<jupyter_text>>  Graphing on separate scales<jupyter_code>by_year_country %>%
    filter(country %in% c("United States", "United Kingdom", "France", "India")) %>%
        ggplot(aes(x=year, y=percent_yes, color=country)) +
        geom_line() +
        facet_wrap(~country, scales = "free_y")
by_year_country %>%
    filter(country %in% c("United States", "United Kingdom", "France", "India")) %>%
        ggplot(aes(x=year, y=percent_yes, color=country)) +
        geom_point() +
        geom_smooth() +
        facet_wrap(~country, scales = "free_y")<jupyter_output>`geom_smooth()` using method = 'loess' and formula 'y ~ x'<jupyter_text>Faceting the time seriesNow we'll take a look at six countries.<jupyter_code>by_year_country %>%
    filter(country %in% c("United States", "United Kingdom","France", "Japan", "Brazil", "India")) %>%
        ggplot(aes(x=year, y=percent_yes)) +
        geom_line() +
        facet_wrap(~country)
by_year_country %>%
    filter(country %in% c("United States", "United Kingdom","France", "Japan", "Brazil", "India")) %>%
        ggplot(aes(x=year, y=percent_yes)) +
        geom_point() +
        geom_smooth() +
        facet_wrap(~country)<jupyter_output>`geom_smooth()` using method = 'loess' and formula 'y ~ x'<jupyter_text>Faceting with free y-axisIn the previous plot, all six graphs had the same axis limits. This made the changes over time hard to examine for plots with relatively little change.Instead, we may want to let the plot choose a different y-axis for each facet.<jupyter_code>by_year_country %>%
    filter(country %in% c("United States", "United Kingdom","France", "Japan", "Brazil", "India")) %>%
        ggplot(aes(x=year, y=percent_yes)) +
        geom_line() +
        facet_wrap(~country, scales = "free_y")
by_year_country %>%
    filter(country %in% c("United States", "United Kingdom","France", "Japan", "Brazil", "India")) %>%
        ggplot(aes(x=year, y=percent_yes)) +
        geom_point() +
        geom_smooth() +
        facet_wrap(~country, scales = "free_y")<jupyter_output>`geom_smooth()` using method = 'loess' and formula 'y ~ x'<jupyter_text>Choose our own countriesLet's see the trend in East Africa<jupyter_code>by_year_country %>%
    filter(country %in% c("Kenya", "Uganda","Tanzania", "Rwanda", "Burundi", "Ethiopia")) %>%
        ggplot(aes(x=year, y=percent_yes)) +
        geom_line() +
        facet_wrap(~country, scales = "free_y")
by_year_country %>%
    filter(country %in% c("Kenya", "Uganda","Tanzania", "Rwanda", "Burundi", "Ethiopia")) %>%
        ggplot(aes(x=year, y=percent_yes)) +
        geom_point() +
        geom_smooth() +
        facet_wrap(~country, scales = "free_y")<jupyter_output>`geom_smooth()` using method = 'loess' and formula 'y ~ x'<jupyter_text>Tidy modeling with broomWhile visualization helps us understand one country at a time, statistical modeling lets us quantify trends across many countries and interpret them together. Here we'll using tidyr, purrr, and broom packages to fit linear models to each country, and understand and compare their outputs.  Linear regression  Linear regression on the United StatesA linear regression is a model that lets us examine how one variable changes with respect to another by fitting a best fit line. It is done with the `lm()` function in R.We'll fit a linear regression to just the percentage of "yes" votes from the United States.<jupyter_code>summary(US_fit <- lm(percent_yes ~ year, data=by_year_country %>%
               filter(country=="United States")))<jupyter_output><empty_output><jupyter_text>- the estimated slope of this relationship is -0.0062393  Finding the p-value of a linear regressionNot all positive or negative slopes are necessarily real. A p-value is a way of assessing whether a trend could be due to chance. Generally, data scientists set a threshold by declaring that, for example, p-values below .05 are significant.the p-value of the relationship between `year` and `percent_yes` is ${1.367}^{-07}$ It seems `year` is very significant when modeling `percent_yes`.  Tidying models with broom>  A model t is a “messy” object<jupyter_code>summary(afghan_fit <- lm(percent_yes ~ year, data = by_year_country %>%
                        filter(country=="Afghanistan")))<jupyter_output><empty_output><jupyter_text>- Models are difcult to combine<jupyter_code>tidy(afghan_fit)<jupyter_output><empty_output><jupyter_text>>  Tidy models can be combined<jupyter_code>bind_rows(tidy(US_fit),tidy(afghan_fit))<jupyter_output><empty_output><jupyter_text>Combining models for multiple countriesOne important advantage of changing models to tidied data frames is that they can be combined.<jupyter_code>(bind_rows(tidy(US_fit), tidy(UK_fit <- lm(percent_yes~year, by_year_country %>%
                                          filter(country=="United Kingdom")))))<jupyter_output><empty_output><jupyter_text>Nesting for multiple models>  Start with one row per country<jupyter_code>head(by_year_country, 10)<jupyter_output><empty_output><jupyter_text>>  nest() turns it into one row per country- `-country` means “nestall except country”<jupyter_code>head(nested <- by_year_country %>%
     nest(-country), 1)<jupyter_output>Warning message:
"All elements of `...` must be named.
Did you want `data = c(year, total, percent_yes)`?"<jupyter_text>unnest() does the opposite  List columnsThis "nested" data has an interesting structure. The second column, data, is a list,  that allows complicated objects to be stored within each row. This is because each item of the data column is itself a data frame.<jupyter_code>head(# Print the nested data for Brazil
nested$data[[7]])<jupyter_output><empty_output><jupyter_text>Fitting multiple models- nest() turns data into one row per country<jupyter_code>dim(nested)<jupyter_output><empty_output><jupyter_text>- map() applies an operation to each item in a list<jupyter_code>v <- list(1,2,3)
map(v, ~.*10)<jupyter_output><empty_output><jupyter_text>>  map() fits a model to each dataset<jupyter_code>head(nested %>%
     mutate(models=map(data,~lm(percent_yes~year,.))),1)<jupyter_output><empty_output><jupyter_text>>  tidy turns each model into a data frame<jupyter_code>head(nested %>%
     mutate(models=map(data,~lm(percent_yes~year,.))) %>%
     mutate(tidied=map(models, tidy)),1)<jupyter_output><empty_output><jupyter_text>>  unnest() combines the tidied models<jupyter_code>head(nested %>%
     mutate(models=map(data,~lm(percent_yes~year,.))) %>%
     mutate(tidied=map(models, tidy)) %>%
     unnest(tidied),1)
head(country_coefficients <- by_year_country %>%
     nest(-country) %>%
     mutate(models=map(data,~lm(percent_yes~year,.))) %>%
     mutate(tidied=map(models, tidy)) %>%
     unnest(tidied) %>% 
     select(-data,-models),6)<jupyter_output>Warning message:
"All elements of `...` must be named.
Did you want `data = c(year, total, percent_yes)`?"<jupyter_text>The `map()` function from `purrr` works by applying a formula to each item in a list, where . represents the individual item.   Working with many tidy models> Working with many tidy models>  Filter for the year term (slope)<jupyter_code>head(country_coefficients %>%
    filter(term=="year"))<jupyter_output><empty_output><jupyter_text>Multiple hypothesis correction because some p-values will beless than .05 by chance  Filtered by adjusted p-valueNot all slopes are significant, and you can use the p-value to guess which are and which are not.<jupyter_code>head(filtered_countries <- country_coefficients %>%
    filter(term=="year", p.adjust(p.value)<.05) %>%
    mutate(p.adjusted = p.adjust(p.value)),10) 
# Sort for the countries increasing most quickly
head(filtered_countries %>%
  arrange(desc(estimate)),10)
# Sort for the countries decreasing most quickly
head(filtered_countries %>%
  arrange(estimate), 10)<jupyter_output><empty_output><jupyter_text>Joining and tidyingWe'll  combine multiple related datasets, such as incorporating information about each resolution's topic into the vote analysis. We'll also turn untidy data into tidy data, and see how tidy data can guide the exploration of topics and countries over time.  Joining datasets<jupyter_code>head(votes_processed, 10)<jupyter_output><empty_output><jupyter_text>- Each row is one roll call/country pair  Descriptions datasetWe created the `votes_processed` dataset, containing information about each country's votes. We'll now combine that with the new `descriptions` dataset, which includes topic information about each country, so that we can analyze votes within particular topics.<jupyter_code>descriptions <- readRDS(gzcon(url("https://assets.datacamp.com/production/repositories/420/datasets/a438432333a31a6f4aba2d5507df9a44e513b518/descriptions.rds")))
head(descriptions)
head(votes_joined <- votes_processed %>%
        inner_join(descriptions, by=c("rcid", "session")), 10)<jupyter_output><empty_output><jupyter_text>Filtering the joined datasetThere are six columns in the descriptions dataset (and therefore in the new joined dataset) that describe the topic of a resolution:- `me`: Palestinian conflict- `nu`: Nuclear weapons and nuclear material- `di`: Arms control and disarmament- `hr`: Human rights- `co`: Colonialism- `ec`: Economic developmentEach contains a 1 if the resolution is related to this topic and a 0 otherwise.<jupyter_code>head(
    # Filter for votes related to colonialism
    votes_joined %>%
        filter(co==1), 
    10
)<jupyter_output><empty_output><jupyter_text>Visualizing colonialism votesWe graphed the percentage of votes each year where the US voted "yes". Now We'll create that same graph, but only for votes related to colonialism.<jupyter_code># Filter, then summarize by year: US_co_by_year
US_co_by_year <- votes_joined %>%
  filter(country=="United States", co==1) %>%
  group_by(year) %>%
  summarize(percent_yes=mean(vote==1))

# Graph the % of "yes" votes over time
ggplot(US_co_by_year, aes(x=year, y=percent_yes)) +
  geom_line()
ggplot(US_co_by_year, aes(x=year, y=percent_yes)) +
  geom_point()+
    geom_smooth()<jupyter_output>`geom_smooth()` using method = 'loess' and formula 'y ~ x'<jupyter_text>Tidy data>  Topic is spread across six columns- Each topic has one column, so combine into a single variable: topic  Use gather() to bring columns into two<jupyter_code>head(votes_joined %>%
    gather(topic, has_topic, me:ec), 10)<jupyter_output><empty_output><jupyter_text>“topic” is now a variable<jupyter_code>head(votes_gathered <- votes_joined %>%
    gather(topic, has_topic, me:ec) %>%
         filter(has_topic==1), 10)<jupyter_output><empty_output><jupyter_text>Recoding the topicsThere's one more step of data cleaning to make this more interpretable. Right now, topics are represented by two-letter codes:- me: Palestinian conflict- nu: Nuclear weapons and nuclear material- di: Arms control and disarmament- hr: Human rights- co: Colonialism- ec: Economic developmentSo that we can interpret the data more easily, we'll recode the data to replace these codes with their full name. we can do that with dplyr's `recode()` function, which replaces values with ones we specify<jupyter_code># Replace the two-letter codes in topic: votes_tidied
head(votes_tidied <- votes_gathered %>%
  mutate(topic = recode(topic,
                        me = "Palestinian conflict",
                        nu = "Nuclear weapons and nuclear material",
                        di = "Arms control and disarmament",
                        hr = "Human rights",
                        co = "Colonialism",
                        ec = "Economic development")),10)<jupyter_output><empty_output><jupyter_text>Summarize by country, year, and topic<jupyter_code># Summarize the percentage "yes" per country-year-topic
head(by_country_year_topic<-votes_tidied %>%
  group_by(year, country, topic) %>%
    summarize(total=n(), percent_yes=mean(vote==1)) %>%
      ungroup(), 10)<jupyter_output>`summarise()` regrouping output by 'year', 'country' (override with `.groups` argument)<jupyter_text>Visualizing trends in topics for one countryWe can now visualize the trends in percentage "yes" over time for all six topics side-by-side. Here, we'll visualize them just for the United States.<jupyter_code>by_country_year_topic %>%
    filter(country=="United States") %>%
        ggplot(aes(x=year, y=percent_yes)) +
        geom_line()+
        facet_wrap(~topic, scales="free_y")<jupyter_output><empty_output><jupyter_text>Tidy modeling by topic and country>  Nesting by topic and country<jupyter_code># Fit model on the by_country_year_topic dataset
head(country_topic_coefficients <- by_country_year_topic %>%
    nest(-country, -topic) %>%
        mutate(model=map(data, ~lm(percent_yes~year, data=.)),tidied=map(model, tidy))  %>%
            unnest(tidied) %>%
                select(-data, -model), 10)
# Create country_topic_filtered
head(country_topic_filtered <- country_topic_coefficients %>%
  filter(term=='year') %>%
    mutate(p.adjusted=p.adjust(p.value)) %>%
    filter(p.adjusted<.05) %>%
         arrange(estimate), 10)<jupyter_output><empty_output><jupyter_text>Checking models visuallyVanuatu (an island nation in the Pacific Ocean) sharply changed its pattern of voting on the topic of Palestinian conflict.Let's examine this country's voting patterns more closely.  the `by_country_year_topic` dataset contained one row for each combination of country, year, and topic. We can use that to create a plot of Vanuatu's voting, faceted by topic.<jupyter_code>by_country_year_topic %>%
    filter(country=="Vanuatu") %>%
        ggplot(aes(x=year, y=percent_yes)) +
        geom_line() +
        facet_wrap(~topic)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11729950075573672, 'fraction_numerical': 0.008519214033802043, 'mean_word_length': 5.0281612368856985, 'pattern_counts': {'":': 0, '<': 193, '<?xml version=': 0, '>': 286, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '46', 'dir': 'jupyter-structured-clean-dedup', 'id': '583772', 'license': 'MIT', 'n_tokens_mistral': 7302, 'n_tokens_neox': 6715, 'n_words': 1980, 'path': 'Exploring The Historical Voting of USGA.ipynb', 'repo_name': 'VictorOmondi1997/Exploring-The-Historical-Voting-of-USGA', 'size': '21840'}