<jupyter_start><jupyter_text>FiveThirtyEight-Style Visuals and Choropleth Maps for U.S. Immigration  Here, we'll create cool FiveThirtyEight-style subplots using matplotlib and an interactive choropleth map using folium. To do that, we'll work with official UN data on immigration to the United States from 1980 to 2013. If you'd like to explore other countries, you can find the original source data on immigration from around to the world into 45 countries here (https://www.un.org/en/development/desa/population/migration/data/empirical2/migrationflows.asp). Let's break it down: **Part 1**: Cleaning and wrangling, including modifying our columns, setting and resetting our index, adjusting data types using pd.to_numeric, coerce, fillna, and astype, creating a new 'Total' column using sum() and axis**Part 2**: Area plot of immigration for Top 7 countries (using sort values) - Mexico, Philippines, China, India, Vietnam, Dominican Republic, and South Korea **Part 3**: Box and line subplots of immigration for Meixco, China, and India**Part 4**: Box and line subplots of immigration for Philippines, China, India, Vietnam, and South Korea **Part 5**: Choropleth map using folium and geojson file Let's dig in.   Part 1: Cleaning and Wrangling<jupyter_code># Import relevant libraries
import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import numpy as np

!pip install folium
import folium # to create choropleth map

# tranforming json file into a pandas dataframe 
import urllib.request 
import json # library to handle JSON files
from pandas.io.json import json_normalize # tranform JSON file into a pandas dataframe
from urllib.request import urlopen
# The code was removed by Watson Studio for sharing.
# Read in data
usa = pd.read_excel(body, sheet_name="USA by Place of birth", skiprows=range(20),skipfooter=2)
# Check first few rows
pd.set_option("display.max_columns", 100)
pd.set_option("display.max_rows", 220)
usa.head()
# Check dimensions of dataframe. 
usa.shape
# Check column headings. Includes immigration data from 1980 to 2013. Notice the year column headings aren't strings. Let's convert them to strings. 
usa.columns
# Remove unnecessary columns 
usa.drop(['AREA','REG','DEV','Type','Coverage'], axis = 1, inplace = True)
# Make all column labels of type string
usa.columns = list(map(str, usa.columns))
# Check column headings. All column headings are strings. 
usa.columns
# Rename columns
usa.rename(columns = {'OdName':'Country', 'AreaName':'Continent','RegName':'Region'}, inplace = True)
# Set the country name as index, important for looking up countries using .loc method
usa.set_index('Country', inplace = True)
# Create list for years that we can use for plotting later on (remember we need to add 1 to the last, hence 2014 not 2013)
years = list(map(str, range(1980, 2014)))
usa.dtypes
usa.columns
# Try changing column values to numeric using pd.to_numeric. Error shows ".." values. Let's convert those to NaN, convert NaN to zeros, and change data type to integer
usa["1980"] = pd.to_numeric(usa["1980"])
# For each year column, apply pd.to_numeric, coerce errors to NaN values, replace NaN values with zeroes using fillna(0), and change data type to integer
usa["1980"] = pd.to_numeric(usa["1980"], errors="coerce").fillna(0).astype(int)
usa["1981"] = pd.to_numeric(usa["1981"], errors="coerce").fillna(0).astype(int)
usa["1982"] = pd.to_numeric(usa["1982"], errors="coerce").fillna(0).astype(int)
usa["1983"] = pd.to_numeric(usa["1983"], errors="coerce").fillna(0).astype(int)
usa["1984"] = pd.to_numeric(usa["1984"], errors="coerce").fillna(0).astype(int)
usa["1985"] = pd.to_numeric(usa["1985"], errors="coerce").fillna(0).astype(int)
usa["1986"] = pd.to_numeric(usa["1986"], errors="coerce").fillna(0).astype(int)
usa["1987"] = pd.to_numeric(usa["1987"], errors="coerce").fillna(0).astype(int)
usa["1988"] = pd.to_numeric(usa["1988"], errors="coerce").fillna(0).astype(int)
usa["1989"] = pd.to_numeric(usa["1989"], errors="coerce").fillna(0).astype(int)
usa["1990"] = pd.to_numeric(usa["1990"], errors="coerce").fillna(0).astype(int)
usa["1991"] = pd.to_numeric(usa["1991"], errors="coerce").fillna(0).astype(int)
usa["1992"] = pd.to_numeric(usa["1992"], errors="coerce").fillna(0).astype(int)
usa["1993"] = pd.to_numeric(usa["1993"], errors="coerce").fillna(0).astype(int)
usa["1994"] = pd.to_numeric(usa["1994"], errors="coerce").fillna(0).astype(int)
usa["1995"] = pd.to_numeric(usa["1995"], errors="coerce").fillna(0).astype(int)
usa["1996"] = pd.to_numeric(usa["1996"], errors="coerce").fillna(0).astype(int)
usa["1997"] = pd.to_numeric(usa["1997"], errors="coerce").fillna(0).astype(int)
usa["1998"] = pd.to_numeric(usa["1998"], errors="coerce").fillna(0).astype(int)
usa["1999"] = pd.to_numeric(usa["1999"], errors="coerce").fillna(0).astype(int)
usa["2000"] = pd.to_numeric(usa["2000"], errors="coerce").fillna(0).astype(int)
usa["2001"] = pd.to_numeric(usa["2001"], errors="coerce").fillna(0).astype(int)
usa["2002"] = pd.to_numeric(usa["2002"], errors="coerce").fillna(0).astype(int)
usa["2003"] = pd.to_numeric(usa["2003"], errors="coerce").fillna(0).astype(int)
usa["2004"] = pd.to_numeric(usa["2004"], errors="coerce").fillna(0).astype(int)
usa["2005"] = pd.to_numeric(usa["2005"], errors="coerce").fillna(0).astype(int)
usa["2006"] = pd.to_numeric(usa["2006"], errors="coerce").fillna(0).astype(int)
usa["2007"] = pd.to_numeric(usa["2007"], errors="coerce").fillna(0).astype(int)
usa["2008"] = pd.to_numeric(usa["2008"], errors="coerce").fillna(0).astype(int)
usa["2009"] = pd.to_numeric(usa["2009"], errors="coerce").fillna(0).astype(int)
usa["2010"] = pd.to_numeric(usa["2010"], errors="coerce").fillna(0).astype(int)
usa["2011"] = pd.to_numeric(usa["2011"], errors="coerce").fillna(0).astype(int)
usa["2012"] = pd.to_numeric(usa["2012"], errors="coerce").fillna(0).astype(int)
usa["2013"] = pd.to_numeric(usa["2013"], errors="coerce").fillna(0).astype(int)
# Check data types. We see that the data types for years are now correctly integers 
usa.dtypes
# Add total column that sums all immigration for a country
usa['Total'] = usa.sum(axis=1)
usa['Total'].to_frame()<jupyter_output><empty_output><jupyter_text>Part 2: Area Plot of Top 7 Countries<jupyter_code># Change index name "Viet Nam" to "Vietnam" and "Republic of Korea" to "South Korea", make sure to use inplace=True
usa.rename(index={"Viet Nam":"Vietnam"},inplace=True)
usa.rename(index={"Republic of Korea":"South Korea"},inplace=True)
# Top 7 origin countries of immigration based on total immigration from 1980 to 2013
usa.sort_values(['Total'], ascending=False, axis=0, inplace=True)
usa_top7 = usa.head(7)
usa_top7
# Transpose dataframe since for the plot we want years on the x-axis (not countries)
usa_top7 = usa_top7[years].transpose()<jupyter_output><empty_output><jupyter_text>Let's create an area plot of the top 7 countries based on total immigration from 1980 to 2013. The most prominent feature is the spike in immigration from Mexico to nearly 1 million in the early 1990s due, in large part, to immigration reforms of the late 1980s and early 1990s. There's also been a steady increase in immigration from these top 7 countries from 2003 to 2013.<jupyter_code># Call FiveThirtyEight style for plot
mpl.style.use("fivethirtyeight")

# Area plot of Top 7 countries based on total immigration
usa_top7.plot(kind='area', alpha = 0.5, figsize=(20, 10))
plt.title('Immigration Trend of Top 7 Countries')
plt.ylabel('Number of Immigrants')
plt.xlabel('Years')
plt.show()<jupyter_output><empty_output><jupyter_text>Part 3: Box and Line Subplots of Mexico, China, and India<jupyter_code># Dataframe for select countries. Transpose because for the plot later we want years on the x-axis (not countries)
usa_mci= usa.loc[['Mexico','China', 'India'], years].transpose()
usa_mci.describe()<jupyter_output><empty_output><jupyter_text>Let's create subplots, one a box plot and another a line plot, of immigration from Mexico, China, and India from 1980 to 2013. From our box plot, we see that the spread and median of immigration from Mexico is significantly higher with high-end outliers than those of China and India. The spreads of China and India are similar. Our line plot once again highlights the spike in immigration from Mexico in the early 1990s.<jupyter_code># Plot box plot and line plot of select countries in two subplots 
fig = plt.figure() 

ax0 = fig.add_subplot(1, 2, 1) # add subplot 1 (1 row, 2 columns, first plot)
ax1 = fig.add_subplot(1, 2, 2) # add subplot 2 (1 row, 2 columns, second plot). 

# Subplot 1: Box plot
usa_mci.plot(kind='box', color='blue', vert=False, figsize=(20, 6), ax=ax0) 
ax0.set_title('Box Plots of Mexico, China, and India')
ax0.set_xlabel('Number of Immigrants')
ax0.set_ylabel('Countries')

# Subplot 2: Line plot
usa_mci.plot(kind='line', figsize=(20, 6), ax=ax1) 
ax1.set_title ('Line Plots of Mexico, China, and India')
ax1.set_ylabel('Number of Immigrants')
ax1.set_xlabel('Years')

plt.show()<jupyter_output><empty_output><jupyter_text>Part 4: Box and Line Subplots for Philippines, China, India, Vietnam, and South Korea<jupyter_code># Dataframe of select countries. Transpose because for the plot later, we want years on x-axis 
usa_pcivk= usa.loc[['Philippines','China', 'India', 'Vietnam','South Korea'], years].transpose()
usa_pcivk.describe()<jupyter_output><empty_output><jupyter_text>Let's create subplots, one a box plot and another a line plot, of immigration from select Asian countries from 1980 to 2013. From our box plot, Philippines has the highest median annual immigration. The spread of China and India are considerably larger than those of the others. Vietnam has two outliers. We can see them on the right in the line plot with immigration spikes from Vietnam in 1982 and 1992. Our line plot also illustrates a notable rise in immigration from Philippines, China, and India from 1999 to 2013.<jupyter_code># Plot box plot and line plot of select countries in two subplots 
fig = plt.figure() 

ax0 = fig.add_subplot(1, 2, 1) # add subplot 1 (1 row, 2 columns, first plot)
ax1 = fig.add_subplot(1, 2, 2) # add subplot 2 (1 row, 2 columns, second plot). 

# Subplot 1: Box plot
usa_pcivk.plot(kind='box', color='blue', vert=False, figsize=(20, 6), ax=ax0) 
ax0.set_title('Box Plots of Select Asian Countries')
ax0.set_xlabel('Number of Immigrants Annually')
ax0.set_ylabel('Countries')

# Subplot 2: Line plot
usa_pcivk.plot(kind='line', figsize=(20, 6), ax=ax1) 
ax1.set_title ('Line Plots of Select Asian Countries')
ax1.set_ylabel('Number of Immigrants Annually')
ax1.set_xlabel('Years')

plt.show()<jupyter_output><empty_output><jupyter_text>Part 5: Choropleth Map of Immigration to the U.S.<jupyter_code># create a plain world map
world_map = folium.Map(location=[0, 0], zoom_start=2)
world_map
# Load json data from url
with urllib.request.urlopen("https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/DV0101EN/labs/Data_Files/world_countries.json") as url:
    world_geojson = json.loads(url.read().decode())
world_geo = world_geojson['features']
world_geo[0]
# Transform geojson data into a dataframe of just country names

# Empty array with column names
column_names = ['Country']

# Instantiate dataframe
countries = pd.DataFrame(columns = column_names)

countries
# Build countries dataframe

# Pull in respective data from json using for loop. In the json, boroughs are under 'Gemeinde_name'
for data in world_geo:
    country = data['properties']['name'] 
    
    # Fill boroughs dataframe we created above with data
    countries = countries.append({'Country': country}, ignore_index=True)
    
countries
# Check to see if the country names from our usa dataframe match those from the countries dataframe created from the geojson
set(usa.index) - set(countries.Country)
# Change select country names so they match the property name in the geojson file, make sure to use inplace=True
usa.rename(index={"Russian Federation":"Russia"},inplace=True)
usa.rename(index={"Bolivia (Plurinational State of)":"Bolivia"},inplace=True)
usa.rename(index={"Venezuela (Bolivarian Republic of)":"Venezuela"},inplace=True)
usa.rename(index={"Côte d'Ivoire":"Ivory Coast"},inplace=True)
usa.rename(index={"United Kingdom of Great Britain and Northern Ireland":"United Kingdom"},inplace=True)
usa.rename(index={"Iran (Islamic Republic of)":"Iran"},inplace=True)
usa.rename(index={"Syrian Arab Republic":"Syria"},inplace=True)
usa.rename(index={"The former Yugoslav Republic of Macedonia":"Macedonia"},inplace=True)
usa.rename(index={"Republic of Moldova":"Moldova"},inplace=True)
usa.rename(index={"Lao People's Democratic Republic":"Laos"},inplace=True)
usa.rename(index={"Serbia":"Republic of Serbia"},inplace=True)
# Check the first few rows again. As we'll need "Country" as a column for our plot later, let's reset the index (recall we set country to index earlier)
usa.head()
# Reset index to original
usa = usa.reset_index()
# Check the column headings, we see that we have "Country" back
usa.columns
# Call the geojson file
world_geo = r'world_countries.json'

# Customize scale to be used for colors, called in threshold_scale parameter below
myscale = (usa['Total'].quantile((0,0.1,0.3,0.6,0.9, 1))).tolist()

# Plot choropleth on world map using our geojson
world_map = folium.Map(location=[0, 0], zoom_start=2)
world_map.choropleth(
    geo_data=world_geo,
    data=usa,
    columns=['Country', 'Total'],
    key_on='feature.properties.name',
    threshold_scale=myscale,
    fill_color='GnBu', 
    fill_opacity=0.7, 
    line_opacity=0.2,
    legend_name='Immigration to the United States',
    reset=True
)

world_map<jupyter_output>/opt/conda/envs/Python36/lib/python3.6/site-packages/folium/folium.py:415: FutureWarning: The choropleth  method has been deprecated. Instead use the new Choropleth class, which has the same arguments. See the example notebook 'GeoJSON_and_choropleth' for how to do this.
  FutureWarning
{'fraction_non_alphanumeric': 0.13403313339045986, 'fraction_numerical': 0.03984575835475578, 'mean_word_length': 6.171018945212493, 'pattern_counts': {'":': 13, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '292001', 'license': 'MIT', 'n_tokens_mistral': 4992, 'n_tokens_neox': 4452, 'n_words': 1424, 'path': 'United States Immigration Map.ipynb', 'repo_name': 'hrishipoola/US_Immigration_Visualization_and_Choropleth_Maps', 'size': '14004'}