<jupyter_start><jupyter_text>Introduction In this notebook, we will be mapping the beginning and end coordinates of people who migrated from one location to another after being placed in internment camps. The Tulean Dispatch was the Tule Lake internment camp's official newspaper from 1942 to 1943, and later became the Tule Lake Newell Star newspaper. However, we will be focusing on the data collected during this approximate one year period, and visualizing the movement of individuals across the U.S. as they were forced from one place to another. More information can be found here: http://encyclopedia.densho.org/Tulean%20Dispatch%20(newspaper)/<jupyter_code>from datascience import *
import numpy as np
import pandas as pd
import folium
% matplotlib inline<jupyter_output><empty_output><jupyter_text>1. Data Processing In this section, we will be importing the main csv files `tuleandispatch.csv` and `combined.csv` in the WRA folder, and converting them into tables that we can use in exploratory data analysis (EDA). The cell below does this with the implementation of pandas, a Python package that allows usage of flexible data structures designed to work with analysis of relational data, represented as *pd*.<jupyter_code>dispatch = pd.read_csv('tuleandispatch.csv')
wra = pd.read_csv('WRA/combined.csv')<jupyter_output>/opt/conda/lib/python3.5/site-packages/IPython/core/interactiveshell.py:2717: DtypeWarning: Columns (35) have mixed types. Specify dtype option on import or set low_memory=False.
  interactivity=interactivity, compiler=compiler, result=result)<jupyter_text>Seen in the cell below, the table has many different versions of the same location Tule Lake. How can we clean the data to reconcile this?<jupyter_code>dispatch['Camp Name'].unique()<jupyter_output><empty_output><jupyter_text>The next two cells relabel our columns of interest in uppercase letters in order to match with the WRA dataset. The two tables will then be combined into one, called `merged`.<jupyter_code># Uppercase the columns of interest
dispatch = dispatch.rename(columns={'Last Name': 'LAST NAME', 'First Name': 'FIRST NAME', 'Camp Name':'RELOCATION PROJECT', 'Destination': 'DESTINATION'})

# Make first name and last name values uppercase
for col_name in ['FIRST NAME', 'LAST NAME']:
    dispatch[col_name] = dispatch[col_name].apply((lambda x: str(x).upper()))

dispatch
# Merging with WRA
merged = pd.merge(dispatch, wra, on = ['LAST NAME', 'FIRST NAME', 'RELOCATION PROJECT'])
# merged<jupyter_output><empty_output><jupyter_text>The following cell will define a function `num_occurrences` that takes in three arguments in order to analyze the above data table, identifying all duplicate names in the columns `FIRST NAME` and `LAST NAME` and how many there are of each duplicate.<jupyter_code>def num_occurrences(table, first_name, last_name):
    first_name_s = pd.Series([first_name] * len(table.index))
    last_name_s = pd.Series([last_name]*len(table.index))
    query = table[(table['FIRST NAME'] == first_name_s) &  (table['LAST NAME'] == last_name_s)]
    num_rows = len(query)
    return num_rows

duplicates = set()
for index, row in merged.iterrows():
    first_name, last_name = row['FIRST NAME'], row['LAST NAME']
    n_occurrences = num_occurrences(merged, row['FIRST NAME'], row['LAST NAME'])
    if n_occurrences > 1:
        duplicates.add((first_name, last_name, n_occurrences))
duplicates<jupyter_output><empty_output><jupyter_text>2. Back to Data Science Tables In this section, we are converting all of our data in the `tulean_dispatch.csv` into a readable table that we can use to analyze.<jupyter_code># Convert to a Datascience Table object
tulean_dispatch_joined = Table.from_df(merged)
#tulean_dispatch_joined
# Grouping by first. What does this entail?
get_first = lambda x: x[0]
tulean_dispatch_joined= tulean_dispatch_joined.group('FIRST NAME', collect = get_first).group('LAST NAME', collect = get_first)
# Get Columns of Interest
tulean_dispatch = tulean_dispatch_joined.select(['FIRST NAME', 'LAST NAME', 'RELOCATION PROJECT', 'LAST PERMANENT ADDRESS STATE', 'LAST PERMANENT ADDRESS COUNTY', 'ASSEMBLY CENTER', 'DESTINATION'])
# tulean_dispatch<jupyter_output><empty_output><jupyter_text>Coordinate Data Cleaning Here is the `tulean_dispatch` table that we can finally work with!<jupyter_code># Loading in coordinates
coords = Table.read_table('tuleandispatch_coordinates.csv')
# Group locations
coords = coords.group('Location', collect = lambda x: x[0])
coords
# Join Coordinates and locations
def process_coords(coords):
    return list([float(coord) for coord in coords.split(',')])
locations = ['RELOCATION PROJECT', 'DESTINATION']
for name in locations:
    tulean_dispatch = tulean_dispatch.join(name, coords, 'Location')
    tulean_dispatch[name + ' COORDS'] = tulean_dispatch.apply(process_coords, 'Lat,Long')
    tulean_dispatch = tulean_dispatch.drop(['Lat,Long', 'Notes'])
# Load the table
tulean_dispatch
def get_from_to_coords(table, from_location, to_location):
    result = table.select([from_location + ' COORDS', to_location + ' COORDS'])
    result.relabel(from_location + ' COORDS', 'from')
    result.relabel(to_location + ' COORDS', 'to')
    return result
relocation_to_destination = get_from_to_coords(tulean_dispatch, 'RELOCATION PROJECT', 'DESTINATION')
relocation_to_destination<jupyter_output><empty_output><jupyter_text>3. Plotting with Folium The below cell checks the version type of folium, a mapping package that uses the Leaflet.js library - this lets us know what plug-ins and features are available to us in this Python notebook. More information can be found on https://pypi.python.org/pypi/folium/0.1.5.<jupyter_code>#Version of folium
print(folium.__version__)
import folium
from IPython.display import HTML

def display(m, height=300):
    """Takes a folium instance and embed HTML."""
    m._build_map()
    srcdoc = m.HTML.replace('"', '&quot;')
    embed = HTML('<iframe srcdoc="{0}" '
                 'style="width: 100%; height: {1}px; '
                 'border: none"></iframe>'.format(srcdoc, height))
    return embed<jupyter_output><empty_output><jupyter_text>Now that we have imported folium to make a map displaying the start and end locations of individuals going to internment camps, we can add location markers to pinpoint the areas of interest, with the cell below:<jupyter_code>def plot_locations(table, location_name):
    location_coords = table.column(location_name + ' COORDS')
    # New US Map with Stamen Terrain
    m = folium.Map(location=[39.828175, -98.5795], zoom_start=4, tiles='Stamen Terrain')
    # Loop through table
    for i in range(table.num_rows):
        coords = location_coords[i]
        label = table.column(location_name)[i]
        m.simple_marker(location = coords, popup = label)
    return m
m = plot_locations(tulean_dispatch, 'DESTINATION')
display(m)
# Try zooming out<jupyter_output><empty_output><jupyter_text>Now that we have imported folium to make a map displaying the start and end locations of individuals going to internment camps, we can add location markers to pinpoint the areas of interest, with the cell below:<jupyter_code>def plot_from_to(table, from_location, to_location):
    from_to_array = table.select([from_location + ' COORDS', to_location + ' COORDS']).rows
    # New US Map with Stamen Terrain
    m = folium.Map(location=[39.828175, -98.5795], zoom_start=4, tiles='Stamen Terrain')
    # Loop through from_to_array/table
    for i in range(table.num_rows):
        from_to_coords = from_to_array[i]
        from_label = table.column(from_location)[i]
        to_label = table.column(to_location)[i]
        m.simple_marker(location = from_to_coords[0], popup = from_label)
        m.simple_marker(location = from_to_coords[1], popup = to_label)
        m.line(from_to_coords, line_weight=3, line_color = '#a3c6ff', line_opacity = 0.7)
    return m
m = plot_from_to(tulean_dispatch, 'RELOCATION PROJECT', 'DESTINATION')
display(m)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09766745665460895, 'fraction_numerical': 0.009230385430959212, 'mean_word_length': 5.404153354632588, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 43, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '420544', 'license': 'MIT', 'n_tokens_mistral': 2571, 'n_tokens_neox': 2457, 'n_words': 795, 'path': 'Tulean_Dispatch_Mapping.ipynb', 'repo_name': 'ds-modules/HIST-88', 'size': '8017'}