<jupyter_start><jupyter_text>File: CA-Travel-Trends_DataPrep_EDA.ipynb  Names: <NAME>, <NAME>  Date: 11/12/20   Usage: Program cleans data, generates exploratory visualizations, and saves cleaned data to a csv file.<jupyter_code>import matplotlib
import matplotlib.pyplot as plt

import pandas as pd
import numpy as np

import matplotlib.dates as mdates
from matplotlib.dates import DateFormatter<jupyter_output><empty_output><jupyter_text>Predicting Travel Trends in the United States 2019 - 2020 *Data Prep and Exploratory Data Analysis***Data source:**    Loading and Previewing Data<jupyter_code># Loading data into a Pandas DataFrame
trip_data = pd.read_csv("Trips_by_Distance.csv")
# Checking dimensions
print(trip_data.shape)
# Previewing data
trip_data.head(5)<jupyter_output><empty_output><jupyter_text>Displaying Data Summaries<jupyter_code># Displaying summary information for string columns

print("String Data:\n")
print(trip_data.describe(include=['O']))
# Displaying summary information for numeric columns without scientific notation

with pd.option_context('float_format', '{:f}'.format):
    print("Numeric Data:\n\n", trip_data.describe())
# List unique values in the County Name column
counties_list = trip_data['County Name'].unique().tolist()

print(counties_list)<jupyter_output>['Putnam County', 'Lake and Peninsula Borough', 'Autauga County', 'Baldwin County', 'Barbour County', 'Bibb County', 'Blount County', 'Bullock County', 'Butler County', 'Calhoun County', 'Chambers County', 'Cherokee County', 'Chilton County', 'Choctaw County', 'Clarke County', 'Clay County', 'Cleburne County', 'Coffee County', 'Colbert County', 'Conecuh County', 'Coosa County', 'Covington County', 'Crenshaw County', 'Cullman County', 'Dale County', 'Dallas County', 'DeKalb County', 'Elmore County', 'Escambia County', 'Etowah County', 'Fayette County', 'Franklin County', 'Geneva County', 'Greene County', 'Hale County', 'Matanuska-Susitna Borough', 'Nome Census Area', 'North Slope Borough', 'Northwest Arctic Borough', 'Petersburg Borough', 'Prince of Wales-Hyder Census Area', 'Sitka City and Borough', 'Skagway Municipality', 'Southeast Fairbanks Census Area', 'Valdez-Cordova Census Area', 'Wrangell City and Borough', 'Yakutat City and Borough', 'Yukon-Koyukuk Census Area', 'Apache County[...]<jupyter_text>**Observations so far:**- There are 1,877 different counties, with Washington County being the most common.- It looks like we have an outlier of -38 for "Population Not Staying at Home" and -108 for "Number of Trips 3-5" which doesn't make sense. Negative numbers will need to be removed. - There may be some missing data (NaN)  Missing Data<jupyter_code># Checking missing data sums
trip_data.isna().sum()<jupyter_output><empty_output><jupyter_text>Since we have a good amount of data to work with, we're going to remove the rows with missing data.  Data Cleanup<jupyter_code># Removing missing data
trip_data_clean = trip_data.dropna()

# Checking missing data sums
trip_data_clean.isna().sum()
# Remove outliers (negative numbers)

trip_data_clean = trip_data_clean[trip_data_clean['Population Not Staying at Home'] > -1]
# Converting dates to datetime format
trip_data_clean['Date'] = pd.to_datetime(trip_data_clean['Date'])

trip_data_clean['Date'].head(10)
# Summary information for datetime column
print("Datetime Data:\n")
print(trip_data_clean['Date'].describe())

# Summary information for cleaned string columns
print("String Data:\n")
print(trip_data_clean.describe(include=['O']))

# Summary information for cleaned numeric columns without scientific notation
with pd.option_context('float_format', '{:f}'.format):
    print("\nNumeric Data:\n\n", trip_data_clean.describe())<jupyter_output>Datetime Data:

count                 2060200
unique                    663
top       2020-07-22 00:00:00
freq                     3141
first     2019-01-01 00:00:00
last      2020-10-24 00:00:00
Name: Date, dtype: object
String Data:

          Level State Postal Code        County Name
count   2060200           2060200            2060200
unique        1                51               1876
top      County                TX  Washington County
freq    2060200            165599              19890

Numeric Data:

           State FIPS    County FIPS  Population Staying at Home  \
count 2060200.000000 2060200.000000              2060200.000000   
mean       30.291433   30395.011147                22564.431223   
std        15.142546   15160.851846                77587.660701   
min         1.000000    1001.000000                    8.000000   
25%        18.000000   18177.000000                 2101.000000   
50%        29.000000   29163.000000                 4889.000000   
75%        45[...]<jupyter_text>With the newly updated data, we now have observations up until October 24, 2020. The previous version of the data set only went up until August 29th.  Exploratory Visualizations: All Data<jupyter_code># Boxplot of number of trips above 500 miles
trip_data_clean.boxplot(by ='Date', 
                        column =['Number of Trips >=500'], 
                        figsize=(15,9), 
                        grid = False)<jupyter_output><empty_output><jupyter_text>Using all of the observations in one graph is mostly convoluted and ineffective, so in order to make our data more managable and focused, we're going to narrow our analysis to California counties. Even amidst the overcrowding in the above plot, we can see the general ups and downs of travel displayed. The trend is what we originally expected. There are more trips in the first half of the graph (2019) with a drop across most of 2020, and a dramatic spike during the summer (August 2020). Finally, there is a drop at the end of the data during the Fall. To better explain this trend, we might need to supplement with Covid-19 data.  Filtering Data to California Counties<jupyter_code># Filtering to California counties
trip_data_clean_CA = trip_data_clean[trip_data_clean['State Postal Code']=='CA']

# Previewing filtered data
print('Shape:', trip_data_clean_CA.shape)
trip_data_clean_CA.head(5)<jupyter_output>Shape: (38093, 19)<jupyter_text>Previously, there were 34,863 observations for California, so with the updated data we've gained 3,230 more observations.<jupyter_code># Summary information for datetime column
print("Datetime Data:\n")
print(trip_data_clean_CA['Date'].describe())

# Summary information for cleaned string columns
print("\nString Data:\n")
print(trip_data_clean_CA.describe(include=['O']))

# Summary information for cleaned numeric columns without scientific notation
with pd.option_context('float_format', '{:f}'.format):
    print("\nNumeric Data:\n\n", trip_data_clean_CA.describe())
# List unique values in the 'County Name' column
counties_list_CA = trip_data_clean_CA['County Name'].unique().tolist()

print(counties_list_CA)<jupyter_output>['Alameda County', 'Alpine County', 'Amador County', 'Butte County', 'Calaveras County', 'Colusa County', 'Contra Costa County', 'Del Norte County', 'El Dorado County', 'Fresno County', 'Glenn County', 'Humboldt County', 'Imperial County', 'Inyo County', 'Kern County', 'Kings County', 'Lake County', 'Lassen County', 'Los Angeles County', 'Madera County', 'Marin County', 'Mariposa County', 'Mendocino County', 'Merced County', 'Modoc County', 'Mono County', 'Monterey County', 'Napa County', 'Nevada County', 'Orange County', 'Placer County', 'Plumas County', 'Riverside County', 'Sacramento County', 'San Benito County', 'San Bernardino County', 'San Diego County', 'San Francisco County', 'San Joaquin County', 'San Luis Obispo County', 'San Mateo County', 'Santa Barbara County', 'Santa Clara County', 'Santa Cruz County', 'Shasta County', 'Sierra County', 'Siskiyou County', 'Solano County', 'Sonoma County', 'Stanislaus County', 'Sutter County', 'Tehama County', 'Trinity County', 'Tulare Coun[...]<jupyter_text>Exploratory Visualizations: California Data<jupyter_code># Boxpot of trips greater than 500 miles in California
trip_data_clean_CA.boxplot(by ='Date', 
                           column =['Number of Trips >=500'], 
                           figsize=(15,9), 
                           grid = False)<jupyter_output><empty_output><jupyter_text>This boxplot looks slightly better than the last plot, and conveys the same overall trend, but there are still too many data to be illustrated effectively in a boxplot. Next, we will try a few scatterplots.<jupyter_code># Scatterplot of California trips above 500 miles

# Handle date time conversions between pandas and matplotlib
from pandas.plotting import register_matplotlib_converters
register_matplotlib_converters()

# Create figure and plot space
fig, ax = plt.subplots(figsize=(15, 9))

# Add x-axis and y-axis
ax.scatter(trip_data_clean_CA['Date'],
       trip_data_clean_CA['Number of Trips >=500'],
       color='purple', alpha=0.5)

plt.xlabel('Date', fontsize=15)
plt.ylabel('Number of Trips >=500 miles', fontsize=15)
plt.suptitle('Number of Trips Taken in California (>=500 miles)\nJanuary 2019 - October 2020', fontsize=20)

plt.show()
# Scatterplot of California Population Staying at Home

# Create figure and plot space
fig, ax = plt.subplots(figsize=(15, 9))

# Add x-axis and y-axis
ax.scatter(trip_data_clean_CA['Date'],
       trip_data_clean_CA['Population Staying at Home'],
       color='blue', alpha=0.5)

plt.xlabel('Date', fontsize=15)
plt.ylabel('Population Staying at Home', fontsize=15)
plt.suptitle('Population Staying at Home in California\nJanuary 2019 - October 2020', fontsize=20)

plt.show()<jupyter_output><empty_output><jupyter_text>The first scatterplot depicting trips taken in California follows the expected pattern, while the second scatterplot depicting the population staying at home shows an opposite pattern but still expected. This makes sense that trips taken would be negatively correlated with population staying at home. In general, the less trips taken, the more people are staying at home.  To account for the dramatic change in trips taken and population staying at home beginning in March of 2020, it will be helpful to look at Covid-19 cases in California.  Covid-19 Supplemental Data**Data Source:**   Loading Data<jupyter_code># Loading data into a Pandas DataFrame
covid_data_CA = pd.read_csv("statewide_cases.csv")

# Checking dimensions
print(covid_data_CA.shape)

# Previewing data
covid_data_CA.head(5)<jupyter_output>(13925, 6)<jupyter_text>Previously, our data included 11,225 rows, but now with updated data we have 13,925.  Cleaning Covid-19 Data<jupyter_code># Checking missing data sums
covid_data_CA.isna().sum()
# Removing missing data
covid_data_CA_clean = covid_data_CA.dropna()

# Checking missing data sums
covid_data_CA_clean.isna().sum()
covid_data_CA_clean.dtypes
# Converting dates to datetime format
covid_data_CA_clean['date'] = pd.to_datetime(covid_data_CA_clean['date'])

covid_data_CA_clean['date'].head(10)
# Summary information for datetime column
print("Datetime Data:\n")
print(covid_data_CA_clean['date'].describe())<jupyter_output>Datetime Data:

count                   13922
unique                    233
top       2020-09-01 00:00:00
freq                       60
first     2020-03-18 00:00:00
last      2020-11-05 00:00:00
Name: date, dtype: object<jupyter_text>This updated data set includes up until November 5, 2020.  Plotting Covid-19 Data<jupyter_code># Scatterplot of California Covid-19 cases

# Create figure and plot space
fig, ax = plt.subplots(figsize=(15, 9))

# Add x-axis and y-axis
ax.scatter(covid_data_CA_clean['date'],
       covid_data_CA_clean['totalcountconfirmed'],
       color='orange', alpha=0.5)

plt.xlabel('Date', fontsize=15)
plt.ylabel('Total Count Confirmed', fontsize=15)
plt.suptitle('Total Confirmed Covid-19 Cases in California\nMarch 2020 - November 2020', fontsize=20)

plt.show()<jupyter_output><empty_output><jupyter_text>From this graph we can confirm that the amount of Covid-19 cases started rising in March of 2020, right as the number of trips taken started decreasing and the population staying at home started increasing. With the newly added data, we can see the recent even more dramatic rise in cases during our current month of November 2020.At this point, we'll save our cleaned data as csv files to import into RStudio for modeling.  Saving Cleaned Data<jupyter_code># Writing cleaned CA trip data to csv file
trip_data_clean_CA.to_csv('Trips_by_Distance_CA_clean.csv', index_label=False)
# Writing cleaned CA Covid-19 data to csv file
covid_data_CA_clean.to_csv('covid_data_CA_clean.csv', index_label=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09645513530621934, 'fraction_numerical': 0.04351954423168223, 'mean_word_length': 4.587533156498674, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 59, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '308527', 'license': 'CC-BY-3.0', 'n_tokens_mistral': 4257, 'n_tokens_neox': 3695, 'n_words': 1302, 'path': 'Projects/CA-Travel-Trends/CA-Travel-Trends_DataPrep_EDA.ipynb', 'repo_name': 'corinnemedeiros/corinnemedeiros.github.io', 'size': '12655'}