<jupyter_start><jupyter_text>A4: Common Analysis Chang XuDuring the last two years we all have been experiencing a global pandemic. This has been tragic and disruptive to many countries and has taken a deep personal toll on many individuals and their families. One aspect that has been hard to miss in the last two years is the datafication of the pandemic. That is, many aspects of the individual toll of the pandemic have been collected, aggregated and re-represented as data. This datafication gives us the privilege to examine the pandemic from potentially many different perspectives to understand how it has changed lives and how it has changed society. To be honest, we are actually at the very beginning of understanding and comprehending these impacts.During our Course Project we are going to begin taking a look at some of the social aspects of the pandemic by conducting a human centered data science analysis of some available COVID-19 data. In this Assignment A4: Common Analysis, every student in the course will work from the same datasets. I am assigned to analyze data for one specific County of the United States - New York, New York.  Step 0: Data acquisition The common analysis research question will require several different datasets. In this assignment, we use data from:1. The RAW_us_confirmed_cases.csv file from the Kaggle repository of John Hopkins University COVID-19 data.2. The CDC dataset of masking mandates by county.3. The New York Times mask compliance survey data.The majority of this data is by US County by Day. The mask compliance is a single shot estimator that gives us a compliance estimate for every County in the US. I carefully review the data descriptions that accompany these datasets. They each have some interesting caveats. As well, some of them are explicit with regard to the way I interpret missing data. Import libraries needed<jupyter_code>import os
import sys
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Read data to dataframe<jupyter_code>case_raw = pd.read_csv("raw/RAW_us_confirmed_cases.csv")
case_raw.head()<jupyter_output><empty_output><jupyter_text>For county data, I used the search and export feature on CDC to search for records specific for New York county and export all the output data to a cvs file. That's why the file only contaions data for New York county.<jupyter_code>county_raw = pd.read_csv("raw/U.S._State_and_Territorial_Public_Mask_Mandates_From_April_10__2020_through_August_15__2021_by_County_by_Day.csv")
county_raw.head()
mask_raw = pd.read_csv("raw/mask-use-by-county.csv")
mask_raw.head()<jupyter_output><empty_output><jupyter_text>Step 1: Analyze In this part, we are asked to answer an common question:- How did masking policies change the progression of confirmed COVID-19 cases from February 1, 2020 through October 15, 2021? First, we need to clean and standardize over the three datasets. In order to do that, we need to find FIPS, which is a unique identifier for counties, for New York county.<jupyter_code>fips = case_raw.loc[(case_raw['Province_State'] == 'New York') & (case_raw['Admin2'] == 'New York'), 'FIPS'].values[0]
fips<jupyter_output><empty_output><jupyter_text>Processing "U.S. State and Territorial Public Mask Mandates From April 10, 2020 through August 15, 2021 by County by Day" data Add FIPS code to each row of data from New York county, so we can use it as a key to join tables later<jupyter_code>county = county_raw.copy()
county['FIPS'] = fips
county<jupyter_output><empty_output><jupyter_text>Drop unnecessary columns in the table<jupyter_code>county = county.drop(['FIPS_State', 'FIPS_County'], axis = 1)
county<jupyter_output><empty_output><jupyter_text>Convert date format for filtering convenience<jupyter_code>county['date'] = pd.to_datetime(county['date'], format = '%m/%d/%Y')
county<jupyter_output><empty_output><jupyter_text>Since the data is already between February 1, 2020 through October 15, 2021, we can keep it all.  Processing "Raw U.S. Confirmed Cases" data Before we start the similar processing, we need to make dates and their respective cases columns so we can process.<jupyter_code>case = case_raw.copy()
case = pd.melt(case, id_vars = case.columns[0:11], var_name = 'date', value_name = 'cases')
case<jupyter_output><empty_output><jupyter_text>Drop unwanted columns<jupyter_code>case = case.drop(['Province_State', 'Admin2', 'UID', 'iso2', 'iso3', 'code3', 'Country_Region', 'Lat', 'Long_'], axis = 1)
case<jupyter_output><empty_output><jupyter_text>Select the case data only for New York County, New York<jupyter_code>case = case.loc[case['FIPS'] == fips]
case<jupyter_output><empty_output><jupyter_text>Standardize the time to make it easy for query<jupyter_code>case['date'] = pd.to_datetime(case['date'], format = '%m/%d/%y')
case<jupyter_output><ipython-input-12-669928a77b66>:1: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  case['date'] = pd.to_datetime(case['date'], format = '%m/%d/%y')<jupyter_text>Get only the data from February 1, 2020 to October 15, 2021<jupyter_code>case = case.loc[(case['date'] >= '2020-02-01') & (case['date'] <= '2021-10-15')].reset_index().drop(['index'], axis = 1)
case<jupyter_output><empty_output><jupyter_text>Combioning the two datasets to get a main table that we will mostly use for our analysis<jupyter_code>combined = case.merge(county, how = 'left', on = ['FIPS', 'date'])
combined<jupyter_output><empty_output><jupyter_text>Make binary answers for "Face_Masks_Required_in_Public"<jupyter_code>combined['Face_Masks_Required_in_Public'] = combined['Face_Masks_Required_in_Public'] == 'Yes'
combined<jupyter_output><empty_output><jupyter_text>Rename the column to make it clear that the "cases" is for cumulative confirmed cases<jupyter_code>combined = combined.rename({'cases': 'Cumulative_Cases'}, axis = 1)
combined<jupyter_output><empty_output><jupyter_text>Calculate daily confirmed cases<jupyter_code>combined['Daily_Cases'] = combined['Cumulative_Cases'].diff(periods = 1)
combined<jupyter_output><empty_output><jupyter_text>The infection time series is a set of slopes. We want to get the data for change in slope over time.<jupyter_code>combined['Slop'] = combined['Cumulative_Cases'].diff(periods = 1)
combined<jupyter_output><empty_output><jupyter_text>Calculating the derivative function - change in slope over time<jupyter_code># copy the table to prevent I mess up...
df = combined.copy()
df['Total_Population'] = 1694251<jupyter_output><empty_output><jupyter_text>The initial population is what I look up from the table. Here I calculate the infection rate and the smooth infection rate considering a window of 7 days and take the average (so it's more meaningful when it comes to ploting the figure). There are a few assumptions made:1. Assume the population for the entire time period remains the same2. Assume people have the same probability of getting COVID, regardless of whether they wear masks or notI may need to refine how I calculate this slope later<jupyter_code>df['Susceptible_Population'] = df['Total_Population'] - df['Cumulative_Cases']

df['Infection_Rate'] = (df['Daily_Cases'] / df['Susceptible_Population']) * 100

df['7Day_Average_Cases'] = df['Daily_Cases'].rolling(window = 7).mean()
df['Infection_Rate_Smooth'] = (df['7Day_Average_Cases'] / df['Susceptible_Population']) * 100

df<jupyter_output><empty_output><jupyter_text>Save data to the directory for A5 anlysis<jupyter_code>df.to_csv("a4_data.csv")<jupyter_output><empty_output><jupyter_text>Analyze Mask Compliance Data<jupyter_code>mask = mask_raw.copy()
mask = mask.loc[mask['COUNTYFP'] == fips]
mask<jupyter_output><empty_output><jupyter_text>Step 2:  Visualize In this step, I create a graph that visualizes how the course of the disease was changed by masking policies. For New York County, New York, I create a time series showing the changes in the derivative function of the rate of infection. My graph indicates days where masking policies were in effect (or not) and whether the difference in the derivative function was significant. Optionally, I add a second time series that shows the actual rate of infection.  Plot for Cumulative Confirmed Cases We use green color to indicate the period when face masks are required in public.<jupyter_code>plt.figure(figsize=(20,7))

plt.plot(combined['date'], combined['Cumulative_Cases'], color = 'orange')

sub_x = combined.loc[combined['Face_Masks_Required_in_Public'], 'date']
sub_y = combined.loc[combined['Face_Masks_Required_in_Public'], 'Cumulative_Cases']
plt.plot(sub_x, sub_y, color = 'green', label = 'Face Masks Required in Public')

plt.xlabel('Date', fontsize = 15)
plt.ylabel('Confirmed Cases', fontsize = 15)
plt.title('COVID-19 Cumulative Confirmed Cases in New York County, New York (2/01/2020 - 10/15/2021)', fontsize = 20)

plt.grid()
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Plot for Daily Confirmed Cases<jupyter_code>plt.figure(figsize=(20,7))

plt.plot(combined['date'], combined['Daily_Cases'], color = 'orange')

sub_x = combined.loc[combined['Face_Masks_Required_in_Public'], 'date']
sub_y = combined.loc[combined['Face_Masks_Required_in_Public'], 'Daily_Cases']
plt.plot(sub_x, sub_y, color = 'green', label = 'Face Masks Required in Public')

plt.xlabel('Date', fontsize = 15)
plt.ylabel('Confirmed Cases', fontsize = 15)
plt.title('COVID-19 Daily Confirmed Cases in New York County, New York (2/01/2020 - 10/15/2021)', fontsize = 20)

plt.grid()
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Plot for changes in the derivative function of the rate of infection<jupyter_code>plt.figure(figsize=(20,7))

plt.plot(df['date'], df['Infection_Rate_Smooth'], color = 'orange')

sub_x = df.loc[df['Face_Masks_Required_in_Public'], 'date']
sub_y = df.loc[df['Face_Masks_Required_in_Public'], 'Infection_Rate_Smooth']
plt.plot(sub_x, sub_y, color = 'green', label = 'Face Masks Required in Public')

plt.xlabel('Date', fontsize = 15)
plt.ylabel('Infection Rate (%)', fontsize = 15)
plt.title('COVID-19 Daily Infection Rate in New York County, New York (2/01/2020 - 10/15/2021)', fontsize = 20)

plt.grid()
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Plot for Mask Compliance<jupyter_code>data = {'NEVER':2, 'RARELY':1.6, 'SOMETIMES':4, 'FREQUENTLY':12.1, 'ALWAYS':80.3 }
atitude = list(data.keys())
values = list(data.values())
  
fig = plt.figure(figsize = (10, 5))
 
# creating the bar plot
plt.bar(atitude, values, width = 0.4)
 
plt.xlabel("Mask Use Frequency")
plt.ylabel("Percentage in Population (%)")

plt.title("Mask Use Frequency in New York County, New York")


plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10307678261663471, 'fraction_numerical': 0.020907513923126085, 'mean_word_length': 6.421409214092141, 'pattern_counts': {'":': 0, '<': 102, '<?xml version=': 0, '>': 102, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '476143', 'license': 'MIT', 'n_tokens_mistral': 3662, 'n_tokens_neox': 3388, 'n_words': 1153, 'path': 'a4-common-analysis.ipynb', 'repo_name': 'maruchang/512project', 'size': '10953'}