<jupyter_start><jupyter_text>Task1  Generate weekly statistics (mean, median, mode) for number of cases and deaths across a specific state. Compare the data against other states.  I selected MA to generate weekly statistics. Then, I chose AZ, IN, MO, NJ, TN to compare their data with MA data. The reason that I selected those 5 states is that their population is similar to MA.<jupyter_code># Select states AZ, IN, MO, NJ, TN

covid_dfSt = covid_dfState.iloc[ [3,15,19,24,31,42] , : ]
covid_dfSt
#Seperate Cases and Deaths

StCases = covid_dfSt.filter(regex = "x")
StDeaths = covid_dfSt.filter(regex = "y")
# Calculate new Cases and deaths

StNewCases = StCases.diff(axis = 1)
StNewDeaths = StDeaths.diff(axis = 1)
StNewCases<jupyter_output><empty_output><jupyter_text>Normalizing new cases and new deaths per 1000000 people. Since population of each state is around 7000000, I selected 1000000 for normalizing my data.<jupyter_code># Normalize new cases

StNormCases = StNewCases.transpose()
StNormCases["AZ Cases(Norm)"] = ((StNormCases[3]/7278717)*1000000).round()
StNormCases["IN Cases(Norm)"] = ((StNormCases[15]/6732219)*1000000).round()
StNormCases["MA Cases(Norm)"] = ((StNormCases[19]/6892503)*1000000).round()
StNormCases["MO Cases(Norm)"] = ((StNormCases[24]/6137428)*1000000).round()
StNormCases["NJ Cases(Norm)"] = ((StNormCases[31]/8882190)*1000000).round()
StNormCases["TN Cases(Norm)"] = ((StNormCases[42]/6829174)*1000000).round()
StNormCases.drop(columns =[3,15,19,24,31,42], inplace = True)


# Normalize new deaths

StNormDeaths = StNewDeaths.transpose()
StNormDeaths["AZ Deaths(Norm)"] = ((StNormDeaths[3]/7278717)*1000000).round()
StNormDeaths["IN Deaths(Norm)"] = ((StNormDeaths[15]/6732219)*1000000).round()
StNormDeaths["MA Deaths(Norm)"] = ((StNormDeaths[19]/6892503)*1000000).round()
StNormDeaths["MO Deaths(Norm)"] = ((StNormDeaths[24]/6137428)*1000000).round()
StNormDeaths["NJ Deaths(Norm)"] = ((StNormDeaths[31]/8882190)*1000000).round()
StNormDeaths["TN Deaths(Norm)"] = ((StNormDeaths[42]/6829174)*1000000).round()
StNormDeaths.drop(columns =[3,15,19,24,31,42], inplace = True)

StNormCases<jupyter_output><empty_output><jupyter_text>Calculate weekly mean for new cases and new deaths<jupyter_code>#Calculate mean for Normalized New Cases

StNormCases_week = StNormCases.groupby(np.arange(len(StNormCases))//7).mean().round(0).astype(int).rename_axis('Week')
StNormCases_week.columns = ['AZ', 'IN','MA','MO','NJ','TN']
StNormCases_week = StNormCases_week.reset_index()
StNormCases_week
#Calculate mean for Normalized New Deaths

StNormDeaths_week = StNormDeaths.groupby(np.arange(len(StNormDeaths))//7).mean().round(0).astype(int).rename_axis('Week')
StNormDeaths_week.columns = ['AZ', 'IN','MA','MO','NJ','TN']
StNormDeaths_week = StNormDeaths_week.reset_index()<jupyter_output><empty_output><jupyter_text>Mean, Median and Mode for normalized new cases and new deaths. Value of mode for most of the states is zero. That's why, after calculating mode, I also write a code for ignoring zero and calculating mode.<jupyter_code># Calculate Mean, Median, Mode for Normalized Cases

AZNormCases_mean = round(StNormCases_week["AZ"].mean())
AZNormCases_median = round(StNormCases_week["AZ"].median())
AZNormCases_mode = StNormCases_week["AZ"].mode()
AZNormCases0_mode = StNormCases_week["AZ"].replace(0, np.nan).mode()

INNormCases_mean = round(StNormCases_week["IN"].mean())
INNormCases_median = round(StNormCases_week["IN"].median())
INNormCases_mode = StNormCases_week["IN"].mode()
INNormCases0_mode = StNormCases_week["IN"].replace(0, np.nan).mode()

MANormCases_mean = round(StNormCases_week["MA"].mean())
MANormCases_median = round(StNormCases_week["MA"].median())
MANormCases_mode = StNormCases_week["MA"].mode()
MANormCases0_mode = StNormCases_week["MA"].replace(0, np.nan).mode()

MONormCases_mean = round(StNormCases_week["MO"].mean())
MONormCases_median = round(StNormCases_week["MO"].median())
MONormCases_mode = StNormCases_week["MO"].mode()
MONormCases0_mode = StNormCases_week["MO"].replace(0, np.nan).mode()

NJNormCases_mean = round(StNormCases_week["NJ"].mean())
NJNormCases_median = round(StNormCases_week["NJ"].median())
NJNormCases_mode = StNormCases_week["NJ"].mode()
NJNormCases0_mode = StNormCases_week["NJ"].replace(0, np.nan).mode()

TNNormCases_mean = round(StNormCases_week["TN"].mean())
TNNormCases_median = round(StNormCases_week["TN"].median())
TNNormCases_mode = StNormCases_week["TN"].mode()
TNNormCases0_mode = StNormCases_week["TN"].replace(0, np.nan).mode()
# Create a list of lists for cases mean, median, mode

Cases_statistics =[[AZNormCases_mean, INNormCases_mean, MANormCases_mean, MONormCases_mean, NJNormCases_mean,TNNormCases_mean],
             [AZNormCases_median, INNormCases_median, MANormCases_median, MONormCases_median, NJNormCases_median, TNNormCases_median],
             [AZNormCases_mode[0], INNormCases_mode[0], MANormCases_mode[0], MONormCases_mode[0], NJNormCases_mode[0], TNNormCases_mode[0]]]


#Create a dataframe from list

Cases_sta = pd.DataFrame(Cases_statistics, columns = ['AZ', 'IN', 'MA','MO', 'NJ', 'TN'], 
                         index=['Mean', 'Median', 'Mode'])  
Cases_sta<jupyter_output><empty_output><jupyter_text>In all states mean is larger than median and mode is zero which means the distribution is right skewed.  Larger value of mean shows that there are a few larger value of new cases in the data which drives the mean upward, but they could not effect the median. The difference between mean and median in IN is not very big which shows that the distribution of new cases data can be assumed to be approximately symmetrical compare to other states. TN has the highest mean and MA lowest mean among other states. Since the data is normalized, we can conclude that MA has the lowest new cases and TN has the highest new Cases among other states.<jupyter_code># Calculate Mean, Median, Mode for normalized new Deaths

AZNormDeaths_mean = round(StNormDeaths_week["AZ"].mean())
AZNormDeaths_median = round(StNormDeaths_week["AZ"].median())
AZNormDeaths_mode = StNormDeaths_week["AZ"].mode()
AZNormDeaths0_mode = StNormDeaths_week["AZ"].replace(0, np.nan).mode()

INNormDeaths_mean = round(StNormDeaths_week["IN"].mean())
INNormDeaths_median = round(StNormDeaths_week["IN"].median())
INNormDeaths_mode = StNormDeaths_week["IN"].mode()
INNormDeaths0_mode = StNormDeaths_week["IN"].replace(0, np.nan).mode()

MANormDeaths_mean = round(StNormDeaths_week["MA"].mean())
MANormDeaths_median = round(StNormDeaths_week["MA"].median())
MANormDeaths_mode = StNormDeaths_week["MA"].mode()
MANormDeaths0_mode = StNormDeaths_week["MA"].replace(0, np.nan).mode()

MONormDeaths_mean = round(StNormDeaths_week["MO"].mean())
MONormDeaths_median = round(StNormDeaths_week["MO"].median())
MONormDeaths_mode = StNormDeaths_week["MO"].mode()
MONormDeaths0_mode = StNormDeaths_week["MO"].replace(0, np.nan).mode()

NJNormDeaths_mean = round(StNormDeaths_week["NJ"].mean())
NJNormDeaths_median = round(StNormDeaths_week["NJ"].median())
NJNormDeaths_mode = StNormDeaths_week["NJ"].mode()
NJNormDeaths0_mode = StNormDeaths_week["NJ"].replace(0, np.nan).mode()


TNNormDeaths_mean = round(StNormDeaths_week["TN"].mean())
TNNormDeaths_median = round(StNormDeaths_week["TN"].median())
TNNormDeaths_mode = StNormDeaths_week["TN"].mode()
TNNormDeaths0_mode = StNormDeaths_week["TN"].replace(0, np.nan).mode()

# TNNormDeaths_mean = round(StNormDeaths_week["TN"].mean())
# TNNormDeaths_median = round(StNormDeaths_week["TN"].median())
# TNNormDeaths_mode = StNormDeaths_week["TN"].mode()
# TNNormDeaths0_mode = StNormDeaths_week["TN"].replace(0, np.nan).mode()

Cases_statistics =[[AZNormDeaths_mean, INNormDeaths_mean, MANormDeaths_mean, MONormDeaths_mean, NJNormDeaths_mean, TNNormDeaths_mean],
             [AZNormDeaths_median,INNormDeaths_median, MANormDeaths_median, MONormDeaths_median, NJNormDeaths_median, TNNormDeaths_median],
             [AZNormDeaths_mode[0],INNormDeaths_mode[0],MANormDeaths_mode[0],MONormDeaths_mode[0],NJNormDeaths_mode[0],TNNormDeaths_mode[0]]]


#Create a dataframe from list

Cases_sta = pd.DataFrame(Cases_statistics, columns = ['AZ', 'IN', 'MA','MO', 'NJ', 'TN'], 
                         index=['Mean', 'Median', 'Mode'])  
Cases_sta<jupyter_output><empty_output><jupyter_text>NJ has the highest mean of new deaths and the difference between its median and its mean is remarkable compare to other states which shows the peak is close to the left side of the graph. IN, TN and MO have the lowest Mean but the mean, median and mode of IN are equal which demonestrate that  the distribution of new cases in IN as approximately symmetrical. For some period of time mode = 0 indicates that the state was successful to minimize the number of deathes for a greater number of weeks, however it was temporery.  Identify counties within the previous state with high case and death rates.<jupyter_code>#select counties for MA

MACounties = covid_df.loc[covid_df['State'] == "MA"]
MACounties.head()
#add a new column to MA dataset which shows total confirmed cases for each county
MACounties['Total Cases']= MACounties["10/19/20_x"]

#add a new column to MA dataset which shows total death for each county
MACounties['Total Death']= MACounties["10/19/20_y"]
MACounties.head()
# Highest rate of cases in MA counties

highest_rateCases_MA = MACounties[['County Name', "population", "Total Cases"]] 
highest_rateCases_MA["Rate of Cases"] = ((highest_rateCases_MA["Total Cases"]/highest_rateCases_MA["population"])*100000).round()
highest_rateCases_MA = highest_rateCases_MA.sort_values(['Rate of Cases'], ascending = False) 
highest_rateCases_MA.head()
# Highest rate of death in MA counties without normalizing

highest_rateDeath_MA = MACounties[['County Name', "population", "Total Death"]] 
highest_rateDeath_MA["Rate of Death"] = ((highest_rateDeath_MA["Total Death"]/highest_rateDeath_MA["population"])*100000).round()
highest_rateDeath_MA = highest_rateDeath_MA.sort_values(['Rate of Death'], ascending = False) 
highest_rateDeath_MA.head()<jupyter_output>C:\Users\sanam\Anaconda3\lib\site-packages\ipykernel_launcher.py:4: SettingWithCopyWarning:


A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy<jupyter_text>Plot daily trends (cases and deaths, new cases) of state and top 5 infected counties. Utilize aggregrate, normalized by population, and log normalized values.  New Cases<jupyter_code>stateMAST= covid_dfSt.transpose()
MA = stateMAST[19]
MA_NewCases = MA.filter(regex = "x").to_frame()
MA_NewDeaths = MA.filter(regex = "y").to_frame()
#Select counties with highest rate of cases 
MACounties_caseRate = MACounties.loc[(MACounties['County Name']=="Suffolk County") | (MACounties['County Name']=="Essex County")
                             | (MACounties['County Name']=="Plymouth County")| (MACounties['County Name']=="Bristol County")
                             | (MACounties['County Name']=="Middlesex County")]
MACounties_caseRate.drop(columns = "County Name_y", inplace=True)
#Select number of Cases
CountiesCases = MACounties_caseRate.filter(regex = "x")
CountiesCases = CountiesCases.transpose()

#Rename columns
CountiesCases.columns = ['Bristol County', 'Essex County','Middlesex County','Plymouth County'
                         ,'Suffolk County']
CountiesCases.head()
# Calculate new Cases for highest rated counties

CountiesNewCases = CountiesCases.diff()
MA_NewCases = MA_NewCases.diff()
#Concatenate two MA dataframe with counties dataframe

dailyTrendsCases = pd.concat([MA_NewCases, CountiesNewCases], axis=1)
dailyTrendsCases.head()<jupyter_output><empty_output><jupyter_text>Normalizing Data<jupyter_code># Normalize new cases

dailyTrendsCases["MA Cases(Norm)"] = (((dailyTrendsCases[19]/6892503)*300000).astype(float)).round()
dailyTrendsCases["Bristol Cases(Norm)"] = ((dailyTrendsCases["Bristol County"]/565217)*300000).round()
dailyTrendsCases["Essex Cases(Norm)"] = ((dailyTrendsCases["Essex County"]/789034)*300000).round()
dailyTrendsCases["Middlesex Cases(Norm)"] = ((dailyTrendsCases["Middlesex County"]/1611699)*300000).round()
dailyTrendsCases["Plymouth Cases(Norm)"] = ((dailyTrendsCases["Plymouth County"]/521202)*300000).round()
dailyTrendsCases["Suffolk Cases(Norm)"] = ((dailyTrendsCases["Suffolk County"]/803907)*300000).round()


#Drop unnecessary columns 
dailyTrendsCases.drop(columns =[19,"Bristol County", "Essex County", "Middlesex County", "Plymouth County",
                               "Suffolk County"], inplace = True)
# dailyTrendsCases<jupyter_output><empty_output><jupyter_text>Plot Daily Trends ( New Cases)<jupyter_code>#Plotting Daily trends 

pic1 = go.Figure()

trace0 = go.Scatter(x=dailyTrendsCases.index, y=dailyTrendsCases["MA Cases(Norm)"], 
                    name="MA", 
                    mode="lines+markers") 

trace1 = go.Scatter(x=dailyTrendsCases.index, y=dailyTrendsCases["Bristol Cases(Norm)"], 
                    name="Bristol", 
                    mode="lines+markers")  

trace2 = go.Scatter(x=dailyTrendsCases.index, y=dailyTrendsCases["Essex Cases(Norm)"], 
                    name="Essex", 
                    mode="lines+markers") 

trace3 = go.Scatter(x=dailyTrendsCases.index, y=dailyTrendsCases["Middlesex Cases(Norm)"], 
                    name="Middlesex", 
                    mode="lines+markers") 

trace4 = go.Scatter(x=dailyTrendsCases.index, y=dailyTrendsCases["Plymouth Cases(Norm)"], 
                    name="Plymouth", 
                    mode="lines+markers") 

trace5 = go.Scatter(x=dailyTrendsCases.index, y=dailyTrendsCases["Suffolk Cases(Norm)"], 
                    name="Suffolk", 
                    mode="lines+markers") 


mydata = go.Data([trace0, trace1,trace2, trace3,trace4,trace5])

mylayout = go.Layout(
    title=""
)

fig = go.Figure(data=mydata, layout=mylayout)

plotly.offline.iplot(fig, filename = '')

# fig1 = go.Figure(data=mydata, layout=mylayout)
pic1.write_image("Graphs/State.NewCases.DailyTrends.png")<jupyter_output>C:\Users\sanam\Anaconda3\lib\site-packages\plotly\graph_objs\_deprecations.py:40: DeprecationWarning:

plotly.graph_objs.Data is deprecated.
Please replace it with a list or tuple of instances of the following types
  - plotly.graph_objs.Scatter
  - plotly.graph_objs.Bar
  - plotly.graph_objs.Area
  - plotly.graph_objs.Histogram
  - etc.<jupyter_text>New Deaths<jupyter_code>#Select counties with highest rate of cases 
MACounties_deathsRate = MACounties.loc[(MACounties['County Name']=="Essex County") | (MACounties['County Name']=="Hampden County")
                             | (MACounties['County Name']=="Norfolk County")| (MACounties['County Name']=="Plymouth County")
                             | (MACounties['County Name']=="Suffolk County")]
MACounties_deathsRate.drop(columns = "County Name_y", inplace=True)
#Select number of Deaths
CountiesDeaths = MACounties_deathsRate.filter(regex = "_y")
CountiesDeaths = CountiesDeaths.transpose()

#Rename columns
CountiesDeaths.columns = ['Essex County', 'Hampden County','Norfolk County','Plymouth County'
                         ,'Suffolk County']
# Calculate new Deaths for highest rated counties

CountiesNewDeaths = CountiesDeaths.diff()
MA_NewDeaths = MA_NewDeaths.diff()
#Concatenate two MA dataframe with counties dataframe

dailyTrendsDeaths = pd.concat([MA_NewDeaths, CountiesNewDeaths], axis=1)
dailyTrendsDeaths.head()<jupyter_output><empty_output><jupyter_text>Normalizing Data<jupyter_code># Normalize new deaths

dailyTrendsDeaths["MA Deaths(Norm)"] = (((dailyTrendsDeaths[19]/6892503)*300000).astype(float)).round()
dailyTrendsDeaths["Essex Cases(Norm)"] = ((dailyTrendsDeaths["Essex County"]/789034)*300000).round()
dailyTrendsDeaths["Hampden Cases(Norm)"] = ((dailyTrendsDeaths["Hampden County"]/466372)*300000).round()
dailyTrendsDeaths["Norfolk Cases(Norm)"] = ((dailyTrendsDeaths["Norfolk County"]/706775)*300000).round()
dailyTrendsDeaths["Plymouth Cases(Norm)"] = ((dailyTrendsDeaths["Plymouth County"]/521202)*300000).round()
dailyTrendsDeaths["Suffolk Cases(Norm)"] = ((dailyTrendsDeaths["Suffolk County"]/803907)*300000).round()

#Drop unnecessary columns 
dailyTrendsDeaths.drop(columns =[19, "Essex County", "Hampden County", "Norfolk County", "Plymouth County",
                               "Suffolk County"], inplace = True)
dailyTrendsDeaths.head()<jupyter_output><empty_output><jupyter_text>Plot Daily Trends (New Deaths)<jupyter_code>#Plotting Daily trends 

trace0 = go.Scatter(x=dailyTrendsDeaths.index, y=dailyTrendsDeaths["MA Deaths(Norm)"], 
                    name="MA", 
                    mode="lines+markers") 

trace1 = go.Scatter(x=dailyTrendsDeaths.index, y=dailyTrendsDeaths["Essex Cases(Norm)"], 
                    name="Essex", 
                    mode="lines+markers")  

trace2 = go.Scatter(x=dailyTrendsDeaths.index, y=dailyTrendsDeaths["Hampden Cases(Norm)"], 
                    name="Hampden", 
                    mode="lines+markers") 

trace3 = go.Scatter(x=dailyTrendsDeaths.index, y=dailyTrendsDeaths["Norfolk Cases(Norm)"], 
                    name="Norfolk", 
                    mode="lines+markers") 

trace4 = go.Scatter(x=dailyTrendsDeaths.index, y=dailyTrendsDeaths["Plymouth Cases(Norm)"], 
                    name="Plymouth", 
                    mode="lines+markers") 

trace5 = go.Scatter(x=dailyTrendsDeaths.index, y=dailyTrendsDeaths["Suffolk Cases(Norm)"], 
                    name="Suffolk", 
                    mode="lines+markers") 


mydata = go.Data([trace0, trace1,trace2, trace3,trace4,trace5])

mylayout = go.Layout(
    title=""
)

fig = go.Figure(data=mydata, layout=mylayout)

plotly.offline.iplot(fig, filename = '')<jupyter_output><empty_output><jupyter_text>Task 2  Fit a distribution to the number of COVID-19 cases of a state.<jupyter_code>from decimal import *
from IPython.display import display
from plotly.graph_objs import Bar, Scatter, Marker, Layout 
from plotly.offline import iplot
import numpy as np
from scipy.misc import factorial
%matplotlib inline
from matplotlib import pyplot as plt<jupyter_output><empty_output><jupyter_text>First of all we should draw the histogram of our data<jupyter_code># Histogram of the MA Cases

STstate = StNewCases.transpose()
MA = STstate[19].fillna(0)
MA.diff()
MA.hist(bins = 15, ec = "black")
#Histogram of probability

MA.hist(density = True, ec='white', figsize=(9,5))<jupyter_output><empty_output><jupyter_text>Histogram of the data shows that the data is right skewd. Also, we know data is discrete, then the poisson distribution can fit our data.<jupyter_code>#Calculate pmf
pmf = stats.poisson.pmf(k=MA, mu=MA.mean()).round(4)

plt.bar(MA, pmf)

plt.xlim(MA.mean()-50,MA.mean()+50)
plt.ylim(0, 0.02)
MA.describe()<jupyter_output><empty_output><jupyter_text>Model a poission distribution of COVID-19 cases and deaths of a state and compare to other 5 states.  Normalizing Data by per 1000000 people<jupyter_code># StNormCases_week
# StNormDeaths_week
#Calculate mean of cases for each state 

STCasesMean = StNormCases_week.mean().round().astype(int)
STCasesMean
#Calculate mean of deaths for each state 

STDeathsMean = StNormDeaths_week.mean().round()
STDeathsMean
#calculate different K for different states for cases

K = [50,60,70,80,90,100,120,130]
AZ = []
IN = []
MA = []
MO = []
NJ = []
TN = []

# append calculated k to each corresponding list
for num in K:
    PMF = stats.poisson.pmf(k=num, mu=STCasesMean["AZ"]).round(4)
    AZ.append(PMF)
    
for num in K:
    PMF = stats.poisson.pmf(k=num, mu=STCasesMean["IN"]).round(4)
    IN.append(PMF)

for num in K:
    PMF = stats.poisson.pmf(k=num, mu=STCasesMean["MA"]).round(4)
    MA.append(PMF)
    
for num in K:
    PMF = stats.poisson.pmf(k=num, mu=STCasesMean["MO"]).round(4)
    MO.append(PMF)

for num in K:
    PMF = stats.poisson.pmf(k=num, mu=STCasesMean["NJ"]).round(4)
    NJ.append(PMF)
    
for num in K:
    PMF = stats.poisson.pmf(k=num, mu=STCasesMean["TN"]).round(4)
    TN.append(PMF)
    
#create a dataframe from lists
dfCases = pd.DataFrame(list(zip(K, AZ, IN, MA, MO, NJ, TN)), columns =['k','AZ', 'IN', 'MA', 'MO', 'NJ', 'TN'])
dfCases
#calculate different K for different states for cases

KDeaths = [0,1,2,3,4,5,6,7]
AZdeaths = []
INdeaths = []
MAdeaths = []
MOdeaths = []
NJdeaths = []
TNdeaths = []

# append calculated k to each corresponding list
for num in KDeaths:
    PMF = stats.poisson.pmf(k=num, mu=STDeathsMean["AZ"]).round(4)
    AZdeaths.append(PMF)
    
for num in KDeaths:
    PMF = stats.poisson.pmf(k=num, mu=STDeathsMean["IN"]).round(4)
    INdeaths.append(PMF)

for num in KDeaths:
    PMF = stats.poisson.pmf(k=num, mu=STDeathsMean["MA"]).round(4)
    MAdeaths.append(PMF)
    
for num in KDeaths:
    PMF = stats.poisson.pmf(k=num, mu=STDeathsMean["MO"]).round(4)
    MOdeaths.append(PMF)

for num in KDeaths:
    PMF = stats.poisson.pmf(k=num, mu=STDeathsMean["NJ"]).round(4)
    NJdeaths.append(PMF)
    
for num in KDeaths:
    PMF = stats.poisson.pmf(k=num, mu=STDeathsMean["TN"]).round(4)
    TNdeaths.append(PMF)
    
#create a dataframe from lists
dfDeaths = pd.DataFrame(list(zip(KDeaths, AZdeaths, INdeaths, MAdeaths, MOdeaths, NJdeaths, TNdeaths)),
                  columns =['k','AZ', 'IN', 'MA', 'MO', 'NJ', 'TN'])
dfDeaths<jupyter_output><empty_output><jupyter_text>Model poission distributions for North Carolina counties COVID-19 in cases and deaths.<jupyter_code>#Select rows which are for NC

covid_dfNC = covid_df.loc[covid_df['State'] == "NC"]
covid_dfNC = covid_dfNC.drop(columns = "County Name_y")
covid_dfNC.head()
#Seperate covid_dfNC to cases number and deaths number

NCcases = covid_dfNC.filter(regex = "x")
NCdeaths = covid_dfNC.filter(regex = "_y")

NCcases.head(2)
#Calculae new cases and new deaths

NCNewCases = NCcases.diff(axis = 1)
NCNewDeaths = NCdeaths.diff(axis = 1)
# Normalize new cases

NCNormCases = NCNewCases.transpose()
NCCounties = covid_dfNC["County Name"].tolist()
NCNormCases.columns = NCCounties
CountiesPopulation = covid_dfNC["population"].tolist()
NCNormCases = NCNormCases.rename(columns={col: col.split(' ')[0] for col in NCNormCases.columns})
counties = NCNormCases.columns.tolist()
ExColumns = (NCNormCases.columns).tolist()

for i in range(len(NCNormCases.columns)):
    NCNormCases[NCNormCases.columns[i]+" Cases(Norm)"]=((NCNormCases[NCNormCases.columns[i]]/CountiesPopulation[i])*30000).round()

NCNormCases.drop(columns =ExColumns, inplace = True)


# Normalize new cases

NCNormDeaths = NCNewDeaths.transpose()
# # NCCounties = covid_dfNC["County Name"].tolist()
NCNormDeaths.columns = NCCounties
# # CountiesPopulation = covid_dfNC["population"].tolist()
NCNormDeaths = NCNormDeaths.rename(columns={col: col.split(' ')[0] for col in NCNormDeaths.columns})
ExColumns = (NCNormDeaths.columns).tolist()

for i in range(len(NCNormDeaths.columns)):
    NCNormDeaths[NCNormDeaths.columns[i]+" Deaths(Norm)"]=((NCNormDeaths[NCNormDeaths.columns[i]]/CountiesPopulation[i])*30000).round()

NCNormDeaths.drop(columns =ExColumns, inplace = True)
#Calculate weekly mean for Normalized New Cases

NCNormCases_week = NCNormCases.groupby(np.arange(len(NCNormCases))//7).mean().round(0).astype(int).rename_axis('Week')

NCNormCases_week = NCNormCases_week.reset_index()
#Calculate weekly mean for Normalized New Deaths

NCNormDeaths_week = NCNormDeaths.groupby(np.arange(len(NCNormDeaths))//7).mean().round(0).astype(int).rename_axis('Week')

NCNormDeaths_week = NCNormDeaths_week.reset_index()
# Calculate mean of cases for each county
NCNormCases_mean = NCNormCases_week.mean().round()

#Calculate mean of deaths for each county 
NCNormDeaths_mean = NCNormDeaths_week.mean().round()
#model poisoon for different K for different states for cases

K = [0,1,2,3,4]
NC = []


# append calculated k to each corresponding list
for county in counties:
    countyK = []
    for num in K:
        PMF = stats.poisson.pmf(k=num, mu=NCNormCases_mean[county +" Cases(Norm)"]).round(4)
        countyK.append(PMF)
    NC.append(countyK)
          
# NC

# #create a dataframe from lists

NCCases = pd.DataFrame(NC, columns = ['K=0', 'K=1', 'K=2','K=3','K=4']) 
NCCases["County Name"] = counties
NCCases.set_index("County Name", inplace = True)
NCCases.head()
#model poisoon for different K for different states for deaths

K = [0,1,2,3,4]
NCD = []


# append calculated k to each corresponding list
for county in counties:
    countyK = []
    for num in K:
        PMF = stats.poisson.pmf(k=num, mu=NCNormDeaths_week[county +" Deaths(Norm)"]).round(4)
        countyK.append(PMF)
    NCD.append(countyK)
          
# NC

# #create a dataframe from lists

NCDeaths = pd.DataFrame(NC, columns = ['K=0', 'K=1', 'K=2','K=3','K=4']) 
NCDeaths["County Name"] = counties
NCDeaths.set_index("County Name", inplace = True)
NCDeaths.head()<jupyter_output><empty_output><jupyter_text>Perform corelation between Enrichment data valiables and COVID-19 cases to observe any patterns<jupyter_code># Read csv file decographic

demographic_df = pd.read_csv("../../../src/Stage_I/Sanam/Demographic.csv")
# Remove columns which shows margin of errors, percent estimate and percent margin of error 
demographic_df = demographic_df[demographic_df.columns.drop(list(demographic_df.filter(regex='Margin')))]
demographic_df = demographic_df[demographic_df.columns.drop(list(demographic_df.filter(regex='Percent')))]
demographic_df
#select county FIPS from is columns and append them in the list and create new column with the list 

county_FIPS_list = []
for i in range(len(demographic_df["id"])):
    county_FIPS_list.append(demographic_df.loc[i,'id'][9:])
    
demographic_df["countyFIPS"] = county_FIPS_list

# convert the type of countyFIPS from string to int
demographic_df['countyFIPS'] = demographic_df['countyFIPS'].astype(int)

# Split geographical area name into county and state name
demographic_df[['County Name','State']] = demographic_df["Geographic Area Name"].str.split(",",expand=True)

demographic_df.drop(columns = ['id', 'Geographic Area Name'], inplace = True)
# Get the DataFrame column names as a list
clist = list(demographic_df.columns)
# Rearrange list the way you like 
clist_new = clist[-1:]+clist[:-1] 
# Pass the new list to the DataFrame - like a key list in a dict 
demographic_df = demographic_df[clist_new]

clist = list(demographic_df.columns)
clist_new = clist[-1:]+clist[:-1] 
demographic_df = demographic_df[clist_new]

clist = list(demographic_df.columns)
clist_new = clist[-1:]+clist[:-1] 

demographic_df.head()
#Add new columns of total cases and deaths to the dataframe 
covid_df['Total Cases']= covid_df["10/26/20_x"]
# covid_df['Total Deaths']= covid_df["9/9/20_y"]

#Select necessary columns
covid_dfTotal = covid_df[["countyFIPS","Total Cases"]]
# merge demographic df with modified_confirmed_death

CasesDemographic = pd.merge(covid_dfTotal,demographic_df, on = ["countyFIPS"])
CasesDemographic.head()

#groupby state

CasesDemographic = CasesDemographic.groupby(['State']).sum()
CasesDemographic = CasesDemographic.drop(["countyFIPS"],axis =1)
CasesDemographic.reset_index(inplace=True)
columnsList = CasesDemographic.columns
CasesDemographic
# CasesDemographic.iloc[1]
List = [2, 14, 20, 21, 25, 42]

CasesDemographicST = CasesDemographic.ix[List]
CasesDemographicST
CasesDemographicST
Arizona = CasesDemographicST.ix[2][2:].apply(lambda x: (x/7278717)*1000000)
Indiana = CasesDemographicST.ix[14][2:].apply(lambda x: (x/6732219)*1000000)
Maryland = CasesDemographicST.ix[20][2:].apply(lambda x: (x/6892503)*1000000)
Massachusetts = CasesDemographicST.ix[21][2:].apply(lambda x: (x/6045680)*1000000)
Missouri = CasesDemographicST.ix[25][2:].apply(lambda x: (x/6137428)*1000000)
Tennessee = CasesDemographicST.ix[42][2:].apply(lambda x: (x/6829174)*1000000)
# Demo = CasesDemographic[CasesDemographic.columns[5:]].apply(lambda x: x + 5, axis=1)
AZdf = pd.DataFrame([Arizona])
INdf = pd.DataFrame([Indiana])
MDdf = pd.DataFrame([Maryland])
MAdf = pd.DataFrame([Massachusetts])
MOdf = pd.DataFrame([Missouri])
TNdf = pd.DataFrame([Tennessee])
frames = [AZdf, INdf, MDdf,MAdf, MOdf, TNdf]

df = pd.concat(frames)
#Total cases of each state after normalizing

df["Total Cases"] = [111,293, 185, 175, 277,315]
df
# calculate correlation for each column with total Confirmed cases and then append it to a list

columns =df.columns
columns_modified = columns[8:-2]


correlation_confirmed = []
for i in range(len(columns_modified)-1):
    corr = df["Total Cases"].corr(df[columns_modified[i]])
    correlation_confirmed.append([columns_modified[i], corr])
correlation_confirmed = sorted(correlation_confirmed, key=lambda x: -x[1])
correlation_confirmed<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13470406712665997, 'fraction_numerical': 0.02690614056378073, 'mean_word_length': 5.664048059149723, 'pattern_counts': {'":': 0, '<': 71, '<?xml version=': 0, '>': 71, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '364260', 'license': 'MIT', 'n_tokens_mistral': 11167, 'n_tokens_neox': 10013, 'n_words': 2103, 'path': 'src/Stage_II/Sanam/.ipynb_checkpoints/Stage2-checkpoint.ipynb', 'repo_name': 'HarinB4/Covid-19DataProject', 'size': '28841'}