<jupyter_start><jupyter_text>Team: Valiant Vaccinators Analyzing the Correlation between Covid Vaccinations and Infection/Death Rates  Importing Libraries<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import math
import seaborn as sns
from scipy import stats
from scipy.optimize import curve_fit
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsRegressor
from sklearn.preprocessing import StandardScaler
from sklearn import metrics<jupyter_output><empty_output><jupyter_text>Reading Dataset<jupyter_code>covid_national = pd.read_csv(r"./covidnational.csv")
covid_national.head()<jupyter_output><empty_output><jupyter_text>The dataset is from https://ourworldindata.org/explorers/coronavirus-data-explorer, and data for the United States was isolated. The table has 60 columns and over 400 rows. For this reason, it was important to clean the dataset and extract values that were relevant for the analysis.   Data Cleaning<jupyter_code>covid_national = covid_national.drop(["iso_code", "continent", "location", "weekly_icu_admissions", "weekly_icu_admissions_per_million", "weekly_hosp_admissions", "weekly_hosp_admissions_per_million", "tests_units", "excess_mortality", "handwashing_facilities", "population_density", "aged_65_older", "gdp_per_capita", "cardiovasc_death_rate", "diabetes_prevalence", "female_smokers", "male_smokers", "hospital_beds_per_thousand", "life_expectancy", "human_development_index", "stringency_index", "population", "median_age", "aged_70_older", "extreme_poverty"], axis = 1)
covid_national.head()<jupyter_output><empty_output><jupyter_text>After inspecting the data, columns with constant values or many blank values were dropped. Other columns that seemed irrelevant to the investigation were also dropped. The cleaned dataset was more condensed and only contained columns that would be relevant to the analysis.<jupyter_code>covid_national.shape<jupyter_output><empty_output><jupyter_text>The dataset now has 486 rows, with each row representing a day of data. The data spans from March 3rd, 2020 to July 3rd 2021, which is 486 days. This provides with a good amount of data to thoroughly and effectively analyze trends related to Covid-19.   Data Visualization<jupyter_code>plt.xlabel('Days Since March 3rd, 2020')
plt.ylabel('Total Cases')
plt.rcParams["figure.figsize"] = [10,5]
plt.title('Covid Cases in America over Time')
x_vals = np.linspace(0, 486, 486)
plt.plot(x_vals, covid_national["total_cases"], 'b')<jupyter_output><empty_output><jupyter_text>This graph shows how the number of cases have grown over the past year and a half since the start of the Covid-19 pandemic. As visualized, the cases started growing exponentionally, and then the rate of growth has slowly decreased in recent months. This is good news considering that if the trend continues, the rate of growth will eventually flatten.<jupyter_code>plt.xlabel('Days Since March 3rd, 2020')
plt.ylabel('Cases Per Day')
plt.rcParams["figure.figsize"] = [10,5]
plt.title('Rate of Infection in America over Time')
x_vals = np.linspace(0, 486, 486)
plt.plot(x_vals, covid_national["new_cases"], 'b')<jupyter_output><empty_output><jupyter_text>This visualization shows the rate of infection over time. The rate of infection grew exponentionally overall in the first 200 days, and eventually peaked after the 300th day. This point in the graph represents the most bleak point of the pandemic where the rate of new cases per day was at an all time high. Thankfully, this rate started to decreased past this peak. It is interesting to note that although the overall trend is a nice curve, there are significant fluctations when comparing the rates day by day.<jupyter_code>plt.xlabel('Days Since March 3rd, 2020')
plt.ylabel('Total Deaths')
plt.rcParams["figure.figsize"] = [10,5]
plt.title('Covid Deaths in America Over Time')
x_vals = np.linspace(0, 486, 486)
plt.plot(x_vals, covid_national["total_deaths"], 'b')<jupyter_output><empty_output><jupyter_text>Similar to the cases over time, the deaths over time in America have started flattening out in the later part of the graph. However, it is interesting to note that the graph changes from concave up to concave down multiple times.<jupyter_code>plt.xlabel('Days Since March 3rd, 2020')
plt.ylabel('Deaths Per Day')
plt.rcParams["figure.figsize"] = [10,5]
plt.title('Covid Death Rate in America Over Time')
x_vals = np.linspace(0, 486, 486)
plt.plot(x_vals, covid_national["new_deaths"], 'b')<jupyter_output><empty_output><jupyter_text>Interestingly, the death rate had a significant peak early on in the pandemic, around the 50 day mark after March 3rd, 2020. The next significant peak occurs at about the 350th day after March 3rd, 2020. There also seem to by many day-by-day fluctuations in the death rate.<jupyter_code>plt.xlabel('Days Since March 3rd, 2020')
plt.ylabel('Total Vaccinations (In Hundred Millions)')
plt.rcParams["figure.figsize"] = [10,5]
plt.title('Covid Vaccinations in America Over Time')
x_vals = np.linspace(0, 486, 486)
plt.plot(x_vals, covid_national["people_fully_vaccinated"], 'b')<jupyter_output><empty_output><jupyter_text>Compared to the previous graphs, this graph is extremely promising for the Covid-19 crisis in America since the number of vaccinations has grown greatly in the past 150 days. It is also significant to note that the graph is mostly exponential, but has become more linear in the last 50 days.<jupyter_code>sns.heatmap(covid_national.corr())<jupyter_output><empty_output><jupyter_text>This heatmap shows the correlation between two different columns. It was useful to determine which columns correlate the most with total vaccinations. Since the heatmap shows strong correleation between total vaccinations and new deaths/cases, we decided to visualize and prove the correlation between cases/deaths and total vaccinations.  Visualizing Correlation Between Vaccinations and Infection/Death Rates<jupyter_code>plt.ylabel('Cases')
plt.xlabel('Total Vaccinated')
plt.title('Cases vs People Vaccinated in America')
covid_national = covid_national.dropna()
plt.scatter(covid_national["people_fully_vaccinated"], covid_national["new_cases"], c = "b")
y = np.array(covid_national["new_cases"])
x = np.array(covid_national["people_fully_vaccinated"])
res = stats.linregress(x,y)
plt.plot(x, res.intercept + res.slope*x, 'r')
plt.legend(["Linear Regression", "Actual Values"])
plt.show()<jupyter_output><empty_output><jupyter_text>After performing linear regression on the data and plotting the line of best fit, it is evident that there is a significant relationship between the numbers of new cases per day and total vaccinations. As the number of vaccinations increases, the number of new cases per day tends to be lower. Therefore, this suggests that the number of new cases per day in America would continue to decrease as the number of people vaccinated increases.<jupyter_code>print(f"R-squared: {res.rvalue**2:.6f}")
print(f"R-value: {res.rvalue:.6f}")<jupyter_output>R-squared: 0.638530
R-value: -0.799081<jupyter_text>The R^2 value of .64 or goodness of fit proves that there is a strong correlation between the rate of cases and the number of people vaccinated. Although the scatter plot may seem to fit more to a curve or polynomial fit, it is important to note that many of the 0 vaccinated data points are from when the vaccinations were just starting to roll out. In addition, it likely took a few weeks for the vaccines to start showing an effect in the number of new cases per day.<jupyter_code>plt.ylabel('Deaths')
plt.xlabel('Total Vaccinated')
plt.rcParams["figure.figsize"] = [10,5]
plt.title('Deaths vs People Vaccinated in America')
covid_national = covid_national.dropna()
plt.scatter(covid_national["people_fully_vaccinated"], covid_national["new_deaths"], c = "b")
y = np.array(covid_national["new_deaths"])
x = np.array(covid_national["people_fully_vaccinated"])
res = stats.linregress(x,y)
plt.plot(x, res.intercept + res.slope*x, 'r')
plt.legend(["Linear Regression", "Actual Values"])<jupyter_output><empty_output><jupyter_text>The number of new deaths per day versus total vaccinations shows a similar trend as the previous graph. As visualized, the line of best fit shows correlation between the death rate and total vaccinations. The linear regression seems to not get too influenced by the outliers, however, that definitely seems to have affected the model.<jupyter_code>print(f"R-squared: {res.rvalue**2:.6f}")
print(f"R-value: {res.rvalue:.6f}")<jupyter_output>R-squared: 0.613742
R-value: -0.783417<jupyter_text>The goodness of fit/R^2 value is .61, which suggests that there is a strong correlation between the number of vaccinations and the death rate. This means that it is likely that the death rate will continue to decrease as more vaccinations are pushed out.  Training a Regression Model to Predict Future Deaths<jupyter_code>covid_national_temp = covid_national.dropna()
X = covid_national_temp.filter(['people_fully_vaccinated', 'positive_rate', 'new_cases'])
y = covid_national_temp['new_deaths']
X.head()<jupyter_output><empty_output><jupyter_text>N/A values are dropped in order to provide the regression model with legit values. In addition, the above columns were chosen as features since they seem most closely related or correlated with the number of new deaths per day. New death is the label since that is the value that will be predicted by the regression model.<jupyter_code>y.head()
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.20, random_state = 0)<jupyter_output><empty_output><jupyter_text>An 80-20 train test split is used to train the model.<jupyter_code>knn_reg = KNeighborsRegressor(n_neighbors=5)
regressor = knn_reg.fit(X_train, y_train)

y_pred = regressor.predict(X_test)

print('Mean Absolute Error:', metrics.mean_absolute_error(y_test, y_pred))
print('Mean Squared Error:', metrics.mean_squared_error(y_test, y_pred))
print('Root Mean Squared Error:', np.sqrt(metrics.mean_squared_error(y_test, y_pred)))<jupyter_output>Mean Absolute Error: 237.23636363636362
Mean Squared Error: 139826.37333333332
Root Mean Squared Error: 373.9336483031894<jupyter_text>The KNeighbors Regression technique was used to predict the death rate based on the features provided to the model. The model returned a mean absolute error of about 237. This means that the average error of the death rate predicted by the model is 237. This can be pretty significant for lower death rates, however, since a good portion of the death rates in the data set are above 2000, this is not too bad. However, the model can still be improved.  Improving the Model<jupyter_code>covid_national_temp.describe()
min_threshold, max_threshold = covid_national_temp.new_deaths.quantile([0.1, 0.8])
min_threshold, max_threshold
covid_national_normal = covid_national_temp[covid_national_temp.new_deaths < max_threshold]<jupyter_output><empty_output><jupyter_text>The two lines above serve to get rid of outliers that could be throwing off the model. Specifically, the higher death rates that could be throwing off the model.<jupyter_code>covid_national_normal.head()
X = covid_national_normal.filter(['people_fully_vaccinated', 'positive_rate', 'new_cases'])
y = covid_national_normal['new_deaths']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=0)

sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)
knn_reg = KNeighborsRegressor(n_neighbors=10)
regressor = knn_reg.fit(X_train, y_train)

y_pred = regressor.predict(X_test)

print('Mean Absolute Error:', metrics.mean_absolute_error(y_test, y_pred))
print('Mean Squared Error:', metrics.mean_squared_error(y_test, y_pred))
print('Root Mean Squared Error:', np.sqrt(metrics.mean_squared_error(y_test, y_pred)))<jupyter_output>Mean Absolute Error: 168.70000000000002
Mean Squared Error: 46746.052307692305
Root Mean Squared Error: 216.208353926698
{'fraction_non_alphanumeric': 0.08500541170593623, 'fraction_numerical': 0.025393389393056364, 'mean_word_length': 6.690140845070423, 'pattern_counts': {'":': 0, '<': 74, '<?xml version=': 0, '>': 73, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '208511', 'license': 'MIT', 'n_tokens_mistral': 4042, 'n_tokens_neox': 3598, 'n_words': 1298, 'path': 'Valiant Vaccinators.ipynb', 'repo_name': 'abhi-vinnakota/ValiantVaccinators', 'size': '12011'}