<jupyter_start><jupyter_text><jupyter_code>#Data
import requests
import csv

#EDA and Visualizations
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns

#modeling
from nltk.tokenize import TweetTokenizer, word_tokenize
from nltk.corpus import stopwords
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.metrics import precision_score, accuracy_score, recall_score, f1_score, confusion_matrix, mean_squared_error, roc_curve, auc, roc_auc_score
from sklearn.naive_bayes import GaussianNB, BernoulliNB, MultinomialNB
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier, AdaBoostClassifier
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.feature_selection import chi2
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

import psutil

import datetime<jupyter_output><empty_output><jupyter_text>Business UnderstandingData used in this project came from Kaggle, the CDC, Google Trends, and Twitter. Using basic NLP libraries (NLTK, TweetTokenizer)For the analysis of the tweets I started with a basic, mostly cleaned, dataset off of Kaggle to create text based predictors. I also gathered my own twitter data about vaccinations, which accounted for another 10000 rows. I used other libraries to clean the text (TextHero), and then used a Trigram CountVectorizer. To create my target variable I used data from Google Trends and the CDC. Using percentages of new cases and trending relevance, I created a score between the two that accounts for not only physical outbreaks, but a social one as well.Covid-19 is a problem that does not need much introduction or explanantion, however, for this project I wanted to find if there was any correlation between twitter behavior and possible outbreaks. This is important to understand because this could help hospitals better prepare for a response to an outbreak. This could also be applicable to anybody wanting to avoid close contact with the disease, giving them warning beforehand, which they can then use to make more informed decisions on how to handle their day to day.  Data Preprocessing and Understanding  Google Trends Data<jupyter_code>google = pd.read_csv('data/covid_searches.csv')
google
google.reset_index(inplace=True)

google.info()
google.drop(index=google.index[0], 
        axis=0, 
        inplace=True)
google['date'] = google['index']
google['relevance'] = google['Category: All categories']

google.drop(columns='index', inplace=True)
google.drop(columns='Category: All categories', inplace=True)

google
google['date'] = pd.to_datetime(google['date'])

google['relevance'] = google['relevance'].astype(int)

google['outbreak'] = [x for x in google['relevance'] > 50]<jupyter_output><empty_output><jupyter_text>CDC Data<jupyter_code>CDC = pd.read_csv('data/USA_covid_stats.csv')
CDC
CDC['date'] = CDC['submission_date']

CDC.drop(columns='submission_date', axis=0, inplace=True)


col = ['state', 'conf_cases', 'prob_cases', 'pnew_case', 'conf_death', 'prob_death', 'pnew_death', 'created_at', 'consent_cases', 'consent_deaths']

CDC.drop(columns=col, inplace=True)
CDC.dropna(how='all', inplace=True)

CDC['date'] = pd.to_datetime(CDC['date']).dt.normalize()
CDC<jupyter_output><empty_output><jupyter_text>Twitter Data<jupyter_code>tweets = pd.read_csv('data/covid_tweets.csv')
tweets.head()
column = ['user_name','user_location','user_description','user_created','user_followers','hashtags', 'user_favourites','user_verified','is_retweet', 'source', 'user_friends']

tweets.drop(columns=column, inplace=True)<jupyter_output><empty_output><jupyter_text>> Dropped irrelevant columns because the only data I want from the twitter datasets is text data and the date so I can join them by the same week number later on.<jupyter_code>tweets['date'] = pd.DatetimeIndex(tweets['date']).normalize()
tweets = tweets.groupby(['text', pd.Grouper(key='date', freq='W-SUN')]).sum().reset_index().sort_values('date')
#!pip install texthero<jupyter_output><empty_output><jupyter_text>> Texthero is a python package to work with text data efficiently.It empowers NLP developers with a tool to quickly understand any text-based dataset andit provides a solid pipeline to clean and represent text data, from zero to hero.> Texthero was found doing self research, it helps quickly and effectively clean up text data so you can perform vectorization on it.<jupyter_code>import texthero as hero

tweets['text'] = hero.clean(tweets['text'])
vax_tw = pd.read_csv('data/vaccination_tweets.csv')
vax_tw.info()
cols = ['id','user_name','user_location','user_description','user_created','user_followers','user_favourites',
'user_verified','hashtags','source','retweets','favorites','is_retweet', 'user_friends']

vax_tw.drop(columns=cols, inplace=True)

vax_tw['date'] = pd.DatetimeIndex(vax_tw['date']).normalize()

vax_tw = vax_tw.groupby(['text', pd.Grouper(key='date', freq='W-SUN')]).sum().reset_index().sort_values('date')
vax_tw['text'] = hero.clean(vax_tw['text'])

vax_tw.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 11014 entries, 7007 to 1927
Data columns (total 2 columns):
 #   Column  Non-Null Count  Dtype         
---  ------  --------------  -----         
 0   text    11014 non-null  object        
 1   date    11014 non-null  datetime64[ns]
dtypes: datetime64[ns](1), object(1)
memory usage: 258.1+ KB<jupyter_text>> Here I combined two different Twitter datasets. Both datasets are mostly about covid-19 and vaccinations. I pulled from relevant subjects because I believed that they would be the most correlated with predicting a pandemic. There are some normal tweets imbedded within that serves as an almost baseline.> To review, My data came from multiple sources, First I started with a base, almost cleaned, Kaggle dataset that had about 170k rows of tweets, for the sake of time and computational effectiveness, I trimmed down the dataset so I had more time for model tuning and other aspects of the project. I also gathered another 11k rows of tweets that were about vaccinations. Those two datasets were both used to create text based predictors using a CountVectorizer, to see if there was any direct correlation between the tweets and my target variable.>The target variable was constructed with both data from Google and the CDC. I created a score using the percentages of new cases as well as the relevance of searches. I then scaled the score down so outliers did not effect them as much. This accounts for not only physical outbreaks, but the social aspect as well.  Combining Dataframes<jupyter_code>tweet = tweets.sample(frac=0.5)

df = [tweet, google, vax_tw]

tash = tweet.merge(google, how='inner', on='date')
from functools import reduce

df_merged = reduce(lambda  left,right: pd.merge(left,right,on=['date'],
                                            how='outer'), df)
df_merged['text_x'].fillna(df_merged['text_y'], inplace=True)

df_merged.drop(columns='text_y', inplace=True)

df_merged.dropna(inplace=True)
df_merged.info()<jupyter_output><empty_output><jupyter_text>> After joining together all of my dataframes, I realized there was a major problem, the only join that was functional was an outer join. When I attempted an inner join on the date columns they were deleting the values within and leaving only column names. To combat this I did an outer join with my dataframes, but had to fraction my larger twitter dataframe due to computational effectiveness.  Functions<jupyter_code>def tokenize(tweet):
    tknzr = TweetTokenizer(strip_handles=True, reduce_len=True, 
                           preserve_case=False)
    return tknzr.tokenize(tweet)
def classifiers(x_train, y_train):
    nb  = MultinomialNB()
    nb.fit(x_train, y_train)
    log = LogisticRegression()
    log.fit(x_train, y_train)
    forest = RandomForestClassifier(n_estimators=100, max_depth=5)
    forest.fit(x_train, y_train)
    gradboost = GradientBoostingClassifier(random_state=123, max_depth=5, 
                                          learning_rate = 0.01)
    gradboost.fit(x_train, y_train)
    adaboost = AdaBoostClassifier(n_estimators=100)
    adaboost.fit(x_train, y_train)
    svm = SVC(kernel='linear', probability=True)
    svm.fit(x_train, y_train)
    return [nb, log, forest, gradboost, adaboost, svm]

def classifier_performance(vectorizer, train_data, test_data, y_test):
    accuracy_df = []
    x_train = vectorizer.fit_transform(train_data)
    x_test = vectorizer.transform(test_data)
    
    classifier_list = classifiers(x_train, y_train)
    for i in classifier_list:
        preds = i.predict(x_test)
        accuracy = accuracy_score(y_test, preds)
  
    accuracy_df = pd.DataFrame(accuracy_df)
    classifiers_key = ['Naive Bayes', 'Random Forest', 'Gradient Boost',
                      'AdaBoost', 'Support Vector Machine']
    accuracy_df['Model'] = classifiers_key
    accuracy_df.rename(columns={0: 'Accuracy'}, inplace=True)
    fin_accuracy_df = accuracy_df[['Model', 'Accuracy']]
    return fin_accuracy_df
def model_performance(model, vectorizer):
  
    train_data = vectorizer.fit_transform(x_train)
    test_data = vectorizer.transform(x_test)

    model.fit(train_data, y_train)

    pred = model.predict(test_data)

    return f'{model}', accuracy_score(y_test, pred)
def griddy(model, params, vectorizer):
  X_train = vectorizer.fit_transform(x_train)
  X_test = vectorizer.transform(x_test)

  grid = GridSearchCV(estimator=model, param_grid=params, n_jobs=-1, cv=6)
  grid.fit(X_train, y_train)

  return [grid.best_params_, grid.accuracy_score, grid.recall_score]<jupyter_output><empty_output><jupyter_text>Count Vectorization > In this next section I created baseline models to run through my data on different vectorizations. I used a Count Vectorizer and TFIDF Vectorizer with different n-grams to test the potential of the models.>After taking an iterative modeling approach, I found that one model performed exceptionally better than other models. Using the Multinomial Naive Bayes model I was able to predict outbreaks at a 40 percent accuracy. Part of the reason the Naive Bayes worked better than other models is because it assumes independence between my variables, meaning each of them is counted on its own without any weights, this pairs well with a CountVectorizer because it establishes importance based on word count.<jupyter_code>data = df_merged['text_x']
labels = df_merged['relevance']
x_train, x_test, y_train, y_test = train_test_split(data, labels)
tfidfvec = TfidfVectorizer(stop_words='english', tokenizer=tokenize)
tfidfvec2 = TfidfVectorizer(stop_words='english', tokenizer=tokenize, ngram_range=(1,2))
tfidfvec3 = TfidfVectorizer(stop_words='english', tokenizer=tokenize, ngram_range=(1,3))
countvec = CountVectorizer(stop_words='english', tokenizer=tokenize)
countvec2 = CountVectorizer(stop_words='english', tokenizer=tokenize, ngram_range=(1,2))
countvec3 = CountVectorizer(stop_words='english', tokenizer=tokenize, ngram_range=(1,3))
nb  = MultinomialNB()
model_performance(nb, countvec)
forest = RandomForestClassifier(n_estimators=100, max_depth=5)

model_performance(forest, countvec)
#gradboost = GradientBoostingClassifier(random_state=123, max_depth=5, learning_rate = 0.01)
#model_performance(gradboost, countvec)
#adaboost = AdaBoostClassifier(n_estimators=100)
#model_performance(adaboost, countvec)<jupyter_output><empty_output><jupyter_text>Bigram<jupyter_code>model_performance(nb, countvec2)
model_performance(forest, countvec2)
#model_performance(gradboost, countvec2)
#model_performance(adaboost, countvec2)<jupyter_output><empty_output><jupyter_text>Trigram<jupyter_code>model_performance(nb, countvec3)
model_performance(forest, countvec3)
#model_performance(gradboost, countvec3)
#model_performance(adaboost, countvec3)<jupyter_output><empty_output><jupyter_text>TF-IDF Vectorization  Unigram<jupyter_code>model_performance(nb, tfidfvec)
model_performance(forest, tfidfvec)
#model_performance(gradboost, tfidfvec)
#model_performance(adaboost, tfidfvec)<jupyter_output><empty_output><jupyter_text>Bigram<jupyter_code>model_performance(nb, tfidfvec2)
model_performance(forest, tfidfvec2)
#model_performance(gradboost, tfidfvec2)
#model_performance(adaboost, tfidfvec2)<jupyter_output><empty_output><jupyter_text>Trigram<jupyter_code>model_performance(nb, tfidfvec3)
model_performance(forest, tfidfvec3)
#model_performance(gradboost, tfidfvec3)
#model_performance(adaboost, tfidfvec3)<jupyter_output><empty_output><jupyter_text>Improving models> Trying to improve the best performing models from the previous cells, The two best performing were Random Forests and Naive Bayes, So running grid searches on them to see if there is any hyperparameter tuning that I could do to further improve the models.<jupyter_code>X_train = countvec3.fit_transform(x_train)
X_test = countvec3.transform(x_test)

param_grid = {'alpha': [1, 5, 10, 50, 100]}

clf = GridSearchCV(estimator=nb, param_grid=param_grid, n_jobs=-1, cv=6, scoring='accuracy')
clf.fit(X_train, y_train)


clf.best_params_
#params = {'criterion' : ['gini', 'entropy'],
          #'max_depth' : [3, 4, 5, 6, 7, 8],
          #'min_samples_split' : [2, 4, 6, 8, 10],
          #'min_samples_leaf' : [1, 2, 3, 4, 5],
          #'n_estimators' : [10, 25, 50, 100, 150, 200],
          #'bootstrap' : [True, False]}

#griddy(forest, params, countvec3)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09682176830617212, 'fraction_numerical': 0.012388466927217756, 'mean_word_length': 5.9584402257567985, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 72, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '579533', 'license': 'MIT', 'n_tokens_mistral': 4476, 'n_tokens_neox': 4187, 'n_words': 1236, 'path': 'FinalNotebook.ipynb', 'repo_name': 'ZenBurrito/Predicting-Covid-19-Outbreaks', 'size': '13561'}