<jupyter_start><jupyter_text>**This Exercise shows how to use Google colab to apply regression models.****Copy the Colab notebook and data to your google drive before proceeding.** Step 1: Connect Google drive that stores the data with your Google Colab Notebook<jupyter_code>!pip install plotnine
!pip install -U -q PyDrive
import numpy as np
import numpy.matlib as matlib
import pandas as pd
import matplotlib as mpl
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)<jupyter_output><empty_output><jupyter_text>Replace the ID with ID of file you want to access. To get this ID: 1.   Right-click on the name of the appropriate file in your Google Drive2.   Click "Get link"3.   Copy the characters between `d/` and the following slash4.   Paste them into the dictionary argument of the call to `drive.CreateFile()` as the value associated with the 'id' key. Make sure that this value is a string.<jupyter_code># Replace the ID with ID of file you want to access
# To get this ID, right-click on the name of the appropriate file in your 
downloaded = drive.CreateFile({'id':'1Z6MYK_-FV7J0aTVxoPnjI-E3DdbwMDHG'}) 
downloaded.GetContentFile('time_series_covid_19_confirmed.csv')<jupyter_output><empty_output><jupyter_text>Step 2: Read the Data and verify its format!<jupyter_code>data = pd.read_csv('time_series_covid_19_confirmed.csv') 
print("The shape of data=",data.shape)
data.head()<jupyter_output>The shape of data= (266, 225)<jupyter_text>Step 3: Original data has [266x225] dimensions. Let's use only 50 rows and 100 columns.<jupyter_code># drop the first column
temp = data[list(data.columns)[1:]]
# push the Country, Lat & Long into index
temp.set_index(list(temp.columns)[0:3], inplace=True)
# change columns to integers
temp.columns = range(len(temp.columns))
# stack and make a dataframe
df = pd.DataFrame(temp.stack())
# rename value column
df.columns = ["val"]
# reset index to bring index values 
df.reset_index(inplace=True)
# rename column to day
df.rename(columns={"level_3":"day"}, inplace=True)
df.head()
# trying out using one-hot encoding
if False:
  from sklearn.preprocessing import LabelEncoder
  from sklearn.preprocessing import OneHotEncoder
  # creating instance of one-hot-encoder
  enc = OneHotEncoder(handle_unknown='ignore')

  # passing bridge-types-cat column (label encoded values of bridge_types)
  enc_df = pd.DataFrame(enc.fit_transform(df[['Country/Region']]).toarray())
  enc_df.columns = ["cntry_" + str(c) for c in list(enc_df.columns)]
  enc_df.head()
  edf = df.join(enc_df)
  edfhead()
xvars = ["Lat", "Long", "day"] 
yvar = "val"
X = df[xvars]
Y = df[yvar]
# Goal 1: Fit regression model for (days vs infections)
from sklearn.model_selection import train_test_split
# Split data into train and test
X_train_1, X_test_1, y_train, y_test = train_test_split(X, Y, test_size=0.3, random_state=10)
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
mse_list = []
for degree in range(1,18):
  poly = PolynomialFeatures(degree=degree)
  # Transform both the train and test data
  X_train = poly.fit_transform(X_train_1)
  X_test = poly.fit_transform(X_test_1)
  print("degree={0},  test data shape={1}".format(degree,np.shape(X_test)))
  clf = LinearRegression()
  # Fit the model on train data only
  clf.fit(X_train, y_train)
  #print("Linear Regression Coefficients are=",clf.coef_)
  #print("constant coefficient=",clf.intercept_)
  # Predict regression outcome on test data
  train_pred = clf.predict(X_train) 
  test_pred = clf.predict(X_test)
  from sklearn.metrics import mean_squared_error as mse
  test_mse = mse(y_test, test_pred)
  train_mse = mse(y_train, train_pred)
  mse_vals = {"test": test_mse, "train":train_mse}
  mse_list.append(mse_vals)
mse_df = pd.DataFrame(mse_list)
mse_df.plot(xlabel="degree", ylabel="mse")<jupyter_output><empty_output><jupyter_text>Note that it Seems to make little sense to use Lat and Long as precictor variables

With that said, from the plot above of MSE vs. degree it seems that degree of somewhere between 10 and 12 is best.

It is odd that the test error is lower than the training error.<jupyter_code># I don't really understand what this step is doing
if False:
  # 1D polynomial regression
  model = np.poly1d(np.polyfit(np.transpose(X_train_1[:,2]), np.transpose(y_train[:,0]), 3))
  model_x = np.linspace(0, 100, 100)
  plt.figure(figsize=(20, 20), dpi=80)
  plt.scatter(X[:,2],Y[:])
  plt.plot(model_x, model(model_x),'r')
  plt.show()
#Assignment: Report the mean squared error for prediction using differnt degrers of Polynomial Regression
#https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html
#1. Report the best model for this data

#2. Repeat for complete data (Change step 3) and report the best model<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09962883375659308, 'fraction_numerical': 0.015432701699550693, 'mean_word_length': 5.206060606060606, 'pattern_counts': {'":': 3, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '187312', 'license': 'MIT', 'n_tokens_mistral': 1772, 'n_tokens_neox': 1685, 'n_words': 530, 'path': 'Week0_Regression_assignment_inclass.ipynb', 'repo_name': 'jhmuller/FourthBrain', 'size': '5119'}