<jupyter_start><jupyter_text>Improved feature engineering*<NAME> - 11-02-2020* Ideas I'll be building on- seperating by city (data has different structure between the cities, avoids needing to build a more complex model that captures feature interactions)- using the lifecycle of the mosquito: new mostiquos become adults 1-3 weeks after eggs are laid in water. Therefore, we could expect a lot of cases if the previous 2-3week/~month was humid **Note**: I realized I used *median* absolute error instead of mean absolute error in the previous notebook, which explain why my CV scores were so far from the test set scores!<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns

import matplotlib.pyplot as plt
import sklearn
from os.path import join
DATA_PATH = '../data/raw/'<jupyter_output><empty_output><jupyter_text>Acquiring the data<jupyter_code>X_test_o = pd.read_csv(join(DATA_PATH, 'dengue_features_test.csv'))
X_train_o = pd.read_csv(join(DATA_PATH, 'dengue_features_train.csv'))
y_train_o = pd.read_csv(join(DATA_PATH, 'dengue_labels_train.csv'))<jupyter_output><empty_output><jupyter_text>Preprocessing<jupyter_code>X_train = pd.get_dummies(X_train_o, columns = ['city'], drop_first = True)
X_test = pd.get_dummies(X_test_o, columns = ['city'], drop_first = True)
X_train = X_train.drop('week_start_date', axis = 1)
X_test = X_test.drop('week_start_date', axis = 1)<jupyter_output><empty_output><jupyter_text>Drop features that have correlation 1 with other features<jupyter_code>X_train.drop(
    ['reanalysis_sat_precip_amt_mm', 'reanalysis_dew_point_temp_k',
     'reanalysis_tdtr_k'],
    axis = 1, 
    inplace = True
)
X_test.drop(
    ['reanalysis_sat_precip_amt_mm', 'reanalysis_dew_point_temp_k',
     'reanalysis_tdtr_k'],
    axis = 1, 
    inplace = True
)
y_train<jupyter_output><empty_output><jupyter_text>Precipitation at several time lags First, we split the data by city:<jupyter_code>X_train_sj = X_train[X_train['city_sj'] == 1]
X_train_iq = X_train[X_train['city_sj'] == 0]

X_test_sj = X_test[X_test['city_sj'] == 1]
X_test_iq = X_test[X_test['city_sj'] == 0]

y_train_sj = y_train[y_train['city'] == 'sj']['total_cases']
y_train_iq = y_train[y_train['city'] == 'iq']['total_cases']
def precip_n_weeks(k, n, precips):
    if k - n < 0:
        re turn .0
    else:
        return precips[k - n]    
train_precip_sj = X_train_sj['precipitation_amt_mm']
train_precip_iq = X_train_iq['precipitation_amt_mm']

test_precip_sj = X_test_sj['precipitation_amt_mm']
test_precip_iq = X_test_iq['precipitation_amt_mm']<jupyter_output><empty_output><jupyter_text>We re-index the series for Iquitos so that they start from 0 and our code can run properly<jupyter_code>iq_train_index = list(range(len(train_precip_iq)))
iq_test_index = list(range(len(test_precip_iq)))
train_precip_iq.index = iq_train_index
test_precip_iq.index = iq_test_index
X_train_sj['precip_2'] = [precip_n_weeks(k, 2, train_precip_sj)
                          for k in range(len(train_precip_sj))]
X_train_sj['precip_3'] = [precip_n_weeks(k, 3, train_precip_sj) 
                          for k in range(len(train_precip_sj))]
X_train_sj['precip_4'] = [precip_n_weeks(k, 4, train_precip_sj)
                          for k in range(len(train_precip_sj))]

X_test_sj['precip_2'] = [precip_n_weeks(k, 2, test_precip_sj)
                          for k in range(len(test_precip_sj))]
X_test_sj['precip_3'] = [precip_n_weeks(k, 3, test_precip_sj) 
                          for k in range(len(test_precip_sj))]
X_test_sj['precip_4'] = [precip_n_weeks(k, 4, test_precip_sj)
                          for k in range(len(test_precip_sj))]

X_train_iq['precip_2'] = [precip_n_weeks(k, 2, train_precip_iq)
                          for k in range(len(train_precip_iq))]
X_train_iq['precip_3'] = [precip_n_weeks(k, 3, train_precip_iq) 
                          for k in range(len(train_precip_iq))]
X_train_iq['precip_4'] = [precip_n_weeks(k, 4, train_precip_iq)
                          for k in range(len(train_precip_iq))]

X_test_iq['precip_2'] = [precip_n_weeks(k, 2, test_precip_iq)
                          for k in range(len(test_precip_iq))]
X_test_iq['precip_3'] = [precip_n_weeks(k, 3, test_precip_iq) 
                          for k in range(len(test_precip_iq))]
X_test_iq['precip_4'] = [precip_n_weeks(k, 4, test_precip_iq)
                          for k in range(len(test_precip_iq))]<jupyter_output>C:\Users\Anders\Miniconda3\lib\site-packages\ipykernel_launcher.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  
C:\Users\Anders\Miniconda3\lib\site-packages\ipykernel_launcher.py:4: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  after removing the cwd from sys.path.
C:\Users\Anders\Miniconda3\lib\site-packages\ipykernel_launcher.py:6: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the d[...]<jupyter_text>Let's check that this worked as intended:<jupyter_code>X_test_sj.head(30)<jupyter_output><empty_output><jupyter_text>Building the models<jupyter_code>from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, PolynomialFeatures
from sklearn.linear_model import LinearRegression, ElasticNet
from sklearn.model_selection import (cross_validate, TimeSeriesSplit, 
                                     RandomizedSearchCV)<jupyter_output><empty_output><jupyter_text>ElasticNet with penalty San Jose:<jupyter_code>en_sj = Pipeline([
    ('scale', StandardScaler()),
    ('impute_m', SimpleImputer()),
    ('en', LinearRegression())
])
cv_res_sj = cross_validate(
    estimator = en_sj,
    X = X_train_sj,
    y = y_train_sj,
    cv = TimeSeriesSplit(n_splits = 10),
    scoring = 'neg_mean_absolute_error',
    n_jobs = -1
)
en_sj_score = np.mean(cv_res_sj['test_score'])
en_sj_score
en_sj.fit(X_train_sj, y_train_sj)
y_pred_sj = en_sj.predict(X_train_sj)<jupyter_output><empty_output><jupyter_text>Iquitos:<jupyter_code>en_iq = Pipeline([
    ('scale', StandardScaler()),
    ('impute_m', SimpleImputer()),
    ('en', ElasticNet(alpha = 10))
])
cv_res_iq = cross_validate(
    estimator = en_iq,
    X = X_train_iq,
    y = y_train_iq,
    cv = TimeSeriesSplit(n_splits = 10),
    scoring = 'neg_mean_absolute_error',
    n_jobs = -1
)
en_iq_score = np.mean(cv_res_iq['test_score'])
en_iq_score
y_train_iq.mean()<jupyter_output><empty_output><jupyter_text>Something is really strange here... both models have large MAEs (close to the means values of the targets for each)<jupyter_code>plt.style.use('default')<jupyter_output><empty_output><jupyter_text>We get the date data for each city:<jupyter_code>sj_dates = X_train_o[X_train_o['city'] == 'sj']['week_start_date']
iq_dates = X_train_o[X_train_o['city'] == 'iq']['week_start_date']
ax = plt.axes()
ax.plot(sj_dates, y_pred_sj)
ax.plot(sj_dates, y_train_sj)<jupyter_output><empty_output><jupyter_text>It appears that the model is predicting very close to the mean  Building a submission<jupyter_code>submission = pd.read_csv(join(DATA_PATH, 'submission_format.csv'))
y_pred = poly_model_3.predict(X_test)
submission['total_cases'] = np.round(y_pred).astype(int)
submission
submission.to_csv('../models/baseline.csv', index = False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11210533043589416, 'fraction_numerical': 0.008308832928544037, 'mean_word_length': 4.922785768357305, 'pattern_counts': {'":': 0, '<': 54, '<?xml version=': 0, '>': 53, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '12613', 'license': 'MIT', 'n_tokens_mistral': 3024, 'n_tokens_neox': 2939, 'n_words': 587, 'path': 'notebooks/apoirel-improved-fe-01.ipynb', 'repo_name': 'datascienceucsc/dss-diseasespread', 'size': '7830'}