<jupyter_start><jupyter_text>Model Baseline For this baseline model, we'll use all features without prior selection.Logistic regression is often a good model to use for a classification baseline(reasonably fast and easy to interpret).We'll fit a different model for each of the targets `h1n1_vaccine` and `seasonal_vaccine`as data exploration showed significant differences in how the features related to each,in particular for the respondents' perceived risk and vaccine effectiveness for each disease (e.g. `opinion_h1n1_sick_from_vacc`)<jupyter_code>import pandas as pd
import numpy as np
import os

import matplotlib.pyplot as plt
import seaborn as sns
import matplotlib as mpl

from sklearn.compose import ColumnTransformer
from sklearn.impute import KNNImputer, SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.svm import LinearSVC
from sklearn.metrics import roc_auc_score, balanced_accuracy_score
from sklearn.model_selection import cross_validate
from sklearn.multioutput import MultiOutputClassifier
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder<jupyter_output><empty_output><jupyter_text>Figure prettifying:<jupyter_code>mpl.rcParams.update({
    'figure.autolayout': True,
    'figure.dpi': 150
})
sns.set()<jupyter_output><empty_output><jupyter_text>Data pre-processing<jupyter_code>DATA_PATH = '../../data/raw'
X_train = pd.read_csv(
    os.path.join(DATA_PATH, 'training_set_features.csv')
).drop('respondent_id',axis =1)

X_test = pd.read_csv(
    os.path.join(DATA_PATH, 'test_set_features.csv')
).drop('respondent_id',axis =1)

y_train = pd.read_csv(
    os.path.join(DATA_PATH, 'training_set_labels.csv')
).drop('respondent_id',axis =1)

sub = pd.read_csv(os.path.join(DATA_PATH, 'submission_format.csv'))<jupyter_output><empty_output><jupyter_text>Missing value imputation  ModelsCommon utilities for preprocessing in the pipelines. We can create an special `missing` value to impute categorical features. For numerical features, we'll use K-nearest neighbors imputation, wich in generalproduces better results than imputation along the mean<jupyter_code>num_features = X_train.columns[X_train.dtypes != "object"].values
cat_features = X_train.columns[X_train.dtypes == "object"].values

num_transformer = Pipeline([
    ('scale', StandardScaler()),
    ('impute', KNNImputer(n_neighbors = 10))
])

cat_transformer = Pipeline([
    ('impute', SimpleImputer(strategy = 'constant', fill_value = 'missing')),
    ('encode', OneHotEncoder(drop = 'first'))
])

preprocessor = ColumnTransformer([
    ('num', num_transformer, num_features),
    ('cat', cat_transformer, cat_features)
])<jupyter_output><empty_output><jupyter_text>Logistic RegressionLogistic regression usually makes a decent baseline for classification problems. As we need to predict two different values, we will use a `MultiOutputClassifier`<jupyter_code>X_train.shape<jupyter_output><empty_output><jupyter_text>The choice of $L1$ regularization was a bit arbitrary, but motivated by the high number of features (relative to the number of training examples) after one-hot encoding. The `saga` solver is used becausethere a a significant number of training samples (though not so high that I am not sure whether LFBGS would be faster)<jupyter_code>def get_model(C = 1):
    model = Pipeline([
        ('pre', preprocessor),
        ('model', MultiOutputClassifier(LogisticRegression(
            penalty = 'l1', 
            C = C,
            solver = 'saga'
        )))
    ])
    return model<jupyter_output><empty_output><jupyter_text>Logistic regression only has one major hyperparameter, the regularization $C$. Generate an array of choices of $C$to evaluate performance<jupyter_code>Cs = np.logspace(-2, 1, num = 10, base = 10)
means = []
stds = []
best_auc = 0

for C in Cs:
    cv = cross_validate(
        estimator = get_model(C),
        X = X_train,
        y = y_train,
        cv = 5,
        n_jobs = -1,
        scoring = 'roc_auc',
    )
    means.append(np.mean(cv['test_score']))
    stds.append(np.std(cv['test_score']))
    if means[-1] > best_auc:
        best_C = C
        best_auc = means[-1]
        print('New best AUC: ' + best_auc + ' for C=' + best_C)
print(means)
print(best_auc)
print(best_C)<jupyter_output>0.46415888336127775<jupyter_text>Plot these results for reporting<jupyter_code>fig, ax = plt.subplots()
ax.plot(Cs, means)
ax.vlines(best_C, ymin = 0.82, ymax = 0.86,  colors = 'r', linestyle = 'dotted')
ax.annotate("$C = 0.464$ \n ROC AUC = 0.843", xy = (0.5, 0.835))
ax.set_xscale('log')
ax.set_xlabel('$C$')
ax.grid(axis = 'x')
ax.legend(['AUC', 'best $C$'])
ax.set_title('AUC for different values of $C$')
fig.savefig('../../paper/figures/reg_performance.png')<jupyter_output><empty_output><jupyter_text>Our best AUC score beats the benchmark by a good margin, so we will make a submission usingthe model with $C = 1$ (AUC seems constant for $C > 0.5$)  Building a submissionRefit the logistic regression model with $C = 1$<jupyter_code>model = get_model(1)
model.fit(X_train, y_train)
preds = model.predict_proba(X_test)<jupyter_output><empty_output><jupyter_text>Note that this returns two arrays, each with the predicted probabilities for *each class* for *each* of the two classifiers,we need to be careful in selecting the correct predictions!<jupyter_code>sub['h1n1_vaccine'] = preds[0][:,1]
sub['seasonal_vaccine'] = preds[1][:,1]
sub.to_csv('../../output/baseline_pred.csv', index = False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11062985020754376, 'fraction_numerical': 0.01353546291283162, 'mean_word_length': 5.19217877094972, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 42, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '176195', 'license': 'MIT', 'n_tokens_mistral': 1884, 'n_tokens_neox': 1791, 'n_words': 499, 'path': 'notebooks/apoirel-modeling-01.ipynb', 'repo_name': 'jlehrer1/drivendata-flu', 'size': '5541'}