<jupyter_start><jupyter_text>3. Load and Explore Dataset<jupyter_code>import pandas as pd
import numpy as np
df = pd.read_csv('../data/raw/OnlineNewsPopularity.csv')
df.head()
df.tail()
df.drop(df.tail(1).index,inplace=True)
df.shape
df.info()
df.describe()<jupyter_output><empty_output><jupyter_text>4. Prepare Data Create a copy of df and save it into a variable called df_cleaned<jupyter_code>df_cleaned = df.copy()<jupyter_output><empty_output><jupyter_text>Drop the column url<jupyter_code>df_cleaned.drop('url', axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>Remove leading and trailing space from the column names<jupyter_code>df_cleaned.columns = df_cleaned.columns.str.strip()<jupyter_output><empty_output><jupyter_text>Extract the column shares and save it into variable calletarget = df_cleaned.pop('shares')d target<jupyter_code>target = df_cleaned.pop('shares')
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
df_cleaned = scaler.fit_transform(df_cleaned)
from joblib import dump
dump(scaler, '../models/scaler.joblib')<jupyter_output><empty_output><jupyter_text>Import train_test_split from sklearn.model_selection<jupyter_code>from sklearn.model_selection import train_test_split<jupyter_output><empty_output><jupyter_text>Split randomly the dataset with random_state=8 into 2 different sets: data (80%) and test (20%)<jupyter_code>X_data, X_test, y_data, y_test = train_test_split (df_cleaned, target, test_size=0.2, random_state=8)<jupyter_output><empty_output><jupyter_text>Split the remaining data (80%) randomly with random_state=8 into 2 different sets: training (80%) and validation (20%)<jupyter_code>X_train, X_val, y_train, y_val = train_test_split(X_data, y_data, test_size=0.2, random_state=8)<jupyter_output><empty_output><jupyter_text>Save the different sets in the folder data/processed<jupyter_code>np.save('../data/processed/X_train', X_train)
np.save('../data/processed/X_val',   X_val)
np.save('../data/processed/X_test',  X_test)
np.save('../data/processed/y_train', y_train)
np.save('../data/processed/y_val',   y_val)
np.save('../data/processed/y_test',  y_test)<jupyter_output><empty_output><jupyter_text>5. Get Baseline Model Calculate the average of the target variable for the training set and save it into a variable called y_mean<jupyter_code>y_mean = y_train.mean()<jupyter_output><empty_output><jupyter_text>Create a numpy array called y_base of dimensions (len(y_train), 1) filled with this value<jupyter_code>y_base = np.full((len(y_train), 1), y_mean)<jupyter_output><empty_output><jupyter_text>Import the MSE and MAE metrics from sklearn<jupyter_code>from sklearn.metrics import mean_squared_error as mse
from sklearn.metrics import mean_absolute_error as mae<jupyter_output><empty_output><jupyter_text>Display the RMSE and MAE scores of this baseline model<jupyter_code>print(mse(y_train, y_base, squared=False))
print(mae(y_train, y_base))<jupyter_output>14455.202628642608
2748.776025411728<jupyter_text>6. Train ElasticNet model Import the ElasticNet module from sklearn<jupyter_code>from sklearn.linear_model import ElasticNet<jupyter_output><empty_output><jupyter_text>instantiate the ElasticNet class into a variable called reg<jupyter_code>reg = ElasticNet()<jupyter_output><empty_output><jupyter_text>Fit the model with the prepared data<jupyter_code>reg.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Save the fitted model into the folder models as a file called linearelasticnet_default_reg.joblib<jupyter_code>dump(reg,  '../models/elasticnet_default.joblib')<jupyter_output><empty_output><jupyter_text>Save the predictions from this model for the training and validation sets into 2 variables called y_train_preds and y_val_preds<jupyter_code>y_train_preds = reg.predict(X_train)
y_val_preds = reg.predict(X_val)<jupyter_output><empty_output><jupyter_text>Display the RMSE and MAE scores of this model on the training set<jupyter_code>print(mse(y_train, y_train_preds, squared=False))
print(mae(y_train, y_train_preds))<jupyter_output>14227.22643902199
2770.0178361392896<jupyter_text>Display the RMSE and MAE scores of this model on the validation set<jupyter_code>print(mse(y_val, y_val_preds, squared=False))
print(mae(y_val, y_val_preds))<jupyter_output>4187.350172079278
2425.5074247346956
{'fraction_non_alphanumeric': 0.10326464459365593, 'fraction_numerical': 0.029636489928224125, 'mean_word_length': 9.093457943925234, 'pattern_counts': {'":': 0, '<': 78, '<?xml version=': 0, '>': 78, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '499480', 'license': 'MIT', 'n_tokens_mistral': 1650, 'n_tokens_neox': 1557, 'n_words': 312, 'path': 'adv_dsi_lab_1/notebooks/1_elasticnet.ipynb', 'repo_name': 'Kallikrates/adv_dsi_lab_1', 'size': '4319'}