<jupyter_start><jupyter_text>Model Details Features:- One hot encoded **day of week** and **month** (not year)- Weather feature (OHE):    - Icons (cloudy, partial cloudy, ...)    - Precipitates Type (None, Snow, Sleet, Rain)- Station info:    - community area (OHE)    - capacity    - long, lat     Target- Log scale/normal scale     Work Flow Training Preprocessing- Merge station community area (Join tables)- Drop id after merging- Add weather info (temp_high/low, and OHE ICONs and Precipitates Types)- Convert to numpy matrix Pipeline- OHE on date time (Remember column indices)- Scaling for `year, lon_ave, lat_ave, dp_max, temp_high, temp_low` (`MinMaxScaler`)- Regressor() Test Preprocessing- Start with Pandas template (station_id, lon_ave, lat_ave, dp_max, OHE community area)- Add weather info (temp_high/low, and OHE ICONs and Precipitates Types)- Convert to numpy matrix Post prediction- Rescale if trained on log- Hard cap negative (activation function)<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import os
import pickle<jupyter_output><empty_output><jupyter_text>Load data<jupyter_code>INFO_verbose = False
%%time
st_template = pd.read_pickle('../data/model_v1.0_template.pk')
if INFO_verbose:
    st_template.info()
%%time
## load preprocessed data

if os.path.exists('../data/divvy_data_model_'+major_VERSION+'.pk'):
    print("Loading from previous pickle file.")
    data = pd.read_pickle('../data/divvy_data_model_'+major_VERSION+'.pk')
else:
    print("Create data set for this model... ")
    data_lst = []

    for year in [2013, 2014, 2015, 2016, 2017, 2018]:
        dt_tmp = pd.read_feather('../data/Final_Divvy_data_'+str(year)+'.feather')
        data_lst.append(dt_tmp)

    data = pd.concat(data_lst, ignore_index=True)


    data.to_pickle('../data/divvy_data_model_'+major_VERSION+'.pk')
    print("Data saved to pickle file")
if INFO_verbose:
    data.info()<jupyter_output><empty_output><jupyter_text>Get target value<jupyter_code>target_in = data.total_in
target_out = data.total_out
target_in_log = np.log(target_in+1)<jupyter_output><empty_output><jupyter_text>Prepare features<jupyter_code># Prescreening for useful features
feats = pd.merge(data[['station_id', 'month', 'dayofweek', 'year',
                       'icon_clear-day', 'icon_cloudy', 'icon_partly-cloudy-day', 
                       'icon_rain', 'icon_sleet', 'icon_snow',
                       'precipType_None', 'precipType_rain',
                       'precipType_sleet', 'precipType_snow',
                       'temperatureHigh', 'temperatureLow' 
                         ]], st_template, 
                    left_on='station_id', right_on='id').drop(['station_id', 'id'], axis=1)<jupyter_output><empty_output><jupyter_text>Reordering dataframe<jupyter_code># Reordering
cols = []
cols.extend(feats.columns[15:])
cols.extend(feats.columns[2:15])
cols.extend(feats.columns[:2])
feats_ro = feats[cols]
if INFO_verbose:
    feats_ro.info()
feats_ro = feats_ro.fillna(0)<jupyter_output><empty_output><jupyter_text>Get/Check indices for numerical columns<jupyter_code>num_col_in = [0, 1, 2, 50, 61, 62]
feats_ro.iloc[:, num_col_in].info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 947515 entries, 0 to 947514
Data columns (total 6 columns):
lon_ave            947515 non-null float64
lat_ave            947515 non-null float64
dp_max             947515 non-null float64
year               947515 non-null float64
temperatureHigh    947515 non-null float64
temperatureLow     947515 non-null float64
dtypes: float64(6)
memory usage: 50.6 MB<jupyter_text>One hot encoding date<jupyter_code>feats_fnl = pd.get_dummies(feats_ro, columns=['month', 'dayofweek'])<jupyter_output><empty_output><jupyter_text>Save model details into txt<jupyter_code>import io

buffer = io.StringIO()
feats_fnl.info(buf=buffer)
s = buffer.getvalue()

with open("../model_features_details/features_"+major_VERSION+'.txt', "w", encoding="utf-8") as f: 
    f.write(s)<jupyter_output><empty_output><jupyter_text>Convert to numpy matrix<jupyter_code>features_ = feats_fnl.to_numpy()
features_.shape<jupyter_output><empty_output><jupyter_text>Building pipelines<jupyter_code>import sklearn
from sklearn.preprocessing import MinMaxScaler
from sklearn.preprocessing import OneHotEncoder 
from sklearn.linear_model import Ridge
from sklearn.ensemble import RandomForestRegressor

from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline

from sklearn.model_selection import train_test_split
sklearn.__version__
%%time
model = Pipeline([
    ('mms', MinMaxScaler(feature_range=(0, 1))),
    ('rf', RandomForestRegressor(n_estimators=10, min_samples_split=5))
])

# Train Test split
X_train, X_test, y_train, y_test = train_test_split(
    features_, 
    target_out, 
    test_size=0.2, 
    random_state=42)
%%time
model.fit(X_train, y_train)
# Save model, features and targets
Model_name = 'random_forest_'

with open('../model_data/'+Model_name+'model_'+VERSION+'.pk', 'wb') as p:
    pickle.dump(model, p)

with open('../model_data/'+Model_name+'testfeature_'+VERSION+'.pk', 'wb') as ptf:
    pickle.dump(X_test, ptf)

with open('../model_data/'+Model_name+'testarget_'+VERSION+'.pk', 'wb') as ptg:
    pickle.dump(y_test, ptg)
model.score(X_test, y_test)
# First check
y_pred = model.predict(X_test)
plt.plot(y_test, y_pred, 'o', alpha=0.1)
# Deeper Look
plt.plot(y_test, y_pred, 'o', alpha=0.01)
plt.xlim(0, 300)
plt.ylim(0, 300)<jupyter_output><empty_output><jupyter_text>Grid Search<jupyter_code>%%time
# Train Test split
X_train, X_test, y_train, y_test = train_test_split(
    features_, 
    target_out, 
    test_size=0.2, 
    random_state=42)
print("Train/Test splitted...")

model_gs = GridSearchCV(
    model,
    {'rf__n_estimators': [20, 50], 'rf__max_depth':[5, 10, 20]},
    cv=5,
    n_jobs=4
)


model_gs.fit(X_train, y_train)
print("Best params: ", model_gs.best_params_)
y_pred = model_gs.predict(X_test)
plt.plot(y_test, y_pred, 'o', alpha=0.1)
# Save model, features and targets
Model_name = 'random_forest_'

with open('../model_data/'+Model_name+'model_'+VERSION+'.pk', 'wb') as p:
    pickle.dump(model_gs, p)

with open('../model_data/'+Model_name+'testfeature_'+VERSION+'.pk', 'wb') as ptf:
    pickle.dump(X_test, ptf)

with open('../model_data/'+Model_name+'testraget_'+VERSION+'.pk', 'wb') as ptg:
    pickle.dump(y_test, ptg)<jupyter_output><empty_output><jupyter_text>Reload model<jupyter_code>%%time
# Load previous data
Model_name = 'random_forest_'
with open('../model_data/'+Model_name+'model_'+VERSION+'.pk', 'rb') as pp:
    loaded_model = pickle.load(pp)

with open('../model_data/'+Model_name+'testfeature_'+VERSION+'.pk', 'rb') as ptfl:
    loaded_test_feature = pickle.load(ptfl)

with open('../model_data/'+Model_name+'testarget_'+VERSION+'.pk', 'rb') as ptgl:
    loaded_test_target = pickle.load(ptgl)
loaded_model.score(loaded_test_feature, loaded_test_target)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13400884829456258, 'fraction_numerical': 0.02197802197802198, 'mean_word_length': 4.903959561920809, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '64946', 'license': 'MIT', 'n_tokens_mistral': 2633, 'n_tokens_neox': 2490, 'n_words': 517, 'path': 'machine_learning_models/ML_v1.0.ipynb', 'repo_name': 'mohanliu/divvy-chicago-data', 'size': '7007'}