<jupyter_start><jupyter_text>Custom model implementations and functions are stored in `src/dairyml.py`<jupyter_code>from dairyml import *<jupyter_output><empty_output><jupyter_text>Import the DataLoad the data from the pickle files created in `preproccess.ipynb`<jupyter_code>with open("../pkl/data/data_outliers_removed", "rb" ) as f:
    [X, Y] = pkl.load(f)

Y_reshaped = Y.values.reshape(-1,1)<jupyter_output><empty_output><jupyter_text>Modelling with Lasso Define a splitter- this produces the data splits used for cross-validation. `random_state` is defined for repeatability.<jupyter_code>from splitting import *<jupyter_output><empty_output><jupyter_text>Choose statistics used to evaluate the models<jupyter_code>from scoring import *<jupyter_output><empty_output><jupyter_text>Load CSV file to store test results<jupyter_code>try:
    overall_results = pd.read_csv('../reports/cv_results.csv',index_col=0)
except FileNotFoundError:
    overall_results = pd.DataFrame(columns = scoring.keys())<jupyter_output><empty_output><jupyter_text>Lasso - Conduct a grid search on the regularization parameter alpha on a log scale between 1e-3 and 1 (np.logspace takes exponents of 10 as arguments)- For each value of alpha in this range, we perform 10-fold cross validation (CV) on the training data- Choose the best model based on R^2 score in CV using the `refit` argument of `GridSearchCV`<jupyter_code>param_grid = {'alpha': np.logspace(-3, 0, 50)}

refit = 'r2'

search = GridSearchCV(estimator=Lasso(random_state=7),param_grid=param_grid,scoring=scoring,refit=refit,cv=ten_fold_no_repeat,return_train_score=True)
fitted = search.fit(X,Y_reshaped)<jupyter_output>/Users/gabe/opt/miniconda3/envs/DairyML_old/lib/python3.6/site-packages/sklearn/linear_model/coordinate_descent.py:492: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Fitting data with very small alpha may cause precision problems.
  ConvergenceWarning)
/Users/gabe/opt/miniconda3/envs/DairyML_old/lib/python3.6/site-packages/sklearn/linear_model/coordinate_descent.py:492: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Fitting data with very small alpha may cause precision problems.
  ConvergenceWarning)
/Users/gabe/opt/miniconda3/envs/DairyML_old/lib/python3.6/site-packages/sklearn/linear_model/coordinate_descent.py:492: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Fitting data with very small alpha may cause precision problems.
  ConvergenceWarning)
/Users/gabe/opt/miniconda3/envs/DairyML_old/lib/python3.6/site-packages/skle[...]<jupyter_text>Save model to file<jupyter_code>model = fitted.best_estimator_
with open("../models/lasso.model", "wb" ) as f:
    pkl.dump(model,f)<jupyter_output><empty_output><jupyter_text>Best value of alpha for LassoPrint the best value of the regularization parameter, as chosen by 10-fold CV R^2 score above<jupyter_code>search.best_estimator_.alpha<jupyter_output><empty_output><jupyter_text>Lasso test performanceHighest R^2 score for the Lasso model from 10-fold CV (this is the performance we can expect on the test set)<jupyter_code>search.cv_results_['mean_test_r2'][np.argmax(search.cv_results_['mean_test_'+refit])]
overall_results = scores_to_df(overall_results,'Lasso',search.cv_results_,refit)<jupyter_output><empty_output><jupyter_text>Bounded LassoThis attempts to fix an obvious issue with the above predictor, that the model predicts negative lactose content for some foods. BoundedLasso wraps sklearn's Lasso model, and "clips" the model output so that negative output values become 0.<jupyter_code>param_grid = {'alpha': np.logspace(-3, 0, 100)}

refit = 'r2'

search = GridSearchCV(estimator=BoundedLasso(random_state=7),param_grid=param_grid,scoring=scoring,refit=refit,cv=ten_fold_no_repeat,return_train_score=True)
fitted = search.fit(X,Y_reshaped)<jupyter_output>/Users/gabe/opt/miniconda3/envs/DairyML_old/lib/python3.6/site-packages/sklearn/linear_model/coordinate_descent.py:492: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Fitting data with very small alpha may cause precision problems.
  ConvergenceWarning)
/Users/gabe/opt/miniconda3/envs/DairyML_old/lib/python3.6/site-packages/sklearn/linear_model/coordinate_descent.py:492: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Fitting data with very small alpha may cause precision problems.
  ConvergenceWarning)
/Users/gabe/opt/miniconda3/envs/DairyML_old/lib/python3.6/site-packages/sklearn/linear_model/coordinate_descent.py:492: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Fitting data with very small alpha may cause precision problems.
  ConvergenceWarning)
/Users/gabe/opt/miniconda3/envs/DairyML_old/lib/python3.6/site-packages/skle[...]<jupyter_text>Save model to file<jupyter_code>model = fitted.best_estimator_
with open("../models/lasso_bounded.model", "wb" ) as f:
    pkl.dump(model,f)<jupyter_output><empty_output><jupyter_text>Best value of alpha for Bounded LassoPrint the best value of the regularization parameter, as chosen by 10-fold CV R^2 score above<jupyter_code>search.best_estimator_.alpha<jupyter_output><empty_output><jupyter_text>Bounded Lasso test results<jupyter_code>overall_results = scores_to_df(overall_results,'Bounded Lasso',search.cv_results_,refit)<jupyter_output><empty_output><jupyter_text>Feature importancesHere we plot the non-zero coefficients of the Bounded Lasso model. The magnitude of the coefficient for each feature indicates how much each feature contributes to the overall lactose estimate. We can see that choline, sugar, and potassium contribute positively to the lactose estimate, while fiber, cholesterol, and protein contribute negatively.  Save feature imporantances to file<jupyter_code>coefficients = pd.DataFrame()
coefficients['Feature'] = X.columns
coefficients = coefficients.set_index('Feature')
coefficients['coef'] = model.get_coef()
coefficients.to_csv('lasso_coef.csv')
plot_coefficients(search.best_estimator_,X)<jupyter_output><empty_output><jupyter_text>Bounded Lasso plus Classifier- Add a binary classifier to the model to predict whether a food is zero or non-zero lactose. - In the case that the classifier predicts 0 lactose, the overall model output is 0. - In the case that the classifier predicts non-zero, the output from the bounded lasso model is used  Evaluate the classifier by itself<jupyter_code>Y_binary = Y != 0

param_grid_LR = {'C': np.logspace(-4, 4, 30)}

search_LR = GridSearchCV(estimator=LogisticRegression(solver='lbfgs',max_iter=2000,random_state=7),param_grid=param_grid_LR,scoring=('accuracy','recall'),refit='accuracy',cv=ten_fold_no_repeat,return_train_score=True)
fitted = search_LR.fit(X,Y_binary)
overall_results = scores_to_df(overall_results,'Logistic Regression Classifier',search_LR.cv_results_,refit='accuracy')<jupyter_output><empty_output><jupyter_text>Classifier accuracyThe classifier achieves 89% accuracy in predicting 0 vs non-zero lactose.<jupyter_code>search_LR.best_score_<jupyter_output><empty_output><jupyter_text>Evaluate the Bounded Lasso + Classifier Model<jupyter_code>param_grid = {'alpha': np.logspace(-3, -1, 10), 'C': np.logspace(4, 6, 10)}
refit='r2'
search = GridSearchCV(estimator=BoundedLassoPlusLogReg(random_state=7),param_grid=param_grid,scoring=scoring,refit=refit,cv=ten_fold_no_repeat)
fitted = search.fit(X,Y_reshaped)<jupyter_output>/Users/gabe/opt/miniconda3/envs/DairyML_old/lib/python3.6/site-packages/sklearn/linear_model/coordinate_descent.py:492: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Fitting data with very small alpha may cause precision problems.
  ConvergenceWarning)
/Users/gabe/opt/miniconda3/envs/DairyML_old/lib/python3.6/site-packages/sklearn/utils/validation.py:761: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
  y = column_or_1d(y, warn=True)
/Users/gabe/opt/miniconda3/envs/DairyML_old/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:758: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.
  "of iterations.", ConvergenceWarning)
/Users/gabe/opt/miniconda3/envs/DairyML_old/lib/python3.6/site-packages/sklearn/linear_model/coordinate_descent.py:492: ConvergenceWarning: Objective did not converge.[...]<jupyter_text>Plot actual vs predicted results<jupyter_code>y_pred = cross_val_predict(search.best_estimator_,X,Y_reshaped,cv=ten_fold_no_repeat)
fig, ax = plt.subplots(figsize=[8,8])
ax.scatter(Y, y_pred)
ax.plot([Y.min(), Y.max()], [Y.min(), Y.max()], 'k--', lw=4)
ax.set_xlabel('Actual')
ax.set_ylabel('Predicted')
ax.set_title('Bounded Lasso plus Logistic Regression \n Actual vs Predicted lactose')
plt.show()<jupyter_output><empty_output><jupyter_text>Best parameter values for the combined model<jupyter_code>best_c = search.best_estimator_.get_params()['C']
search.best_estimator_.get_params()<jupyter_output><empty_output><jupyter_text>Model test performance<jupyter_code>overall_results = scores_to_df(overall_results,'Bounded Lasso + LogReg',search.cv_results_,refit)
overall_results.to_csv('../reports/cv_results.csv')<jupyter_output><empty_output><jupyter_text>Save model to file<jupyter_code>model = fitted.best_estimator_
with open("../models/lasso_combined.model", "wb" ) as f:
    pkl.dump(model,f)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10043532338308457, 'fraction_numerical': 0.013577943615257049, 'mean_word_length': 7.2895189003436425, 'pattern_counts': {'":': 0, '<': 82, '<?xml version=': 0, '>': 82, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '660008', 'license': 'Apache-2.0', 'n_tokens_mistral': 3343, 'n_tokens_neox': 3161, 'n_words': 729, 'path': 'src/lasso_predictor.ipynb', 'repo_name': 'IBPA/dairyML', 'size': '9648'}