<jupyter_start><jupyter_text>Applying regression models with pipelinesFor this notebook we will:* Bring over the dataframe we generated in the previous step (Generating features and wrangling)* Creating a preprocessing pipeline with sklearn to feed our data and apply different models* Test our first hypothesis:  We can guess the number of monthly interactions a user have with the characteristics of an uploaded recipe, and the history of the user who uploaded it.* Test our second hypothesis: Changing our dependent variable to be the number of interactions that will happen only in the month after the recipe is submitted, to be able to predict short term activity in our site.First we import general libraries for data manipulation and plotting, and we will bring sklearn libraries later on as we use them.<jupyter_code>#Libraries for tables manipulation
import pandas as pd
import numpy as np
import datetime

#Libraries for plotting
import matplotlib.pyplot as plt
import seaborn as sns 
%matplotlib inline

#To not receive warning messages
import warnings
warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>Let's take a look to our data<jupyter_code>data = pd.read_csv("../data/mvp_export.csv", index_col=0)
data
data.describe()<jupyter_output><empty_output><jupyter_text>We start transforming categoricals into dtype_objects, as well as our index<jupyter_code>data['month_posted'] = data['month_posted'].astype(object)
data['recipe_id'] = data['recipe_id'].astype(object)<jupyter_output><empty_output><jupyter_text>Looking at the distributionsWe first take a look in the distribution of our target variables<jupyter_code>plt.figure(figsize=(15,7))
sns.distplot(data['comm_rec_month'], color='orange')
plt.figure(figsize=(15,7))
sns.distplot(data['in_first30d'], color='orange')
plt.figure(figsize=(15,7))
sns.distplot(data['rating'], color='orange')<jupyter_output><empty_output><jupyter_text>checking the distribution of the numerical variables we can see that some scale transformation and we we'll have to deal with some ouliers, as there are very extreme ones, as well as the distributions being very skewed to the right, or to the left in the case of the average ratings.<jupyter_code>for col in data.select_dtypes([np.number]).columns:
    sns.distplot(data[col], color='teal')
    plt.suptitle(col)
    plt.show()<jupyter_output><empty_output><jupyter_text>Seeing the column: days_since_posted we see the distribution of when the recipes were being posted. The activity in the page lowered considerably after 2010, that's why we see a sudden rise in this variable. We will remove all recipes posted after 1500 days ago (bigger than 1500) and remove those generated shortly when the page was created, when it was less known (taking all smaller than 6500 days)<jupyter_code>plt.figure(figsize=(15,7))
plt.axvline(1500, color='red')
plt.axvline(6500, color='red')
plot = sns.distplot(data['days_since_posted'], color='teal')
plot.set_xlabel('days since posted',fontsize=20)
plot.set_ylabel('density',fontsize=20)
plt.show();
data = data[data['days_since_posted'].between(1500,6500)]<jupyter_output><empty_output><jupyter_text>Removing outliersBy the look of the distribution of the variables, we will need to clean some outliers and apply some transformation as steps for preprocessing.We start by checking the numerical variables:<jupyter_code>for col in data.select_dtypes([np.number]).columns:
    plot = sns.boxplot(data[col], color='teal')
    plot.set_xlabel(col, fontsize=15)
    plt.show()<jupyter_output><empty_output><jupyter_text>We have to be very careful with outliers that correspond to columns other than minutes, n_ingredients and n_steps, as those columns correspond to characteristics of the user that uploaded the recipe, and if we remove users that submitted a lot of recipes, we will remove too many of our database data points. If a good result is not achieved, it might be good to bin these columns and treat them as categoricals<jupyter_code>data = data[data['no_recipes_submitted']<2000]
data.shape
data = data[data['avg_comments_recipes']<200]
data.shape
data = data[data['avg_sentiment_recipes']>-0.75]
data.shape<jupyter_output><empty_output><jupyter_text>We'll take out all recipes that take over 12 hours to cook<jupyter_code>data = data[data['minutes']<720]
data.shape
data = data[data['n_steps']<40]
data.shape
data = data[data['n_ingredients']<20]
data.shape
data = data[data['no_recipes_submitted']<1000]
data.shape<jupyter_output><empty_output><jupyter_text>Looking for multicollinearity<jupyter_code>def corr_matr_show(data):
    correlations_matrix = data.corr()
    mask = np.zeros_like(correlations_matrix)
    mask[np.triu_indices_from(mask)] = True
    fig, ax = plt.subplots(figsize=(13, 10))
    ax = sns.heatmap(correlations_matrix, mask=mask, annot=True)
    plt.show()
    
corr_matr_show(data)<jupyter_output><empty_output><jupyter_text>From this chart we could take interesting information on how the web works:* Both dependent variables we want to look for, have a strong correlation. This could indicate that the comments that a recipe receives monthly is strongly determined by the ones received in the first month. * Apart from our index and our dependent variables, we can't see any other strong correlation happening, so we won't have to drop any features  Creating a pipelineFirst we need to import all transformers and models we want to use in our pipeline, as well as the pipeline itself<jupyter_code># Preprocessing libraries
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.compose import make_column_selector as selector
from sklearn.preprocessing import StandardScaler, OneHotEncoder, PowerTransformer

# Libraries different regression modelling and evaluation
from sklearn.model_selection import train_test_split
from sklearn import linear_model
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error, r2_score<jupyter_output><empty_output><jupyter_text>Then we need to define a pipeline of preprocessing for categoricals and numericals separately, as the process we want to put them through is different. For categoricals we will use:* OneHotEncoder: creates a new feature per category and will be labeled as 1 if it belongs to that feature<jupyter_code>cat_transf = Pipeline(steps=[('onehot',OneHotEncoder(handle_unknown='ignore'))
                             ])<jupyter_output><empty_output><jupyter_text>Then for numericals we will use:* PowerTransformer: This transformer will alter the scale in a way that heavily skewed data becomes more normalised* StandardScaler: will scale our data to have a mean of 0 and standard deviation of 1, preserving the proportions between all data<jupyter_code>num_transf = Pipeline(steps=[('transformer', PowerTransformer()),
                             ('scaler', StandardScaler())
                             ])<jupyter_output><empty_output><jupyter_text>This part of the pipeline will choose columns from our data, and put them through one or another pipeline. We use selector function to choose columns based on the type.<jupyter_code>preprocessor = ColumnTransformer(transformers=[('cat', cat_transf, selector(dtype_include="object")),
                                               ('num', num_transf, selector(dtype_exclude="object"))
                                               ])<jupyter_output><empty_output><jupyter_text>Split X and y to pedict the number of interactions in the first 30 days<jupyter_code>X = data.drop(['recipe_id','comm_rec_month','rating'], axis=1)
y = data['in_first30d']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=66)<jupyter_output><empty_output><jupyter_text>Prototyping pipelines for different modelsPrototyping the pipeline: Here comes all together, we will first preprocess our data through the previous parts of the pipeline, and then apply a regression model. For this project we will not go deep into improving the models and adjusting hyperparameters, as the purpose here was to learn how to deploy these pipelines and make them work. To achieve better models we would have to research and work more in the specific preprocessing steps, and apply them to each one of the regressors  Linear Regression<jupyter_code>pipe_lr = Pipeline([('preprocessing', preprocessor),
                    ('regressor',linear_model.LinearRegression())
                    ])
pipe_lr.fit(X_train, y_train)
print("model score: %.3f" % pipe_lr.score(X_test, y_test))
y_test_predict = pipe_lr.predict(X_test)
print("RMSE:      {:.3f}".format(mean_squared_error(y_test, y_test_predict, squared=False)))
print("R2 score: {:.3f}".format(r2_score(y_test, y_test_predict)))<jupyter_output>RMSE:      15.356
R2 score: 0.128<jupyter_text>Stochastic Gradient Descent (SGD) Regression<jupyter_code>pipe_SGDreg = Pipeline([('preprocessing', preprocessor),
                        ('SGDregressor',linear_model.SGDRegressor())
                        ])
pipe_SGDreg.fit(X_train, y_train)
print("model score: %.3f" % pipe_SGDreg.score(X_test, y_test))
y_test_predict = pipe_SGDreg.predict(X_test)
print("RMSE:      {:.3f}".format(mean_squared_error(y_test, y_test_predict, squared=False)))
print("R2 score: {:.3f}".format(r2_score(y_test, y_test_predict)))<jupyter_output>RMSE:      15.366
R2 score: 0.127<jupyter_text>Poisson Regression<jupyter_code>pipe_Preg = Pipeline([('preprocessing', preprocessor),
                      ('Poissonregressor',linear_model.PoissonRegressor())
                      ])
pipe_Preg.fit(X_train, y_train)
print("model score: %.3f" % pipe_Preg.score(X_test, y_test))
y_test_predict = pipe_Preg.predict(X_test)
print("RMSE:      {:.3f}".format(mean_squared_error(y_test, y_test_predict, squared=False)))
print("R2 score: {:.3f}".format(r2_score(y_test, y_test_predict)))<jupyter_output>RMSE:      14.191
R2 score: 0.256<jupyter_text>Random Forest Regressor<jupyter_code>pipe_RFreg = Pipeline([('preprocessing', preprocessor),
                       ('random forest reg', RandomForestRegressor(max_depth=2, random_state=66))
                       ])
pipe_RFreg.fit(X_train, y_train)
print("model score: %.3f" % pipe_RFreg.score(X_test, y_test))
y_test_predict = pipe_RFreg.predict(X_test)
print("RMSE:      {:.3f}".format(mean_squared_error(y_test, y_test_predict, squared=False)))
print("R2 score: {:.3f}".format(r2_score(y_test, y_test_predict)))<jupyter_output>RMSE:      5.919
R2 score: 0.871<jupyter_text>* Plotting the predictions vs the real test set<jupyter_code>plt.figure(figsize=(10,10))
plt.scatter(y_test, y_test_predict, c='orange')

p1 = max(max(y_test_predict), max(y_test))
p2 = min(min(y_test_predict), min(y_test))
plt.plot([p1, p2], [p1, p2], 'teal')
plt.xlabel('True Values', fontsize=15)
plt.ylabel('Predictions', fontsize=15)
plt.axis('equal')
plt.show()<jupyter_output><empty_output><jupyter_text>Predicting the rating We can use the pipelines earlier set to try to predict rating now, we only have to do the split again:<jupyter_code>X = data.drop(['recipe_id','comm_rec_month','in_first30d'], axis=1)
y = data['rating']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=66)<jupyter_output><empty_output><jupyter_text>Random Forest Regressor<jupyter_code>pipe_RFreg = Pipeline([('preprocessing', preprocessor),
                       ('random forest reg', RandomForestRegressor(max_depth=2, random_state=66))
                       ])
pipe_RFreg.fit(X_train, y_train)
print("model score: %.3f" % pipe_RFreg.score(X_test, y_test))
y_test_predict = pipe_RFreg.predict(X_test)
print("RMSE:      {:.3f}".format(mean_squared_error(y_test, y_test_predict, squared=False)))
print("R2 score: {:.3f}".format(r2_score(y_test, y_test_predict)))<jupyter_output>RMSE:      0.239
R2 score: 0.942<jupyter_text>* Plotting the predictions vs the real test set<jupyter_code>plt.figure(figsize=(10,10))
plt.scatter(y_test, y_test_predict, c='orange')

p1 = max(max(y_test_predict), max(y_test))
p2 = min(min(y_test_predict), min(y_test))
plt.plot([p1, p2], [p1, p2], 'teal')
plt.xlabel('True Values', fontsize=15)
plt.ylabel('Predictions', fontsize=15)
plt.axis('equal')
plt.show()<jupyter_output><empty_output><jupyter_text>Linear Regression<jupyter_code>pipe_lr = Pipeline([('preprocessing', preprocessor),
                    ('regressor',linear_model.LinearRegression())
                    ])
pipe_lr.fit(X_train, y_train)
print("model score: %.3f" % pipe_lr.score(X_test, y_test))
y_test_predict = pipe_lr.predict(X_test)
print("RMSE:      {:.3f}".format(mean_squared_error(y_test, y_test_predict, squared=False)))
print("R2 score: {:.3f}".format(r2_score(y_test, y_test_predict)))
plt.figure(figsize=(10,10))
plt.scatter(y_test, y_test_predict, c='orange')

p1 = max(max(y_test_predict), max(y_test))
p2 = min(min(y_test_predict), min(y_test))
plt.plot([p1, p2], [p1, p2], 'teal')
plt.xlabel('True Values', fontsize=15)
plt.ylabel('Predictions', fontsize=15)
plt.axis('equal')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10509821906290606, 'fraction_numerical': 0.017274325460521288, 'mean_word_length': 5.723535457348407, 'pattern_counts': {'":': 0, '<': 98, '<?xml version=': 0, '>': 93, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '124158', 'license': 'Apache-2.0', 'n_tokens_mistral': 4357, 'n_tokens_neox': 4095, 'n_words': 1163, 'path': 'code/2-Applying-regression-models-with-pipelines.ipynb', 'repo_name': 'Alex-Skp/Recipe-Popularity-Predictor', 'size': '13083'}