<jupyter_start><jupyter_text>Retrieve data<jupyter_code>import pandas as pd
import numpy as np
from plotnine import *

import warnings
warnings.filterwarnings('ignore')
# TODO use a relative path here
train_path = "C:\\Users\\Rishi\\Documents\\Kaggle\\Mercari\\data\\train.tsv"
train = pd.read_csv(train_path, sep='\t')

pd.np.random.seed(369)<jupyter_output><empty_output><jupyter_text>Preprocessing Parse out individual category levelsProduct category names consist of subcategory trees (Men/Tops/T-shirts). There don't appear to be more than 5 levels and the vast majority of them have just 3 levels. Below, we create a variable for each level.<jupyter_code>from sklearn import preprocessing

train_x = train[["brand_name", "shipping", "category_name"]]
train_y = train.price

def piece(string, delim, n):
    string = str(string)
    if string.count(delim) < n:
        return pd.np.NaN
    return string.split(delim)[n]

train_x["category1"] = train_x.category_name.map(lambda x: piece(x, "/", 0))
train_x["category2"] = train_x.category_name.map(lambda x: piece(x, "/", 1))
train_x["category3"] = train_x.category_name.map(lambda x: piece(x, "/", 2))
train_x["category4"] = train_x.category_name.map(lambda x: piece(x, "/", 3))
train_x["category5"] = train_x.category_name.map(lambda x: piece(x, "/", 4))
train_x = train_x.drop(columns='category_name')
train_x.head()<jupyter_output><empty_output><jupyter_text>Fill in missing dataFor now we will be representing any missing data with the string 'zMissing.'<jupyter_code>train_x = train_x.fillna('zMissing')
train_x.head()<jupyter_output><empty_output><jupyter_text>One hot encodehttp://scikit-learn.org/stable/modules/preprocessing.htmlencoding-categorical-featuresCreate dummy variables out of categorical variables. First we use LabelEncoder to swap out category names with numbers. Then we use OneHotEncoder to pivot out those values in to columns.**OneHotEncoder*** Encode categorical integer features using a one-hot aka one-of-K scheme.* The input to this transformer should be a matrix of integers, denoting the values taken on by categorical (discrete) features.* The output will be a sparse matrix where each column corresponds to one possible value of one feature.* It is assumed that input features take on values in the range [0, n_values).* This encoding is needed for feeding categorical data to many scikit-learn estimators, notably linear models and SVMs with the standard kernels.<jupyter_code># Pandas is proving to be too slow
# dummies = pd.get_dummies(train_x)
# lets try it with sklearn
encoder = preprocessing.OneHotEncoder()
label_encoder = preprocessing.LabelEncoder()
data_label_encoded = train_x.apply(label_encoder.fit_transform)
train_x_encoded = encoder.fit_transform(data_label_encoded)<jupyter_output><empty_output><jupyter_text>Supervised LearningLets start by experimenting with a simple SGD regressor becasue of how fast it is.http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDRegressor.htmlsklearn.linear_model.SGDRegressorFurther down the line we should incorporate matrix factorization as the collinearity and correlations are likely quite high. We'll also want to perform cross validation to pick optimal parameters for SGD.<jupyter_code># This competition uses root mean squared log error.
# Lets use this to evaluate our models as well.
from sklearn.metrics import make_scorer

# vectorized error calc
def rmsle(y, y0):
    assert len(y) == len(y0)
    return np.sqrt(np.mean(np.power(np.log1p(y)-np.log1p(y0), 2)))

rmsle_score = make_scorer(rmsle, greater_is_better=False)<jupyter_output><empty_output><jupyter_text>Stochastic Gradient Descent - SGDStochastic gradient descent is a simple yet very efficient approach to fit linear models. It is particularly useful when the number of samples (and the number of features) is very large. The partial_fit method allows only/out-of-core learning. The classes SGDClassifier and SGDRegressor provide functionality to fit linear models for classification and regression using different (convex) loss functions and different penalties.  Grid SearchSGD has a lot of parameters that need to be optimized. Lets try a 10-fold-cross-validation on several potential values and figure out whats giving us the best rmsle.<jupyter_code>from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import SGDRegressor
from sklearn.model_selection import cross_val_score

param_grid = [
  {'loss': ["squared_loss", "huber", "epsilon_insensitive"], 
   'penalty': ['elasticnet'],
   'alpha' : 10.0**-pd.np.arange(1,7),
   'l1_ratio' : [0, .15, .3, .5, .75, 1],
   'learning_rate' : ['constant', 'optimal', 'invscaling']
  }
 ]
sgd_cv = GridSearchCV(estimator=SGDRegressor(), param_grid=param_grid, cv=5, scoring=rmsle_score)
sgd_cv.fit(train_x_encoded, train_y)
sgd_cv.best_estimator_<jupyter_output><empty_output><jupyter_text>Optimal SGDWe have found our optimal parameters. Lets use them on the whole data set and see how we're doing.<jupyter_code>sgd = SGDRegressor(alpha=1.0000000000000001e-05, average=False, epsilon=0.1,
       eta0=0.01, fit_intercept=True, l1_ratio=1, learning_rate='optimal',
       loss='epsilon_insensitive', max_iter=None, n_iter=None,
       penalty='elasticnet', power_t=0.25, random_state=None, shuffle=True,
       tol=None, verbose=0, warm_start=False)
scores = cross_val_score(sgd, train_x_encoded, train_y, cv=10, scoring=rmsle_score)
print("Average CV performance: {0}".format(scores.mean()))
ggplot(pd.DataFrame({'scores': scores}), aes('scores')) + geom_density() + geom_rug() + ggtitle("10 fold cross validation performance on traing set")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10075744231107979, 'fraction_numerical': 0.013211203100228995, 'mean_word_length': 6.0272277227722775, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '59787', 'license': 'MIT', 'n_tokens_mistral': 1850, 'n_tokens_neox': 1744, 'n_words': 539, 'path': '.ipynb_checkpoints/Simple model pipeline (No NLP)-checkpoint.ipynb', 'repo_name': 'blamarre19/kaggle-mercari', 'size': '5677'}