<jupyter_start><jupyter_text>Allstate Claims Severity[Kaggle](https://www.kaggle.com/c/allstate-claims-severity)  When you’ve been devastated by a serious car accident, your focus is on the things that matter the most: family, friends, and other loved ones. Pushing paper with your insurance agent is the last place you want your time or mental energy spent. This is why Allstate, a personal insurer in the United States, is continually seeking fresh ideas to improve their claims service for the over 16 million households they protect. Allstate is currently developing automated methods of predicting the cost, and hence severity, of claims.<jupyter_code>data = pd.read_csv('train.csv')
data.info()
data.head()<jupyter_output><empty_output><jupyter_text>You must predict the value for the `loss` column. Variables prefaced with `cat` are categorical, while those prefaced with `cont` are continuous. Let's check is there a missing data?<jupyter_code>data.isnull().any().any()<jupyter_output><empty_output><jupyter_text>Analyze target variable<jupyter_code>data['loss'].describe()<jupyter_output><empty_output><jupyter_text>- There're 188 318 rows (objects).- Minumum is 0.67 and maxiumum 121 012.25.- The avarage (mean) is 3037.338- There's a big difference between 75th and 100th(max), should be long tail by right side.<jupyter_code>data['loss'].hist(bins=100)<jupyter_output><empty_output><jupyter_text>As usually, for those tasks, distrubution is right skewed (long tail by right side) and there're some outliers.There're few approaches how to manage outliers:1. remove 1 or 2 % the biggest values (usually this is outliers)2. transform data by function which is work fine with outliers (e.g. log transformation)<jupyter_code>data[ data.loss < np.percentile(data.loss, 99) ]['loss'].hist(bins=100)
data['log_loss'] = np.log( data['loss'] + 1 )
data['log_loss'].hist(bins=100)<jupyter_output><empty_output><jupyter_text>Categorical variablesHas prefix `cat` in the name<jupyter_code>cat_feats = [feat for feat in data.columns if 'cat' in feat]
print(len(cat_feats),  cat_feats)<jupyter_output>(116, ['cat1', 'cat2', 'cat3', 'cat4', 'cat5', 'cat6', 'cat7', 'cat8', 'cat9', 'cat10', 'cat11', 'cat12', 'cat13', 'cat14', 'cat15', 'cat16', 'cat17', 'cat18', 'cat19', 'cat20', 'cat21', 'cat22', 'cat23', 'cat24', 'cat25', 'cat26', 'cat27', 'cat28', 'cat29', 'cat30', 'cat31', 'cat32', 'cat33', 'cat34', 'cat35', 'cat36', 'cat37', 'cat38', 'cat39', 'cat40', 'cat41', 'cat42', 'cat43', 'cat44', 'cat45', 'cat46', 'cat47', 'cat48', 'cat49', 'cat50', 'cat51', 'cat52', 'cat53', 'cat54', 'cat55', 'cat56', 'cat57', 'cat58', 'cat59', 'cat60', 'cat61', 'cat62', 'cat63', 'cat64', 'cat65', 'cat66', 'cat67', 'cat68', 'cat69', 'cat70', 'cat71', 'cat72', 'cat73', 'cat74', 'cat75', 'cat76', 'cat77', 'cat78', 'cat79', 'cat80', 'cat81', 'cat82', 'cat83', 'cat84', 'cat85', 'cat86', 'cat87', 'cat88', 'cat89', 'cat90', 'cat91', 'cat92', 'cat93', 'cat94', 'cat95', 'cat96', 'cat97', 'cat98', 'cat99', 'cat100', 'cat101', 'cat102', 'cat103', 'cat104', 'cat105', 'cat106', 'cat107', 'cat108', 'cat109', 'cat110', '[...]<jupyter_text>There're 116 categorical variables.Let's check cardinality (how many unique values each category has)<jupyter_code>unq_values = [data[cat].nunique() for cat in cat_feats]
id_cat_feats = range(len(cat_feats))
plt.figure(figsize=(30, 10))
plt.plot(id_cat_feats, unq_values)
plt.xticks(id_cat_feats, cat_feats, rotation='vertical')
plt.show()<jupyter_output><empty_output><jupyter_text>Let's remove last 20 elements, to verify how looks like smaller data.<jupyter_code>cut_last_elems = 20
plt.figure(figsize=(30, 10))
plt.plot(id_cat_feats[:-cut_last_elems], unq_values[:-cut_last_elems])
plt.xticks(id_cat_feats[:-cut_last_elems], cat_feats[:-cut_last_elems], rotation='vertical')
plt.show()<jupyter_output><empty_output><jupyter_text>Questions1. Are there hight cardinality (let say more than 50 unique values)?2. How many unique values have average categorical variables?3. How this information could help you?  Continuous variablesHas prefix `cont` in the name<jupyter_code>cont_feats = [feat for feat in data.columns if 'cont' in feat]
print(len(cont_feats),  cont_feats)
_ = data[ cont_feats ].hist(bins=30, figsize=(20, 20))<jupyter_output><empty_output><jupyter_text>What you can say about:- distributions- range (min and max)- outliers  Evaluation [Mean absolute error](https://en.wikipedia.org/wiki/Mean_absolute_error)In practise the means, that you will calculate difference between your prediction and actual value for each points.Letter sum up all of them and devide by count of points.What the problem do you know about mae?  Base LineLet's build the simplest model<jupyter_code>from sklearn.dummy import DummyRegressor
from sklearn.metrics import mean_absolute_error
def base_line_model(data, target_variable='loss', strategy='mean'):
    X = data[cont_feats].values
    y = data[target_variable].values

    model = DummyRegressor(strategy=strategy)
    model.fit(X, y)
    y_pred = model.predict(X)

    if target_variable == 'log_loss':
        y = np.exp(y) - 1
        y_pred = np.exp(y_pred) - 1
        y_pred[ y_pred < 0 ] = 0

    score = mean_absolute_error(y, y_pred)
        
    return score

for strategy in ['mean', 'median']:
    for target_variable in ['loss', 'log_loss']:
        score = base_line_model(data, target_variable=target_variable, strategy=strategy)
        print(strategy, target_variable, score)<jupyter_output>('mean', 'loss', 1966.1652855607203)
('mean', 'log_loss', 1809.8469823690546)
('median', 'loss', 1809.0487864675706)
('median', 'log_loss', 1809.0487864675708)<jupyter_text>EvaluationTo avoid overfitting is need more advaned way to validate result (e.g. cross validation) Task implement cross-validationyou can use DummyRegressor like a model Tips<jupyter_code>from sklearn.cross_validation import KFold

nfolds = 3
folds = KFold(data.shape[0], n_folds=nfolds, shuffle = True, random_state = 2017)

for num_iter, (train_index, test_index) in enumerate(folds):
    pass
    #X_train, y_train = X[train_index], y[train_index]
    #X_test, y_test   = X[test_index], y[test_index]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.143042071197411, 'fraction_numerical': 0.060841423948220064, 'mean_word_length': 6.288915094339623, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 43, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '43632', 'license': 'Apache-2.0', 'n_tokens_mistral': 2395, 'n_tokens_neox': 2094, 'n_words': 606, 'path': 'step2.ipynb', 'repo_name': 'konradbjk/MIO-allstate', 'size': '6180'}