<jupyter_start><jupyter_text>Привет!Это ноутбук-бейзлайн к конкурсу на каггл: https://www.kaggle.com/t/38353d5b42dd423e82f26794bed57bc1Вам нужно постараться улучшить его, чтобы скор на каггле стал побольше =)Тренировочные и тестовые данные лежат рядом с ноутбуком или их можно скачать с каггла.<jupyter_code>import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>1. Get data Загрузим данные и посмотрим на них:<jupyter_code>train_data = pd.read_csv("train.csv")
train_data.head()
X_train, y_train = train_data.drop("open_account_flg", axis=1), train_data["open_account_flg"]<jupyter_output><empty_output><jupyter_text>2. Preprocessing<jupyter_code>train_data.isna().any()<jupyter_output><empty_output><jupyter_text>fill nans in "living region"<jupyter_code>X_train["living_region"].value_counts()<jupyter_output><empty_output><jupyter_text>^ действительно, какой еще самый частый регион можно было ожидать.<jupyter_code>X_train["living_region"].fillna("ОБЛ МОСКОВСКАЯ", inplace=True)
data_region_feats = X_train[['living_region']].reset_index(drop=True)

data_region_feats['living_region'] = data_region_feats['living_region'].str.lower().str.strip()
data_region_feats.loc[data_region_feats['living_region']=='москва г', 'living_region'] = 'москва'
data_region_feats.loc[data_region_feats['living_region']=='санкт-петербург г', 'living_region'] = 'санкт-петербург'

data_region_feats['living_region'] = data_region_feats['living_region'].str.replace(' г.', '')
data_region_feats['living_region'] = data_region_feats['living_region'].str.replace('г. ', '')
data_region_feats['living_region'] = data_region_feats['living_region'].str.replace(' г', '')
data_region_feats['living_region'] = data_region_feats['living_region'].str.replace('г ', '')
data_region_feats['living_region'] = data_region_feats['living_region'].str.replace('край', '')
data_region_feats['living_region'] = data_region_feats['living_region'].str.replace('область', '')
data_region_feats['living_region'] = data_region_feats['living_region'].str.replace('аобл', '')
data_region_feats['living_region'] = data_region_feats['living_region'].str.replace('обл', '')
data_region_feats['living_region'] = data_region_feats['living_region'].str.replace('республика', '')
data_region_feats['living_region'] = data_region_feats['living_region'].str.replace('респ', '')
data_region_feats['living_region'] = data_region_feats['living_region'].str.replace(' ао', '')
data_region_feats['living_region'] = data_region_feats['living_region'].str.replace('ао ', '')
data_region_feats['living_region'] = data_region_feats['living_region'].str.replace('автономный', '')
data_region_feats['living_region'] = data_region_feats['living_region'].str.replace('автономная', '')
data_region_feats['living_region'] = data_region_feats['living_region'].str.replace('округ', '')
data_region_feats['living_region'] = data_region_feats['living_region'].str.replace('.', '')
data_region_feats['living_region'] = data_region_feats['living_region'].str.replace(',', '')
data_region_feats['living_region'] = data_region_feats['living_region'].str.replace('-', '')
data_region_feats['living_region'] = data_region_feats['living_region'].str.replace(' ', '')
data_region_feats['living_region'] = data_region_feats['living_region'].str.replace('москвоская', 'московская')
data_region_feats['living_region'] = data_region_feats['living_region'].str.replace('годинцовомосковская', 'московская')
data_region_feats['living_region'] = data_region_feats['living_region'].str.replace('гмосква', 'москва')
data_region_feats['living_region'] = data_region_feats['living_region'].str.replace('чувашиячувашская', 'чувашская')
data_region_feats['living_region'] = data_region_feats['living_region'].str.replace('чувашия', 'чувашская')
data_region_feats['living_region'] = data_region_feats['living_region'].str.strip()

reg_cnt = data_region_feats['living_region'].value_counts()
X_train.head()
data_region_feats
import seaborn as sns
fig, ax = plt.subplots(figsize=(20, 10))
plt.xticks(rotation=90)
sns.countplot(data_region_feats['living_region'], ax=ax,)<jupyter_output><empty_output><jupyter_text>fill nans in "credit_count"<jupyter_code>X_train["credit_count"].value_counts()
X_train["credit_count"].fillna(0., inplace=True)<jupyter_output><empty_output><jupyter_text>fill nans in "overdue_credit_count"<jupyter_code>X_train["overdue_credit_count"].value_counts()
X_train["overdue_credit_count"].fillna(0., inplace=True)<jupyter_output><empty_output><jupyter_text>fill nans in "monthly income"<jupyter_code>X_train["monthly_income"].median()
X_train["monthly_income"].fillna(X_train["monthly_income"].median(), inplace=True)<jupyter_output><empty_output><jupyter_text>Dealing with categorical features  In a baseline lets delete "living_region" feature as its categorical and has too many values)<jupyter_code>X_train.drop("living_region", axis=1, inplace=True)
cat_features = [i[0] for i in dict(X_train.dtypes).items() if 'obj' in str(i[1])]+["tariff_id"]
cat_features<jupyter_output><empty_output><jupyter_text>Convert categirical features to numerical and binarize them<jupyter_code>X_train.head()
# https://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html
X_train = pd.get_dummies(X_train, columns=cat_features)
X_train.head()
X_train.drop(["gender_F", "client_id"], axis=1, inplace=True)
X_train["year_credit_sum"] = X_train["credit_sum"]/X_train["credit_month"]<jupyter_output><empty_output><jupyter_text>3. Build the model<jupyter_code>from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
import xgboost as xgb
from sklearn.model_selection import cross_val_score
from sklearn.metrics import roc_auc_score
from sklearn.grid_search import GridSearchCV<jupyter_output>/anaconda3/lib/python3.6/site-packages/sklearn/cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)
/anaconda3/lib/python3.6/site-packages/sklearn/grid_search.py:42: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. This module will be removed in 0.20.
  DeprecationWarning)<jupyter_text>На каггле наши ответы будут оцениваться метрикой AUC ROC.  Натренируем Random Forest<jupyter_code>rf = RandomForestClassifier()

params = {
    'n_estimators' : [10, 100],
    'max_depth' : [None, 3, 9],
    
}

gsv_rf = GridSearchCV(rf, param_grid=params, scoring='roc_auc', cv=5, verbose=2)
gsv_rf.fit(X_train, y_train)
gsv_rf.best_params_, gsv_rf.best_score_<jupyter_output>Fitting 5 folds for each of 6 candidates, totalling 30 fits
[CV] max_depth=None, n_estimators=10 .................................
[CV] ........................ max_depth=None, n_estimators=10 -   1.6s
[CV] max_depth=None, n_estimators=10 .................................<jupyter_text>XGBoost с оберткой над sklearn:<jupyter_code>import xgboost as xgb

clf = xgb.XGBClassifier(n_jobs=-1)
params = {
    'n_estimators' : [10, 50, 100, 500],
    'max_depth' : [3, 5],
}

gsv_xgb = GridSearchCV(clf, param_grid=params, scoring='roc_auc', cv=5, verbose=1)
gsv_xgb.fit(X_train, y_train)
gsv_xgb.best_params_, gsv_xgb.best_score_<jupyter_output>Fitting 5 folds for each of 8 candidates, totalling 40 fits<jupyter_text>XGBoost без обертки<jupyter_code>dtrain = xgb.DMatrix(X_train[:4*len(X_train)//5], y_train[:4*len(X_train)//5])
dvalid = xgb.DMatrix(X_train[4*len(X_train)//5:], y_train[4*len(X_train)//5:])

params = {
    'n_estimators' : 1000,
    'max_depth' : 3,
    'eval_metric': 'auc'
}

watchlist = [(dtrain, 'train'), (dvalid, 'valid')]
bst = xgb.train(params, dtrain, 1000, evals = watchlist, 
    early_stopping_rounds=2,verbose_eval=1,
    maximize=True)<jupyter_output>[18:04:58] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 0 pruned nodes, max_depth=3
[0]	train-auc:0.667229	valid-auc:0.66784
Multiple eval metrics have been passed: 'valid-auc' will be used for early stopping.

Will train until valid-auc hasn't improved in 2 rounds.
[18:04:58] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 0 pruned nodes, max_depth=3
[1]	train-auc:0.683364	valid-auc:0.682402
[18:04:58] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 0 pruned nodes, max_depth=3
[2]	train-auc:0.698822	valid-auc:0.698966
[18:04:58] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 0 pruned nodes, max_depth=3
[3]	train-auc:0.712274	valid-auc:0.712276
[18:04:58] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 0 pruned nodes, max_depth=3
[4]	train-auc:0.715483	valid-auc:0.714154
[18:04:58] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 0 pru[...]<jupyter_text>Натренируем на всех примерах:<jupyter_code>rf = RandomForestClassifier(n_estimators=100, max_depth=9)
rf.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>4. Process test data<jupyter_code>X_test = pd.read_csv("test.csv")
X_test.head()
ids = X_test['client_id']<jupyter_output><empty_output><jupyter_text>Нам нужно предобработать тестовые данные точно так же, как тренировочные, чтобы можно было засовывать их в модели, обученные на тренировочных данных.<jupyter_code>X_test.isna().any()
X_test.drop("living_region", axis=1, inplace=True)
X_test["monthly_income"].fillna(X_train["monthly_income"].median(), inplace=True)
X_test["credit_count"].fillna(0., inplace=True)
X_test["overdue_credit_count"].fillna(0., inplace=True)
X_test = pd.get_dummies(X_test, columns=cat_features)
X_test.head()
X_test.drop("gender_F", axis=1, inplace=True)
X_test["year_credit_sum"] = X_test["credit_sum"]/X_test["credit_month"]
test_columns = [column for column in X_train.columns if column in X_test.columns]
X_test = X_test[test_columns]<jupyter_output><empty_output><jupyter_text>Выделим колонки, которых нет в X_test, но есть в X_train<jupyter_code>missing_columns = [column for column in X_train.columns if column not in X_test.columns]
missing_columns
for column in missing_columns:
    X_test[column] = np.zeros(len(X_test))<jupyter_output><empty_output><jupyter_text>5. Get file with answers Для roc auc нужны предсказания **вероятностей** классов (точнее, первого класса -- единицы), а не просто 0 или 1. Поэтому для моделей из sklearn нужно мспользовать .predict_proba(), чтобы получить вероятности:<jupyter_code># получим предсказания Random Forest
preds = rf.predict_proba(X_test[X_train.columns])[:, 1]
preds.mean()<jupyter_output><empty_output><jupyter_text>А xgboost, который не обернут в sklearn, выдает вероятности принадлежности 1 классу просто с помощью .predict():<jupyter_code>dtest = xgb.DMatrix(X_test[X_train.columns])
preds = bst.predict(dtest)
preds
X_test['client_id'] = ids
X_test['open_account_flg'] = preds
X_test[['client_id', 'open_account_flg']].to_csv("sampleSubmission.csv", index=None)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14250089063056645, 'fraction_numerical': 0.025561097256857856, 'mean_word_length': 8.121852152721365, 'pattern_counts': {'":': 0, '<': 77, '<?xml version=': 0, '>': 77, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '643177', 'license': 'MIT', 'n_tokens_mistral': 4484, 'n_tokens_neox': 4324, 'n_words': 721, 'path': 'Homeworks/Homework2/Winning_the_competition_solution.ipynb', 'repo_name': 'JacksonGibsonESP/Sberbank-ML', 'size': '11228'}