<jupyter_start><jupyter_text>info  発展1. ★教師あり学習や教師なし学習で新たな特徴を生成している1. ★生成したモデルの誤差と説明変数の相関分析し、どのようなデータに対して予測が上手くいっていないのかを確認している1. ★2の誤差の分析で得た知見から新たな特徴を生成し、精度向上を図っている1. 非線形モデルを使用し、ハイパーパラメーターチューニングを行っている1. 自身が生成したモデルを用いてアンサンブルに予測し、精度を確認している  init<jupyter_code>import sys, os, copy
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import japanize_matplotlib
import seaborn as sns
plt.style.use("seaborn-pastel")
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold
from sklearn.externals import joblib
def config_plot(h=10):
    plt.rcParams["font.size"] = 1.5*h
    return (1.62*h, h)<jupyter_output><empty_output><jupyter_text>load<jupyter_code>path_doc = "../doc/"
path_data = "../data/"
path_clns = path_data + "clns/"
path_mid = path_data + "mid/"
_dic_data = dict()
for d in os.listdir(path_clns):
    fn = d.split(".")[0]
    _dic_data[fn] = pd.read_csv(path_clns+d, index_col=0)
def get_mart(log=False, oh=False, tgt=False, fam=False, ps=False):
    dic_data = copy.deepcopy(_dic_data)
    yname = "survived"
    y = dic_data["y"]
    test_id = y[y[yname].isna()].index
    
    nums = dic_data["num_z"]
    cats = list()

    onehot = [v for k, v in dic_data.items() if ("onehot" in k) and not ("family" in k or "parch_sibsp" in k)]
    onehot_fam = [v for k, v in dic_data.items() if ("onehot" in k) and ("family" in k)]
    onehot_ps = [v for k, v in dic_data.items() if ("onehot" in k) and ("parch_sibsp" in k)]
    
    tgtenc = [v for k, v in dic_data.items() if ("tgt" in k) and not ("family" in k or "parch_sibsp" in k)]
    tgt_fam = [v for k, v in dic_data.items() if ("tgt" in k) and ("family" in k)]
    tgt_ps = [v for k, v in dic_data.items() if ("tgt" in k) and ("parch_sibsp" in k)]

    if log:
        nums = dic_data["num_logz"]
    if oh:
        cats += onehot
    if tgt:
        cats += tgtenc
    if fam:
        if oh:
            cats += onehot_fam
        if tgt:
            cats += onehot_ps
    if ps:
        if oh:
            cats += tgt_fam
        if tgt:
            cats += tgt_ps
            
    mart = pd.concat([y, nums, *cats], axis=1)
    train, test = mart[~mart.index.isin(test_id)], mart[mart.index.isin(test_id)]
    train_X, train_y, test_X = train.drop(yname, 1), train[yname], test.drop(yname, 1)
    return train_X, train_y, test_X
X, y, _ = get_mart(1,0,1,1,1)
model = joblib.load("../model/lr_10111.pkl")
params = model.get_params()<jupyter_output><empty_output><jupyter_text>pred<jupyter_code>kf = StratifiedKFold(n_splits=3)
res = list()
for train_index, test_index in kf.split(X, y):
    X_train, X_test = X.iloc[train_index], X.iloc[test_index]
    y_train, y_test = y.iloc[train_index], y.iloc[test_index]
    lr = LogisticRegression(**params)
    lr.fit(X_train, y_train)
    p = lr.predict_proba(X_test)[:,1]
    proba = y_test.copy().to_frame()
    proba["p"] = p
    res.append(proba)<jupyter_output><empty_output><jupyter_text>**MEMO**- 学習に使用していないデータに対するprobaを分析対象にしている。<jupyter_code>likelihood = pd.concat(res)
likelihood["likelihood"] = (likelihood.p**likelihood.survived)*((1-likelihood.p)**(1-likelihood.survived))
likelihood.T
likelihood[["likelihood"]].to_csv(path_clns+"num_likelihood.csv")<jupyter_output><empty_output><jupyter_text>チェックポイント1: 教師あり学習や教師なし学習で新たな特徴を生成している- ロジスティック回帰の出力を説明変数に加える  plot<jupyter_code>cats = pd.read_csv(path_mid+"cats.csv", index_col=0)
nums = pd.read_csv(path_mid+"nums.csv", index_col=0)
bools = pd.read_csv(path_mid+"bools.csv", index_col=0)
cats = cats.join(nums[["parch", "sibsp"]])
y = likelihood[["likelihood"]]<jupyter_output><empty_output><jupyter_text>cat<jupyter_code>data = cats.copy()
data = data.dropna()
data = data.astype(str)
data = data.join(y, how="inner")
def plot_cat(data, x):
    p = path_doc + "img/error_cat_%s.png"%x
    plt.figure(figsize=config_plot())
    for v in data[x].unique():
        subs = data[data[x]==v]
        sns.distplot(subs["likelihood"], label=v)
    plt.legend()
    print("###", x)
    print("![](%s)"%p)
    plt.savefig(p)
for x in data.drop("likelihood",1):
    plot_cat(data, x)<jupyter_output>C:\Users\keisu\Anaconda3\lib\site-packages\scipy\stats\stats.py:1713: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  return np.add.reduce(sorted[indexer] * weights, axis=axis) / sumval<jupyter_text>- pcalss=1stに尤度の小さい集団が存在する  bool<jupyter_code>data = bools.copy()
data = data.dropna()
data = data.astype(str)
data = data.join(y, how="inner")
for x in data.drop("likelihood",1):
    plot_cat(data, x)<jupyter_output>C:\Users\keisu\Anaconda3\lib\site-packages\scipy\stats\stats.py:1713: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  return np.add.reduce(sorted[indexer] * weights, axis=axis) / sumval<jupyter_text>num<jupyter_code>data = nums.copy()
data = data.dropna()
data = data.join(y, how="inner")
sns.pairplot(data.join(cats.pclass), hue="pclass")<jupyter_output>C:\Users\keisu\Anaconda3\lib\site-packages\scipy\stats\stats.py:1713: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  return np.add.reduce(sorted[indexer] * weights, axis=axis) / sumval<jupyter_text>advance - pcalss=1stに絞った集合を分析する<jupyter_code>data = cats.copy()
data = data.dropna()
data = data.astype(str)
data = data.join(y, how="inner")
subs = data[data.pclass=="1st"]
subs.T
# for x in subs.drop("likelihood",1):
plot_cat(subs, "sex")<jupyter_output>C:\Users\keisu\Anaconda3\lib\site-packages\scipy\stats\stats.py:1713: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  return np.add.reduce(sorted[indexer] * weights, axis=axis) / sumval<jupyter_text>- 1stかつmaleの尤度が軒並み低いことが分かった    - 1stかつmaleのフラグを生成することで精度が向上する可能性がある。<jupyter_code>subs[subs.sex=="male"].T<jupyter_output><empty_output><jupyter_text>チェックポイント2: 生成したモデルの誤差と説明変数の相関分析し、どのようなデータに対して予測が上手くいっていないのかを確認している- 1stかつ男性の予測が上手くいっていない  get feature<jupyter_code>is_1st_male = (cats.pclass=="1st")&(cats.sex=="male")
is_1st_male = is_1st_male * 1
is_1st_male.name = "is_1st_male"
is_1st_male = is_1st_male.to_frame()
is_1st_male.to_csv(path_clns+"onehot_is_1st_male.csv")
is_1st_male.T<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14349261013057826, 'fraction_numerical': 0.011192423590185106, 'mean_word_length': 4.957264957264957, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '477532', 'license': 'MIT', 'n_tokens_mistral': 3096, 'n_tokens_neox': 2986, 'n_words': 561, 'path': 'notebooks/4.ModelAnalysis.ipynb', 'repo_name': 'mnm-analytics/titanic', 'size': '6969'}