<jupyter_start><jupyter_text>基礎情報
- 各breath_idの配列長は80固定。吸気フェーズの配列長は25~32。<jupyter_code>print("tr unique breath_id", len(tr_df["breath_id"].unique()))
print("ts unique breath_id", len(ts_df["breath_id"].value_counts()))
print("tr seq len", tr_df["breath_id"].value_counts().unique())
print("ts seq len", ts_df["breath_id"].value_counts().unique())
print("tr seq len (u_out==0)", tr_df["breath_id"][tr_df["u_out"] == 0].value_counts().unique())
print("ts seq len (u_out==0)", ts_df["breath_id"][ts_df["u_out"] == 0].value_counts().unique())
print("tr first index u_out==0", tr_df[tr_df["u_out"] == 0].groupby("breath_id")["index_in_breath"].first().unique())
print("ts first index u_out==0", ts_df[ts_df["u_out"] == 0].groupby("breath_id")["index_in_breath"].first().unique())
print("tr last index u_out==0", tr_df[tr_df["u_out"] == 0].groupby("breath_id")["index_in_breath"].last().unique())
print("ts last index u_out==0", ts_df[ts_df["u_out"] == 0].groupby("breath_id")["index_in_breath"].last().unique())<jupyter_output>tr last index u_out==0 [29 28 31 30 27 24 25]
ts last index u_out==0 [30 28 29 31 27 24]<jupyter_text>値
- R: 5, 20, 50
- C: 10, 20, 50
- u_in: 0~100
- pressure: 0~40ぐらい<jupyter_code>tr_df[["u_in", "pressure"]].hist(bins=100)
plt.show()
np.log(tr_df[["u_in", "pressure"]] + 1).hist(bins=100)
plt.show()
tr_df[["u_in", "pressure"]].describe()
print("R", tr_df["R"].unique(), "C", tr_df["C"].unique())
ts_df[["u_in"]].hist(bins=100)
plt.show()
print("R", ts_df["R"].unique(), "C", ts_df["C"].unique())
r_map = {5: 0, 20:1, 50: 2}
c_map = {10: 0, 20:1, 50: 2}<jupyter_output><empty_output><jupyter_text>dataset split
- R, Cの組み合わせパターンは9とおり。出現頻度にあまり偏りはない。
- breath_idでsplitすれば良さそう。<jupyter_code>grdf = tr_df.groupby("breath_id").first()

R = grdf["R"].map(r_map)
C = grdf["C"].map(c_map)
val_R = sorted(R.unique().tolist())
val_C = sorted(C.unique().tolist())

cm = confusion_matrix(R, C)
cm = pd.DataFrame(cm, index=val_R, columns=val_C)
cm.index.name = "R"
cm.columns.name = "C"
cm
grdf = ts_df.groupby("breath_id").first()

R = grdf["R"].map(r_map)
C = grdf["C"].map(c_map)
val_R = sorted(R.unique().tolist())
val_C = sorted(C.unique().tolist())

cm = confusion_matrix(R, C)
cm = pd.DataFrame(cm, index=val_R, columns=val_C)
cm.index.name = "R"
cm.columns.name = "C"
cm<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.2303107705406556, 'fraction_numerical': 0.039165602383993185, 'mean_word_length': 6.885906040268456, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '372899', 'license': 'MIT', 'n_tokens_mistral': 1147, 'n_tokens_neox': 1075, 'n_words': 162, 'path': 'eda/eda001.ipynb', 'repo_name': 'statsu1990/kaggle_ventilator_pressure_prediction', 'size': '2349'}