<jupyter_start><jupyter_text>4장 누락된 데이터> 누락된 데이터를 분석하는 방법을 다룹니다.- author: "<NAME>"- toc: true- comments: true- permalink: /chapter4/- badges: true- sticky_rank: 4<jupyter_code>#hide

import warnings
warnings.filterwarnings('ignore')
#hide

!pip install pandas==0.23.4
!pip install sklearn==0.21.0
!pip install yellowbrick==0.9<jupyter_output>Collecting pandas==0.23.4
[?25l  Downloading https://files.pythonhosted.org/packages/67/a7/12261a51ac2e7be4c698ca27cbe364ca5f16d64999456ee47ea8c7b44417/pandas-0.23.4-cp37-cp37m-manylinux1_x86_64.whl (8.8MB)
[K     |████████████████████████████████| 8.8MB 5.2MB/s 
[?25hRequirement already satisfied: pytz>=2011k in /usr/local/lib/python3.7/dist-packages (from pandas==0.23.4) (2018.9)
Requirement already satisfied: python-dateutil>=2.5.0 in /usr/local/lib/python3.7/dist-packages (from pandas==0.23.4) (2.8.1)
Requirement already satisfied: numpy>=1.9.0 in /usr/local/lib/python3.7/dist-packages (from pandas==0.23.4) (1.19.5)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.5.0->pandas==0.23.4) (1.15.0)
[31mERROR: xarray 0.15.1 has requirement pandas>=0.25, but you'll have pandas 0.23.4 which is incompatible.[0m
[31mERROR: plotnine 0.6.0 has requirement pandas>=0.25.0, but you'll have pandas 0.23.4 which is incompatible.[0m
[3[...]<jupyter_text>필요한 패키지<jupyter_code>#hide

import matplotlib.pyplot as plt
import pandas as pd
from sklearn import (
  ensemble,
  preprocessing,
  tree
)
from sklearn.metrics import (
  auc,
  confusion_matrix,
  roc_auc_score,
  roc_curve
)
from sklearn.model_selection import (
  train_test_split,
  StratifiedKFold
)
from yellowbrick.classifier import (
  ConfusionMatrix,
  ROCAUC
)
from yellowbrick.model_selection import (
  LearningCurve
)<jupyter_output>/usr/local/lib/python3.7/dist-packages/sklearn/utils/deprecation.py:144: FutureWarning: The sklearn.metrics.classification module is  deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.metrics. Anything that cannot be imported from sklearn.metrics is now part of the private API.
  warnings.warn(message, FutureWarning)<jupyter_text>데이터의 수집<jupyter_code>url = "https://biostat.app.vumc.org/wiki/pub/Main/DataSets/titanic3.xls"
df = pd.read_excel(url)
orig_df = df<jupyter_output><empty_output><jupyter_text>4.1 누락된 데이터의 분석<jupyter_code>df.isnull().mean() * 100
import missingno as msno
ax = msno.matrix(orig_df.sample(500))
plt.show()
fig, ax = plt.subplots(figsize=(6, 4))
(1 - df.isnull().mean()).abs().plot.bar(ax=ax)
plt.show()
ax = msno.bar(orig_df.sample(500))
plt.show()
ax = msno.heatmap(df, figsize=(6, 6))
plt.show()
ax = msno.dendrogram(df)
plt.show()<jupyter_output><empty_output><jupyter_text>4.2 누락된 데이터의 삭제<jupyter_code>df1 = df.dropna()
df1 = df.drop(columns="cabin")
df1 = df.dropna(axis=1)<jupyter_output><empty_output><jupyter_text>4.3 데이터의 대치<jupyter_code>from sklearn.impute import SimpleImputer

num_cols = df.select_dtypes(
  include="number"
).columns

im = SimpleImputer() # 평균
imputed = im.fit_transform(df[num_cols])<jupyter_output><empty_output><jupyter_text>4.4 지시자 열의 추가<jupyter_code>def add_indicator(col):
  def wrapper(df):
    return df[col].isna().astype(int)

  return wrapper

df1 = df.assign(cabin_missing=add_indicator("cabin"))
df1.head(10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14809384164222875, 'fraction_numerical': 0.060703812316715546, 'mean_word_length': 5.636186770428016, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 37, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '354194', 'license': 'Apache-2.0', 'n_tokens_mistral': 1541, 'n_tokens_neox': 1463, 'n_words': 256, 'path': '_notebooks/2021_03_09_ML_Pocket_Chapter4.ipynb', 'repo_name': 'yujinchu04/pocket-ml-reference-korean', 'size': '3417'}