<jupyter_start><jupyter_text>檢視與處理 Outliers 為何會有 outliers, 常見的 outlier 原因* 未知值，隨意填補 (約定俗成的代入)，如年齡常見 0,999* 可能的錯誤紀錄/手誤/系統性錯誤，如某本書在某筆訂單的銷售量 = 1000 本<jupyter_code># Import 需要的套件
import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

%matplotlib inline

# 設定 data_path
dir_data = './data'
f_app = os.path.join(dir_data, 'application_train.csv')
print('Path of read in data: %s' % (f_app))
app_train = pd.read_csv(f_app)
app_train.head()<jupyter_output>Path of read in data: ./data/application_train.csv<jupyter_text>請參考 HomeCredit_columns_description.csv 的欄位說明，觀察並列出三個你覺得可能有 outlier 的欄位並解釋可能的原因<jupyter_code># 先篩選數值型的欄位
"""
YOUR CODE HERE, fill correct data types (for example str, float, int, ...)
"""
num_df = app_train.select_dtypes(include=[np.number])

drop_list = list()
for colname in num_df:
    unique_values = num_df[colname].unique()
    if len(unique_values) == 2:
        drop_list.append(colname)
        #del num_df[colname] # It can also delete columns
        
numeric_df = num_df.drop(drop_list, axis=1)

print("Numbers of remain columns : " + str(numeric_df.shape[1]))

ncols = 4
nrows = np.ceil(numeric_df.shape[1]/4).astype(np.int)
fig, ax_arr = plt.subplots(nrows, ncols, figsize=(20, nrows*1.5))
for col_name, ax in zip(numeric_df, ax_arr.ravel()):
    numeric_df.boxplot(column=col_name, ax=ax, vert=False)
    ax.set_yticklabels(labels=[])
    ax.set_title(label=col_name)
plt.tight_layout()
plt.show()
# 從上面的圖檢查的結果，至少這三個欄位好像有點可疑

# AMT_INCOME_TOTAL
# REGION_POPULATION_RELATIVE
# OBS_60_CNT_SOCIAL_CIRCLE
def empirical_distribute_function(data):
    '''
    Computing empirical distribution
    : data: 1-D numpy array
    '''
    sort_data = np.sort(data)
    cumulate_prob = np.arange(1, len(data) + 1)/len(data)
    
    return sort_data, cumulate_prob<jupyter_output><empty_output><jupyter_text>Hints: Emprical Cumulative Density Plot, [ECDF](https://zh.wikipedia.org/wiki/%E7%BB%8F%E9%AA%8C%E5%88%86%E5%B8%83%E5%87%BD%E6%95%B0), [ECDF with Python](https://stackoverflow.com/questions/14006520/ecdf-in-python-without-step-function)<jupyter_code># 最大值離平均與中位數很遠
print(app_train['AMT_INCOME_TOTAL'].describe())

# 繪製 Empirical Cumulative Density Plot (ECDF)
"""
YOUR CODE HERE
"""
sort_data, cum_prob = empirical_distribute_function(app_train['AMT_INCOME_TOTAL'])

plt.plot(sort_data, cum_prob)
plt.xlabel('Value')
plt.ylabel('ECDF')
#plt.xlim([sort_data.min(), sort_data.max()]) # 限制顯示圖片的範圍
plt.ylim([-0.05,1.05]) # 限制顯示圖片的範圍

plt.show()

# 改變 y 軸的 Scale, 讓我們可以正常檢視 ECDF
plt.plot(np.log(sort_data), cum_prob)
plt.xlabel('Value (log-scale)')
plt.ylabel('ECDF')

plt.ylim([-0.05,1.05]) # 限制顯示圖片的範圍

plt.show()

sub = app_train['AMT_INCOME_TOTAL']<jupyter_output><empty_output><jupyter_text>補充：Normal dist 的 ECDF![ecdf_normal](https://au.mathworks.com/help/examples/stats/win64/PlotEmpiricalCdfAndCompareWithSamplingDistributionExample_01.png)<jupyter_code># 最大值落在分布之外
print(app_train['REGION_POPULATION_RELATIVE'].describe())

# 繪製 Empirical Cumulative Density Plot (ECDF)
#"""
#Your Code Here
#"""
sort_data, cum_prob = empirical_distribute_function(app_train['REGION_POPULATION_RELATIVE']) 

plt.plot(sort_data, cum_prob)
plt.xlabel('Value')
plt.ylabel('ECDF')
plt.ylim([-0.05,1.05]) # 限制顯示圖片的範圍
plt.show()

app_train['REGION_POPULATION_RELATIVE'].hist()
plt.show()

app_train['REGION_POPULATION_RELATIVE'].value_counts()

# 就以這個欄位來說，雖然有資料掉在分布以外，也不算異常，僅代表這間公司在稍微熱鬧的地區有的據點較少，
# 導致 region population relative 在少的部分較為密集，但在大的部分較為疏漏
# 最大值落在分布之外
print(app_train['OBS_60_CNT_SOCIAL_CIRCLE'].describe())

# 繪製 Empirical Cumulative Density Plot (ECDF)
"""
Your Code Here
"""
sort_data, cum_prob = empirical_distribute_function(app_train['OBS_60_CNT_SOCIAL_CIRCLE'])

plt.plot(sort_data, cum_prob)
plt.xlabel('Value')
plt.ylabel('ECDF')
#plt.xlim([sort_data.min() * 0.95, sort_data.max() * 1.05])
plt.ylim([-0.05,1.05]) # 限制顯示圖片的範圍
plt.show()

app_train['OBS_60_CNT_SOCIAL_CIRCLE'].hist()
plt.show()
print(app_train['OBS_60_CNT_SOCIAL_CIRCLE'].value_counts().sort_index(ascending = False))<jupyter_output>count    306490.000000
mean          1.405292
std           2.379803
min           0.000000
25%           0.000000
50%           0.000000
75%           2.000000
max         344.000000
Name: OBS_60_CNT_SOCIAL_CIRCLE, dtype: float64<jupyter_text>注意：當 histogram 畫出上面這種圖 (只出現一條，但是 x 軸延伸很長導致右邊有一大片空白時，代表右邊有值但是數量稀少。這時可以考慮用 value_counts 去找到這些數值<jupyter_code># 把一些極端值暫時去掉，在繪製一次 Histogram
# 選擇 OBS_60_CNT_SOCIAL_CIRCLE 小於 20 的資料點繪製
"""
Your Code Here
"""
loc_a = app_train['OBS_60_CNT_SOCIAL_CIRCLE'] < 20
loc_b = 'OBS_60_CNT_SOCIAL_CIRCLE'

app_train.loc[loc_a, loc_b].hist()
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13751611516974646, 'fraction_numerical': 0.03781693167168027, 'mean_word_length': 6.031722054380665, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 19, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '473787', 'license': 'MIT', 'n_tokens_mistral': 2414, 'n_tokens_neox': 2312, 'n_words': 298, 'path': 'Day_006_HW.ipynb', 'repo_name': 'hengbinxu/ML100-Days', 'size': '4654'}