<jupyter_start><jupyter_text>作業在鐵達尼資料集中，可以用今天範例中提到的三種辨識異常值的方法，以 training data 為資料集。  1.觀察票價是否有異常值的現象?  2.你覺得找出的異常是真的異常? 你覺得需要做處理嗎?  (試著講出自己的想法。)    資料鐵達尼號 train 資料集<jupyter_code># library
import numpy as np
import pandas as pd
import seaborn  as sns
from scipy import stats
import math
import statistics
import matplotlib.pyplot as plt
#讓圖可以在 jupyter notebook顯示
%matplotlib inline
df_train = pd.read_csv( 'Titanic_train.csv' )
df_train<jupyter_output><empty_output><jupyter_text>探索性分析:*  觀察-是否有異常值  我們來觀察一下 Fare 變數吧!!<jupyter_code># 檢查訓練資料集的 Fare 欄位中是否有遺失值 ?
def Missing_Counts( Data ) : 
    missing = Data.isnull().sum()  
    missing = missing[ missing>0 ]
    missing.sort_values( inplace=True ) 
    
    Missing_Count = pd.DataFrame( { 'ColumnName':missing.index, 'MissingCount':missing.values } ) 
    Missing_Count[ 'Percentage(%)' ] = Missing_Count['MissingCount'].apply( lambda x:round(x/Data.shape[0]*100,2) )
    return  Missing_Count

Missing_Counts(df_train)
g = sns.FacetGrid( df_train, col='Survived' )
g.map( sns.histplot, 'Fare', bins=20 )
plt.show()<jupyter_output><empty_output><jupyter_text>Q1:觀察票價是否有異常值的現象?  方法1* 進行一個描述性的統計（規定範圍），從而檢視哪些值是不合理的（範圍以外的值）。<jupyter_code>df_train['Fare'].describe()<jupyter_output><empty_output><jupyter_text>方法 2* 進行 3 倍標準差原則的計算，從而檢視哪些值是可疑的異常值。<jupyter_code># 創建一個函數，尋找在資料中超出設定倍數的標準差的異常值。
# ys: 資料、threshold: 設定幾倍標準差
def outliers_z_score( ys, threshold ):
    mean_y = np.mean(ys)
    stdev_y = np.std(ys)
    z_scores = [(y - mean_y) / stdev_y for y in ys]
    return np.where(np.abs(z_scores)>threshold)
outlier_index = outliers_z_score( df_train['Fare'], 3 )
outlier_index
print( "'Fare' 欄位中疑似的異常值: " )
df_train.loc[ outlier_index[0], 'Fare' ]<jupyter_output>'Fare' 欄位中疑似的異常值:<jupyter_text>方法 3 - 盒鬚圖判別法(IQR method)<jupyter_code>sns.boxplot( x=df_train['Fare'], whis=1.5, orient='h', color='skyblue' )
plt.xlabel( 'Fare' )
plt.show()
def outliers_iqr( ys, times ):
    # 注意: 使用 np.percentile 要去除 nan ，不然計算出來會錯誤!
    # quartile_1, quartile_3 = np.percentile(ys, [25, 75])
    quartile_1, quartile_3 = np.nanpercentile( ys, [25,75] )
    iqr = quartile_3 - quartile_1
    lower_bound = quartile_1 - (iqr * times)
    upper_bound = quartile_3 + (iqr * times)
    return np.where( (ys > upper_bound) | (ys < lower_bound) )
outliers_index2 = outliers_iqr( df_train['Fare'], 1.5 )
outliers_index2
print( "'Fare' 欄位中疑似的異常值(超出 1.5 倍 IQR): " )
df_train.loc[ outliers_index2[0], 'Fare' ]<jupyter_output>'Fare' 欄位中疑似的異常值(超出 1.5 倍 IQR):
{'fraction_non_alphanumeric': 0.13599677158999193, 'fraction_numerical': 0.018159806295399514, 'mean_word_length': 5.575596816976128, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '205696', 'license': 'MIT', 'n_tokens_mistral': 1367, 'n_tokens_neox': 1367, 'n_words': 190, 'path': 'Homework/Day_36_HW.ipynb', 'repo_name': 'YenLinWu/DataScienceMarathon', 'size': '2478'}