<jupyter_start><jupyter_text>Pandas dtypes (數據類型)<jupyter_code>import os
import numpy as np
import pandas as pd
# 設定 data_path

dir_data = '../data/'
f_app_train = os.path.join(dir_data, 'application_train.csv')
f_app_test = os.path.join(dir_data, 'application_test.csv')

app_train = pd.read_csv(f_app_train)
app_test = pd.read_csv(f_app_test)
# 檢視資料中各個欄位類型的數量

# app_train.dtypes.value_counts()
app_train.get_dtype_counts()
# 檢視資料中類別型欄位各自類別的數量

app_train.select_dtypes(include=["object"]).apply(pd.Series.nunique, axis = 0)<jupyter_output><empty_output><jupyter_text>Label encoding有仔細閱讀[參考資料](https://medium.com/@contactsunny/label-encoder-vs-one-hot-encoder-in-machine-learning-3fc273365621)的人可以發現，Label encoding 的表示方式會讓同一個欄位底下的類別之間有大小關係 (0<1<2<...)，所以在這裡我們只對有類別數量小於等於 2 的類別型欄位示範使用 Label encoding，但不表示這樣處理是最好的，一切取決於欄位本身的意義適合哪一種表示方法<jupyter_code>from sklearn.preprocessing import LabelEncoder

le = LabelEncoder()  # Create a label encoder object
le_count = 0
le_column = []

for col in app_train:  # Iterate through the columns
    
    if app_train[col].dtype == 'object':
        if len(list(app_train[col].unique())) <= 2:  # If 2 or fewer unique categories
            
            # Train on the training data
            le.fit(app_train[col])
            
            # Transform both training and testing data
            app_train[col] = le.transform(app_train[col])
            app_test[col] = le.transform(app_test[col])
            
            # Keep track of how many columns were label encoded
            le_count += 1
            le_column.append(col)
            
print('{} columns {} were label encoded.'.format(le_count, ", ".join(le_column)))<jupyter_output>3 columns NAME_CONTRACT_TYPE, FLAG_OWN_CAR, FLAG_OWN_REALTY were label encoded.<jupyter_text>One Hot encodingpandas 中的 one hot encoding 非常方便，一行程式碼就搞定  可以觀察到原來的類別型欄位都轉為 0/1 了<jupyter_code>app_train = pd.get_dummies(app_train)
app_test = pd.get_dummies(app_test)

print(app_train.filter(regex=("CODE_GENDER_*")).head())
print()

print(app_train['NAME_EDUCATION_TYPE_Higher education'].head())<jupyter_output>CODE_GENDER_F  CODE_GENDER_M  CODE_GENDER_XNA
0              0              1                0
1              1              0                0
2              0              1                0
3              1              0                0
4              0              1                0

0    0
1    1
2    0
3    0
4    0
Name: NAME_EDUCATION_TYPE_Higher education, dtype: uint8<jupyter_text>HW將下列部分資料片段 sub_train 使用 One Hot encoding, 並觀察轉換前後的欄位數量 (使用 shape) 與欄位名稱 (使用 head) 變化<jupyter_code>app_train = pd.read_csv(f_app_train)
sub_train = pd.DataFrame(app_train['WEEKDAY_APPR_PROCESS_START'])

print(sub_train.shape)
sub_train.head(n=5)
# Use One Hot encoding to handle categorical data

import json

sub_train = pd.get_dummies(sub_train)
print("DataFrame size: {}".format(sub_train.shape))
print("DataFrame columns: {}".format(json.dumps(sub_train.columns.values.tolist(), indent=4)))

sub_train.head(n=6)<jupyter_output>DataFrame size: (307511, 7)
DataFrame columns: [
    "WEEKDAY_APPR_PROCESS_START_FRIDAY",
    "WEEKDAY_APPR_PROCESS_START_MONDAY",
    "WEEKDAY_APPR_PROCESS_START_SATURDAY",
    "WEEKDAY_APPR_PROCESS_START_SUNDAY",
    "WEEKDAY_APPR_PROCESS_START_THURSDAY",
    "WEEKDAY_APPR_PROCESS_START_TUESDAY",
    "WEEKDAY_APPR_PROCESS_START_WEDNESDAY"
]
{'fraction_non_alphanumeric': 0.09910714285714285, 'fraction_numerical': 0.01875, 'mean_word_length': 3.4753661784287617, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 14, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '506230', 'license': 'MIT', 'n_tokens_mistral': 1410, 'n_tokens_neox': 1409, 'n_words': 218, 'path': 'homework/Day_004_HW.ipynb', 'repo_name': 'hackerYM/2nd-ML100Days', 'size': '3360'}