<jupyter_start><jupyter_text>New York City Taxi Trip Duration 02 - Machine Learning     목표 : 데이터를 가지고 택시 주행거리 예측 모델 구축        1. 데이터 전처리<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

train = pd.read_csv("train.csv", parse_dates = ['pickup_datetime', 'dropoff_datetime'])
test = pd.read_csv("test.csv", parse_dates = ['pickup_datetime'])
submission = pd.read_csv("sample_submission.csv")

train['pickup_year'] = train['pickup_datetime'].dt.year  
train['pickup_month'] = train['pickup_datetime'].dt.month
train['pickup_day'] = train['pickup_datetime'].dt.day
train['pickup_hour'] = train['pickup_datetime'].dt.hour
train['pickup_minute'] = train['pickup_datetime'].dt.minute
train['pickup_second'] = train['pickup_datetime'].dt.second
train["pickup_dayofweek"] = train["pickup_datetime"].dt.dayofweek 

test['pickup_year'] = test['pickup_datetime'].dt.year  
test['pickup_month'] = test['pickup_datetime'].dt.month
test['pickup_day'] = test['pickup_datetime'].dt.day
test['pickup_hour'] = test['pickup_datetime'].dt.hour
test['pickup_minute'] = test['pickup_datetime'].dt.minute
test['pickup_second'] = test['pickup_datetime'].dt.second
test["pickup_dayofweek"] = test["pickup_datetime"].dt.dayofweek 


train['store_and_fwd_flag_Y'] = train['store_and_fwd_flag'] == 'Y'
train['store_and_fwd_flag_N'] = train['store_and_fwd_flag'] == 'N'

test['store_and_fwd_flag_Y'] = test['store_and_fwd_flag'] == 'Y'
test['store_and_fwd_flag_N'] = test['store_and_fwd_flag'] == 'N'


train['vendor_1'] = train['vendor_id'] == 1
train['vendor_2'] = train['vendor_id'] == 2

test['vendor_1'] = test['vendor_id'] == 1
test['vendor_2'] = test['vendor_id'] == 2


train['passenger_0'] = train['passenger_count'] == 0
train['passenger_1'] = train['passenger_count'] == 1
train['passenger_2'] = train['passenger_count'] == 2
train['passenger_3'] = train['passenger_count'] == 3
train['passenger_4'] = train['passenger_count'] == 4
train['passenger_5'] = train['passenger_count'] == 5
train['passenger_6'] = train['passenger_count'] == 6
train['passenger_7'] = train['passenger_count'] == 7
train['passenger_8'] = train['passenger_count'] == 8
train['passenger_9'] = train['passenger_count'] == 9

test['passenger_0'] = test['passenger_count'] == 0
test['passenger_1'] = test['passenger_count'] == 1
test['passenger_2'] = test['passenger_count'] == 2
test['passenger_3'] = test['passenger_count'] == 3
test['passenger_4'] = test['passenger_count'] == 4
test['passenger_5'] = test['passenger_count'] == 5
test['passenger_6'] = test['passenger_count'] == 6
test['passenger_7'] = test['passenger_count'] == 7
test['passenger_8'] = test['passenger_count'] == 8
test['passenger_9'] = test['passenger_count'] == 9

print("완료!")<jupyter_output>완료!<jupyter_text>2. 변수 선택 및 데이터 조정  1) 변수 선택<jupyter_code>print(train.shape)
train.columns
print(test.shape)
test.columns<jupyter_output>(625134, 30)<jupyter_text>모델링에 필요한 변수들 선택<jupyter_code>feature_names = [ 'pickup_longitude', 'pickup_latitude',  
                  'store_and_fwd_flag_Y', 'store_and_fwd_flag_N',
                  'pickup_year', 'pickup_month', 'pickup_day',
                  'pickup_hour', 'pickup_minute', 'pickup_second', 'pickup_dayofweek',
                  'vendor_id', 'vendor_1', 'vendor_2',
                  'passenger_count', 'passenger_0', 'passenger_1', 'passenger_2', 'passenger_3',
                  'passenger_4', 'passenger_5', 'passenger_6', 'passenger_7', 'passenger_8', 
                  'passenger_9']<jupyter_output><empty_output><jupyter_text>'id'는 단순 구분을 위한 값이고,  'pickup_datetime'의 경우 날짜형태라 제외 'dropoff_longitude', 'dropoff_latitude', 'dropoff_datetime'는 test 데이터에 없음 'store_and_fwd_flag'의 경우 Y와 N이 문자열이기 때문에 계산을 못함. 원핫인코딩만 사용 'trip_duration'은 목표값이니 제외<jupyter_code>basic_features : 모든 변수 포함
feature_names =['pickup_longitude', 'pickup_latitude',  
                  'store_and_fwd_flag_Y', 'store_and_fwd_flag_N',
                  'pickup_year', 'pickup_month', 'pickup_day',
                  'pickup_hour', 'pickup_minute', 'pickup_second', 'pickup_dayofweek',
                  'vendor_id', 'vendor_1', 'vendor_2',
                  'passenger_count', 'passenger_0', 'passenger_1', 'passenger_2', 'passenger_3',
                  'passenger_4', 'passenger_5', 'passenger_6', 'passenger_7', 'passenger_8', 
                  'passenger_9']

feautres_1 : vendor_id와 passenger_count의 경우 원핫인코딩 제외
feautres_1 = ['pickup_longitude', 'pickup_latitude',  
              'store_and_fwd_flag_Y', 'store_and_fwd_flag_N',
              'pickup_year', 'pickup_month', 'pickup_day',
              'pickup_hour', 'pickup_minute', 'pickup_second', 'pickup_dayofweek',
              'vendor_id',
              'passenger_count']
    
feautres_2 : vendor_id와 passenger_count의 경우 원핫인코딩만 사용
feautres_2 = ['pickup_longitude', 'pickup_latitude',  
              'store_and_fwd_flag_Y', 'store_and_fwd_flag_N',
              'pickup_year', 'pickup_month', 'pickup_day',
              'pickup_hour', 'pickup_minute', 'pickup_second', 'pickup_dayofweek',
              'vendor_1', 'vendor_2',
              'passenger_0', 'passenger_1', 'passenger_2', 'passenger_3',
              'passenger_4', 'passenger_5', 'passenger_6', 'passenger_7', 'passenger_8', 
              'passenger_9']
    
feautres_3 :  초 제외 /vendor_id와 passenger_count의 경우 원핫인코딩만 사용
feautres_3 = ['pickup_longitude', 'pickup_latitude',  
              'store_and_fwd_flag_Y', 'store_and_fwd_flag_N',
              'pickup_year', 'pickup_month', 'pickup_day',
              'pickup_hour', 'pickup_minute', 'pickup_dayofweek',
              'vendor_1', 'vendor_2',
              'passenger_0', 'passenger_1', 'passenger_2', 'passenger_3',
              'passenger_4', 'passenger_5', 'passenger_6', 'passenger_7', 'passenger_8', 
              'passenger_9']

feautres_4 :  날짜 제외 / vendor_id와 passenger_count의 경우 원핫인코딩만 사용
feautres_4 = ['pickup_longitude', 'pickup_latitude',  
              'store_and_fwd_flag_Y', 'store_and_fwd_flag_N',
              'pickup_hour', 'pickup_minute', 'pickup_dayofweek',
              'vendor_1', 'vendor_2',
              'passenger_0', 'passenger_1', 'passenger_2', 'passenger_3',
              'passenger_4', 'passenger_5', 'passenger_6', 'passenger_7', 'passenger_8', 
              'passenger_9']
feature_names =['pickup_longitude', 'pickup_latitude',  
                  'store_and_fwd_flag_Y', 'store_and_fwd_flag_N',
                  'pickup_year', 'pickup_month', 'pickup_day',
                  'pickup_hour', 'pickup_minute', 'pickup_second', 'pickup_dayofweek',
                  'vendor_id', 'vendor_1', 'vendor_2',
                  'passenger_count', 'passenger_0', 'passenger_1', 'passenger_2', 'passenger_3',
                  'passenger_4', 'passenger_5', 'passenger_6', 'passenger_7', 'passenger_8', 
                  'passenger_9']  # basic

# RF<jupyter_output><empty_output><jupyter_text>2) 데이터 조정<jupyter_code>x_train = train[feature_names]
print(x_train.shape)
x_train.head()
x_test = test[feature_names]
print(x_test.shape)
x_test.head()
label_name = "trip_duration"

y_train = train[label_name]
print(y_train.shape)
y_train.head()
y_train = np.log1p(y_train)
y_train.head()<jupyter_output><empty_output><jupyter_text>3. Linear Regression  기본<jupyter_code>from sklearn import linear_model

LR_model = linear_model.LinearRegression()
LR_model.fit(x_train, y_train)

predictions = LR_model.predict(x_test)
predictions = np.expm1(predictions)
predictions[0:10]<jupyter_output><empty_output><jupyter_text>hyperparameters 조정<jupyter_code>from sklearn import linear_model
LR_model = linear_model.LinearRegression()
LR_model
from sklearn import linear_model
LR_model = linear_model.LinearRegression(fit_intercept=False)
LR_model.fit(x_train, y_train)

predictions = LR_model.predict(x_test)
predictions = np.expm1(predictions)
predictions[0:10]<jupyter_output><empty_output><jupyter_text>제출<jupyter_code>submission['trip_duration'] = predictions
submission.to_csv('LR8.csv', index = False)<jupyter_output><empty_output><jupyter_text>결과보고서<jupyter_code>LR / basic_features : 0.77677
LR / features_1     : 0.77714
LR / features_2     : 0.77677
LR / features_3     : 0.77677
LR / features_4     : 0.77762
    
LR / basic_features / normalize = True : 0.95757
LR / basic_features / normalize=True  fit_intercept=False : 0.77677
LR / basic_features / fit_intercept=False : 0.77677<jupyter_output><empty_output><jupyter_text>4. Decision Tree Regressor  기본<jupyter_code>from sklearn.tree import DecisionTreeRegressor

DT_model = DecisionTreeRegressor()
DT_model.fit(x_train, y_train)

predictions = DT_model.predict(x_test)
predictions = np.expm1(predictions)
predictions[0:10]<jupyter_output><empty_output><jupyter_text>hyperparameters 조정<jupyter_code>from sklearn.tree import DecisionTreeRegressor

DT_model = DecisionTreeRegressor()
DT_model
from sklearn.tree import DecisionTreeRegressor

DT_model = DecisionTreeRegressor(criterion='mae',
                                 max_depth = 10, random_state = 50)     # 12시 35분 시작
DT_model.fit(x_train, y_train)

predictions = DT_model.predict(x_test)
predictions = np.expm1(predictions)
predictions[0:10]<jupyter_output><empty_output><jupyter_text>제출<jupyter_code>submission['trip_duration'] = predictions
submission.to_csv('DC19.csv', index = False)<jupyter_output><empty_output><jupyter_text>결과보고서<jupyter_code>DT / basic_features : 1.07356
DT / features_1     : 1.07007
DT / features_2     : 1.07381
DT / features_3     : 1.07070
DT / features_4     : 1.06416
    
DT / basic_features / max_depth = 12 random_state = 50 : 0.74603
DT / basic_features / max_depth = 10 random_state = 50 : 0.74505
DT / basic_features / max_depth = 8 random_state = 50 :  0.74690
DT / basic_features / max_depth = 9 random_state = 50 :  0.74551

DT / features_4  / max_depth = 10 random_state = 50 : 0.74460
DT / features_4  / max_depth = 10 random_state = 50 max_features = 'auto' :  0.74460
DT / features_4  / max_depth = 10 random_state = 50 max_features = 'sqrt' :  0.75377
DT / features_4  / max_depth = 10 random_state = 50 max_features = 'log2' :  0.75377
    
DT / features_4  / max_depth = 10 random_state = 50 : 0.74460 criterion='friedman_mse'  : 0.75377   
DT / features_4  / max_depth = 10 random_state = 50 : 0.74460 criterion='mae'  : 너무 오래걸려 실패<jupyter_output><empty_output><jupyter_text>5. Random Forest Regressor  기본<jupyter_code>from sklearn.ensemble import RandomForestRegressor

RF_model = RandomForestRegressor()
RF_model.fit(x_train, y_train)

predictions = RF_model.predict(x_test)
predictions = np.expm1(predictions)
predictions[0:10]<jupyter_output><empty_output><jupyter_text>hyperparameters 조정<jupyter_code>from sklearn.ensemble import RandomForestRegressor

RF_model = RandomForestRegressor()
RF_model
import time
start = time.time()

from sklearn.ensemble import RandomForestRegressor

RF_model = RandomForestRegressor(n_estimators = 80, random_state =  50, max_depth = 22, max_features = 12)

RF_model.fit(x_train, y_train)

predictions = RF_model.predict(x_test)
predictions = np.expm1(predictions)


print('Running Time : %.02f초' % (time.time() - start))
predictions[0:10]<jupyter_output>Running Time : 492.23초<jupyter_text>제출<jupyter_code>submission['trip_duration'] = predictions
submission.to_csv('RF27.csv', index = False)<jupyter_output><empty_output><jupyter_text>결과보고서<jupyter_code>RF / basic_features : 0.77799
RF / features_1     : 0.77918
RF / features_2     : 0.77863
RF / features_3     : 0.77942
RF / features_4     : 0.78734
    
RF / basic_features / random_state =  50 : 0.77764
RF / basic_features / n_estimators = 100 random_state =  50 : 0.74300
RF / basic_features / n_estimators = 100 random_state =  50 max_depth = 20 max_features = 8 : 0.73541
RF / basic_features / n_estimators = 100 random_state =  50 max_depth = 20 max_features = 10 : 0.73423
RF / basic_features / n_estimators = 100 random_state =  50 max_depth = 30 max_features = 10 : 0.73765   
RF / basic_features / n_estimators = 100 random_state =  50 max_depth = 20 max_features = 12 : 0.73380 ☆
RF / basic_features / n_estimators = 100 random_state =  40 max_depth = 20 max_features = 12 : 0.73385
RF / basic_features / n_estimators = 100 random_state =  60 max_depth = 20 max_features = 12 : 0.73388

    
RF / basic_features / n_estimators = 200 random_state =  50 : 0.74087
RF / basic_features / n_estimators = 200 random_state =  50 max_depth = 20 max_features = 10 : 0.73379 
    
RF / basic_features / n_estimators = 300 random_state =  50 : 메모리 오류
    
RF / basic_features / n_estimators = 50 random_state =  50 max_depth = 20 max_features = 12 : 0.73416  
RF / basic_features / n_estimators = 75 random_state =  50 max_depth = 20 max_features = 12 : 0.73359 ☆ 
RF / basic_features / n_estimators = 75 random_state =  50 max_depth = 20 max_features = 13 : 0.73382
RF / basic_features / n_estimators = 75 random_state =  50 max_depth = 20 max_features = 11 : 0.73446 
RF / basic_features / n_estimators = 75 : 0.74591

RF / basic_features / n_estimators = 70 random_state =  50 max_depth = 20 max_features = 12 : 0.73417
RF / basic_features / n_estimators = 80 random_state =  50 max_depth = 20 max_features = 12 : 0.73403
RF / basic_features / n_estimators = 75 random_state =  50 max_depth = 15 max_features = 12 : 0.73583
RF / basic_features / n_estimators = 75 random_state =  50 max_depth = 22 max_features = 12 : 0.73425<jupyter_output><empty_output><jupyter_text>6. Gradient Boost Regressor  기본<jupyter_code>from sklearn.ensemble import GradientBoostingRegressor

GB_model = GradientBoostingRegressor()
GB_model.fit(x_train, y_train)

predictions = GB_model.predict(x_test)
predcitions = np.expm1(predictions)
predictions[0:10]<jupyter_output><empty_output><jupyter_text>hyperparameters 조정<jupyter_code>from sklearn.ensemble import GradientBoostingRegressor

GB_model = GradientBoostingRegressor()
GB_model
import time
start = time.time()

from sklearn.ensemble import GradientBoostingRegressor

GB_model = GradientBoostingRegressor(n_estimators = 550, random_state =  50)
GB_model.fit(x_train, y_train)

predictions = GB_model.predict(x_test)
predcitions = np.expm1(predictions)

print('Running Time : %.02f초' % (time.time() - start))
predictions[0:10]<jupyter_output>Running Time : 1045.19초<jupyter_text>제출<jupyter_code>submission['trip_duration'] = predictions
submission.to_csv('GB5.csv', index = False)<jupyter_output><empty_output><jupyter_text>결과보고서<jupyter_code>GB / basic_features : 4.52659
GB / features_1     : 4.52659
    
GB / basic_features / n_estimators = 750 random_state =  50 : 4.52634
GB / basic_features / n_estimators = 550 random_state =  50 : 4.52636<jupyter_output><empty_output><jupyter_text>7. XGBoost Regressor   기본<jupyter_code>import xgboost as xgb

XGB_model = xgb.XGBRegressor()
XGB_model.fit(x_train, y_train)

predictions = XGB_model.predict(x_test)
predictions = np.expm1(predictions)
predictions[0:10]<jupyter_output><empty_output><jupyter_text>hyperparameters 조정<jupyter_code>import xgboost as xgb

XGB_model = xgb.XGBRegressor()
XGB_model
import time
start = time.time()

import xgboost as xgb

XGB_model = xgb.XGBRegressor(max_depth = 12, n_estimators = 300, min_child_weight = 8, gamma = 0.4,
                             colsample_bytree = 0.6, nthread = 5, subsample = 0.9, seed = 24, learning_rate = 0.05 )

XGB_model.fit(x_train, y_train)

predictions = XGB_model.predict(x_test)
predictions = np.expm1(predictions)

print('Running Time : %.02f초' % (time.time() - start))
predictions[0:10]<jupyter_output>C:\Users\Jeong\Anaconda3\lib\site-packages\sklearn\cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)<jupyter_text>제출<jupyter_code>submission['trip_duration'] = predictions
submission.to_csv('XGB43.csv', index = False)<jupyter_output><empty_output><jupyter_text>결과보고서<jupyter_code>XGB / basic_features : 0.74462
XGB / features_1     : 0.74462
XGB / features_2     : 0.74448 ☆
XGB / features_3     : 0.74448 ☆
XGB / features_4     : 0.74536

    
XGB / feautres_2 / max_depth = 12, n_estimators = 2000, min_child_weight = 8, gamma = 0.4,
colsample_bytree = 0.6, nthread = 5, subsample = 0.6, seed = 25, learning_rate = 0.05       : 0.73437
XGB / feautres_2 / max_depth = 12, n_estimators = 1500, min_child_weight = 8, gamma = 0.4,
colsample_bytree = 0.6, nthread = 5, subsample = 0.6, seed = 25, learning_rate = 0.05       : 0.73231 
XGB / feautres_2 / n_estimators = 1500, min_child_weight = 8, gamma = 0.4,
colsample_bytree = 0.6, nthread = 5, subsample = 0.6, seed = 25, learning_rate = 0.05       : 0.73822
XGB / feautres_2 / max_depth = 14, n_estimators = 1500, min_child_weight = 8, gamma = 0.4,
colsample_bytree = 0.6, nthread = 5, subsample = 0.6, seed = 25, learning_rate = 0.05       : 0.73705
    
# n_estimators만 조정 
max_depth = 12, min_child_weight = 8, gamma = 0.4, learning_rate = 0.05
colsample_bytree = 0.6, nthread = 5, subsample = 0.6, seed = 25

XGB / feautres_3 / n_estimators = 1500 : 0.73231
XGB / feautres_3 / n_estimators = 1400 : 0.73093
XGB / feautres_3 / n_estimators = 1200 : 0.73031
XGB / feautres_3 / n_estimators = 1000 : 0.72973
XGB / feautres_3 / n_estimators = 500  : 0.72891   
XGB / feautres_3 / n_estimators = 300  : 0.72880 
XGB / feautres_3 / n_estimators = 100  : 0.73256    
XGB / feautres_3 / n_estimators = 200  : 0.72922 
XGB / basic_features / n_estimators = 300  : 0.72880
    
# learning_rate만 조정
max_depth = 12, n_estimators = 300, min_child_weight = 8, gamma = 0.4,
colsample_bytree = 0.6, nthread = 5, subsample = 0.6, seed = 25

XGB / feautres_3 / learning_rate = 0.01       : 0.79178
XGB / feautres_3 / learning_rate = 0.1        : 0.73046  
    
# subsample 만 조정
max_depth = 12, n_estimators = 300, min_child_weight = 8, gamma = 0.4,
colsample_bytree = 0.6, nthread = 5,  seed = 25, learning_rate = 0.05    

XGB / basic_features / subsample = 0.5 : 0.72937
XGB / basic_features / subsample = 0.7 : 0.72841
XGB / basic_features / subsample = 0.8 : 0.72802 
XGB / basic_features / subsample = 1 : 0.72814    
XGB / basic_features / subsample = 0.9 : 0.72789 ☆
    
# colsample_bytree 만 조정
max_depth = 12, n_estimators = 300, min_child_weight = 8, gamma = 0.4,
nthread = 5, subsample = 0.9, seed = 25, learning_rate = 0.05

XGB / basic_features / colsample_bytree = 0.7 : 0.72822
XGB / basic_features / colsample_bytree = 0.5 : 0.72881

# gamma 만 조정
max_depth = 12, n_estimators = 300, min_child_weight = 8, 
colsample_bytree = 0.6, nthread = 5, subsample = 0.9, seed = 25, learning_rate = 0.05

XGB / basic_features / gamma = 0.5 : 0.72794
XGB / basic_features / gamma = 0.3 : 0.72791
    
# min_child_weight 만 조정
max_depth = 12, n_estimators = 300, gamma = 0.4, 
colsample_bytree = 0.6, nthread = 5, subsample = 0.9, seed = 25, learning_rate = 0.05

XGB / basic_features / min_child_weight = 7 : 0.72797
XGB / basic_features / min_child_weight = 9 : 0.72810
    
# seed만 조정
    
max_depth = 12, n_estimators = 300, min_child_weight = 8, gamma = 0.4,
colsample_bytree = 0.6, nthread = 5, subsample = 0.9, learning_rate = 0.05 

XGB / basic_features / seed = 30 : 0.72804
XGB / basic_features / seed = 20 : 0.72801
XGB / basic_features / seed = 26 : 0.72800
XGB / basic_features / seed = 24 : 0.72806   
    
0.72789 ☆☆☆<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12073289318798301, 'fraction_numerical': 0.06504938840268182, 'mean_word_length': 4.279654147527695, 'pattern_counts': {'":': 0, '<': 96, '<?xml version=': 0, '>': 96, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '336983', 'license': 'Apache-2.0', 'n_tokens_mistral': 8344, 'n_tokens_neox': 7708, 'n_words': 1442, 'path': 'New York City Taxi Trip Duration/New York City Taxi Trip Duration 02 -  Machine Learning.ipynb', 'repo_name': 'todagi/DataScience_with_Kaggle', 'size': '19539'}