<jupyter_start><jupyter_text>Load DataSet<jupyter_code>train_df = pd.read_csv("Train_Processed_4.23.csv", nrows = 9750645)
train_df.dtypes<jupyter_output><empty_output><jupyter_text>Data Exploration<jupyter_code>print(train_df.isnull().sum())
import matplotlib.pyplot as plt
# Plot variables using only 1000 rows for efficiency
train_df.iloc[:1000].plot.scatter('pickup_longitude', 'pickup_latitude')
train_df.iloc[:1000].plot.scatter('dropoff_longitude', 'dropoff_latitude')
plt.show()
# Get distribution of values
train_df.describe()<jupyter_output><empty_output><jupyter_text>Feature engineeringAdd some interesting features in the dataset:total_distance: distance from pickup to dropoff. The longer the trip, the more expensive.Extract information from datetime( day of week, month, hour, day). Taxi fares change day/ night or on weekdays/ holidays.<jupyter_code>def sphere_dist(pickup_lat, pickup_lon, dropoff_lat, dropoff_lon):
    """
    Return distance along great radius between pickup and dropoff coordinates
    """
    # Define earth radius (km)
    R_earth = 6371
    # Convert degrees to radians
    pickup_lat, pickup_lon, dropoff_lat, dropoff_lon = map(np.radians, 
                                                           [pickup_lat, pickup_lon, 
                                                            dropoff_lat, dropoff_lon])
    # Compute distance along lat, lon dimensions
    dlat = dropoff_lat - pickup_lat
    dlon = dropoff_lon - pickup_lon
    
    # Compute haversine distance
    a = np.sin(dlat / 2.0) ** 2 + np.cos(pickup_lat) * np.cos(dropoff_lat) * np.sin(dlon / 2.0) ** 2
    return 2 * R_earth * np.arcsin(np.sqrt(a))
    

def add_datetime_info(dataset):
    #Convert to datetime format
    dataset['pickup_datetime'] = pd.to_datetime(dataset['pickup_datetime'],format="%Y-%m-%d %H:%M:%S UTC")
    
    dataset['hour'] = dataset.pickup_datetime.dt.hour
    dataset['day'] = dataset.pickup_datetime.dt.day
    dataset['month'] = dataset.pickup_datetime.dt.month
    dataset['weekday'] = dataset.pickup_datetime.dt.weekday
    dataset['year'] = dataset.pickup_datetime.dt.year
    
    return dataset
train_df = add_datetime_info(train_df)
train_df['distance'] = sphere_dist(train_df['pickup_latitude'], train_df['pickup_longitude'], 
                                   train_df['dropoff_latitude'] , train_df['dropoff_longitude'])

train_df.head() 
train_df.drop(columns=['Unnamed: 0', 'key', 'pickup_datetime', 'abs_diff_longitude', 'abs_diff_latitude'], inplace=True)
train_df.head()<jupyter_output><empty_output><jupyter_text>Model trainingNow that we have the dataframe that we wanted we can start to train the XGBoost model. First we will split the dataset into train(99%) and test(1%). With this amount of data 1% should be enough to test performance.<jupyter_code>y = train_df['fare_amount']
train = train_df.drop(columns=['fare_amount'])

x_train, x_test, y_train, y_test = train_test_split(train, y, random_state=0, test_size=0.01)
%%time
XGB_model = XGBRegressor(learning_rate=0.3, max_depth=6, n_jobs=-1, silent=False)
XGB_model.fit(x_train, y_train)
%%time
y_XGB_predict = XGB_model.predict(x_test)

XGB_model_error = np.sqrt(mean_squared_error(y_test, y_XGB_predict))

print('XGBoost Root Mean Squared Error - {XGB_model_error}')
print(XGB_model_error)<jupyter_output>XGBoost Root Mean Squared Error - {XGB_model_error}
3.660760959812958
CPU times: user 646 ms, sys: 10.7 ms, total: 657 ms
Wall time: 683 ms<jupyter_text>PredictionFinally we can use our trained model to predict the submission. First we will need to load and preprocess the test dataset just like we did for the training dataset.<jupyter_code># Read and preprocess test set
test_df = pd.read_csv('test_copy.csv')
test_df = add_datetime_info(test_df)
test_df['distance'] = sphere_dist(test_df['pickup_latitude'], test_df['pickup_longitude'], 
                                   test_df['dropoff_latitude'] , test_df['dropoff_longitude'])

test_key = test_df['key']
x_pred = test_df.drop(columns=['key', 'pickup_datetime'])

#Predict from test set
prediction = XGB_model.predict(x_pred)
#Create submission file
import pickle


print(prediction)

#XGB_model.save_model('alldata_allfeatures.model')
#np.savetxt('pre.txt',prediction)

pkl_filename = "pickle_model.pkl"  
with open(pkl_filename, 'wb') as file:  
    pickle.dump(XGB_model, file)
    
with open(pkl_filename, 'rb') as file:
    pickle_model = pickle.load(file)

pre1 = pickle_model.predict(x_pred)
print(pre1)

submission = pd.DataFrame({
        "key": test_key,
        "fare_amount": prediction.round(2)
})
submission = submission[['key', 'fare_amount']]
submission.to_csv('zhibindata_taxi_fare_submission.csv',index=False)
submission.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11307271963009669, 'fraction_numerical': 0.016183270281630937, 'mean_word_length': 4.685782556750299, 'pattern_counts': {'":': 2, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '339947', 'license': 'MIT', 'n_tokens_mistral': 1678, 'n_tokens_neox': 1603, 'n_words': 349, 'path': 'src/Xgboost_With_ZhibinData.ipynb', 'repo_name': 'qzhizhou/New-York-Taxi-Fare-Prediction', 'size': '4758'}