<jupyter_start><jupyter_text>Predicting Cab Booking CancellationsNotebook by [<NAME>](http://github.com/numb3r33/)Data hosted at [Kaggle](https://inclass.kaggle.com/c/predicting-cab-booking-cancellations/data) Table of contents1. [Introduction](Introduction)2. [Required libraries](Required-libraries)3. [Step 1: Answering the question](Step-1:-Answering-the-question)4. [Step 2: Checking the data](Step-2:-Checking-the-data)5. [Step 3: Exploratory analysis](Step-3:-Exploratory-analysis)6. [Step 4: Data Preparation](Step-4:-Data-Preparation)7. [Step 5: Feature Extraction](Step-5:-Feature-Extraction)8. [Step 6: Fitting Model](Step-6:-Fitting-Model)    - [Validation](Validation)    - [Learning Curves](Learning-Curves)9. [Step 7: Performance on Test Set](Step-7:-Performance-on-test-set)10. [Step 8: Model Training on full dataset](Step-8:-Model-Training-on-full-dataset)11. [Step 9: Prepare Submission](Step-9:-Prepare-Submission)12. [Further improvements](Further-Improvements) Introduction[[ go back to the top ]](Table-of-contents)The business problem tackled here is trying to improve customer service for [YourCabs](http://www.yourcabs.com/), a cab company in Bangalore. The problem of interest is booking cancellations by the company due to unavailability of a car. The challenge is that cancellations can occur very close to the trip start time, thereby causing passengers inconvenience.Competition GoalThe goal of the competition is to create a predictive model for classifying new bookings as to whether they will eventually get cancelled due to car unavailability. This is a classification task that includes misclassification costs. The winning entry will be the one with the lowest **average-cost-per-booking.**Participants need to upload their classifications (0=no cancellation or 1=cancellation), and the system will compute the average-cost-per-booking based on these classifications. Required libraries[[ go back to the top ]](Table-of-contents)If you don't have Python on your computer, you can use the [Anaconda Python distribution](http://continuum.io/downloads) to install most of the Python packages you need. Anaconda provides a simple double-click installer for your convenience.This notebook uses several Python packages that come standard with the Anaconda Python distribution. The primary libraries that we'll be using are:* **NumPy**: Provides a fast numerical array structure and helper functions.* **pandas**: Provides a DataFrame structure to store data in memory and work with it easily and efficiently.* **scikit-learn**: The essential Machine Learning package in Python.* **matplotlib**: Basic plotting library in Python; most other Python plotting libraries are built on top of it.* **Seaborn**: Advanced statistical plotting library.To make sure you have all of the packages you need, install them with `conda`:    conda install numpy pandas scikit-learn matplotlib seaborn`conda` may ask you to update some of them if you don't have the most recent version. Allow it to do so. Step 1: Answering the question[[ go back to the top ]](Table-of-contents)The first step to any data analysis project is to define the question or problem we're looking to solve, and to define a measure (or set of measures) for our success at solving that task.>Did you specify the type of data analytic question (e.g. exploration, association causality) before touching the data?The goal of the competition is to create a predictive model for classifying new bookings as to whether they will eventually get cancelled due to car unavailability, so this is a classification task.>Did you define the metric for success before beginning?Let's do that now. Since we're taking into account penalties for misclassification, we can use [Weighted Mean Absolute Error](https://www.kaggle.com/wiki/WeightedMeanAbsoluteError)>Did you understand the context for the question and the scientific or business application?The business problem tackled here is trying to improve customer service for YourCabs.com, a cab company in Bangalore. The problem of interest is booking cancellations by the company due to unavailability of a car. The challenge is that cancellations can occur very close to the trip start time, thereby causing passengers inconvenience.>Did you record the experimental design?The cab bookings data are made available through a collaboration between Prof. <NAME> at the Indian School of Business and YourCabs co-founder Mr. <NAME> and IDRC managing partner, Mr. <NAME>.>Did you consider whether the question could be answered with the available data?The data set we currently have is only for three types of *Iris* flowers. The model built off of this data set will only work for those *Iris* flowers, so we will need more data to create a general flower classifier. Step 2: Checking the data[[ go back to the top ]](Table-of-contents)The next step is to look at the data we're working with.Generally, we're looking to answer the following questions:* Is there anything wrong with the data?* Are there any quirks with the data?* Do I need to fix or remove any of the data?Let's start by reading the data into a pandas DataFrame.Data fields* id - booking *ID** user_id - the *ID* of the customer (based on mobile number)* vehicle_model_id - vehicle model type.* package_id - type of package (1=4hrs & 40kms, 2=8hrs & 80kms, 3=6hrs & 60kms, 4= 10hrs &   100kms, 5=5hrs & 50kms, 6=3hrs &     30kms, 7=12hrs & 120kms)* travel_type_id - type of travel (1=long distance, 2= point to point, 3= hourly rental).* from_area_id - unique identifier of area. Applicable only for point-to-point travel and packages* to_area_id - unique identifier of area. Applicable only for point-to-point travel* from_city_id - unique identifier of city* to_city_id - unique identifier of city (only for intercity)* from_date - time stamp of requested trip start* to_date - time stamp of trip end* online_booking - if booking was done on desktop website* mobile_site_booking - if booking was done on mobile website* booking_created - time stamp of booking* from_lat - latitude of from area* from_long -  longitude of from area* to_lat - latitude of to area* to_long - longitude of to area* Car_Cancellation (available only in training data) - whether the booking was cancelled (1) or not (0) due to unavailability of   a car.* Cost_of_error (available only in training data) - the cost incurred if the booking is misclassified. The cost of misclassifying an uncancelled booking as a cancelled booking (cost=1 unit). The cost associated with misclassifying a cancelled booking as uncancelled, This cost is a function of how close the cancellation occurs relative to the trip start time. The closer the trip, the higher the cost. Cancellations occurring less than 15 minutes prior to the trip start incur a fixed penalty of 100 units.<jupyter_code>import pandas as pd
cars_cancel_train = pd.read_csv('./data/Kaggle_YourCabs_training.csv')
cars_cancel_train.head()<jupyter_output><empty_output><jupyter_text>**As you can see there are various columns with missing values and some columns which need to be converted to date-time.**<jupyter_code># convert from_date column to date-time
# Note: when we try to convert NaN to date-time, it gets converted to NaT( Not a timestamp )

cars_cancel_train['from_date'] = pd.to_datetime(cars_cancel_train['from_date'])
# convert to_date column to date-time
cars_cancel_train['to_date'] = pd.to_datetime(cars_cancel_train['to_date'])
# convert booking_created column to date-time
cars_cancel_train['booking_created'] = pd.to_datetime(cars_cancel_train['booking_created'])<jupyter_output><empty_output><jupyter_text>Next thing, we could do is to print out the summary statistics about the dataset<jupyter_code>cars_cancel_train.describe()<jupyter_output><empty_output><jupyter_text>**We can see that many of the columns are missing values like package_id, to_area_id etc.** Let's also set id to be the index of the dataset<jupyter_code>cars_cancel_train.set_index('id', inplace=True)
%matplotlib inline

import matplotlib.pyplot as plt
import seaborn as sb
# lets find out the class balance
cars_cancel_train.Car_Cancellation.value_counts()<jupyter_output><empty_output><jupyter_text>Step-3:-Exploratory-analysis[[ go back to the top ]](Table-of-contents) ** Major class imbalance, very few cancellations as compared to large amount of non-cancellations. **<jupyter_code># Lets see the distribution of package_id
cars_cancel_train.package_id.value_counts()
cars_cancel_train.package_id.hist();<jupyter_output><empty_output><jupyter_text>** Most of the packages that people opt for are for a journey of 4hrs and around 40kms, followed by 8hrs and 80kms.**<jupyter_code># Let's see whether package type affect car cancellations
cars_cancel_train.hist(column='package_id', by='Car_Cancellation');<jupyter_output><empty_output><jupyter_text>**As you can see most of the times 1st package ( 4hrs & 40kms ) gets cancelled followed by packages ( 3hrs & 30kms ) and ( 8hrs & 80kms ).**<jupyter_code># Lets take a look at travel_type variable
cars_cancel_train.hist(column='travel_type_id', by='Car_Cancellation');<jupyter_output><empty_output><jupyter_text>** Not surprisingly, most people rent car from point to point travel and around ( 1 / 10 )th of these bookings get cancelled. **<jupyter_code># Lets take a look at from_area_id variable and see if it has any relation with car cancellation.
g = cars_cancel_train.groupby(['from_area_id', 'Car_Cancellation']).size().unstack(1)
g.head()
g['percent_cancelled'] = (g[1] / (g[1] + g[0])) * 100.
g.percent_cancelled.order(ascending=False).iloc[:20]<jupyter_output><empty_output><jupyter_text>**So as you can see there are certain areas ( from area ) for which more than 50% of the bookings were cancelled.**<jupyter_code>to = cars_cancel_train.groupby(['to_area_id', 'Car_Cancellation']).size().unstack(1)
to.head()
to['percent_cancelled'] = (to[1] / (to[1] + to[0])) * 100.
to.percent_cancelled.order(ascending=False).iloc[:20]
from_to = cars_cancel_train.groupby(['from_area_id', 'to_area_id', 'Car_Cancellation']).size()
f_t = from_to.unstack(2)
f_t.head()
f_t['percent_cancelled'] = (f_t[1] / (f_t[0] + f_t[1])) * 100.
f_t.percent_cancelled.order(ascending=False).iloc[:20]<jupyter_output><empty_output><jupyter_text>** So there are some routes for which the cancellation rate is pretty high, e.g. from area-626 to-122 almost 90% of the bookings were cancelled **<jupyter_code>import numpy as np
# Lets see if online or mobile booking has any effect on cancellation
pd.pivot_table(cars_cancel_train[['online_booking', 'mobile_site_booking', 'Car_Cancellation']]
              , values='Car_Cancellation', index=['online_booking', 'mobile_site_booking'], aggfunc= np.count_nonzero)<jupyter_output><empty_output><jupyter_text>**So most of the cancellations are of orders that were booked online.**<jupyter_code>booking_diff = cars_cancel_train[['from_date', 'to_date', 'booking_created', 'Car_Cancellation']]
booking_diff.loc[:, 'difference'] = booking_diff.loc[:, ('from_date')] - booking_diff.loc[:, ('booking_created')]
booking_diff.head()
def convert_to_days(x):
    seconds = int(x / (1e9))
    days = seconds / ( 24. * 60 * 60 )
    return days

booking_diff.loc[:, 'diff_in_days'] = booking_diff.difference.map(convert_to_days)
booking_diff.head()
booking_diff.hist(column='diff_in_days', by='Car_Cancellation', bins=100);<jupyter_output><empty_output><jupyter_text>** There seems to be no relation between number of days between date of booking and trip's start date with cancellation.Generally people tend to cancel their booking 5 days prior to their trip's start date which is not unusual ** Step-4:-Data-Preparation[[ go back to the top ]](Table-of-contents) ** Following features will be considered for analysis *** vehicle_model_id* package_id* travel_type_id* from_area_id* to_area_id* from_city_id* to_city_id* days_before_booking* online_booking* mobile_site_booking* from_month* from_weekday* booking_month* booking_weekday** Many of them have missing values which have to be dealt accordingly **<jupyter_code>cars_cancel_train.columns
cars_cancel_train.info()
cars_cancel_train.loc[:, 'from_month'] = cars_cancel_train.from_date.dt.month
cars_cancel_train.loc[:, 'from_weekday'] = cars_cancel_train.from_date.dt.weekday

cars_cancel_train.loc[:, 'booking_month'] = cars_cancel_train.booking_created.dt.month
cars_cancel_train.loc[:, 'booking_weekday'] = cars_cancel_train.booking_created.dt.weekday
cars_cancel_train.loc[:, 'days_before_booking'] = booking_diff['diff_in_days']
features_cols = ['vehicle_model_id', 'travel_type_id', 'from_area_id',
                 'to_area_id','from_month', 'from_weekday',
                 'days_before_booking', 'online_booking', 'mobile_site_booking',
                 'booking_month', 'booking_weekday']
features = cars_cancel_train[features_cols]
y = cars_cancel_train.Car_Cancellation
# Add a new level ( -9999 ) for missing values
features.from_area_id.fillna(-9999, inplace=True)
features.to_area_id.fillna(-9999, inplace=True)<jupyter_output><empty_output><jupyter_text>Step-5:-Feature-Extraction[[ go back to the top ]](Table-of-contents)<jupyter_code>from sklearn.cross_validation import StratifiedShuffleSplit

sss = StratifiedShuffleSplit(y, n_iter=2, test_size=0.3)
train_index, test_index = next(iter(sss))

X_train = features.iloc[train_index]
y_train = y.iloc[train_index]

X_test = features.iloc[test_index]
y_test = y.iloc[test_index]
print 'Shape of training and test dataset %s %s ' %(X_train.shape, X_test.shape)
## Take a sample from the training data to do feature selection
sss = StratifiedShuffleSplit(y_train, n_iter=2, test_size=.2)
train_index, test_index = next(iter(sss))

X_train_features = X_train.iloc[test_index]
y_train_features = y_train.iloc[test_index]

X_train_rest = X_train.iloc[train_index]
y_train_rest = y_train.iloc[train_index]
print 'Shape of the training data used for feature selection %s and rest of the dataset %s ' %(X_train_features.shape, X_train_rest.shape)
from sklearn.linear_model import LogisticRegression
from sklearn.cross_validation import StratifiedKFold
from sklearn.feature_selection import RFECV

logreg = LogisticRegression()
rfecv = RFECV(estimator=logreg, step=1, cv=StratifiedKFold(y_train_features, 5), scoring='roc_auc')

rfecv.fit(X_train_features, y_train_features)
print("Optimal number of features : %d" % rfecv.n_features_)
# Plot number of features VS. cross-validation scores
plt.figure()
plt.xlabel("Number of features selected")
plt.ylabel("Cross validation score (nb of correct classifications)")
plt.plot(range(1, len(rfecv.grid_scores_) + 1), rfecv.grid_scores_)
plt.show()
rfecv.support_
features_cols_np = np.array(features_cols)
features_to_consider = features_cols_np[rfecv.support_]
print 'Selected features are %s ' %(features_to_consider)<jupyter_output>Selected features are ['vehicle_model_id' 'travel_type_id' 'from_area_id' 'to_area_id'
 'from_month' 'from_weekday' 'days_before_booking' 'online_booking'
 'mobile_site_booking' 'booking_month' 'booking_weekday']<jupyter_text>Step-6:-Fitting-Model[[ go back to the top ]](Table-of-contents)<jupyter_code>Xt = X_train_rest[features_cols]
yt = y_train_rest
from sklearn.cross_validation import train_test_split

Xtr, Xv, ytr, yv = train_test_split(Xt, yt, test_size=0.3)
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import ExtraTreesClassifier, GradientBoostingClassifier, RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier

logreg = LogisticRegression(C=13., penalty='l1', class_weight='auto')
etr = ExtraTreesClassifier(n_estimators=75, class_weight='auto')
knn = KNeighborsClassifier(weights='distance', n_neighbors=10)
rf = RandomForestClassifier(n_estimators=50)
gbc = GradientBoostingClassifier(subsample=0.8, learning_rate=0.05 , n_estimators=160, random_state=5, max_depth=9, max_leaf_nodes=100)
from sklearn.cross_validation import cross_val_score
sss = StratifiedShuffleSplit(ytr, 5, test_size=.2)

cv_scores = cross_val_score(gbc, Xtr, ytr, scoring='roc_auc', cv=sss)
print 'cross validation scores are %s ' %(cv_scores)
print 'Mean cross validation score %s ' %(cv_scores.mean())
print 'Standard deviation in cv scores %s ' %(cv_scores.std())<jupyter_output>Standard deviation in cv scores 0.00525780070108<jupyter_text>Validation[[ go back to the top ]](Table-of-contents)<jupyter_code>from sklearn.learning_curve import validation_curve
C = np.linspace(0.1, 50, 5)
val_train, val_test = validation_curve(LogisticRegression(penalty='l1', class_weight='auto'),
                                       Xtr, ytr, 'C', C, cv=5, scoring='roc_auc')
def plot_with_err(x, data, **kwargs):
    mu, std = data.mean(1), data.std(1)
    lines = plt.plot(x, mu, '-', **kwargs)
    plt.fill_between(x, mu - std, mu + std, edgecolor='none',
                     facecolor=lines[0].get_color(), alpha=0.2)

plot_with_err(C, val_train, label='training scores')
plot_with_err(C, val_test, label='validation scores')
plt.xlabel('C'); plt.ylabel('ROC AUC score')
plt.legend();<jupyter_output><empty_output><jupyter_text>Learning-Curves[[ go back to the top ]](Table-of-contents)<jupyter_code>from sklearn.learning_curve import learning_curve

def plot_learning_curve(C=13):
    train_sizes = np.linspace(0.05, 1, 10)
    N_train, val_train, val_test = learning_curve(LogisticRegression(C=C, penalty='l2', class_weight='auto'),
                                                  Xtr, ytr, train_sizes, cv=5,
                                                  scoring='roc_auc')
    plot_with_err(N_train, val_train, label='training scores')
    plot_with_err(N_train, val_test, label='validation scores')
    plt.xlabel('Training Set Size'); plt.ylabel('ROC AUC Score')
    plt.legend()
plot_learning_curve()<jupyter_output><empty_output><jupyter_text>Step-7:-Performance-on-test-set[[ go back to the top ]](Table-of-contents)<jupyter_code># knn.fit(Xtr, ytr)
# logreg.fit(Xtr, ytr)
gbc.fit(Xtr, ytr)
# rf.fit(Xtr, ytr)
# ytest_pred_knn = knn.predict(Xv)
# ytest_pred_logreg = logreg.predict_proba(Xv)[:, 1]
ytest_pred_gbc = gbc.predict_proba(Xv)[:, 1]
ytest_preds_threshold = [1 if pred > 0.10 else 0 for pred in ytest_pred_gbc]
# ytest_pred_rf = rf.predict(Xv)
from sklearn.metrics import roc_auc_score
print 'ROC AUC score on the test dataset ', roc_auc_score(yv, ytest_preds_threshold)<jupyter_output>ROC AUC score on the test dataset  0.746771205593<jupyter_text>Step-8:-Model-Training-on-full-dataset[[ go back to the top ]](Table-of-contents)<jupyter_code>final_features = features[features_cols]
# logreg.fit(final_features, y)
# knn.fit(final_features, y)
gbc.fit(final_features, y)<jupyter_output><empty_output><jupyter_text>Step-9:-Prepare-Submission[[ go back to the top ]](Table-of-contents)<jupyter_code>cars_cancel_test = pd.read_csv('./data/Kaggle_YourCabs_score.csv', index_col='id')
cars_cancel_test.head()
cars_cancel_test['from_date'] = pd.to_datetime(cars_cancel_test['from_date'])
cars_cancel_test['to_date'] = pd.to_datetime(cars_cancel_test['to_date'])
cars_cancel_test['booking_created'] = pd.to_datetime(cars_cancel_test['booking_created'])
cars_cancel_test.loc[:, 'difference'] = cars_cancel_test.loc[:, ('from_date')] - cars_cancel_test.loc[:, ('booking_created')]
cars_cancel_test.loc[:, 'from_month'] = cars_cancel_test.from_date.dt.month
cars_cancel_test.loc[:, 'from_weekday'] = cars_cancel_test.from_date.dt.weekday

cars_cancel_test.loc[:, 'booking_month'] = cars_cancel_test.booking_created.dt.month
cars_cancel_test.loc[:, 'booking_weekday'] = cars_cancel_test.booking_created.dt.weekday
cars_cancel_test.loc[:, 'days_before_booking'] = cars_cancel_test.difference.map(convert_to_days)
features_test = cars_cancel_test[features_cols]
features_test.info()
features_test.from_area_id.fillna(-9999, inplace=True)
features_test.to_area_id.fillna(-9999, inplace=True)
# preds_logreg = logreg.predict(features_test)
# preds_knn = knn.predict(features_test)
preds_gbc = gbc.predict_proba(features_test)[:, 1]
preds_threshold = [1 if pred > 0.10 else 0 for pred in preds_gbc]
submission = pd.DataFrame({'id': features_test.index.values, 'Car_Cancellation': preds_threshold})
submission.to_csv('./submissions/GBC_threshold.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10251107828655835, 'fraction_numerical': 0.013589364844903988, 'mean_word_length': 6.104232249038125, 'pattern_counts': {'":': 0, '<': 78, '<?xml version=': 0, '>': 85, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '177809', 'license': 'MIT', 'n_tokens_mistral': 6722, 'n_tokens_neox': 6387, 'n_words': 1919, 'path': 'Kaggle-Competitions/cars-cancellation/cars_cancellation.ipynb', 'repo_name': 'gopala-kr/ds-notebooks', 'size': '20339'}