<jupyter_start><jupyter_text>**Introduction**Exploratory Data Analysis of Sendy Logistics Challenge by EXPLORE Data Science AcademyLogistics in Sub-Saharan Africa increases the cost of manufactured goods by up to 320%; while in Europe, it only accounts for up to 90% of the manufacturing cost. Economies are better when logistics is efficient and affordable.Sendy is hosting a Zindi challenge to predict the estimated time of delivery of orders, from the point of driver pickup to the point of arrival at final destination.The solution will improve customer experience. The solution will need to enable Sendy to realise cost savings, and ultimately reduce the cost of doing business, through improved resource management and planning for order scheduling.The analysis will assist in understanding the data and get the insights about the variables that affect the time taken from pick up to arrival, making better predicts. **Body****1. Exploratory Data Analysis (EDA)****1.1. Importing libraries**<jupyter_code>pip install reverse_geocoder catboost
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
import warnings
import requests
import reverse_geocoder as rg
np.random.seed(123)
warnings.filterwarnings('ignore')
%matplotlib inline<jupyter_output><empty_output><jupyter_text>**1.2. Uploading the dataset files**<jupyter_code>train_1 = pd.read_csv("Train.csv")
test_1 = pd.read_csv("Test.csv")
riders_1 = pd.read_csv("Riders.csv")
train_df = train_1.copy()
train_df.head()<jupyter_output><empty_output><jupyter_text>The dataset have the columns Vehicle Type, User Id and Personal or Business, Precipitation in millimeters which we believe will not have so much impact on the estimated time from pick up to arrival. Out of the remaining variables, Order No, User Id are categorical variables and the remaining variables i.e. Placement - Day of Month, Arrival at Destination - Time, Distance (KM) and Temperature are numerical variables.<jupyter_code>mean_t = train_df['Temperature'].mean()
train_df['Temperature'] = train_df['Temperature'].fillna(mean_t)
mean_p = train_df['Precipitation in millimeters'].mean()
train_df['Precipitation in millimeters'] = train_df['Precipitation in millimeters'].fillna(mean_p)
train_df["Rider Id"] = train_df["Rider Id"].str.get(-1)
train_df["Order No"] = train_df["Order No"].str.split("_").str.get(-1)
train_df["User Id"] = train_df["User Id"].str.split("_").str.get(-1)
train_df = train_df.set_index(keys = ["Order No"])
dataset = train_df[["Platform Type", "Placement - Day of Month", "Placement - Weekday (Mo = 1)", "Confirmation - Day of Month","Confirmation - Weekday (Mo = 1)", "Arrival at Pickup - Day of Month","Arrival at Pickup - Weekday (Mo = 1)", "Pickup - Day of Month","Pickup - Weekday (Mo = 1)", "Distance (KM)", "Temperature", "Time from Pickup to Arrival"]]
dataset
dataset.describe().T<jupyter_output><empty_output><jupyter_text>**1.3. Checking the mathematical relationships**We firsty check linearity which is the mathematical relationship between each predictor variable and the response variable.> Indented blockIf we fit the regression model to a dataset that is non-linear, it will fail to adequately capture the relationship in the data - resulting in a mathematically inappropriate model. In order to check for linearity, we can produce scatter plots of each individual predictor against the response variable. The intuition here is that we are looking for obvious linear relationships.<jupyter_code>fig, axs = plt.subplots(2,6, figsize=(14,6),)
fig.subplots_adjust(hspace = 0.5, wspace=.2)
axs = axs.ravel()

for index, column in enumerate(dataset.columns):
    axs[index-1].set_title("{} vs. Arrival".format(column),fontsize=8)
    axs[index-1].scatter(x=dataset[column],y=dataset['Time from Pickup to Arrival'],color='blue',edgecolor='k')

fig.tight_layout(pad=3)<jupyter_output><empty_output><jupyter_text>It appears that at least half of the variables have an approximately linear relationship. However, Temperature exhibit no linearity with Time from Pickup to Arrival.<jupyter_code>from seaborn import pairplot
g = pairplot(dataset.drop('Time from Pickup to Arrival', axis='columns'))
g.fig.set_size_inches(11,11)
corr = dataset.drop('Time from Pickup to Arrival', axis='columns').corr()
from statsmodels.graphics.correlation import plot_corr
fig=plot_corr(corr,xnames=corr.columns)<jupyter_output><empty_output><jupyter_text>From both the pairwise scatterplot and correlation heatmap, we can see a number of strong correlations among predictors:    Placement - Day of Month and Arrival at Pickup - Day of Month;    Arrival at Pickup - Day of Month and Pickup - Day of Month;    Confirmation - Day of Month and Pickup - Day of Month;We'll keep these in mind when we build and continue to check the quality of our model. **1.4. OLS Regression**<jupyter_code>import statsmodels.api as sm
import statsmodels.formula.api as smf
X = dataset.iloc[:, :-1]
y = dataset.iloc[:, -1]

## fit a OLS model with intercept

X = sm.add_constant(X)
est = sm.OLS(y, X).fit()
est.summary()
plt.figure(figsize=(8,3))
p=plt.scatter(x=est.fittedvalues,y=est.resid,edgecolor='k')
xmin = min(est.fittedvalues)
xmax = max(est.fittedvalues)
plt.hlines(y=0,xmin=xmin*0.9,xmax=xmax*1.1,color='red',linestyle='--',lw=3)
plt.xlabel("Fitted values",fontsize=15)
plt.ylabel("Residuals",fontsize=15)
plt.title("Fitted vs. residuals plot",fontsize=18)
plt.grid(True)
plt.show()<jupyter_output><empty_output><jupyter_text>It appears the points are scattered quite a bit densely throughout and not forming a funnel shape, indicating a strong assumption of Homoscedasticity. The chances of outliers disproportionately influencing the model's performance are quite slim.In conclusion, there are variables that correlate to assist in estimating the Time from Pickup to Arrival. This is properly showed when we train our model even further and add more variables from the dataset given. **2. Data Preprocessing**<jupyter_code>riders = pd.read_csv("Riders.csv", index_col = 'Rider Id')
train = pd.read_csv('Train.csv', index_col='Order No')
test = pd.read_csv('Test.csv', index_col='Order No')<jupyter_output><empty_output><jupyter_text>**2.1. Preprocessing of the train data**<jupyter_code>y_train = train['Time from Pickup to Arrival'].values
train.drop(["Arrival at Destination - Day of Month","Arrival at Destination - Weekday (Mo = 1)","Arrival at Destination - Time"],axis = 1,inplace =True)

##START coding rider meta data
avg_rating = []
no_ratings = []
days_worked = []
orders = []
rid = train['Rider Id'].values 
for x in rid:
    avg_rating.append(riders.at[x, 'Average_Rating']) #creates a list of average rider rating
    no_ratings.append(riders.at[x, 'No_of_Ratings']) #creates a list of total numbe rof rider ratings
    days_worked.append(riders.at[x, 'Age']) #creates a list of rider age (Days delivering)
    
train['Average Rating'] = avg_rating
train['Age'] = days_worked
train['No. of Ratings'] = no_ratings
##END coding rider meta data
train.pop('Temperature')
##START coding time hours (24hr period)
holder = []
time_binary = ['Placement - Time', 'Confirmation - Time', 'Arrival at Pickup - Time', 'Pickup - Time']
for x in train[time_binary]:
    holder = pd.to_datetime(train[x]).dt.hour
    train[x] = holder
##END coding time hours (24hr period)


##START reverse geocoding
#Creating PICK_UP name DF
pick_up_la = train['Pickup Lat'].values 
pick_up_lo = train['Pickup Long'].values

pic_dest = pd.DataFrame(np.concatenate((pick_up_la.reshape(len(pick_up_la),1), pick_up_lo.reshape(len(pick_up_lo),1)),1))
pic_dest['GPS'] = pic_dest.apply(tuple, axis=1) #Create useable search tuple

pick_up = pic_dest['GPS'].values
searcher_pic = rg.search(list(pick_up))
pick_df = pd.DataFrame(searcher_pic) #final frame with PICK_UP LOC


#Creating DESTINATION name DF

dest_la = train['Destination Lat'].values 
dest_lo = train['Destination Long'].values

del_dest = pd.DataFrame(np.concatenate((dest_la.reshape(len(dest_la),1), dest_lo.reshape(len(dest_lo),1)),1))
del_dest['GPS'] = del_dest.apply(tuple, axis=1) #Create useable search tuple

deliveries = del_dest['GPS'].values
searcher_del = rg.search(list(deliveries))
dest_df = pd.DataFrame(searcher_del) #final frame with DEST LOC

train['Destination'] = dest_df['admin1'].values
train['Pick_up'] = pick_df['admin1'].values
## END reverse geocoding


##STAR df features to be removed
train.pop('Time from Pickup to Arrival')
train.pop('User Id')
train.pop('Placement - Day of Month')
train.pop('Confirmation - Day of Month')
train.pop('Arrival at Pickup - Day of Month')
train.pop('Pickup - Day of Month')
train.pop('Personal or Business')
train.pop('Pickup Lat')
train.pop('Pickup Long')
train.pop('Destination Lat')
train.pop('Destination Long')
train.pop('Rider Id')
train.pop('Precipitation in millimeters')
train.pop('Pickup - Weekday (Mo = 1)')
train.pop('Placement - Weekday (Mo = 1)')
train.pop('Destination')
train.pop('Vehicle Type')
train.pop('Pick_up')
##END df features to be removed

x_train = train.values<jupyter_output><empty_output><jupyter_text>**Checking the preprocessed data**<jupyter_code>train.columns
test.columns
train.head()
train.shape
dataset = train[['Platform Type', 'Placement - Time', 'Confirmation - Weekday (Mo = 1)',
       'Confirmation - Time', 'Arrival at Pickup - Weekday (Mo = 1)',
       'Arrival at Pickup - Time', 'Pickup - Time', 'Distance (KM)',
       'Average Rating', 'Age', 'No. of Ratings']]
dataset.describe().T<jupyter_output><empty_output><jupyter_text>**2.3. Preprocessing of the test data**<jupyter_code>##START coding time hours (24hr period)
holder2 = []
time_binary2 = ['Placement - Time', 'Confirmation - Time', 'Arrival at Pickup - Time', 'Pickup - Time']
for x in test[time_binary2]:
    holder2 = pd.to_datetime(test[x]).dt.hour
    test[x] = holder2
##END coding time hours (24hr period)

##START coding rider meta data
avg_rating2 = []
no_ratings2 = []
days_worked2 = []
rid2 = test['Rider Id'].values 
for x in rid2:
    avg_rating2.append(riders.at[x, 'Average_Rating']) #creates a list of average rider rating
    no_ratings2.append(riders.at[x, 'No_of_Ratings']) #creates a list of total numbe rof rider ratings
    days_worked2.append(riders.at[x, 'Age']) #creates a list of rider age (Days delivering)

test['Average Rating'] = avg_rating2
test['Age'] = days_worked2
test['No. of Ratings'] = no_ratings2
##END coding rider meta data


##START reverse geocoding
#Creating PICK_UP name DF
pick_up_la_test = test['Pickup Lat'].values 
pick_up_lo_test = test['Pickup Long'].values

pic_dest_test = pd.DataFrame(np.concatenate((pick_up_la_test.reshape(len(pick_up_la_test),1), pick_up_lo_test.reshape(len(pick_up_lo_test),1)),1))
pic_dest_test['GPS'] = pic_dest_test.apply(tuple, axis=1) #Create useable search tuple

pick_up_test = pic_dest_test['GPS'].values
searcher_pic_test = rg.search(list(pick_up_test))
pick_df_test = pd.DataFrame(searcher_pic_test) #final frame with PICK_UP LOC


#Creating DESTINATION name DF
dest_la_test = test['Destination Lat'].values 
dest_lo_test = test['Destination Long'].values

del_dest_test = pd.DataFrame(np.concatenate((dest_la_test.reshape(len(dest_la_test),1), dest_lo_test.reshape(len(dest_lo_test),1)),1))
del_dest_test['GPS'] = del_dest_test.apply(tuple, axis=1) #Create useable search tuple

deliveries_test = del_dest_test['GPS'].values
searcher_del_test = rg.search(list(deliveries_test))
dest_df_test = pd.DataFrame(searcher_del_test) #final frame with DEST LOC
test['Destination'] = dest_df_test['admin1'].values
test['Pick_up'] = pick_df_test['admin1'].values
## END reverse geocoding

test.pop('Temperature')

##START df features to be reomved
test.pop('Placement - Day of Month')
test.pop('Confirmation - Day of Month')
test.pop('Arrival at Pickup - Day of Month')
test.pop('Pickup - Day of Month')
test.pop('Personal or Business')
test.pop('Precipitation in millimeters')
test.pop('User Id')
test.pop('Pickup Lat')
test.pop('Pickup Long')
test.pop('Destination Lat')
test.pop('Destination Long')
test.pop("Rider Id")
test.pop('Pickup - Weekday (Mo = 1)')
test.pop('Placement - Weekday (Mo = 1)')
test.pop('Destination')
test.pop('Vehicle Type')
test.pop('Pick_up')
##END df features to be removed

x_test2 = test.values




test.shape<jupyter_output><empty_output><jupyter_text>**3. Modelling** **3.1. Feature scaling**<jupyter_code>from sklearn.preprocessing import StandardScaler

sc = StandardScaler()
x_train = sc.fit_transform(x_train)
x_test2 = sc.fit_transform(x_test2)
# Final check of the data for NaN values
for x in train:
    print(x, train[x].isna().mean())
x_train.shape
x_test2.shape<jupyter_output><empty_output><jupyter_text>**3.2. Splitting the train dataset**<jupyter_code>from sklearn.model_selection import train_test_split
x_trainer, x_tester, y_trainer, y_tester = train_test_split(x_train, y_train, test_size=0.20)<jupyter_output><empty_output><jupyter_text>**3.3. Using catboost to evaluate the features**<jupyter_code>from catboost import CatBoostRegressor
import catboost
cata = CatBoostRegressor(iterations=600,
                              learning_rate=0.1,
                              depth=10,
                             l2_leaf_reg=10,
                             bootstrap_type='Bernoulli',
                              loss_function = 'RMSE',
                              subsample=0.7,
                              metric_period=20,
                              random_seed=32,
                              allow_writing_files=False)

cata.fit(x_trainer, y_trainer)

from catboost import Pool
feature_score = pd.DataFrame(list(zip(train.dtypes.index, cata.get_feature_importance(Pool(x_train, label=y_train)))),
                columns=['Feature','Score'])

feature_score = feature_score.sort_values(by='Score', ascending=False, inplace=False, kind='quicksort', na_position='last')<jupyter_output><empty_output><jupyter_text>**Viewing the appropriateness of the fit of variables in the dataset**<jupyter_code>plt.rcParams["figure.figsize"] = (12,7)
ax = feature_score.plot('Feature', 'Score', kind='bar', color='green')
ax.set_title("Catboost Feature Importance Ranking", fontsize = 20)
ax.set_xlabel('')

rects = ax.patches

labels = feature_score['Score'].round(2)

for rect, label in zip(rects, labels):
    height = rect.get_height()
    
    ax.text(rect.get_x() + rect.get_width()/2, height + 0.35, label, ha='center', va='bottom')

plt.show()<jupyter_output><empty_output><jupyter_text>**3.4. Fitting the models**<jupyter_code>from sklearn.model_selection import cross_val_score
from sklearn.linear_model import ElasticNet

regEN = ElasticNet(alpha=0.00009, tol=0.00001, l1_ratio = 0.9,
                 max_iter=100, normalize = True, random_state=32)
regEN.fit(x_trainer, y_trainer)
cvsEN = cross_val_score(regEN, x_trainer, y_trainer, cv=5)

print(cvsEN)
print(np.mean(cvsEN))
from sklearn.linear_model import Lasso

regR = Lasso(tol=0.000001,  max_iter=1000,  random_state=32)
regR.fit(x_trainer, y_trainer)
cvsR = cross_val_score(regEN, x_trainer, y_trainer, cv=100)

print(cvsR)
print(np.mean(cvsR))
from sklearn.ensemble import ExtraTreesRegressor

regET = ExtraTreesRegressor(n_estimators = 100, criterion = 'mse', min_weight_fraction_leaf=0.001 , random_state=32, n_jobs=-1)
regET.fit(x_trainer, y_trainer)
cvsET = cross_val_score(regET, x_trainer, y_trainer, cv=10)

print(cvsET)
print(np.mean(cvsET))<jupyter_output>[0.36224605 0.36612046 0.38255472 0.31881604 0.37449474 0.33092038
 0.35740665 0.38608666 0.33655276 0.36289096]
0.35780894287309023<jupyter_text>**3.5. Combining the fitted models to increase our model performance**<jupyter_code>from sklearn.ensemble import StackingRegressor
from sklearn.linear_model import LinearRegression

models = [("LA",regR),('CA', cata),('ET', regET)]
meta_learner_reg = regEN

s_reg = StackingRegressor(estimators=models, final_estimator=meta_learner_reg, passthrough = True)
s_reg.fit(x_trainer, y_trainer)
s_cvs = cross_val_score(s_reg, x_trainer, y_trainer, cv=5)

print(s_cvs)
print(np.mean(s_cvs))<jupyter_output>0:	learn: 961.5762262	total: 28.6ms	remaining: 17.1s
20:	learn: 789.7731285	total: 296ms	remaining: 8.15s
40:	learn: 767.8885102	total: 587ms	remaining: 8s
60:	learn: 756.7323420	total: 882ms	remaining: 7.8s
80:	learn: 748.1254657	total: 1.19s	remaining: 7.61s
100:	learn: 738.3575786	total: 1.5s	remaining: 7.41s
120:	learn: 729.0654888	total: 1.81s	remaining: 7.16s
140:	learn: 719.1911164	total: 2.12s	remaining: 6.9s
160:	learn: 709.1576204	total: 2.42s	remaining: 6.59s
180:	learn: 701.7946147	total: 2.72s	remaining: 6.29s
200:	learn: 693.1440742	total: 3.02s	remaining: 6s
220:	learn: 685.7273202	total: 3.33s	remaining: 5.71s
240:	learn: 678.6161516	total: 3.63s	remaining: 5.42s
260:	learn: 671.9918086	total: 3.94s	remaining: 5.12s
280:	learn: 665.2676530	total: 4.24s	remaining: 4.81s
300:	learn: 658.4350826	total: 4.54s	remaining: 4.51s
320:	learn: 651.3994449	total: 4.85s	remaining: 4.21s
340:	learn: 645.0188302	total: 5.16s	remaining: 3.92s
360:	learn: 639.4194351	total: 5.47s	remai[...]<jupyter_text>4. **Performance Evaluation** **4.1. Comparing the prected values by each model**<jupyter_code>'''
****
*****
******
********
'''
#Super Learner 
y_predS = s_reg.predict(x_test2)

y_predS
#Extra Trees 
y_predET = regET.predict(x_tester)

y_predET
#RLasso 
y_predR = regR.predict(x_tester)

y_predR
#Elastic Net
y_predEN = regEN.predict(x_tester)

y_predEN
#CatBoost
zoo = cata.predict(x_tester)

zoo<jupyter_output><empty_output><jupyter_text>**Conclusion****Model Analysis**<jupyter_code>from sklearn import metrics

print('r2 Score:', metrics.r2_score(y_tester, y_predS))
print('Mean Absolute Error:', metrics.mean_absolute_error(y_tester, y_predS))
print('Mean Squared Error:', metrics.mean_squared_error(y_tester, y_predS))
print('Root Mean Squared Error:', np.sqrt(metrics.mean_squared_error(y_tester, y_predS)))<jupyter_output><empty_output><jupyter_text>**Writing Results Data**<jupyter_code>testid = test.index
testid
d = {'ID': testid, 'Delivery Time':np.round(y_predS, 0)}
sub = pd.DataFrame(data=d)
sub = sub[['ID', 'Delivery Time']]
sub

sub.to_csv("sub_1.csv",index  = False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12454734993964665, 'fraction_numerical': 0.03687040491605399, 'mean_word_length': 5.630411058566752, 'pattern_counts': {'":': 0, '<': 75, '<?xml version=': 0, '>': 76, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '359375', 'license': 'MIT', 'n_tokens_mistral': 6852, 'n_tokens_neox': 6282, 'n_words': 1640, 'path': 'Regression/Sendy Logistics Zindi Prac/Regression_predict_CT5.ipynb', 'repo_name': 'k-skaap/portfolio', 'size': '18226'}