<jupyter_start><jupyter_text>Welcome to the Sendy Logistics Challenge Group 9: JHBLogistics in **Sub-Saharan Africa** increases the cost of manufactured goods by up to `320%`; while in **Europe**, it only accounts for up to `90%` of the manufacturing cost. The aim of this hackathon is to help Sendy **predictdelivery time from driver picking up the item to the time of delivery**. This will help Sendy to enhance customer `communication`, `reliability`, and `customer experience` in general. In addition, the solution will enable Sendy to realise cost savings, and ultimately reduce the cost of doing business, through improvedresource management and planning for order scheduling. In this assignment, we shall be applying machine learning regression models to make the preditions. Let us begin by loading the python modules necessaryfor data handling, statistics and also for visualizations:<jupyter_code>#import numpy module
import numpy as np
# import the pandas module
import pandas as pd

#importing library for stats
import scipy.stats as st

# import the seaborn module
import seaborn as sns
# import the matplotlib module
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Train Dataset Dataset used to train prediction modelWe begin by loading the training and the riders data sets: we do this so that we can merge the two tables to get a full picture of every delivery item. We continue by ensuring that all column names are in lower letter cases. Furthermore, we use the pandas info method to gather details of the data set: It captures column names, count of non-null entries per column and also the data type of the column.<jupyter_code># Train Dataset
train_df = pd.read_csv('data/Train.csv')

riders = pd.read_csv("data/Riders.csv")
riders['RidersID'] = range(len(riders))
train_df = pd.merge(train_df,riders,on="Rider Id")
train_df.columns = [name.lower() for name in train_df.columns]

train_df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 21201 entries, 0 to 21200
Data columns (total 34 columns):
 #   Column                                     Non-Null Count  Dtype  
---  ------                                     --------------  -----  
 0   order no                                   21201 non-null  object 
 1   user id                                    21201 non-null  object 
 2   vehicle type                               21201 non-null  object 
 3   platform type                              21201 non-null  int64  
 4   personal or business                       21201 non-null  object 
 5   placement - day of month                   21201 non-null  int64  
 6   placement - weekday (mo = 1)               21201 non-null  int64  
 7   placement - time                           21201 non-null  object 
 8   confirmation - day of month                21201 non-null  int64  
 9   confirmation - weekday (mo = 1)            21201 non-null  int64  
 10  confirmation - time   [...]<jupyter_text>Feature EngineeringFrom the output of the info method, we can already see issues with the temperature and precipitations columns: the precipitation column only has ~3%  while the temperature column has ~80% of data  as compared to the length of the dataframe. This lead to the decision to drop the precipitation column and then fill the temperature column by means of imputation. The next stop was to look at categorical data: a trick is to look at all columns of type object from the pandas info method. In this section, we changed the time to seconds: this was done to keep the data precise to the second. The following columns were dropped:1. precipitation in millimeters : column only has ~3% of valid entries2. arrival at destination - day of month : variable we are trying to predict3. arrival at destination - time : variable we are trying to predict4. arrival at destination - weekday (mo = 1) : variable we are trying to predict5. order no : every entry is unique6. user id : We did not think it has much effect7. vehicle type: there was only one type of vehicle i.e Bikes<jupyter_code>def remove_colon_AM_PM(number):
    """
    Takes a number as a string and removes 
    the semicolon :, letter A, and letter P
    at end of the word
    
    Parameters:
    -----------
        number: str
            string that may have A, P or :
            
    returns:
    --------
        number without the number integer
            
    Example: 11A == 11
             09: == 09    
    
    """
    if number[-1]==':' or number[-1]=='A' or number[-1]=='P' :
        number = number[:-1]
        
    return int(number)

def convert_dates_to_seconds(df, date,time):
    """
    takes in pandas dataframe and converts time stamps
    into seconds 
    
    parameters:
    -----------
        df: Pandas dataframe
            
        date: str
            column name with day of month e.g 5 
            
        time: str
            column name with time strings e.g 11:03:12 AM
            
    returns:
    --------
        dataframe with dates represented in seconds    
    
    """
    date_in_seconds = []
    days_in_seconds = (df[date]-1)*3600*24
    
    for i in range(len(df)):
        hours = int(remove_colon_AM_PM(df[time][i][:2]))
        mins =  int(remove_colon_AM_PM(df[time][i][3:5]))
        seconds =  int(remove_colon_AM_PM(df[time][i][6:9]))
        
        if df[time][i][-2:] =='PM':
            date_in_seconds.append((hours+12)*3600+mins*60+seconds)
            
        else:
            date_in_seconds.append((hours)*3600+mins*60+seconds)
                
        
    
    return np.array(date_in_seconds)+np.array(days_in_seconds)


def correct_data_format_train(df):
    df = df.copy()
    """
    takes in model training dataframe and transforms
    datetimes into seconds integers, drop unnecesary 
    columns.
    
    parameters:
    -----------
        df: pandas dataframe
        
    returns:
    --------
        transformed dataframe   
    
    
    """
    
    convert_columns = {"placement - day of month":"placement - time",
                       "confirmation - day of month":"confirmation - time",
                      "pickup - day of month":"pickup - time",
                       "arrival at pickup - day of month":"arrival at pickup - time"}
    

    drop_columns = ["precipitation in millimeters",
                    "arrival at destination - day of month",
                    "arrival at destination - time",
                    "arrival at destination - weekday (mo = 1)",
                    "order no","user id","vehicle type","order no","rider id"]
    for column_name in df.columns:
        if column_name in convert_columns:
            df[convert_columns[column_name]] = convert_dates_to_seconds(df,date=column_name ,time =convert_columns[column_name])
        elif column_name in drop_columns:
            df.drop(column_name,axis=1,inplace=True)
            
    #Fillingna and dummy variable
    df['temperature'].fillna(df['temperature'].mean(),inplace=True)

    
    return df
train_df = correct_data_format_train(train_df)
train_df.head()<jupyter_output><empty_output><jupyter_text>**The head of the dataframe reveals that we have done our preprocessing excellently. We have all the columns in numerical format except the column "personal or business" since we want to use that information for our visualizations in the next step.  We make comments on the visuals right after they appear.**  VisualizationLet us plot the raw data and explore the relations that exists in the dataset. We shall look at a couple of scatter plots, boxplots and probability distributions bar graphs and correlation matrix heat map.<jupyter_code>fig, ((ax1, ax2, ax3), (ax4, ax5 ,ax6),(ax7, ax8,ax9)) = plt.subplots(nrows=3, ncols=3, sharex=False, sharey=False,figsize = (20,15))


#Distribution
ax1.hist(train_df["time from pickup to arrival"], 150, histtype = 'step',density = True, label = "data");
kde_xs = np.linspace(0, train_df["time from pickup to arrival"].max(), 301)
kde = st.gaussian_kde(train_df["time from pickup to arrival"])
ax1.plot(kde_xs, kde.pdf(kde_xs), label="Gaussian PDF")
ax1.legend(loc="upper right")
ax1.set_xlabel("time in seconds")
ax1.set_ylabel("percentage");
ax1.set_title("Histogram");

#Boxplot
ax2.boxplot(x=train_df["time from pickup to arrival"])
ax2.set_ylabel("time in seconds");

# Distance vs time taken 
ax3.scatter(train_df["distance (km)"],train_df["time from pickup to arrival"])
ax3.set_xlabel("distance (km)")
ax3.set_ylabel("time in seconds")

# Temperature vs time taken
ax4.scatter(train_df["temperature"],train_df["time from pickup to arrival"])
ax4.set_xlabel("Temperature")
ax4.set_ylabel("time in seconds")

# personal or Business?
ax5.hist(train_df["personal or business"])
ax5.set_ylabel("count")

#Box plots filtered by personal and business purchases
ax6.boxplot([train_df[train_df["personal or business"]=="Personal"]["time from pickup to arrival"],
             train_df[train_df["personal or business"]=="Business"]["time from pickup to arrival"]])
ax6.set_ylabel("time in seconds(personal deliveries)");
ax6.set_xlabel(["personal","Business"])


#pickup time vs time taken
ax8.scatter(train_df["pickup - time"],train_df["time from pickup to arrival"])
ax8.set_xlabel("pickup date in seconds")
ax8.set_ylabel("time from pickup to destination in seconds")

# histogram platform type
ax7.hist(train_df["platform type"], bins = 10)
ax7.set_xlabel("Platform type")


#saving images
fig.savefig('images/full_figure.png',transparent = True)
extent = ax1.get_window_extent().transformed(fig.dpi_scale_trans.inverted())
fig.savefig('images/distribution', bbox_inches=extent)<jupyter_output><empty_output><jupyter_text>**Take away points from the above visuals:***  `Temperature` does not seem to play a role in time taken to complete deliveries* `Time taken` for deliveries follows a normal distribution: hence we can use corresponding mathematics to improve our predictions* A lot of deliveries were for `businesses` rather than individuals.* Looking at the box plot tell us that there were `deliveries` made in `10 seconds and less`: this is actually concerning* There seem to be a positive linear relation between `distance` and `time taken`<jupyter_code>#Get dummy variables for personal or business
train_df["personal or business"] = pd.get_dummies(train_df["personal or business"],drop_first=True)<jupyter_output><empty_output><jupyter_text>Correlation matrix heat map<jupyter_code>from statsmodels.graphics.correlation import plot_corr

fig = plt.figure(figsize=(20,10));
ax = fig.add_subplot(111);
plot_corr(train_df.corr(), xnames = train_df.corr().columns, ax = ax)
fig.savefig('images/correlation.jpg',optimize=True,transparent=True,bbox_inches='tight')<jupyter_output><empty_output><jupyter_text>The `correlation heat map` tell us how `different` features `correlate` with each other. The dark `meroon` says a strong correlation while a dark `blue` suggest there is no correlation between those features whatsoever. The following insights can be captured from the map:* There is a strong self correlation of features (as expected and shown by the diagonal from top left to bottom right corners)* There is a mild correlation between `time taken` and `distance`  Test Dataset Dataset used to test prediction accuracy of modelWe transformed the training set, `it is also required to transform the testing data aswell`. This is a requirement for machine learning. Finally we ensure that the training set have the same format: ensure that the two sets have the same features with the same data format and feature names.<jupyter_code># Test Dataset
test_df = pd.read_csv('data/Test.csv')
#test = correct_data_format(test)
test_df = pd.merge(test_df,riders,on="Rider Id")
test_df.columns = [name.lower() for name in test_df.columns]
test_df.head()
def correct_data_test(df):
    """
    takes in model training dataframe and transforms
    datetimes into seconds integers, drop unnecesary 
    columns.
    
    parameters:
    -----------
        df: pandas dataframe
        
    returns:
    --------
        transformed dataframe   
    """
    df = df.copy()
    
    #convert dates 
    convert_columns = {"placement - day of month":"placement - time","confirmation - day of month":"confirmation - time",
                      "pickup - day of month":"pickup - time","arrival at pickup - day of month":"arrival at pickup - time"}

    drop_columns = ["placement - day of month","confirmation - day of month", "precipitation in millimeters",
                    "pickup - day of month","arrival at destination - day of month",
            "arrival at destination - time","user id","vehicle type","rider id"]
    
    #converting time columns and dropping unnecessary columns
    for column_name in df.columns:
        if column_name in convert_columns:
            df[convert_columns[column_name]] = convert_dates_to_seconds(
                df,date=column_name ,time =convert_columns[column_name])
        elif column_name in drop_columns:
            df.drop(column_name,axis=1,inplace=True)
            
    #Filling nulls and creating dummy variable
    df['temperature'].fillna(df['temperature'].mean(),inplace=True)
    df["personal or business"] = pd.get_dummies(df["personal or business"],drop_first=True)
    
    return df
test_df = correct_data_test(test_df)
submission = pd.DataFrame({"Order_No":test_df.pop("order no")})
test_df.head()
train_df.drop("time from pickup to arrival",axis = 1).columns == test_df.columns<jupyter_output><empty_output><jupyter_text>Fit regression models to datawe are now ready to fit the regression models to the data. We will fit three models to the data: * Multiple Linear Regression* Polynomial Regression* Decision tree* Random forestwe fit these models to train data and then compare them using validation set which we obtain from `train_test_split` method which we obtain from the `sklearn model selection` library. We end thissection by looking at the statistical metrics to see which method works best for us Multiple linear regression MLRMultiple linear regression is an ensemble of simple linear regressions. Simple linear regression takes the form $$y=mx + c$$ where $y$ is the dependent variable, $x$ is the independent, $m$ is the slope and $c$ is the intercept. This is when we are considering only a single feature. When we have, and are considering $n$ features, the model becomes $$y=m_1x_1+m_2x_2+\ldots+m_nx_n + c$$ where $x_1,\ldots,x_n$represent the features in the data, and $m_1 \ldots m_n$<jupyter_code>#separate independent and dependant variables

x_train = train_df.drop("time from pickup to arrival",axis = 1).values

y_train = train_df.pop("time from pickup to arrival").values
#from sklearn.model_selection import train_test_split
#x_train,x_validation,y_train,y_validation = train_test_split(x_train,y_train,test_size=0.1)
#fit the multiple linear regression to the training set
from sklearn.linear_model import LinearRegression
regressor = LinearRegression()
regressor.fit(x_train,y_train)
#make the prediction
y_pred = regressor.predict(x_validation)
from sklearn.metrics import mean_squared_error, r2_score


mean_square_MLR_vali = mean_squared_error(y_validation,y_pred)**0.5
mean_square_MLR_train = mean_squared_error(y_train,regressor.predict(x_train))**0.5


r2_MLR_train = r2_score(y_train,regressor.predict(x_train))
r2_MLR_valid = r2_score(y_validation,y_pred)
mean_square_MLR_train
#r2_MLR_valid
#submission["Time from Pickup to Arrival"] = y_pred.round(0)
#submission.to_csv('Submissions/submission_multilinear.csv', index=False)<jupyter_output><empty_output><jupyter_text>Polynomial regression model<jupyter_code>#build the polynomial regression model
from sklearn.preprocessing import PolynomialFeatures
polynomial_regressor = PolynomialFeatures(degree = 3)
x_poly = polynomial_regressor.fit_transform(x_train)
linear_regressor2 = LinearRegression()
linear_regressor2.fit(x_poly,y_train)
#make the prediction
y_pred = linear_regressor2.predict(polynomial_regressor.fit_transform(x_validation))


mean_square_pol_vali = mean_squared_error(y_validation,y_pred)**0.5
mean_square_pol_train = mean_squared_error(y_train,linear_regressor2.predict(polynomial_regressor.fit_transform(x_train)))**0.5


r2_pol_train = r2_score(y_train,linear_regressor2.predict(polynomial_regressor.fit_transform(x_train)))
r2_pol_valid = r2_score(y_validation,y_pred)
#submission["Time from Pickup to Arrival"] = y_pred.round(0)
#submission.to_csv('Submissions/submission_polynomial.csv', index=False)<jupyter_output><empty_output><jupyter_text>Decision Tree RegressionDecision tree builds regression models in the form of a tree structure. It breaks down a dataset into smaller and smaller subsets while at the same time an associated decision tree is incrementally developed. The final result is a tree with decision nodes and leaf nodes. The nodes are developed such that we have binary decisions at each node.<jupyter_code>#fit the decision tree regressor
from sklearn.tree import DecisionTreeRegressor 
DT_regressor = DecisionTreeRegressor()
DT_regressor.fit(x_train,y_train)
y_pred = DT_regressor.predict(x_validation)
mean_square_dt_vali = mean_squared_error(y_validation,y_pred)**0.5
mean_square_dt_train = mean_squared_error(y_train,DT_regressor.predict(x_train))**0.5


r2_dt_train = r2_score(y_train,DT_regressor.predict(x_train))
r2_dt_valid = r2_score(y_validation,y_pred)
#submission["Time from Pickup to Arrival"] = y_pred.round(0)
#submission.to_csv('Submissions/submission_DT.csv', index=False)<jupyter_output><empty_output><jupyter_text>Random forest regression: RFThe main idea behind random forest is to combine predictions made by many decision trees into a single model.A prediction made by a single decision tree may not be accurate, however, a prediction by an ensemble of decision trees tends to the true value on average.<jupyter_code>from sklearn.ensemble import RandomForestRegressor

RF = RandomForestRegressor(n_estimators=350, max_depth=15)
RF.fit(x_train,y_train)


y_pred =RF.predict(test_df)

submission["Time from Pickup to Arrival"] = y_pred.round(0)
submission.to_csv('Submissions/submission_RF.csv', index=False)
mean_square_RF_vali = mean_squared_error(y_validation,y_pred)**0.5
mean_square_RF_train = mean_squared_error(y_train,RF.predict(x_train))**0.5


r2_RF_train = r2_score(y_train,RF.predict(x_train))
r2_RF_valid = r2_score(y_validation,y_pred)<jupyter_output><empty_output><jupyter_text>Model assessmentWe will test all our method now. We need to look at the errors and also talk about underfitting and/or overfittingchoose a model which is best suited for our data.<jupyter_code>Error_df = pd.DataFrame({"Model":["MLR","pol","DT","RF"],
                         "MSE train":[mean_square_MLR_train,mean_square_pol_train,mean_square_dt_train,mean_square_RF_train],
                        "MSE test":[mean_square_MLR_vali,mean_square_pol_vali,mean_square_dt_vali,mean_square_RF_vali ],
                        "R^2 train":[r2_MLR_train,r2_pol_train,r2_dt_train,r2_RF_train ],
                        "R^2 test":[r2_MLR_valid,r2_pol_valid,r2_dt_valid,r2_RF_valid]})
Error_df.round(3)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09049773755656108, 'fraction_numerical': 0.014770895095438706, 'mean_word_length': 3.9151329243353783, 'pattern_counts': {'":': 14, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '484335', 'license': 'MIT', 'n_tokens_mistral': 5900, 'n_tokens_neox': 5548, 'n_words': 1849, 'path': 'Sendy_Logistics_Challenge_Regression_Predict_Final.ipynb', 'repo_name': 'explore-thulani/sendy-logistics-challenge', 'size': '19227'}