<jupyter_start><jupyter_text>Minimum Distance Travelled by Ambulance in a DayFind the optimal position to place 6 ambulances within a day (3 hour interval) in which they will travel the least amount of distance. Notes- Each ambulance is committed every three hours- Ambulances travel within allocated "sectors"<jupyter_code># load required libraries
import pandas as pd
import numpy as np
import geopandas as gpd 
from google.colab import files

import contextily as ctx
import matplotlib.pyplot as plt
import time

# processing geojson files
from geovoronoi.plotting import subplot_for_map, plot_voronoi_polys_with_points_in_area
from geovoronoi import voronoi_regions_from_coords, points_to_coords

# clustering + visualisation
from sklearn.cluster import KMeans
import seaborn as sns; sns.set()

from scipy.spatial import Voronoi, voronoi_plot_2d
# load data
pip install geovoronoi
# upload the data
uploaded = files.upload()<jupyter_output><empty_output><jupyter_text>Exploratory Data AnalysisLook into the characteristics of the data to pick up patterns for use in determining ambulance placement within the city of Nairobi.<jupyter_code># load data
df = pd.read_csv('Train.csv', parse_dates=['datetime'])
print(df.shape)
df.head()

## Notes: The accidents identify areas where an ambulance must reach, creating a voronoi diagram gives us the central point of these accident 
# 1. visualize the accident points
# 2. time distribution of accidents
# 2. cluster the accidents
# 3. create voronoi diagram for the points


# 2. timeline

## per month
df['idx'] = 1
df['hour'] = df['datetime'].dt.hour

df_month = df.resample('M', on='datetime')['idx'].count().reset_index()
df_pd_month = pd.DataFrame(df_month)

## per day
df_day = df.resample('D', on='datetime')['idx'].count().reset_index()
df_pd_day = pd.DataFrame(df_day)

## per hour
df_hour = df.groupby('hour').count().reset_index()
df_pd_hour = pd.DataFrame(df_hour)

## per 3 hours
df_3hours = df.resample('3H', on='datetime')['idx'].count().reset_index()
df_3hours['datetime'] = df_3hours['datetime'].dt.hour

df_3hours = df_3hours.groupby('datetime').sum().reset_index()
df_pd_3hours = pd.DataFrame(df_3hours)

#---- axes
x_day = df_pd_day['datetime']
y_day = df_pd_day['idx']

x_month = df_pd_month['datetime']
y_month = df_pd_month['idx']

x_hour = df_pd_hour['hour']
y_hour = df_pd_hour['idx']

x_3hours = df_pd_3hours['datetime']
y_3hours = df_pd_3hours['idx']

fig, axs = plt.subplots(2, 2,figsize=(20, 18))

axs[0, 0].plot(x_month, y_month)
axs[0, 0].set_title("Accident Distribution per Month")

axs[1, 0].plot(x_day, y_day)
axs[1, 0].set_title("Accident Distribution per Day")

axs[0, 1].plot(x_hour, y_hour)
axs[0, 1].set_title("Accident Distribution per Hour")

axs[1, 1].plot(x_3hours, y_3hours)
axs[1, 1].set_title("Accident Distribution per 3Hours")

# clustering

K_clusters = range(1,10)

kmeans = [KMeans(n_clusters=i) for i in K_clusters]

Y_axis = df[['latitude']]
X_axis = df[['longitude']]

score = [kmeans[i].fit(Y_axis).score(Y_axis) for i in range(len(kmeans))]

# Variable with the Longitude and Latitude
X=df.loc[:,['datetime','latitude','longitude']]

kmeans = KMeans(n_clusters = 5, init ='k-means++')
kmeans.fit(X[X.columns[1:3]]) # Compute k-means clustering.
X['cluster_label'] = kmeans.fit_predict(X[X.columns[1:3]])
centers = kmeans.cluster_centers_ # Coordinates of cluster centers.
labels = kmeans.predict(X[X.columns[1:3]]) # Labels of each point

# ===== clusters
#X.plot.scatter(x = 'latitude', y = 'longitude', c=labels, s=50, cmap='viridis')


# ===== road segments locations


fig, axs = plt.subplots(2, 2,figsize=(20, 18))
axs[0, 0].plot(K_clusters, score)
axs[0, 0].set_title('Elbow Curve')
axs[0, 0].set_ylabel('Score')
axs[0, 0].set_xlabel('Number of Clusters')

axs[1,0].scatter(centers[:, 0], centers[:, 1], c='black', s=200, alpha=0.5,cmap='viridis')
axs[1, 0].set_title('Cluster Centres of Accidents')
axs[1, 0].set_ylabel('Longitude')
axs[1, 0].set_xlabel('Latitude')

axs[0,1].scatter(x = X['latitude'], y = X['longitude'], c=labels, s=50, cmap='viridis')
axs[0, 1].set_title('Scatter of Accident Zones')
axs[0, 1].set_ylabel('Longitude')
axs[0, 1].set_xlabel('Latitude')

points = pd.DataFrame([centers[:, 0], centers[:, 1]]).transpose()
vor = Voronoi(points)

# plot
voronoi_plot_2d(vor,ax=axs[1,1],point_size=20)
axs[1, 1].set_title('Voronoi Regions')
axs[1, 1].set_ylabel('Longitude')
axs[1, 1].set_xlabel('Latitude')


# Variable with the Longitude and Latitude
X=df.loc[:,['datetime','latitude','longitude']]
X.head(10)

kmeans = KMeans(n_clusters = 5, init ='k-means++')
kmeans.fit(X[X.columns[1:3]]) # Compute k-means clustering.
X['cluster_label'] = kmeans.fit_predict(X[X.columns[1:3]])
centers = kmeans.cluster_centers_ # Coordinates of cluster centers.
labels = kmeans.predict(X[X.columns[1:3]]) # Labels of each point

points['cluster_label'] = range(0,5)
points.columns = ['lat','lon','cluster_label']

point = points

merged = X.merge(point,on='cluster_label',how='left')
merged['hour'] = merged['datetime'].dt.hour

df_pd_3hours.columns = ['hour','idx']

mg = merged.merge(df_pd_3hours,on='hour',how='left')
mg.head()


df_pd_3hours.head()<jupyter_output><empty_output><jupyter_text>Rather than trying to predict accident locations or the number of accidents in a given timeframe, we are instead asked to come up with an ambulance deployment schedule that will minimise distance travelled. This is of course a simplification of the problem, but by solving this you will hopefully develop strategies that can be used going forward.<jupyter_code># View the submission format
ss = pd.read_csv('Data/SampleSubmission.csv', parse_dates=['date'])
ss.head()<jupyter_output><empty_output><jupyter_text>ScoringYou can re-create the scoring process by creating your own test set and using the following function:<jupyter_code>def score(sub, ref):
    total_distance = 0
    for date, c_lat, c_lon in ref[['datetime', 'latitude', 'longitude']].values:
        row = sub.loc[sub.date < date].tail(1) # Prior to Oct 2 this was incorrectly .head(1)
        dists = []
        for a in range(6):
            dist = ((c_lat - row[f'A{a}_Latitude'].values[0])**2+(c_lon - row[f'A{a}_Longitude'].values[0])**2)**0.5 
            dists.append(dist)
        total_distance += min(dists)
    return total_distance

# Example:

# Test set
reference = df.loc[df.datetime > '2019-01-01'] # Using 2019 as our test set

# Date rane covering test set
dates = pd.date_range('2019-01-01', '2020-01-01', freq='3h')

# Create submission dataframe
sub = pd.DataFrame({
    'date':dates
})
for ambulance in range(6):
    sub['A'+str(ambulance)+'_Latitude'] = 0
    sub['A'+str(ambulance)+'_Longitude'] = 0

# Place an ambulance in the center of the city:
sub['A'+str(ambulance)+'_Latitude'] = 36.82
sub['A'+str(ambulance)+'_Longitude'] = -1.3



score(sub, reference)<jupyter_output><empty_output><jupyter_text>This scoring function can be used for local testing without needing to submit on Zindi. Lower is better. Experiment with specifying set locations for all 6 ambulances, and see how low you can get your score.  Weather DataSome weatehr variables form the GFS dataset are provided, covering the greater Nairobi area. Let's take a quick look at the dataset before moving on:<jupyter_code>weather = pd.read_csv('Data/Weather_Nairobi_Daily_GFS.csv', parse_dates=['Date'])
weather.head()
# Interestingly, Kenya has two rainy seasons per year!
weather.set_index('Date')['precipitable_water_entire_atmosphere'].rolling(10).mean().plot()<jupyter_output><empty_output><jupyter_text>Road Survey DataThe road survey data contains many different measurements for each surveyed road segment. Unfortunately we can't access the raw data, and must work 'in the dark' in terms of what the column headings mean. However, the naming convention does at least show which columns may be related.To locate the road segments geographically, a geojson file is provided. Here is an example of merging the survey data with the segments and visualizing the result.<jupyter_code># Load the survey data
road_surveys = pd.read_csv('Data/Segment_info.csv')
road_surveys.head()
# Load the map
road_segment_locs = gpd.read_file('Data/segments_geometry.geojson')
road_segment_locs.head()
# Because this is a geodataframe, we can plot it
road_segment_locs.plot()
# Let's merge the two and color by one of the survey columns
segments_merged = pd.merge(road_segment_locs, road_surveys, on='segment_id', how='left')
segments_merged.plot(column='55_90', figsize=(20, 8))<jupyter_output><empty_output><jupyter_text>This data will come in useful if you're trying to predict the number of crashes along a particular road, or within a specified region.   Adding Uber DataYou can access speed data from https://movement.uber.com/explore/nairobi/speeds and clicking 'Download Data'. Here we'll load in the speeds from January 2019 and explore the data briefly.<jupyter_code>speeds = pd.read_csv('../uber_data/movement-speeds-hourly-nairobi-2019-1.csv', parse_dates=['utc_timestamp'])
speeds.head()
# Plot speeds over time for a specific OSM way
way = speeds.loc[speeds.osm_way_id == 133901081]
way = way.set_index('utc_timestamp')
way.resample('1h').mean().plot(y='speed_kph_mean')
way.resample('1D').mean().plot(y='speed_kph_mean')<jupyter_output><empty_output><jupyter_text>You can use any and all data available through Uber Movement, including linking the speeds data to OSM segments, downloading quarterly statistics, using the Movement Data Toolkit etc. You can find much more info about working with the speeds data here: https://medium.com/uber-movement/working-with-uber-movement-speeds-data-cc01d35937b3 - including how to link the segment_id with OSM data  What Next?There are many ways you could approach this challenge. For example, you could:- Create a model to predict the liklihood of an accident given a location, the features of the nearby road segments, the weather, the traffic speed and the time of day- Use this model to predict the probability of crashes for different times+locations over the test period, and then sample from this probability distribution to generate plausible crash locations.- Use an optimization technique to minimise the travel distance for the ambulances to your fake crash points. This might seem too complicated - perhaps simply picking 6 fixed locations based on the existing data will be enough? Or maybe it's worth setting separate locations during busy traffic times? Perhaps a Reinforcement Learning aproach will do well? Or maybe you can see a totally different way to solve this!Share your ideas on the discussion boards, and let's see how innovative we can be :)<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12163044492118273, 'fraction_numerical': 0.019867549668874173, 'mean_word_length': 5.838010204081633, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 34, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '302543', 'license': 'MIT', 'n_tokens_mistral': 3727, 'n_tokens_neox': 3509, 'n_words': 1079, 'path': 'Uber_Nairobi_Ambulance_Perambulation_Challenge_with_Clustering.ipynb', 'repo_name': 'Paul-mwaura/ML-Hackathons', 'size': '10721'}