<jupyter_start><jupyter_text>SAMUR Emergency Frequencies This notebook explores how the frequency of different types of emergency changes with time in relation to different periods (hours of the day, days of the week, months of the year...) and locations in Madrid. This will be useful for constructing a realistic emergency generator in the city simulation.Let's start with some imports and setup, and then read the table.<jupyter_code>import pandas as pd
import datetime
import matplotlib.pyplot as plt
import yaml
%matplotlib inline
df = pd.read_csv("../data/emergency_data.csv")
df.head()<jupyter_output><empty_output><jupyter_text>The column for the time of the call is a string, so let's change that into a timestamp.<jupyter_code>df["time_call"] = pd.to_datetime(df["Solicitud"])<jupyter_output><empty_output><jupyter_text>We will also need to assign a numerical code to each district of the city in order to properly vectorize the distribution an make it easier to work along with other parts of the project.<jupyter_code>district_codes = {
    'Centro': 1, 
    'Arganzuela': 2, 
    'Retiro': 3, 
    'Salamanca': 4, 
    'Chamartín': 5, 
    'Tetuán': 6, 
    'Chamberí': 7, 
    'Fuencarral - El Pardo': 8, 
    'Moncloa - Aravaca': 9, 
    'Latina': 10, 
    'Carabanchel': 11, 
    'Usera': 12, 
    '<NAME>': 13, 
    'Moratalaz': 14, 
    'Ciudad Lineal': 15, 
    'Hortaleza': 16, 
    'Villaverde': 17, 
    'Villa de Vallecas': 18, 
    'Vicálvaro': 19, 
    'San Blas - Canillejas': 20, 
    'Barajas': 21,
    }

df["district_code"] = df.Distrito.apply(lambda x: district_codes[x])<jupyter_output><empty_output><jupyter_text>Each emergency has already been assigned a severity level, depending on the nature of the reported emergency.<jupyter_code>df["severity"] = df["Gravedad"]<jupyter_output><empty_output><jupyter_text>We also need the hour, weekday and month of the event in order to assign it in the various distributions.<jupyter_code>df["hour"] = df["time_call"].apply(lambda x: x.hour)  # From 0 to 23
df["weekday"] = df["time_call"].apply(lambda x: x.weekday()+1)  # From 1 (Mon) to 7 (Sun)
df["month"] = df["time_call"].apply(lambda x: x.month)<jupyter_output><empty_output><jupyter_text>Let's also strip down the dataset to just the columns we need right now.<jupyter_code>df = df[["district_code", "severity", "time_call", "hour", "weekday", "month"]]
df.head()<jupyter_output><empty_output><jupyter_text>We are going to group the distributions by severity.<jupyter_code>emergencies_per_grav = df.severity.value_counts().sort_index().rename("total_emergencies")
emergencies_per_grav<jupyter_output><empty_output><jupyter_text>We will also need the global frequency of the emergencies:<jupyter_code>total_seconds = (df.time_call.max()-df.time_call.min()).total_seconds()
frequencies_per_grav = (emergencies_per_grav / total_seconds).rename("emergency_frequencies")
frequencies_per_grav<jupyter_output><empty_output><jupyter_text>Each emergency will need to be assigne a district. Assuming independent distribution of emergencies by district and time, each will be assigned to a district according to a global probability based on this dataset, as follows.<jupyter_code>prob_per_district = (df.district_code.value_counts().sort_index()/df.district_code.value_counts().sum()).rename("distric_weight")
prob_per_district<jupyter_output><empty_output><jupyter_text>In order to be able to simplify the generation of emergencies, we are going to assume that the distributions of emergencies per hour, per weekday and per month are independent, sharing no correlation. This is obiously not fully true, but it is a good approximation for the chosen time-frames.<jupyter_code>hourly_dist = (df.hour.value_counts()/df.hour.value_counts().mean()).sort_index().rename("hourly_distribution")
daily_dist = (df.weekday.value_counts()/df.weekday.value_counts().mean()).sort_index().rename("daily_distribution")
monthly_dist = (df.month.value_counts()/df.month.value_counts().mean()).sort_index().rename("monthly_distribution")<jupyter_output><empty_output><jupyter_text>We will actually make one of these per severity level. This will allow us to modify the base emergency density of a given severity as follows:<jupyter_code>def emergency_density(gravity, hour, weekday, month):
    base_density = frequencies_per_grav[gravity]
    density = base_density * hourly_dist[hour] * daily_dist[weekday] * monthly_dist[month]
    return density
emergency_density(3, 12, 4, 5)  # Emergency frequency for severity level 3, at 12 hours of a thursday in May<jupyter_output><empty_output><jupyter_text>In order for the model to read these distributions we will need to store them in a dict-like format, in this case YAML, which is easily readable by human or machine.<jupyter_code>dists = {}
for severity in range(1, 6):
    sub_df = df[df["severity"] == severity]
    
    frequency = float(frequencies_per_grav.round(8)[severity])
    
    hourly_dist  = (sub_df.hour.   value_counts()/sub_df.hour.   value_counts().mean()).sort_index().round(5).to_dict()
    daily_dist   = (sub_df.weekday.value_counts()/sub_df.weekday.value_counts().mean()).sort_index().round(5).to_dict()
    monthly_dist = (sub_df.month.  value_counts()/sub_df.month.  value_counts().mean()).sort_index().round(5).to_dict()
    
    district_prob = (sub_df.district_code.value_counts()/sub_df.district_code.value_counts().sum()).sort_index().round(5).to_dict()
    
    dists[severity] = {"frequency": frequency,
                      "hourly_dist": hourly_dist,
                      "daily_dist": daily_dist,
                      "monthly_dist": monthly_dist,
                      "district_prob": district_prob}
    
f = open("../data/distributions.yaml", "w+")
yaml.dump(dists, f, allow_unicode=True)<jupyter_output><empty_output><jupyter_text>We can now check that the dictionary stored in the YAML file is the same one we have created.<jupyter_code>with open("../data/distributions.yaml") as dist_file:
    yaml_dict = yaml.safe_load(dist_file)
yaml_dict == dists<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11513157894736842, 'fraction_numerical': 0.008881578947368421, 'mean_word_length': 5.192464358452138, 'pattern_counts': {'":': 5, '<': 54, '<?xml version=': 0, '>': 54, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '1253', 'license': 'MIT', 'n_tokens_mistral': 2068, 'n_tokens_neox': 1935, 'n_words': 526, 'path': 'notebooks/emergency_frequencies.ipynb', 'repo_name': 'samurai-madrid/reinforced-learning', 'size': '6092'}