<jupyter_start><jupyter_text>Machine  Learning Assessment Input:•   The input of your model should take a vector with at least the date/timeOutput:•   The output should contain the call volume (count) of 911 calls predicted for this time period.•	Please  provide a backtesting plot comparing the actual (measured)  call volume vs. predicted call volume for a month and a week.•   Please  think about measures to judge the quality and certainty of predictions  and providethem as well.  NOTES Data is presented in USA format MM-DD-YYYY, it will be parsed to fit European time notation YYYY- MM-DD.Data has been simplified and adjusted to a vector input.<jupyter_code>import tensorflow as tf
import os
import pandas as pd
import numpy as np
import logging<jupyter_output><empty_output><jupyter_text>DATA INPUT ("getting data in"  as requested in Rule 2.) Vector with at least the date/time The Dataset of 911 calls:  https://data.seattle.gov/Public- Safety/Seattle-Real-Time-Fire-911-Calls/kzjm-xkqj is sent to DataPrep and edited using BigQuery as suggested, in order to obtain vector with dates only.<jupyter_code>#Read data from preprocessed db
def a(data):
    data = pd.read_csv(r"C:\Users\Dell\Niologic\Seattle_Real_Time_Fire_911_Calls-v.csv")
    return 'Data loaded'
a(pd.read_csv(r"C:\Users\Dell\Niologic\Seattle_Real_Time_Fire_911_Calls-v.csv"))

# Shows how the db looks like from the beginning in steps of 1000
df = data[1::1000]
df<jupyter_output><empty_output><jupyter_text>Processing data (scoring) as requested in Rule 2. and 3)<jupyter_code>#counts the number of calls per day from database, db is ordered
dgroup=data.groupby('Datetime').size()
dgroup
dgroup.plot(figsize=(15,5), kind='line',x=0, y=1)

def df2c(df, window_size=5):
    df2np=df.to_numpy()
    X=[]
    y=[]
    for i in range(len(df2np)-window_size):
        row=[[a] for a in df2np[i:i+window_size]]
        X.append(row)
        label=df2np[i+window_size]
        y.append(label)
    return np.array(X), np.array(y)
import convert2np
import numpy as np
WINDOW_SIZE = 5
X1, y1 = df2c(dgroup, WINDOW_SIZE)
X1.shape, y1.shape<jupyter_output><empty_output><jupyter_text>Training as requested in rule 3<jupyter_code>#%%writefile train.py
#splits the set with 70% for training, 20 % for validation and 10% for testing
train_range=round(X1.shape[0]*0.7)
val_range=round(X1.shape[0]*0.2)+train_range
test_range=round(X1.shape[0]*0.1)+val_range


X_train1, y_train1 = X1[:train_range], y1[:train_range]
X_val1, y_val1 = X1[train_range:val_range], y1[train_range:val_range]
X_test1, y_test1 = X1[val_range:test_range], y1[val_range:test_range]
X_train1.shape, y_train1.shape, X_val1.shape, y_val1.shape, X_test1.shape, y_test1.shape<jupyter_output><empty_output><jupyter_text>Model<jupyter_code>#%%writefile modellstm.py

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import *
from tensorflow.keras.callbacks import ModelCheckpoint
from tensorflow.keras.losses import MeanSquaredError
from tensorflow.keras.metrics import RootMeanSquaredError
from tensorflow.keras.optimizers import Adam

model1 = Sequential()
model1.add(InputLayer((5, 1)))
model1.add(LSTM(32))
model1.add(Dense(8, 'relu'))
model1.add(Dense(1, 'linear'))

model1.summary()
cp1 = ModelCheckpoint('model1/', save_best_only=True)
model1.compile(loss=MeanSquaredError(), optimizer=Adam(learning_rate=0.1), metrics=[RootMeanSquaredError()])
model1.fit(X_train1, y_train1, validation_data=(X_val1, y_val1), epochs=10, callbacks=[cp1])
from tensorflow.keras.models import load_model
model1 = load_model('model1/')
train_predictions = model1.predict(X_train1).flatten()
train_results = pd.DataFrame(data={'Prediction':train_predictions, 'Data':y_train1})
train_results<jupyter_output><empty_output><jupyter_text>Output:•   The output should contain the call volume (count) of 911 calls predicted for this time period.•	Please  provide a backtesting plot comparing the actual (measured)  call volume vs. predicted call volume for a month and a week.•   Please  think about measures to judge the quality and certainty of predictions  and providethem as well.  Predicted calls<jupyter_code>val_predictions = model1.predict(X_val1).flatten()
val_results = pd.DataFrame(data={'Prediction':val_predictions, 'Data':y_val1})
val_results<jupyter_output><empty_output><jupyter_text>Backtesting plot   Call volume vs. predicted call volume for a month<jupyter_code>import matplotlib.pyplot as plt
plt.plot(train_results['Prediction'][50:80])
plt.plot(train_results['Data'][50:80])
val_predictions = model1.predict(X_val1).flatten()
val_results = pd.DataFrame(data={'Prediction':val_predictions, 'Data':y_val1})
val_results<jupyter_output><empty_output><jupyter_text>Call volume vs. predicted call volume for a week.<jupyter_code>plt.plot(val_results['Prediction'][50:58])
plt.plot(val_results['Data'][50:58])
test_predictions = model1.predict(X_test1).flatten()
test_results = pd.DataFrame(data={'Prediction':test_predictions, 'Data':y_test1})
test_results<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10934709267711848, 'fraction_numerical': 0.027783290335384004, 'mean_word_length': 6.5675675675675675, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '467092', 'license': 'MIT', 'n_tokens_mistral': 1809, 'n_tokens_neox': 1736, 'n_words': 437, 'path': 'MLAssessment.ipynb', 'repo_name': 'EstebanPozo/MLAssesment', 'size': '5039'}