<jupyter_start><jupyter_text>Problem Date: 25 June 2020Time : 16:30Main problem: Arrival-time estimation  Problem statement: Umniah aims at estimating the arrival times of its employees..- An appropriate holistic approach to solve this problem is "Profiling": building a profile/model that describes a person's behavior.- Simple examples will demonstrate the concept below. Note:I should point out that there are several ways to solve this problem and selecting an appropriate approach depends on some business related issues such as:- Who the client is and why s/he needs this- where this system will be deployed (the bigger system)- The available data  The profile/model: the probability of a delay in minutes (The prior*).*That model is the back-pone for solving such problems.  Employee 1: Long tail distribution - This example tests the effectiveness of this method.- A well know behavor is assumed (An employee who usually arrives on time: a standard delay 'Poisson'). - The arrival times were generated from a well-known model, then the  generated data were used to reconstruct the origional model<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
from numpy import random

lamb = 10
x = np.linspace(0, 100, 10000)
pdf = np.exp(-x/lamb)/lamb

plt.plot(x, pdf, 'bs')
pyplot.xlabel(" Delay in minutes")

plt.show()
#Generate arrival times from the distribution using MC method
y = -lamb*np.log(1 - random.rand(10000))
min(y)
max(y)
hist = plt.hist(y, bins=100)<jupyter_output><empty_output><jupyter_text>Estimate the arrival time prior<jupyter_code>from sklearn.neighbors import KernelDensity

# instantiate and fit the KDE model
kde = KernelDensity(bandwidth=2, kernel='linear')
kde.fit(y[:, None])

# score_samples returns the log of the probability density
logprob = kde.score_samples(x[:, None])
yp = np.exp(logprob)

#plot
plt.fill_between(x, yp, alpha=0.5)
plt.plot(y, np.full_like(y, -0.01), '|k', markeredgewidth=1)
pyplot.xlabel("Employee1 : delay in minutes")
plt.ylim(-0.02, 0.1)<jupyter_output><empty_output><jupyter_text>Evaluation: mean square error<jupyter_code>print('MSE:',np.mean((pdf-yp)**2))<jupyter_output>MSE: 1.0395537676094016e-05<jupyter_text>Employee 2: Actual distribution An employee arrives 10 minutes late on average.Only 100 records are available<jupyter_code># Generate from Poisson distribution
employee2_delay = random.poisson(lam=10, size=100)
employee2_delay
from matplotlib import pyplot
pyplot.hist(employee2_delay, bins=13)
pyplot.show()
# fit density
sample = employee2_delay
model2 = KernelDensity(bandwidth=2, kernel='exponential')
sample = sample.reshape((len(sample), 1))
model2.fit(sample)


# sample probabilities for a range of outcomes
values = np.asarray([value for value in range(1, 25)])
values = values.reshape((len(values), 1))
probabilities = model2.score_samples(values)
probabilities = np.exp(probabilities)


# plot the histogram and pdf
pyplot.plot(values[:], probabilities)
pyplot.xlabel("Employee2 : delay in minutes")
pyplot.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09832292009207498, 'fraction_numerical': 0.029924366984544558, 'mean_word_length': 5.929384965831435, 'pattern_counts': {'":': 1, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '409570', 'license': 'MIT', 'n_tokens_mistral': 1005, 'n_tokens_neox': 920, 'n_words': 330, 'path': 'Arrival_time.ipynb', 'repo_name': 'Demonstartion/Arrivaal-time', 'size': '3041'}