<jupyter_start><jupyter_text>The `Dataset` objectTo illustrate extreme value theory, this tutorial will focus on an example time-series dataset.The example dataset will contain normally distributed time-series, but the positive tail has a Pareto(2.5) distribution superimposed.The behaviour of this power law tail is under investigation in this tutorial series.The `Dataset` object forms the basis of extreme value analysis in this package.At its core is a `pd.Series` in the `self.series` attribute.The dataset performs checks on the series so that the rest of the package can safely make assumptions about data quality.The checks are:* the data cannot contain `nan`s,* the data cannot contain non-finite values,* the data cannot contain duplicate values.In this notebook, the dataset will be generated and its properties will be investigated using generic plots.<jupyter_code>from evt.dataset import Dataset
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from scipy.stats import pareto, norm

N_DATAPOINTS = 100000  # number of datapoints in the example set
NORMAL_STD = 5  # standard deviation of the normal distribution
PARETO_SHAPE = 2.5  # shape parameter of the Pareto distribution
EXAMPLE_NAME = 'Values'  # for nicer plots
EXAMPLE_INDEX_NAME = 'Index'

np.random.seed(0)  # enforce deterministic behaviour

series = pd.Series(
    norm.rvs(scale=NORMAL_STD, size=N_DATAPOINTS) + pareto.rvs(PARETO_SHAPE, size=N_DATAPOINTS),
    name=EXAMPLE_NAME
)
series.index.name = EXAMPLE_INDEX_NAME

dataset = Dataset(series)<jupyter_output><empty_output><jupyter_text>The series is stored in the `.series` attribute.<jupyter_code>dataset.series<jupyter_output><empty_output><jupyter_text>When working with data, always plot the dataset. The `Dataset` supports simply plotting against its index. Moreover, it is possible to plot a boxplot of the values, to get a grip on quantiles.<jupyter_code>fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 6))

dataset.plot_dataset(ax1)
dataset.plot_boxplot(ax2)

fig.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>The boxplot raises the suspicion the right side of the distribution could be fat tailed.The convergence behaviour of sample moments can be investigated as a next step.<jupyter_code>fig, ax = plt.subplots()

dataset.plot_maximum_to_sum(ax)

fig.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>The third and fourth moment do not seem to converge.This could be a signal of a tail with an index between 2 and 3.Finally, let's determine what a good threshold would be for the peaks over threshold method.This can be done using a mean excess plot.<jupyter_code>fig, ax = plt.subplots()

dataset.plot_mean_excess(ax)

fig.tight_layout()
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07137733142037303, 'fraction_numerical': 0.00824964131994261, 'mean_word_length': 5.819070904645477, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '304248', 'license': 'MIT', 'n_tokens_mistral': 853, 'n_tokens_neox': 819, 'n_words': 321, 'path': 'tutorial/0_the_dataset_object.ipynb', 'repo_name': 'spmvg/evt', 'size': '2788'}