<jupyter_start><jupyter_text>Loading toy datasetShould be time sorted<jupyter_code># loading the dataset
df = pd.read_csv("example_dataset/trilux_dataset.csv")
# filtering
df = (
    df
    .loc[df["trabajos_id"]==1]
    .filter(items=["fecha", "chla", "tby", "pc"])
    .set_index("fecha")
    .sort_index()
)
df<jupyter_output><empty_output><jupyter_text>Detecting a single sample<jupyter_code>detect_outliers?
# first 100 samples as past data
past_data = df.iloc[:100]
# next sample as current data
current_data = df.iloc[101].to_frame().T

# detect if is outlier with flexibility 3 (a relaxed approach)
detect_outliers(past_data, current_data, 3)<jupyter_output><empty_output><jupyter_text>returns 1, which means that sample 101 isn't an outlier (based on the previous 100 samples)  Detecting outliers on a dataset, with a rolling window<jupyter_code># select 10k samples for time efficiency
N_SAMPLES = 10000 # let's perform outlier detection on 10k samples
WINDOW_SIZE = 100 # analyzed 100 samples in order to classify the next
ELASTICITY = 3 # relaxed detector

mini_df = df.iloc[:N_SAMPLES]
results = rolling_outlier_detector(mini_df, WINDOW_SIZE, ELASTICITY)<jupyter_output>100%|██████████| 10000/10000 [09:30<00:00, 17.54it/s]<jupyter_text>Plotting results<jupyter_code>px.scatter(mini_df.reset_index(), x="fecha" , y="chla", color=results)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11811594202898551, 'fraction_numerical': 0.043478260869565216, 'mean_word_length': 6.715083798882682, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '364407', 'license': 'MIT', 'n_tokens_mistral': 562, 'n_tokens_neox': 486, 'n_words': 120, 'path': 'example.ipynb', 'repo_name': 'martin-conur/timeseries_outlier_detection', 'size': '1380'}