<jupyter_start><jupyter_text>Generate some time series data For this tutorial, I will use the TwoPatterns dataset that I downloaded here:http://www.timeseriesclassification.com/description.php?Dataset=TwoPatternsAfter downloading and storing the data, I will first load them from the text files and plot them.<jupyter_code>ts_path = '/Users/vincentvercruyssen/projects/maintained_repositories/anomatools/data/time_series/TwoPatterns/'

# load the train data
with open(os.path.join(ts_path, 'TwoPatterns_TRAIN.txt'), 'r') as f:
    lines = f.readlines()
ts_train = []
for line in lines:
    nl = [float(l.strip()) for l in line.split(' ') if not(l == '')]
    ts_train.append(nl)
ts_train = np.array(ts_train)
print('Size of the training data:', ts_train.shape)

# load the test data
with open(os.path.join(ts_path, 'TwoPatterns_TEST.txt'), 'r') as f:
    lines = f.readlines()
ts_test = []
for line in lines:
    nl = [float(l.strip()) for l in line.split(' ') if not(l == '')]
    ts_test.append(nl)
ts_test = np.array(ts_test)
print('Size of the testing data:', ts_test.shape)

# combine the train and test set and extract the labels
ts = np.vstack((ts_train, ts_test))
print('Size of the combined data:', ts.shape)
original_labels = ts[:, 0].flatten().astype(int)
ts = ts[:, 1:]
print('Number of labels per category:', Counter(original_labels))
c = 0.05
normal_class = 1
size=500

# construct an anomaly detection dataset (by selecting one class as normal and subsample the other classes to get the anomalies)
ts_normal = ts[original_labels == normal_class, :]
ts_anomaly = ts[original_labels != normal_class, :]
np.random.shuffle(ts_anomaly)
ts_normal = ts_normal[:size, :]
nn = ts_normal.shape[0]
na = int(nn * c)
X = np.vstack((ts_normal, ts_anomaly[:na, :])).astype(np.double)
y = np.ones(X.shape[0])
y[:nn] = -1

# plot the result (10 normals and 10 anomalies)
f, axes = plt.subplots(1, 2, figsize=(15, 4))
Xn = X[y == -1, :]
Xa = X[y == 1, :]
for i in range(4):
    axes[0].plot(np.arange(0, X.shape[1], 1), Xn[i, :], color='blue')
    axes[1].plot(np.arange(0, X.shape[1], 1), Xa[i, :], color='red')
for i in range(2):
    axes[i].grid(alpha=0.4)<jupyter_output><empty_output><jupyter_text>Anomaly detection with different detectors and DTW distance We need to add our custom (DTW) distance metric to sklearn methods.First, let's try the different detectors.<jupyter_code>#detector = kNNO(k=10, metric='euclidean', contamination=0.05)
#detector = iNNE(metric='dtw', sample_size=8, contamination=0.05)
detector = SSkNNO(k=10, metric='dtw', contamination=0.05)
#detector = SSDO(metric='dtw',  contamination=0.05)
ytr = detector.fit_predict(X)
test_probs = detector.predict_proba(X, method='unify')[:, 1]  # anomaly probabilities

# plot the result (10 normals and 10 anomalies)
f, axes = plt.subplots(1, 2, figsize=(15, 4))
Xn = X[ytr == -1, :]
Xa = X[ytr == 1, :]
for i in range(4):
    axes[0].plot(np.arange(0, X.shape[1], 1), Xn[i, :], color='blue')
    axes[1].plot(np.arange(0, X.shape[1], 1), Xa[i, :], color='red')
for i in range(2):
    axes[i].grid(alpha=0.4)<jupyter_output><empty_output><jupyter_text>Add some labels<jupyter_code>detector = SSkNNO(k=10, metric='dtw', contamination=0.05)
yl = np.zeros(len(y), dtype=int)
yl[:5] = -1
ytr = detector.fit_predict(X, yl)
test_probs = detector.predict_proba(X, method='unify')[:, 1]  # anomaly probabilities

# plot the result (10 normals and 10 anomalies)
f, axes = plt.subplots(1, 2, figsize=(15, 4))
Xn = X[ytr == -1, :]
Xa = X[ytr == 1, :]
for i in range(4):
    axes[0].plot(np.arange(0, X.shape[1], 1), Xn[i, :], color='blue')
    axes[1].plot(np.arange(0, X.shape[1], 1), Xa[i, :], color='red')
for i in range(2):
    axes[i].grid(alpha=0.4)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17045150948437082, 'fraction_numerical': 0.028586695164306706, 'mean_word_length': 5.157894736842105, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '80389', 'license': 'Apache-2.0', 'n_tokens_mistral': 1521, 'n_tokens_neox': 1429, 'n_words': 364, 'path': 'notebooks/Time_series_anomaly_detection_example.ipynb', 'repo_name': 'Vincent-Vercruyssen/anomatools', 'size': '3743'}