Implementation details For preprocessing, we set window size \(W\)  to be 120 for each dataset. The sequential VAE based models, STORN, and SISVAE have a similar network architecture, which is defined by two varying hyper-parameters, h_dim for dimension of hidden layer, and z_dim for dimension of latent variable \(\mathbf {z}\) . We set h_dim to be 200, and z_dim to be 40 for all datasets. We empirically set the smoothness regularization hyper-parameter \(\lambda =0.5\)  for SISVAE during training process. The encoder, decoder, and RNN of models use the same hidden dimension. The three models are implemented using PyTorch [1]} 0.4, and trained on a single Nvidia RTX 2080ti GPU. We train the three models 200 epochs using Adam [2]} with learning rate 0.001. Note for Donut, the M-ELBO technique proposed by Donut may require some anomaly labels to learn `normal' patterns. For fair comparison, we provide no anomaly label to the model, and the experiments are conducted under fully unsupervised setting. We use the code open sourced by the authors. We set h_dim to be 200, and z_dim to be 40, same as our model. The model is fundamentally designated for uni-variate time series and it is nontrivial to extend it to the multi-dimensional case. We test two settings: 1) train a model for each sequence individually, and compute anomaly score individually; 2) flatten the multi-dimensional time series data after preprocessing to accommodate the model, the anomaly score is computed together. We find the second protocol exhibits better performance. This may be because the normal patterns are common along different sequences. With more sequences as input, the model can be less influenced by anomalies. We adopt the second protocol for Donut in the experiments. For all deep latent variable models, we use negative reconstruction probability as anomaly score output. For all three traditional statistical models, we use the same preprocessing method. For history average (HA), the anomaly score is absolute observation value since all series have zero mean. For ARMA and LDS, we use absolute reconstruction error as anomaly score.
<TABLE><FIGURE>