ASAT: Adaptively Scaled Adversarial Training in Time Series

Adversarial training is a method for enhancing neural networks to improve the robustness against adversarial examples. Besides the security concerns of potential adversarial examples, adversarial training can also improve the generalization ability of neural networks, train robust neural networks, and provide interpretability for neural networks. In this work, we introduce adversarial training in time series analysis to enhance the neural networks for better generalization ability by taking the finance field as an example. Rethinking existing research on adversarial training, we propose the adaptively scaled adversarial training (ASAT) in time series analysis, by rescaling data at different time slots with adaptive scales. Experimental results show that the proposed ASAT can improve both the generalization ability and the adversarial robustness of neural networks compared to the baselines. Compared to the traditional adversarial training algorithm, ASAT can achieve better generalization ability and similar adversarial robustness.

Adversarial examples are generated by adding small malicious perturbations to the input data that can mislead models, which reveal the vulnerability of neural networks with respect to the input data. Adversarial training [2,5,6,7,8] is a method for enhancing neural networks that can improve both the adversarial robustness and the generalization ability at the same time. Besides the security concerns that neural network faces potential risks of adversarial example attack, researches on adversarial training focus on improving the accuracy of the neural networks [6,7,8], training robust neural networks [9,10,11] Our contributions are summarized as follows: • Rethinking existing adversarial training approaches, we propose to rescale dimensions of the perturbation according to their importance and adopt timedependent or adaptive scales with multi-step risk averaging. We propose an adaptively scaled adversarial training (ASAT) algorithm.
• We adopt ASAT on various baselines, including our newly proposed Transformer baselines. Results show that ASAT can improve both the generalization ability and the adversarial robustness of neural networks compared to the baselines. Compared to the traditional adversarial training algorithm, ASAT can achieve better generalization ability and similar adversarial robustness.
• With the proposed dimension-wise adversarial sensitivity indicator, we probe the sensitivities of different input dimensions and explain the decision bases of black-box neural networks, which provides more interpretability for neural network decisions.

Adversarial Training
In this section, we first introduce the concept of adversarial examples and adversarial training. Then, we describe the proposed dimension-wise adversarial sensitivity indicator inspired by the adversarial sensitivity.

Adversarial Examples
Let D denote the dataset, x = (x 1 , x 2 , · · · , x k ) T ∈ R k and y stand for a data input and its ground truth, f denote the neural network with the parameter vector θ and L denote the loss function. The adversarial example [1,2] can be defined as the small perturbation δ = (δ 1 , δ 2 , · · · , δ k ) T ∈ R k on the data input x that can mislead the model and cause the maximum increases in the loss function: where S denotes the constraint set, the perturbation is specified by S. For example, S = {δ : δ 2 ≤ } specifies that the perturbation should be in a k-dimensional hyperball with a radius of . Typical constraints are L p -norm bounded (1 ≤ p ≤ +∞) and S = {δ : δ p ≤ }, where the L p -norm is defined as: Since neural networks are sensitive to malicious perturbations in inputs, i.e., adversarial attacks, the adversarial risk is defined as the loss after adversarial attacks and the empirical risk is defined as the initial loss: The loss increases caused by adversarial attacks can be defined as the adversarial sensitivity: The adversarial risk or sensitivity can be defined on a dataset or a single data instance. The adversarial risk or sensitivity on the test dataset can evaluate the robustness of neural networks with respect to adversarial attacks, a lower adversarial sensitivity or risk means a better adversarial robustness.

Generating Adversarial Attacks
To generate adversarial attacks, the fast gradient methods [2], including the fast gradient sign method (FGSM) and the fast gradient method (FGM), propose to optimize Eq.(1) via optimizing the first-order Taylor Expansion term of the loss function instead when S = {δ : δ p ≤ }, namely maximizing the inner product of δ and the gradient g = ∇ x L(f (x, θ), y): where denotes element-wise product and sgn denotes element-wise sign function. FGSM adopts p = +∞ and FGM adopts p = 2.
Another representative line of existing studies for generating adversarial examples is PGD-based attacks [28,29]. Suppose S = {δ : δ p ≤ }. The projected gradient descent (PGD) algorithm optimizes Eq.(1) via a multi-step variant of fast gradient methods [2]: where δ 0 is usually set to the zero vector and u k is the k-th update in PGD. It is usually solved in Eq. (7) and {u : u p ≤ τ } and the step size is set to τ . Π S (v) is the projection function that projects v into the set S. In two common cases, L 2 -norm [8] and L +∞ -norm [28], the projection functions are: where the clip(h) function clips each dimension

Traditional Adversarial Training
Since neural networks are sensitive to adversarial examples, adversarial training [2,5] algorithms are designed to improve both the adversarial robustness and the generalization ability of neural networks. The target of traditional adversarial training is seeking to find the parameter vector θ with lower adversarial risks: [2] estimate the risk for adversarial attacks via the fast gradient methods. Other existing studies also estimate the risk via PGD-based attacks [5,6,7,8,29]. Fast gradient methods, including FGSM and FGM, can be seen as PGDbased attacks with the step number K = 1. Besides, some PGD-based adversarial training algorithms [7,8] minimize the adversarial sensitivities of multiple perturbations generated in different steps.

Dimension-wise Adversarial Sensitivity
Inspired by the adversarial sensitivity, we propose to probe the sensitivities of different input dimensions. Besides the adversarial sensitivity of the whole model, we can also define the dimension-wise adversarial sensitivity indicator, namely the sensitivity of the i-th input dimen- adv ( ) denotes the sensitivities of the i-th dimension, namely the expectation of the maximum loss change when the i-th dimension of the input changes no more than , which is defined as: The sensitivities of input dimensions reveal how the perturbations on different dimensions can cause the loss to increase. Neural networks pay more attention to dimensions with higher sensitivities and these dimensions tend to be relatively important dimensions for models to make decisions. Therefore, dimension-wise sensitivities can help reveal the decision basis of the model f and the importance of different dimensions in the prediction process. In traditional factor analysis methods, the factor loading [30] measures the importance of a factor, which can be seen as , (i = 1, 2, · · · , k), (13) namely R where g = ∇ x L(f (x, θ), y) and its i-th dimension is g i .
Namely R

Methodology
In this section, we first rethink the constraint set in adversarial training. Then, we propose to rescale different dimensions of the perturbation in the constraint set.
Last, we propose the adaptively scaled adversarial training (ASAT) algorithm.

Rescaling the Constraint in Adversarial Training
In this section, we rethink the constraint in adversarial training in time series and propose to rescale the constraint. 1 Note that L is only required to be convex and M -smooth in the neighborhood of θ instead of the entire R k .

Rethinking the Constraint in Adversarial Training
Adversarial training algorithms are widely adopted in the CV [1,2,5,6] and NLP [10,8,11] fields. The widelyadopted L p -norm constraint, S = {δ : δ p ≤ }, is invariant about different dimensions of the perturbation, which is reasonable in both the CV and NLP fields.
In the CV field, an important hypothesis of convolution neural networks is shift-invariant [25,26] and different pixels in an image can be treated of equal significance. In the NLP field, adversarial attacks can be conducted on the word embeddings and different dimensions of word embeddings are of similar significance [27,8].
However, in time series, different dimensions of inputs are not symmetrical. Therefore, directly adopting the L pnorm constraint in adversarial training in time series is not reasonable.

Rescaling the Constraint in Adversarial Training
Scales of all dimensions of the L p -norm bounded constraint are the same. However, in time series, dimensions with different timestamps may be of different significance.
The time series is usually defined as X = (v (1) , v (2) , · · · , v (T ) ), where v (t) ∈ R V is the input vector at the t-th timestamp and a smaller t means an earlier timestamp. For example, is the input vector at the first timestamp 9:00, Jan. 21, and v (2) is the input vector at the second timestamp 10:00, Jan. 21. In our work, we flatten a time series j is from the input vector with the t-th timestamp. Suppose x i is from the input vector with the t i -th timestamp, namely t i = i V , where · is the ceil function.
Suppose α = (α 1 , α 2 , · · · , α k ) T , we multiply the scale of dimension i of the perturbation with α i to rescale the radius of dimension i from to α i . As illustrated in Fig. 1, the L 2 -norm bounded constraint specifies a k-dimensional hyperball in R k (a circle in R 2 , as shown in Fig. 1a) and the L +∞ -norm bounded constraint specifies a k-dimensional hypercube in R k (a square in R 2 , as shown in Fig. 1b).
After rescaling, the hyperball is transformed into a hyperellipsoid (an ellipse in R 2 , as shown in Fig. 1c) and the hypercube is transformed into a hypercuboid (a rectangle in R 2 , as shown in Fig. 1d). The rescaling process assigns different importance to different dimensions of perturbations in adversarial training. Generally, the L p -norm constraint δ p ≤ is transformed to the constraint:

Fast Gradient Methods and PGD Algorithms after
Rescaling After rescaling, the constraint set is S = {δ : α −1 δ p ≤ }. For the fast gradient method target, namely maximizing the inner product of δ and the gradient g k−1 , the solution Eq.(7) becomes: For the projection function Π S (v) that projects v into the set S, we adopt the variants of Eq.(9) and Eq.(10): The proof and theoretical details are in Appendix.

Time-dependently Scaled and Adaptively Scaled Adversarial Training
In this section, we propose two lines of adversarial training with time-dependent or adaptive scales.

Multi-step Risk Averaging Adversarial Training
To

Time-dependently Scaled Adversarial Training
Adaptive scales α i are determined by the t i , we require α i ∈ [0, 1]. Since in time series analysis, data of closer timestamps with larger t i may be more important, We may assume α i ≤ α j when t i < t j , and max 1≤i≤k α i = 1. The scale α i decays when t i decreases.

Algorithm 1 Multi-step Risk Averaging Adversarial Training Algorithm with Dimension-wise Scales
Require: Network f ; parameters θ ∈ R k ; loss L and dataset D; constraint set S; step number K and step size τ ; optimizer O; iterations; batch size |B|.

7:
Calculate time-dependent or adaptive scales α i .
where the hyperparameter γ controls the decaying speed, and T denotes the number of different timestamps. Traditional adversarial training can be seen as adversarial training with constant scales.

Adaptively Scaled Adversarial Training
Time-dependent scales α i are determined by the i-th dimension in the data instance x i and its gradient g i = Similarly to time-dependent scales, we may assume that 0 ≤ α i ≤ 1 and max The ideal attack scales should relate to the scales of data instance, which will be analyzed later in detail in Theorem 2 and Sec. 3.3.2 that the ideal attack scales β should be proportional to the shifts between the training and test data. We may assume that the ideal attack scales or the data shifts are proportional to the scale of data instance |x i |, namely: As analyzed in Proposition 2, when conducting adaptive scales, the dimension-wise adversarial sensitivities are combined with Eq.(20), we may assume α i ∝ |xi| |gi| , namely In our implementation, we adopt: where we add a small positive real number e = 10 −6 on |x i | and |g i | for numerical stability.
As shown in Algorithm 1, we calculate adaptive scales

Theoretical Analysis
In this section, we first analyze the adversarial sensitivity under our proposed adaptively scaled adversarial attacks. Then, we reveal that the adversarial sensitivity under our proposed adaptively scaled adversarial attacks determines the generalization bound when the training and test distributions are not IID.

Adversarial Sensitivity after Rescaling
We show the adversarial sensitivity or risk under our proposed adaptively scaled adversarial attacks after rescaling in Theorem 1.

Theorem 1. Consider a neural network that is convex
and M -smooth in the neighborhood of θ, suppose g denotes the gradient, g = ∇ x L(f (x, θ), y), then its adversarial sensitivity or risk under our proposed adaptively scaled adversarial attacks where S = {δ : α −1 δ p ≤ } is:

Generalization Bound and Adversarial Sensitivity
Let z = (x, y), we may assume the training distribution P := p(z) and the test distribution Q := q(z) are two close distributions but there exists a distributional shift between , and R adv (S) is the adversarial sensitivity under adaptively scaled adversarial attacks on P. Define the l-th order distributional shift distance as: For example, when the training and test distributions are two multivariate Gaussian distributions, namely N (µ p , Σ p ) and N (µ q , Σ q ) respectively, then . When training and test distributions are close, namely µ p − µ q 2 and Σ q Σ −1 p − I 2 are bounded, Based on previous PAC-Bayes bounds [31], we provide a theoretical analysis in Theorem 2.
where S = {δ : β −1 δ p ≤ }, and the remainder term When |D| → +∞, we have Rem → 0, then the generalization bound is mainly determined by 2 . Theorem 1 and Theorem 2 reveal that the generalization bound L Q relates to the adversarial sensitivity R adv (S) under adaptively scaled adversarial attacks.
Traditional adversarial training algorithms with constant scales optimize the adversarial sensitivities under L p constrained adversarial attacks, while the proposed ASAT optimizes the adversarial sensitivities under adaptively scaled adversarial attacks, which are more precise estimations of the generalization bound. It implies that adaptive scales may help alleviate overfitting and improve the generalization ability compared to traditional adversarial training.
Since β is unknown, we adopt adaptive scales determined by data scales and gradients in Sec. 3.2.3.

Experiments
In this section, we first introduce the task and dataset details, then introduce the baseline models and experimental settings. Last, we report experimental results.  Table 1. and y is the ground truth, then these metrics are defined as: here x last denotes the volume of the last time slot, and ACC is the accuracy of whether the volume increases or decreases compared to the last time slot.

Baselines
We implement multiple moving average baselines and three neural network baselines in our experiments.

Moving average baselines
We adopt moving average baselines as our baselines, which are commonly used in technical analysis in finance.
In statistics, suppose a series of volumes v 1 , v 2 , · · · , v T , three commonly adopted estimations are: • Naive forecasting. The naive forecasting algorithm uses v T as the prediction. In our experiments, we try to adopt the volumes of yesterday or last time slot as the prediction.

Linear
The linear model is formulated as where x ∈ R 160 is the flattened vector of the concatenated input of both 12-slot and 20-day history.

LSTM
The long-short term memory (LSTM) [32] networks can capture features and long-term dependency in the entire sequences of data. Following [33], we implement two one-layer LSTM models with the global attention mechanism [34] to generate the representation vectors of previous time slots and 20-day history. Before feeding input into LSTM, we first adopt two linear layers to project the inputs into two higher dimensional spaces, respectively.
Then, we concatenate the representation vectors and feed them into a linear layer to regress the log volume. The input size, hidden size, and output size of LSTM models are 200.

Transformer
We also implement a six-layer Transformer [ of the last layer into a linear layer to regress the volume.
The input size, hidden size, and output size are 200, where the hidden states are split into 8 heads.

Settings and Choices of Hyperparameters
We train every model for 5 epochs and report the test performance on the checkpoint with the lowest valid loss. We

Experimental Results
After finding the best configurations of hyperparameters, the experimental results are shown in Table 2

Analysis
In this section, we first conduct an ablation study.
Then, we compare the adversarial robustness of neural networks under adversarial training. We also adopt ASAT to enhance existing methods. Last, we probe the sensitivities of different input dimensions and analyze the decision bases of neural networks, both on the dataset level and single instance level.

Ablation Study
We conduct an ablation study to investigate the influence of different mechanisms in our proposed ASAT.
In ASAT, as analyzed in Sec. 3.2.3, scales α i are determined by both |x i | and |g i | in Eq. (22). We try both α i = |x i |/max |x i | and α i = |g i | −1 /max |g i | −1 to verify the effectiveness of the proposed adaptive scales. The proposed ASAT algorithm minimizes the multi-step risk averaging target in Eq. (19). We try to adopt the traditional

Adversarial Robustness with ASAT
We compare the adversarial robustness of baseline models and models with adversarial training under multiple adversarial attacks on the five-minute dataset.
We first adopt multiple adversarial sensitivity indicators, including adversarial sensitivities under L 2 or L +∞ attacks and mean dimension-wise adversarial sensitivity.
Results are shown in We also visualize multiple adversarial attacks with several larger attacking scales. Experimental results are shown in Fig. 2. Similarly, we can conclude that ASAT can improve both the generalization ability and the adversar-    can improve the robustness of neural networks and achieve similar robustness to the 20-day average.

ASAT for Enhancing Existing Methods
We also implement two typical existing high-performance time-series prediction methods on the hourly dataset: Deep Auto-Regression (DeepAR) [36] and Multiple time scale LSTM (M-LSTM) [37]. The base models of both DeepAR and M-LSTM methods are LSTM models.
In DeepAR [36], we choose the Gaussian distribution as the prior distribution and adopt two linear layers to predict µ and σ of the predicted distribution respectively.    Namely, the prediction process and the training loss are: (y|µ, σ) = − log p(y|µ, σ) where h is the representation vector of the encoder, matrices U 1 , U 2 , and vectors b 1 , b 2 are learnable parameters.
µ, σ is the output of the neural networks, and y is the ground truth. In the inference phase, the prediction of the volume is µ. In M-LSTM, hidden states are split into four parts, and the time steps to propagate hidden states of four parts are 1, 2, 4, and 8, respectively.
The experimental results are in Table 5. Results show that DeepAR can enhance LSTM models in the volume prediction task, while M-LSTM cannot. ASAT can enhance these methods, which validates its effectiveness. Besides, the Transformer model enhanced with ASAT has a similar performance to the high-performance DeepAR [36] method enhanced with ASAT. The Transformer model is a high-performance time-series prediction model.

Probing Dimension-wise Sensitivities
We probe the dimension-wise sensitivities of different input dimensions in linear models, trained with three random seeds, the visualization results are shown in Fig. 3 and As shown in Fig. 3a and Fig. 3b    However, as shown in Fig. 3c, Fig. 4c and Fig. 4d, models with ASAT tend to pay close attention to some recent time slots and are sensitive to them, which accords with human intuition and is reasonable since recent time slots are more important in time series regularization. It also indicates that ASAT can alleviate overfitting since models with ASAT often focus on similar dimensions during multiple training.

Analysing Decision Basis on Individual Instance
In Eq.(12), the dimension-wise sensitivity can be defined on a dataset D or a single data instance (x, y). When  We choose the five-minute dataset, = 1, and randomly choose a data instance in the dataset to plot the dimension-wise sensitivities on the dataset and a single data instance. As shown in Fig. 5, Fig. 5a shows the dimension-wise sensitivities of a Transformer model trained with ASAT on the dataset. We can see that the model mainly concerns the volumes of nearest time slots and volumes in 20-day history. We also probe the decision basis on the data instance, whose volumes are shown in Fig. 5b.
The volumes of the 9-th nearest time slots, the 14-th nearest day in history, and the 4-th nearest day in history are relatively low compared to other time slots or days in his-tory. Both the baseline Transformer (shown in Fig. 5c) and the ASAT Transformer (shown in Fig. 5d) are sensitive to these time slots or days. They pay close attention to these abnormal volumes and the corresponding prices.
Therefore, their decisions on this data instance can capture these abnormal fluctuations. Moreover, the model with ASAT tends to be more sensitive to abnormal fluctuations, which indicates that ASAT can help the decision bases of models be more reasonable and more explainable.  [1,2,3,4,5,38,27,39], existing studies also concern improving the adversarial robustness and generalization ability of neural networks via adversarial training [2,5,6,7,8]. Adversarial training is widely adopted in both the computer vision field [2,5,6,7,40,41] and the natural language processing field [11,10,8,9,42].

Generative Adversarial Networks
Generative Adversarial Networks (GANs) [24] are also applications of adversarial training, which train both the generator model and the discriminator model with an adversarial learning target. GANs have been widely used in multiple machine learning fields for generating highquality synthetic data, such as computer vision [43,44], natural language processing [45,46], and time series synthesis [14,15]. In the finance field, several GAN models [14,15,16,17,18,19,20,21,22,23] are proposed to generate or synthesize better time-series data that can preserve temporal dynamics in time-series.

Conclusion
In this work, we propose adaptively scaled adversarial Proof. Since when |δ i | ≤ , δ =i = 0, according to Taylor series with Lagrange form of the remainder, there exists η near x, such that: Appendix A.2. Proofs of Proposition 2 Proposition 2. Consider a neural network that its loss function is convex and M -smooth in the neighborhood of θ 2 . Then, where g = ∇ x L(f (x, θ), y) and its i-th dimension is g i . Namely R (i) adv ( ) ∝ E|g i | holds approximately when is small.
Proof. Since when |δ i | ≤ , δ =i = 0, according to Taylor series with Lagrange form of the remainder, there exists η near x, such that: therefore, Appendix A.3. Proofs of Theorem 1 Theorem 1. Consider a neural network that is convex and M -smooth in the neighborhood of θ, suppose g denotes the gradient, g = ∇ x L(f (x, θ), y), then its adversarial sensitivity or risk under our proposed adaptively scaled adversarial attacks where S = {δ : α −1 δ p ≤ } is: Proof. Since L is convex and M -smooth near θ, According to Holder Inequality,   Proof. Suppose 1 p + 1 q = 1, according to the Holder Inequality, δ T g = (α −1 δ) T (α g) ≤ α −1 δ p α g q = α g q .

Appendix B. Hypothesis Tests
We also conduct student-t tests on the hourly dataset to verify that the proposed ASAT outperforms baselines statistically significantly (p < 0.05). The null hypothesis is that the outperform of ASAT is worse than the baseline.
The degree of freedom is df= 5 + 5 − 2 = 8 and the critical t-value is 1.86. Experimental results of hypothesis tests are in Table B.6. We can see that t-values of MLP and Transformer models are larger than 1.86. For the LSTM model, the improvements in ASAT are not significant.