A framework for low-resource video domain adaptation using a supervised disentangled learning strategy that is particularly well-suited to keystroke inference attacks.
Background
Keystroke Inference Attacks
Much of the early works in vision-based keystroke inference attacks have focused on direct line of sight and reflective surfaces
[1]}, [2]}, [3]}, [4]}, [5]} to infer sensitive data.
These data driven approaches are necessary because the attacker can not recover the text using off-the-shelf optical character recognition software (OCR) at low resolutions [4]}.
Loosely speaking, the attackers train models that account for various capture angles by aligning the user's mobile phone to a template keyboard.
Collectively, these works showed that attackers are able to successfully recover pins and, sometimes, even full sentences.
In this paper, we advance the state-of-the-art under the direct line of sight model wherein the attacker uses a mobile camera to record a victim's mobile phone usage. Most germane is the work of
Lim et al. [5]} that created a simulator that generates synthetic data for keystroke inference attacks. The authors showed that training with both synthetic and real data, in a supervised domain adaptation framework, yielded a CNN that generalized to a real-life test set. Unfortunately, that work is limited in scope due to the restricted threat scenario they target: analyzing single keypresses.
By contrast, we assess the ability of an attacker (armed with deep-learning techniques) to recover complete sequences in more demanding settings.

Style and Content Disentanglement in Videos
Tenenbaum and Freeman ([8]} [9]}) observe that by learning to factor observations of data into two independent factors of variation, style and content, models learn separate representations that can extrapolate style into novel content, classify content in different styles, and translate new content into new styles.
Others have disentangled videos into a time-dependent style representation and time-independent content with adversarial training [10]} or with variational autoencoders [11]}, [12]}. These methods are all unsupervised methods to disentangle style and content.
In our setting, style and content are both time-dependent.
Style encapsulates the trajectory of the finger in between keys or speed of the user typing.
The difference in texture on a per-frame basis is also encapsulated by style.
Content represents the entire trajectory as that determines the sentence that was typed. 
Since we have labels, we take heed of statements made by Locatello et al. [13]}): learning disentangled representations is impossible without supervision, and unsupervised methods using temporal inductive biases do not lead to improved disentangled representations.

Low Resource Domain Adaptation
We operate in a low resource setting in which we have abundant labels in the source domain and have very few, albeit labeled, data points in the target domain.
[14]} extend the CyCada [15]} and CycleGAN [16]} frameworks to the low resource domain adaptation setting by adding a semantic consistency loss.
[17]} addresses this problem by learning a feature space that is domain invariant, but is semantically aligned across both domains by introducing a pairing process that augments the datapoints in the target domain.

Video Domain Adaptation
Domain adaptation for videos has been under explored relative to images, with nearly all methods being limited to human action recognition [18]}, [19]}.
Domain adaptation techniques used for action recognition are not easily transferable to our setting because action recognition methods typically only need a small fraction of frames from the entire video [20]}.
However, in our settings, we need to process every frame in order to predict the entire sequence that a user typed.
Video translation methods such as [21]}, [22]}, and [23]} show some potential for video domain adaptation tasks, but these methods require that the two domains (RGB images to semantic labels, for example) have the same temporal dynamics.

Our Approach
<FIGURE>We first provide a brief introduction to keystroke inference attacks and then describe our framework to disentangle the style and content latent spaces to train on all style-content pairs.
An overview of our method is shown in Figure REF .
Keystroke Inference Attacks
We model the keystroke inference attack as a Seq2Seq [24]} problem where the input \(X = \lbrace  x_{1}, x_{2}, ..., x_{k} \rbrace \)  is a video with \(k\)  frames and \(Y = \lbrace  y_{1}, y_{2}, ..., y_{j} \rbrace \)  is a sequence of \(j\)  characters.
The videos are of users typing on their mobile phones that are cropped and aligned to a template image.
The tokens are a sequence of characters of the sentence the user typed.
We do not use any paired data (i.e. the synthetic and real-life datasets do not contain the same sentences), and do not have access to any auxiliary labels such as the exact frame in which a key was pressed.
Our goal is to learn the parameters of a model that maximizes the conditional probability of \(Y\)  given \(X\) .
We use a Transformer [25]} encoder-decoder as our model.
In our setting, we have a dataset of synthetic videos, \(\mathcal {D}_s = \lbrace  (X^s_i, Y^s_i) \rbrace  \) , and a dataset of real-life videos \(\mathcal {D}_t = \lbrace  (X^t_i, Y^t_i) \rbrace \) , where the number of real-life videos is significantly less than the synthetic.
While a large synthetic dataset can be easily generated, there is a distribution shift between the two domains (Figure REF ).
Moreover, when the amount of labeled data is scarce, it can be challenging to train neural networks that generalize to samples outside of the training set.

Disentangling Style and Content
To address the lack of real-life data, we train on combinations of style and content representation pairs from the synthetic and real domains. Additionally,
we introduce auxiliary losses to enforce disentanglement of style and content, ensuring that the style latent space does not contain any information about the content, and vice versa.
Our training framework consists of a Content Encoder, a Style Encoder, a Decoder, a Feature Aggregation Module, a Style Discriminator, a Content Discriminator, and a Domain-Class Discriminator (see Fig REF ). In what follows, we only discuss the intuition and higher level details necessary for understanding how our method works. The loss functions and low-level training specifics are given in the Appendix.
Pretraining Synthetic Model
We first pretrain an Encoder-Decoder Transformer on synthetic data only.
We train this network with a multi-class cross entropy loss where the goal is to predict the correct sentence for a given video.
Then the Content Encoder, Style Encoder, and Content Discriminator are initialized with the weights of the pretrained Encoder, and the Decoder is initialized with the weights of the pretrained Decoder.

Style Disentanglement
Style disentanglement ensures that style information is removed from the content latent space.
The content latent space is defined as the output of the content encoder given a synthetic or real video.
The content encoder is trained to produce content feature representations that are domain invariant.
For example, encoding synthetic and real videos of the sentence “hello, how are you?” should be close together in the feature space since they have the same semantic information.
To achieve this, we train this network in an adversarial fashion [26]}.
Specifically, the Style Discriminator is trained to classify whether a content embedding is real or synthetic, and the Content Encoder is trained to spoof the Style Discriminator.
Further information can be found in Section REF  of the Appendix.

Content Disentanglement
Content disentanglement ensures that content information is removed from the style latent space.
The style latent space is defined as the output of the Style Encoder given a real or synthetic video.
The Content Discriminator is a Transformer Decoder that is trained to predict the correct sentence given the input style representation.
The Style Encoder is trained to spoof the Content Discriminator.
We do so by producing a style feature representation such that the Content Discriminator can not predict the correct sentence.
We achieve this by maximizing the entropy, \(H\) , of the predictions of the Content Discriminator.

Feature Aggregation
A Feature Aggregation Module combines the disentangled representations from the previous two steps.
The aggregation module combines any given pair of style and content embeddings to produce one embedding.
For the experiments that follow, we use the LayerNorm [27]} operation with learnable parameters as our feature aggregation module.
There are four different possible pairs that can be the input to our model, since there are two factors of variation (style and content) and two domains (synthetic and real-life).
For any given input pair, the output feature representation can be thought as the content in the style of the specified domain.

Prediction
The Decoder takes in the output of the feature aggregation module and outputs the predicted sentence, and is trained with cross-entropy loss.
At test time, the model outputs the most likely sentence given a real-life video.

Semantic Alignment
Lastly, to further encourage style and content separation, we extend the framework of [17]} to create training pairs to compensate for limited data in one domain.
We create four pairs \(\mathcal {G}_k, k \in \lbrace 1,2,3,4\rbrace  \) .
\(\mathcal {G}_1\)  and \(\mathcal {G}_2\)  are outputs of \(M\)  that share synthetic content: (Synthetic Style, Synthetic Content) and (Real Style, Synthetic Content).
\(\mathcal {G}_3\)  and \(\mathcal {G}_4\)  share real content: (Synthetic Style, Real Content) and (Real Style, Real Content).
A multi-class discriminator is trained to correctly identify which group every output of \(M\)  belongs to.
The Content Encoder, Style Encoder, and Feature Aggregation module are trained adversarially such that the multi-class discriminator can not distinguish outputs of the feature aggregation module that are in \(\mathcal {G}_1\)  and \(\mathcal {G}_2\)  and outputs of \(M\)  that are in \(\mathcal {G}_3\)  and \(\mathcal {G}_4\) .
The high level overview is given in Algorithm REF .The revised loss function used to train our model is given in the appendix.
Content Encoder, Style Encoder, Feature Aggregation Module, Style Discriminator, Content Discriminator, Decoder, Multi-Class Discriminator, Synthetic Dataset, Real-life Dataset.
Not Converged
sample mini-batch of b synthetic samples, {\((X_1^s, Y_1^s)\) , \(\cdots \) , \((X_b^s, Y_b^s)\)  } from the synthetic dataset.
sample mini-batch of b real-life samples, {\((X_1^t, Y_1^t)\) , \(\cdots \) , \((X_b^t, Y_b^t)\)  } from the real-life dataset.
Style Disentanglement: Remove Style information from the Content Space
update the Style Discriminator and Content Encoder
Content Disentanglement: Remove Content information from the Style Space
update Content Decoder
update Style Encoder
Sequence Prediction
update Content Encoder, Style Encoder, Decoder, and Feature Aggregation Module
Semantic Alignment
update Multi Class Discriminator 
update Feature Aggregation Module, Content Encoder, and Style Encoder
Learning Algorithm for Disentangling Style and Content.

Experiments
Next, we describe the datasets we used, the motivation and interpretation of our chosen evaluation metrics, and our experimental results. To support reproducible research, additional details regarding our data collection methodology and network architectures are given in the Appendix.
<FIGURE>Datasets
Figure REF  shows different statistics for the synthetic and real datasets.
We set aside 10% of the training set as a validation set.
The real-life dataset was collected by recording participants typing sentences into a mobile phone.
Three participants were asked to type conversational text messages into their mobile devices while we recorded them in both indoor and outdoor settings, with the screen brightness varying according to the environment.
We asked the participants to type only with their right thumb.
We used a mobile camera and captured from a distance of 3 meters.
The synthetic data was generated using a simulator [5]} we built.
The simulator outputs aligned videos of a synthetic thumb typing a given set of sentences.
We generated sentences from the “A Million News Headlines” datasethttps://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/SYBGZL
We added a START and STOP token to the beginning and end of a sentence, respectively.
In total, there are 30 tokens in which the decoder can predict: 26 letters and 4 special tokens (START, STOP, SPACE, PAD).
In both settings, we used a QWERTY keyboard layout.

Evaluation Metrics
We use a variety of metrics to quantify the amount of information the attacker is able to recover from the user because there is no single, agreed-upon, metric for keystroke inference attacks.
First, we postprocess the outputs of our model with a language model, similar to that done elsewhere [2]}, [3]}, [32]}, [33]}.
Appropriate metrics for this scenario are Bleu-n [34]}, ROUGE [35]}, and METEOR [36]}.
Bleu-n scores are scored on n-gram precision, i.e., the n-grams in the predicted sentence that are also in the ground truth sentence.
ROUGE scores are scored on n-gram recall, i.e., the n-grams in the ground truth that are also in the predicted sentence.
METEOR is a metric that is scored on the harmonic mean of unigram precision and recall and was developed to address some of the drawbacks of ROUGE and Bleu-n.
METEOR scores range from 0 to 1.
Scores above 0.5 reflect understandable translations and scores above 0.7 reflect fluent ones [37]}.
While these scores have merit in the context of keystroke inference attacks, they are not without shortcomings.
These scores are especially harsh for predictions that contain slight typographical errors (e.g., “hello” vs. “hellp”), and there is no guarantee that the previously mentioned postprocessing steps will address every error.
Also, there are settings in which the applicability of these metrics does not make sense — e.g., recovering alphanumeric passwords.
Thus, we also need evaluation metrics for the raw outputs of our model.
Two appropriate metrics are Translation Edit Rate (TER) [38]} and a QWERTY-keyboard-based edit distance.
Both metrics measure the number of edits required for a hypothesis sentence to be translated to the ground truth.
The latter is a form of the Damerau–Levenshtein (DL) distance [39]} that penalizes the edit operations (i,e., insertions, deletions, substitutions, character swapping) conditioned on the QWERTY keyboard layout.
For example, if "hello" was the ground truth word, "hellp" should be less penalized than "hellv" as the former is a more likely output than the latter given the assumed keyboard layout.

Network Architectures
For all experiments, the Encoders (\(E_C\) , \(E_S\) ) and Decoders (\(D_C\) , \(G\) ) are both Transformers with 4 layers, 4 attention heads, an embedding size of 128 and a hidden size of 256.
\(D_M\)  and \(D_S\)  are both 1-layer fully connected layers.
Since the output of the Encoder is a sequence of \(n\)  continuous representations, where \(n\)  is the input sequence length, we do a max pooling operation along the temporal dimension so that we have a fixed vector representation.
These fixed vector representations are the direct inputs to \(D_M\)  and \(D_S\) .
The max sequence length is set at 300, and the max phrase length is set at 70.
If an input sequence has more than 300 frames, we randomly sample 300 frames at each epoch.
If a video in the testing or validation set has more than 300 frames, we fix the indices of the sampled frames to remove any randomness for evaluation.
For input sequences that are shorter than 300 frames, we zero-pad the remaining sequence.
Table REF  shows the results for a model trained and tested on synthetic data.
The model performs very well on the synthetic test set across all proposed evaluation metrics.
To lessen the compute cost of processing over 45k raw videos, we extract a fixed 128-dimensional feature representation as a preprocessing step by training a CNN for single key press classification.
We use the simulator to generate single key press images and train a CNN to predict the correct key.
<TABLE>
Baselines
To evaluate our approach, we compare against several alternative ideas: finetuning, ADDA, [40]}, CyCADA [15]}, and Vid2Vid [21]}.
All methods are evaluated on the real-life test set and use the model trained on synthetic data.
[noitemsep,topsep=0pt,leftmargin=*]
Finetuning. We finetune a model trained only on synthetic with the real-life training set.

ADDA. We use ADDA to generate Encoder output feature representations that are domain invariant to a Discriminator, but are also discriminative for the Decoder.

CyCADA.
We learn a pixel-wise transformation that transforms data from one domain to another.
We apply this transformation to every real frame to a synthetic one.
Then, we finetune the synthetic model with the transformed real training set and test on the transformed real test set.
Finetuning is needed because [15]} do not address the temporal shift as the transformations are conducted on a per-frame basis.

Vid2Vid
We leverage a video translation framework that learns to map videos from one domain to another (e.g., labels to RGB images).
Our aim is to translate the real-life videos in our training set into synthetic versions, finetune the model trained only on synthetic data, translate the real-life videos in our testing set into real-life versions, and test on the transformed real-life test set.

Findings
Although we were successful in applying ADDA to the task of single key press classification [5]} when the number of labeled data is scarce,
we found that simply applying ADDA to our sequence prediction task leads to severe overfitting due to the limited real-life data, indicating that this task is more challenging than the single keypress classification task.
While CyCADA and ADDA are common approaches for visual domain adaptation, we did not find them suitable for our sequence prediction problem.
This is because these approaches are tailored to domains in which the domain shift is limited to textures.
Recall that in our case, we are facing both a kinematic and texture domain shift.
This is especially true for CyCADA and other pixel-wise transformation methods.
We carried out numerous experiments to tune our baselines and maximize their performance, but despite an extensive search of hyperparameters, the models still overfit.
We report the best results in Table REF .
Another important fact is that Vid2Vid is trained with pairs that are composed of a video in one domain (RGB) paired with the same with video in another domain (Semantic Labels). This provides both global and local supervision.
By global supervision, we mean that the video for the two domains are of the same event.
By local supervision, we mean that there is supervision on a per-frame basis.
For example, every RGB frame corresponds to a semantic label frame.
While such supervision exists for the datasets ([45]}, [46]}, [47]}) used by Wang et al. [21]}, we are unable to simulate such supervision.
We can generate synthetic versions of any real-life video (global supervision), but we are unable to simulate the synthetic thumb trajectories such that the temporal dynamics are the same (local supervision).
Despite this, we still attempt to generate realistic videos with only global supervision.
Our aim is to transform every real-life video into a synthetic one.
For Vid2Vid, we used the official implementation provided by the authors.https://github.com/NVIDIA/vid2vid
First, we generate a synthetic video for each of the real-life videos in our training and test sets.
Next, we clip the lengths of the videos so that the number of frames is equal for a given (synthetic, real) video pair.
Then, we train using this set of real and synthetic pairs using the default hyperparameters used in [21]}.
Once the generator is trained, we transform each real-life training video to a synthetic version.
We finetune the pretrained synthetic model using the transformed real-life training set.
Finally, we test on transformed real-life videos. Even so, this method failed to yield any results as the generator was unable to generate plausible looking synthetic videos – underscoring the need for a new approach like ours.

Adapting to Real-Life Videos
<TABLE><FIGURE>Our method, unlike the above baselines, does not overfit to the real-life training set.
Our results show that training with our pairing mechanism with disentangled representations across domains is an effective form of data augmentation.
We outperform the baselines in both raw output evaluations and post-processed evaluations as shown in Table REF .
We found that our training was not sensitive to the hyperparameters and weightings of the loss terms (in Equation REF  in the Appendix), and use the same hyperparameters for all experiments.
While a direct comparison to the state of the art in direct line of sight attacks is difficult due to the differences in datasets, it is worth noting how our model performs relative to others.
Raguram et al. [2]} achieve a METEOR score of 0.89 whereas Xu et al. [3]} achieve a score of 0.71, albeit with recordings taken from much farther distances.
To measure an attacker's ability to recover passwords, Raguram et al. [2]} report precision and recall for individual word units and characters.
They achieve word-level precision and recall of 75% and 78%, respectively, and character-level scores of 94% and 98%.
We achieve a word-level precision and recall of 78% and 79%, respectively, and a precision and recall of 96% and 95%, respectively, for characters.
[2]} does not report METEOR scores for this scenario.
In their experiments, Raguram et al. [2]} use three different cameras in their experiments: Canon VIXIA HG21 Camcorder, Kodak PlayTouch
and Sanyo VPC-CG20.
Xu et al. [3]}, on the other hand, use a Canon 60D DSLR with 400mm lens and a Canon VIXIA HG21 Camcorder.
<FIGURE>Feature Visualization
For pedagogical purposes, the t-sne [56]} plots in Figure REF  show the feature representations of \(E_S\) , \(E_C\) , and \(M\)  on synthetic and real test data. Notice how that sentences with different styles have a noticeable separation, whereas the content representations are intertwined.
The last figure on the right shows outputs of our feature aggregation module, \(M\) , and shows the transfer of styles in the feature space.
Notice the clear separation between styles, while the datapoints within one style cluster are mixed.
To obtain inputs suitable for t-sne, we perform a max pooling operation along the temporal dimension of the outputs of the networks.

Ablation
Lastly, we conducted a series ablation studies to explore the effectiveness of our proposed framework.
We introduce seven different models: [label=]
I is our base method without the use of any adversarial losses, just the pairing mechanism.

II uses style disentanglement, i.e., I \(+\)  style disentanglement.

III uses content disentanglement, i.e., I \(+\)  content disentanglement.

IV uses both style and content disentanglement.

V is the base method with the modified semantic alignment loss.

VI uses style and content disentanglement, along with the semantic alignment. This is our proposed method trained with Algorithm REF , i.e., IV \(+\)  semantic alignment.
First, we find that our base model (I) achieves competitive results without any losses on the latent spaces.
This indicates that training on paired representations across domains is an effective method for data augmentation.
Second, we find that adding auxiliary losses on the latent spaces to enforce style and content disentanglement improves performance.
The performance for models II and III shows the base model is benefiting from the added loss terms.
The results for Model IV aligns with our hypothesis that explicitly disentangling style and content allows us to overcome the lack of training data in the target domain by training with all combinations of the factors of variation.
Finally, we trained model V to apply the semantic alignment step on our paired outputs without any additional adversarial losses.
This is quite competitive with IV, but we find the greatest performance boost when training model VI using both semantic alignment and disentanglement.
A closer look into the distribution of the scores in Figure REF  shows that the distribution of scores for Model VI (Full) indicates higher overall performance compared to Model I (Base).
Our results show that explicitly disentangling style and content by adding the adversarial losses on the latent spaces, supplements the pairing mechanism to achieve the highest performance under the evaluation metrics.
Implications
Taken as a whole, our results show an adversary seeking to deploy keystroke inference attacks can leverage deep learning methods, despite the difficulties in curating training data.
While we are unable to directly show whether such an adversary would outperform those in prior attempts, we show that we can train a deep learning model using the same amount of real-life data used in previous studies.
Thus, the settings for vision-based keystroke inference attacks should be revisited as the realism and threat capacity of these attacks are most likely greater than initially thought. In particular, the lack of training was a significant impediment to many earlier proposals, but having a synthetic-to-real domain adaptation framework like ours — to augment the limited real data — would likely lend itself to a more capable adversary.
Furthermore, research over the past few years has shown that deep learning methods have outperformed shallow methods in most computer vision tasks. Arguably, if we hold all the parameters of a threat scenario constant (distance, camera model, phone model), an attacker using a deep learning method should outperform one with a shallow method. Similarly, our style and content disentanglement techniques can be used to help an attacker thwart certain defenses.
For example, [32]} proposed the use of random device perturbations as a way to mitigate an attacker's ability to map backside perturbations to keystrokes. While this is an effective defense, an attacker can undermine it by training a model to disentangle the fake perturbations from the real backside perturbations.

Conclusion
Our work provides the important initial step needed to formulate defenses for keystroke inference attacks in the age of deep learning.
Specifically, we provide the first assessment of an attacker's ability to recover sentence level information using deep learning systems.
We address the problem of limited training data by introducing a framework for low resource video domain adaptation, that disentangles the style and content across both domains, and creates representations from all pairs of style and content combinations.
Our results indicate that training with these pairs, along with auxiliary losses to explicitly disentangle style and content, serves as an effective form of data augmentation that prevents overfitting.
We evaluate our method using a number of metrics to quantify the amount of information the attacker is able to recover, and our results show that an attacker armed with a deep learning system is able to recover enough information to pose a significant threat to unsuspecting victims.
Our framework can also be used to assess other keystroke inference attacks such as those that focus on device perturbations [32]} or eye gaze [33]}.

Availability
The code and data used in this paper can be found at https://github.com/jlim13/keystroke-inference-attack-deep-learning.

Additional Training Details
Synthetic Single Key Press Classifier
We train a CNN, \(\phi (\cdot )\) , for the task of single key press classification in order to learn a \(d-\) dimensional (\(d = 128\) ) feature extractor.
Once this network is fully trained for the task of single key press classification, we can extract the features of each video on a per-frame basis. We use the simulator by [5]} to generate 70,000 single key press images.
These images contain the synthetic thumb over one of 27 keys on the QWERTY keyboard (26 letters \(+\)  the space bar).
Once generated, these images are preprocessed in a similar fashion to the synthetic video dataset.
We resize the images to size 200 X 100 and crop the phone such that only the keyboard is showing.
We use 50,000 images for training and 10,000 images for testing and validation, respectively. We use a CNN where each layer consists of a Convolution Layer, ReLU activation, and MaxPool operation.
We use 3 layers and 2 fully connected layers.
We train a network to achieve 95% accuracy on a held out test set without much hyperparameter or architecture search, as this is a fairly simple 27-way classification task.
We use this final model to preprocess every frame in our synthetic video dataset.
Every video is a now a sequence of these \(d-\) dimensional feature representations.
We use the Adam optimizer with a learning rate of 0.0001.

Real-Life Single Key Press Classifier
When extracting the visual features for the real-life videos, we can not use a feature extractor that was trained only on synthetic data.
There is a distribution shift between the synthetic and real-life data, so the features we extract would be not be informative.
Instead of using \(\phi (\cdot )\)  that was trained for single keypress classification on just synthetic data, we train \(\phi (\cdot )\)  with a combination of synthetic and real-life data.
Specifically, we adopt the ADDA [40]} framework for unsupervised domain adaptation to train \(\phi (\cdot )\) .
We treat the individual frames for all of the videos in our real-life training set as unlabeled data.
Even though we do not have labels for individual keypresses for real-life data, we can leverage the fact that we have abundant labels for synthetic data by adopting the unsupervised domain adaptation technique ADDA.
We use the CNN for single key press classification on synthetic data as our pretrained network.
The Discriminator is a 1 layer, 128-dimensional fully connected layer followed by a sigmoid.
We follow the same guidelines to train ADDA as the original paper [40]}, and refer the reader to this work for the full description of their training process.
We use the Adam optimizer and a learning rate of 0.0001 for both the Discriminator and CNN.

Network Architectures
For all experiments, the Encoders (\(E_C\) , \(E_S\) ) and Decoders (\(D_C\) , \(G\) ) are both Transformers with 4 layers, 4 attention heads, an embedding size of 128 and a hidden size of 256.
\(D_M\)  and \(D_S\)  are both 1-layer fully connected layers.
Since the output of the Encoder is a sequence of \(n\)  continuous representations, where \(n\)  is the input sequence length, we do a max pooling operation along the temporal dimension so that we have a fixed vector representation.
These fixed vector representations are the direct inputs to \(D_M\)  and \(D_S\) .
The max sequence length is set at 300, and the max phrase length is set at 70.
If an input sequence has more than 300 frames, we randomly sample 300 frames at each epoch.
If a video in the testing or validation set has more than 300 frames, we fix the indices of the sampled frames to remove any randomness for evaluation.
For input sequences that are shorter than 300 frames, we zero-pad the remaining sequence.

Loss functions
Style Disentanglement
\(D_S\)  is trained using Equation REF .
\(E_C\)  is trained using the same equation, but the labels are flipped and \(D_S\)  is not updated.
\(\mathcal {L}_{{Adv}_{D_S}} =-\mathop {\mathbb {E}} [\log ( D_S(E_C(X_i^s)) ) - \log (1 - D_S(E_C(X_i^t)) )]\) 

Content Disentanglement
\(D_C\)  is trained by minimizing Equation REF .
\(E_S\)  is trained by maximizing Equation REF  with the weights of \(D_C\)  kept frozen.
\(\mathcal {L}_{{Adv}_{D_C}} = - \log p(Y^z_i \vert D_C( E_S(X^f_i) ) )\) 
\(\mathcal {L}_{{Adv}_{E_S}} = H(Y_i^z \vert D_C(E_S(X_i^s)) )\) 

Feature Aggregation
A Feature Aggregation Module, \(M\) , combines the disentangled representations from the previous two steps.
For any given pair of style and content representations we have:
\(M(z_{style}^f , z_{content}^{f\prime }) = \mathbf {m}( z_{style}^f + z_{content}^{f\prime })\) 
In Equation REF , \(\mathbf {m}\)  is the LayerNorm operation [27]}, \(f \in \lbrace s,t\rbrace \)  and \(f^{\prime } \in \lbrace s,t\rbrace \) .

Prediction
The Decoder, \(G\) , takes in the output of \(M\) , \(h_{f\prime }^f\) , and outputs the predicted sentence \(\hat{Y}^{f\prime }\) , and is trained with the cross-entropy loss using the labels \(Y^{f\prime }\) . The objective is:
\(\mathcal {L}_{cls} = - \log p(Y^{f\prime } \vert G( M(E_S(X^f), E_C(X^{f\prime })) ))\) 
At test time, the model outputs the most likely sentence given a real-life video:
\(\operatornamewithlimits{argmax}_{Y} p(Y^{t} \vert G( h^t_t ))\) 

Semantic Alignment
We create four pairs \(\mathcal {G}_k, k \in \lbrace 1,2,3,4\rbrace  \) .
\(\mathcal {G}_1\)  and \(\mathcal {G}_2\)  are outputs of \(M\)  that share synthetic content: (Synthetic Style, Synthetic Content) and (Real Style, Synthetic Content).
\(\mathcal {G}_3\)  and \(\mathcal {G}_4\)  share real content: (Synthetic Style, Real Content) and (Real Style, Real Content).
A multi-class discriminator, \(D_M\) , is trained using Equation REF  to correctly identify which group every output of \(M\)  belongs to.
\(l_k\)  is the corresponding label for a given \(\mathcal {G}_k\) .
\(E_C\) , \(E_S\) , and \(M\)  are updated with Equation REF  such that \(D_M\)  can't distinguish outputs of \(M\)  that are in \(\mathcal {G}_1\)  and \(\mathcal {G}_2\)  and outputs of \(M\)  that are in \(\mathcal {G}_3\)  and \(\mathcal {G}_4\) .
\(\mathcal {L}_{{Adv}_{D_M}} =- \mathop {\mathbb {E}} [\sum \limits _{k=1}^4 l_k \log (D_M(M (\mathcal {G}_k) ))]\) 
\( \mathcal {L}_{{Adv}_{M}} =- \mathop {\mathbb {E}}[ l_1 \log (D_M(M (\mathcal {G}_2) )) \\- l_3 \log (D_M(M (\mathcal {G}_4) )) ]\) 
The final loss function to train our model is show in Equation REF  where the weightings for each term are tuned using a validation set.
An overview of the training procedure is shown in Algorithm  REF
\(\mathcal {L} =\lambda _1\mathcal {L}_{cls} +\lambda _2\mathcal {L}_{{Adv}_M} +\lambda _3\mathcal {L}_{{Adv}_{D_M}} + \\\lambda _4\mathcal {L}_{{Adv}_{E_S}} +\lambda _5\mathcal {L}_{{Adv}_{D_C}} +\lambda _6\mathcal {L}_{{Adv}_{D_S}}\) 

Hyperparameters
We use the Adam optimizer with learning rate of 0.0001 for all of our networks.
We train for 60k iterations with a batch size of 8, and use a dropout value of 0.15.
We use the validation set to tune the different weightings for our loss function.
We set \(\lambda _1 = 1.0, \lambda _2 = 0.25, \lambda _3 = 1.0, \lambda _4 = 1.0, \lambda _5 = 1.0\)  and \(\lambda _6 = 1.0\) .
