VI uses style and content disentanglement, along with the semantic alignment. This is our proposed method trained with Algorithm REF , i.e., IV \(+\)  semantic alignment.
First, we find that our base model (I) achieves competitive results without any losses on the latent spaces.
This indicates that training on paired representations across domains is an effective method for data augmentation.
Second, we find that adding auxiliary losses on the latent spaces to enforce style and content disentanglement improves performance.
The performance for models II and III shows the base model is benefiting from the added loss terms.
The results for Model IV aligns with our hypothesis that explicitly disentangling style and content allows us to overcome the lack of training data in the target domain by training with all combinations of the factors of variation.
Finally, we trained model V to apply the semantic alignment step on our paired outputs without any additional adversarial losses.
This is quite competitive with IV, but we find the greatest performance boost when training model VI using both semantic alignment and disentanglement.
A closer look into the distribution of the scores in Figure REF  shows that the distribution of scores for Model VI (Full) indicates higher overall performance compared to Model I (Base).
Our results show that explicitly disentangling style and content by adding the adversarial losses on the latent spaces, supplements the pairing mechanism to achieve the highest performance under the evaluation metrics.
Implications
Taken as a whole, our results show an adversary seeking to deploy keystroke inference attacks can leverage deep learning methods, despite the difficulties in curating training data.
While we are unable to directly show whether such an adversary would outperform those in prior attempts, we show that we can train a deep learning model using the same amount of real-life data used in previous studies.
Thus, the settings for vision-based keystroke inference attacks should be revisited as the realism and threat capacity of these attacks are most likely greater than initially thought. In particular, the lack of training was a significant impediment to many earlier proposals, but having a synthetic-to-real domain adaptation framework like ours — to augment the limited real data — would likely lend itself to a more capable adversary.
Furthermore, research over the past few years has shown that deep learning methods have outperformed shallow methods in most computer vision tasks. Arguably, if we hold all the parameters of a threat scenario constant (distance, camera model, phone model), an attacker using a deep learning method should outperform one with a shallow method. Similarly, our style and content disentanglement techniques can be used to help an attacker thwart certain defenses.
For example, [1]} proposed the use of random device perturbations as a way to mitigate an attacker's ability to map backside perturbations to keystrokes. While this is an effective defense, an attacker can undermine it by training a model to disentangle the fake perturbations from the real backside perturbations.

Conclusion
Our work provides the important initial step needed to formulate defenses for keystroke inference attacks in the age of deep learning.
Specifically, we provide the first assessment of an attacker's ability to recover sentence level information using deep learning systems.
We address the problem of limited training data by introducing a framework for low resource video domain adaptation, that disentangles the style and content across both domains, and creates representations from all pairs of style and content combinations.
Our results indicate that training with these pairs, along with auxiliary losses to explicitly disentangle style and content, serves as an effective form of data augmentation that prevents overfitting.
We evaluate our method using a number of metrics to quantify the amount of information the attacker is able to recover, and our results show that an attacker armed with a deep learning system is able to recover enough information to pose a significant threat to unsuspecting victims.
Our framework can also be used to assess other keystroke inference attacks such as those that focus on device perturbations [1]} or eye gaze [3]}.

Availability
The code and data used in this paper can be found at https://github.com/jlim13/keystroke-inference-attack-deep-learning.

Additional Training Details
Synthetic Single Key Press Classifier
We train a CNN, \(\phi (\cdot )\) , for the task of single key press classification in order to learn a \(d-\) dimensional (\(d = 128\) ) feature extractor.
Once this network is fully trained for the task of single key press classification, we can extract the features of each video on a per-frame basis. We use the simulator by [4]} to generate 70,000 single key press images.
These images contain the synthetic thumb over one of 27 keys on the QWERTY keyboard (26 letters \(+\)  the space bar).
Once generated, these images are preprocessed in a similar fashion to the synthetic video dataset.
We resize the images to size 200 X 100 and crop the phone such that only the keyboard is showing.
We use 50,000 images for training and 10,000 images for testing and validation, respectively. We use a CNN where each layer consists of a Convolution Layer, ReLU activation, and MaxPool operation.
We use 3 layers and 2 fully connected layers.
We train a network to achieve 95% accuracy on a held out test set without much hyperparameter or architecture search, as this is a fairly simple 27-way classification task.
We use this final model to preprocess every frame in our synthetic video dataset.
Every video is a now a sequence of these \(d-\) dimensional feature representations.
We use the Adam optimizer with a learning rate of 0.0001.

Real-Life Single Key Press Classifier
When extracting the visual features for the real-life videos, we can not use a feature extractor that was trained only on synthetic data.
There is a distribution shift between the synthetic and real-life data, so the features we extract would be not be informative.
Instead of using \(\phi (\cdot )\)  that was trained for single keypress classification on just synthetic data, we train \(\phi (\cdot )\)  with a combination of synthetic and real-life data.
Specifically, we adopt the ADDA [5]} framework for unsupervised domain adaptation to train \(\phi (\cdot )\) .
We treat the individual frames for all of the videos in our real-life training set as unlabeled data.
Even though we do not have labels for individual keypresses for real-life data, we can leverage the fact that we have abundant labels for synthetic data by adopting the unsupervised domain adaptation technique ADDA.
We use the CNN for single key press classification on synthetic data as our pretrained network.
The Discriminator is a 1 layer, 128-dimensional fully connected layer followed by a sigmoid.
We follow the same guidelines to train ADDA as the original paper [5]}, and refer the reader to this work for the full description of their training process.
We use the Adam optimizer and a learning rate of 0.0001 for both the Discriminator and CNN.

Network Architectures
For all experiments, the Encoders (\(E_C\) , \(E_S\) ) and Decoders (\(D_C\) , \(G\) ) are both Transformers with 4 layers, 4 attention heads, an embedding size of 128 and a hidden size of 256.
\(D_M\)  and \(D_S\)  are both 1-layer fully connected layers.
Since the output of the Encoder is a sequence of \(n\)  continuous representations, where \(n\)  is the input sequence length, we do a max pooling operation along the temporal dimension so that we have a fixed vector representation.
These fixed vector representations are the direct inputs to \(D_M\)  and \(D_S\) .
The max sequence length is set at 300, and the max phrase length is set at 70.
If an input sequence has more than 300 frames, we randomly sample 300 frames at each epoch.
If a video in the testing or validation set has more than 300 frames, we fix the indices of the sampled frames to remove any randomness for evaluation.
For input sequences that are shorter than 300 frames, we zero-pad the remaining sequence.

Loss functions
Style Disentanglement
\(D_S\)  is trained using Equation REF .
\(E_C\)  is trained using the same equation, but the labels are flipped and \(D_S\)  is not updated.
\(\mathcal {L}_{{Adv}_{D_S}} =-\mathop {\mathbb {E}} [\log ( D_S(E_C(X_i^s)) ) - \log (1 - D_S(E_C(X_i^t)) )]\) 

Content Disentanglement
\(D_C\)  is trained by minimizing Equation REF .
\(E_S\)  is trained by maximizing Equation REF  with the weights of \(D_C\)  kept frozen.
\(\mathcal {L}_{{Adv}_{D_C}} = - \log p(Y^z_i \vert D_C( E_S(X^f_i) ) )\) 
\(\mathcal {L}_{{Adv}_{E_S}} = H(Y_i^z \vert D_C(E_S(X_i^s)) )\) 

Feature Aggregation
A Feature Aggregation Module, \(M\) , combines the disentangled representations from the previous two steps.
For any given pair of style and content representations we have:
\(M(z_{style}^f , z_{content}^{f\prime }) = \mathbf {m}( z_{style}^f + z_{content}^{f\prime })\) 
In Equation REF , \(\mathbf {m}\)  is the LayerNorm operation [7]}, \(f \in \lbrace s,t\rbrace \)  and \(f^{\prime } \in \lbrace s,t\rbrace \) .

Prediction
The Decoder, \(G\) , takes in the output of \(M\) , \(h_{f\prime }^f\) , and outputs the predicted sentence \(\hat{Y}^{f\prime }\) , and is trained with the cross-entropy loss using the labels \(Y^{f\prime }\) . The objective is:
\(\mathcal {L}_{cls} = - \log p(Y^{f\prime } \vert G( M(E_S(X^f), E_C(X^{f\prime })) ))\) 
At test time, the model outputs the most likely sentence given a real-life video:
\(\operatornamewithlimits{argmax}_{Y} p(Y^{t} \vert G( h^t_t ))\) 

Semantic Alignment
We create four pairs \(\mathcal {G}_k, k \in \lbrace 1,2,3,4\rbrace  \) .
\(\mathcal {G}_1\)  and \(\mathcal {G}_2\)  are outputs of \(M\)  that share synthetic content: (Synthetic Style, Synthetic Content) and (Real Style, Synthetic Content).
\(\mathcal {G}_3\)  and \(\mathcal {G}_4\)  share real content: (Synthetic Style, Real Content) and (Real Style, Real Content).
A multi-class discriminator, \(D_M\) , is trained using Equation REF  to correctly identify which group every output of \(M\)  belongs to.
\(l_k\)  is the corresponding label for a given \(\mathcal {G}_k\) .
\(E_C\) , \(E_S\) , and \(M\)  are updated with Equation REF  such that \(D_M\)  can't distinguish outputs of \(M\)  that are in \(\mathcal {G}_1\)  and \(\mathcal {G}_2\)  and outputs of \(M\)  that are in \(\mathcal {G}_3\)  and \(\mathcal {G}_4\) .
\(\mathcal {L}_{{Adv}_{D_M}} =- \mathop {\mathbb {E}} [\sum \limits _{k=1}^4 l_k \log (D_M(M (\mathcal {G}_k) ))]\) 
\( \mathcal {L}_{{Adv}_{M}} =- \mathop {\mathbb {E}}[ l_1 \log (D_M(M (\mathcal {G}_2) )) \\- l_3 \log (D_M(M (\mathcal {G}_4) )) ]\) 
The final loss function to train our model is show in Equation REF  where the weightings for each term are tuned using a validation set.
An overview of the training procedure is shown in Algorithm  REF
\(\mathcal {L} =\lambda _1\mathcal {L}_{cls} +\lambda _2\mathcal {L}_{{Adv}_M} +\lambda _3\mathcal {L}_{{Adv}_{D_M}} + \\\lambda _4\mathcal {L}_{{Adv}_{E_S}} +\lambda _5\mathcal {L}_{{Adv}_{D_C}} +\lambda _6\mathcal {L}_{{Adv}_{D_S}}\) 

Hyperparameters
We use the Adam optimizer with learning rate of 0.0001 for all of our networks.
We train for 60k iterations with a batch size of 8, and use a dropout value of 0.15.
We use the validation set to tune the different weightings for our loss function.
We set \(\lambda _1 = 1.0, \lambda _2 = 0.25, \lambda _3 = 1.0, \lambda _4 = 1.0, \lambda _5 = 1.0\)  and \(\lambda _6 = 1.0\) .
