Analyzing the Use of Influence Functions for Instance-Specific Data Filtering in Neural Machine Translation

Customer feedback can be an important signal for improving commercial machine translation systems. One solution for fixing specific translation errors is to remove the related erroneous training instances followed by re-training of the machine translation system, which we refer to as instance-specific data filtering. Influence functions (IF) have been shown to be effective in finding such relevant training examples for classification tasks such as image classification, toxic speech detection and entailment task. Given a probing instance, IF find influential training examples by measuring the similarity of the probing instance with a set of training examples in gradient space. In this work, we examine the use of influence functions for Neural Machine Translation (NMT). We propose two effective extensions to a state of the art influence function and demonstrate on the sub-problem of copied training examples that IF can be applied more generally than handcrafted regular expressions.


Introduction
Neural Machine Translation (NMT) is the de facto standard for recent high-quality machine translation systems. NMT, however, requires abundant amount of bi-text for supervised training. One common approach to increase the amount of bi-text is via data augmentation (Sennrich et al., 2015;Edunov et al., 2018;He et al., 2019, inter alia). Another approach is the use of web-crawled data (Bañón et al., 2020) but since crawled data is known to be notoriously noisy (Khayrallah and Koehn, 2018;Caswell et al., 2020), a plethora of data filtering techniques (Junczys-Dowmunt, 2018;Wang et al., 2018;Ramírez-Sánchez et al., 2020, inter alia) have been proposed for retaining a cleaner portion of the bi-text for training.
While standard data filtering techniques aim to improve the quality of the overall training data * Work done during an internship at Amazon. without targeting the translation quality of specific instances, instance-specific data filtering focuses on the improvement of translation quality toward a specific set of input sentences via removal of the related training data. In commercial MT, this selected set of sentences can be the problematic translations reported by customers. One simple approach of instance-specific data filtering in NMT is manual filtering. In manual filtering, human annotators identify translation errors on sentences reported by customer and designs filtering scheme, e.g., regular expressions to search related training examples for removal from the training set.
In this work, we attempt to apply a more automatable technique called influence functions (IF) which is shown to be effective on image classification (Koh and Liang, 2017), and certain NLP tasks such as sentiment analysis, entailment and toxic speech detection (Han et al., 2020;Guo et al., 2020). Given a probing example, influence functions (IF) search for the influential training examples by measuring the similarity of the probing example with a set of training examples in gradient space. Schioppa et al. (2021) use a low-rank approximation of the Hessian to speed up the computation of IF and apply the idea of self-influence to NMT. However, self-influence measures if a training instance is an outlier rather than its similarity with another instance. Akyürek et al. (2022) question the back-tracing ability of IF on the facttracing task. They compare IF with heuristics used in Information Retrieval and attribute the worse performance of IF to a problem called saturation. Compared to fact-tracing, the target sides of machine translation can be more diverse which complicates the application of IF.
We apply an effective type of IF called TracIn (Pruthi et al., 2020) to NMT for instance-specific data filtering and analyze its behaviour by constructing synthetic training examples containing simulated translation errors. In particular, we find arXiv:2210.13281v1 [cs.CL] 24 Oct 2022 that • the gradient similarity, also called the influence 1 , is highly sensitive to the network component.
• vanilla IF may not be sufficient to achieve good retrieval performance. We proposed two contrastive methods to further improve the performance.
• training examples consisting of copied source sentences have similar gradients even when they are lexically different. This indicates that the use of influence functions can go beyond what can be achieved with regular expressions.
• an effective automation of the instancespecific data filtering remains challenging.
To the best of our knowledge, we are the first to investigate applying IF for instance-specific data filtering to NMT.

Method
Influence functions IF is a technique from robust statistics (Hampel, 1974;Cook and Weisberg, 1982, inter alia). It aims to trace a model's predictions back to the most responsible training examples without repeated re-training of the model, aka Leave-One-Out. Koh and Liang (2017) extend this idea from robust statistics to deep neural network that requires only the gradient of the loss functions L and Hessian-vector products so that the influence I(z, z ) of two examples z and z is approximated as whereθ is the model parameters at optimum and is the Hessian of the model parameters atθ. Given n number of training instances and p number of model parameters, the inverse of Hessian has a complexity of O(np 2 + p 3 ) which is expensive to compute for deep neural network. There are several proposed methods to speed up the computation of IF, e.g., by computing on a training subset selected by KNN-search (Guo et al., 2020), by approximating the Hessian with LISSA (Agarwal et al., 2017), by computing on a subset of model parameters (Koh and Liang, 2017), or by replacing the Hessian with some other procedures (Pruthi et al., 2020). In this work, we focus on TracIn which is shown to be better than some other variations (Han and Tsvetkov, 2020;Schioppa et al., 2021) in terms of retrieval performance.
TracIn, denoted by I TracIn (z, z ), replaces the computationally costly Hessian matrix with an identity matrix. The remained gradient dot product, or called the gradient similarity, is instead computed over C number of checkpoints, followed by averaging: In NMT, given the same source sentence, the magnitude of the gradient in general is positively correlated to the length of the target sentence. In order to reduce the effect of the target length, we normalize equation 2 by the product of ∇ θ L(z ) and ∇ θ L(z) , or equivalently, we compute the cosine similarity of ∇ θ L(z ) and ∇ θ L(z).
Given a probing instance z and its probing gradient ∇ θ L(z ), instances in the training set that yield a positive value of I TracIn (z, z ) are called the positively influential training instances (+IF-Train) whereas those that yield a negative value of I TracIn (z, z ) are called the negatively influential training instances (-IFTrain). Taking a gradient step on +IFTrain reduces the loss on the probing example while taking a gradient step on -IFTrain increases it. IF can be used for data filtering by removing the +IFTrain examples of low quality probing samples since their gradients have similar direction. Conversely, if the probing sample is of high quality, removing -IFTrain examples from the training data would be expected to increase translation quality w.r.t. the probing sample.

Experimental Setting
Model configuration and training We use Transformer BASE configuration as described in Vaswani et al. (2017) with default setting and implementation in FAIRSEQ. We use a sentence-piece model to create subword units of size 32k. Unless otherwise specified, we pre-trained our NMT on Europarl-v7 data and News Commentary-v12 data in German-English direction from WMT17 for 100 epochs, about 112K updates, using Adam  optimizerion training of 16-bit 2 . The effective mini-batch size is 4096 x 16 tokens and it takes a p3.16xlarge 3 machine on AWS 6 hours for training. We evaluate the MT model on the newstest2017 test set with a checkpoint averaged over the 10-best checkpoints, measured by the validation loss on the newstest2014-2016 dev set. On the test set, our NMT model with non-shared parameters with the two word embeddings and the output layer scores 29.99 BLEU whereas the one with shared parameters scores 29.78 BLEU. We use beam search with beam size of 5 in decoding.
TracIn We select 5 checkpoints, i.e., at epoch 5, 8, 15, 30 and 100 for computing TracIn 4 . We select checkpoints which have relatively large changes in the validation loss, i.e., usually in the earlier phrase of training, and include the last one to cover information at the end of the training. We com-pute the per-sample gradient with a batch size of 1 parallelized over multiple processes with several g4dn.2x 3 machines on AWS.

Experimental results
This section describes our findings on the properties of applying IF on NMT for instance-specific data filtering.

Sensitivity of gradient similarity to the network components
In previous works, the influence, or called the gradient similarity, is usually computed with respect to a small part of the network parameters, especially the last or the last few layers (Han et al. (2020); Barshan et al. (2020); inter alia). In NMT, we found that the resulting influence is highly sensitive to the network components used in computing the gradients (or gradient component). For illustration, we construct a set of perturbed instances, compute its influence by different gradient components and observe their changes. The perturbed instances are not included during the NMT training. This independence between the NMT and the perturbed instances provides a simpler setting for checking how gradient components and the perturbed examples affect the influence. Table 1 shows the gradient similarities of a probing example from newstest2017 with six artificially created instances. We use two NMT models, 1) trained with shared parameters between the two word embeddings and the output layer and 2) trained without parameter sharing, to compute the similarities.
We notice that gradient similarity for the model with shared parameters is more strongly influenced by lexical matches on the target side, as shown by the larger magnitude of influence values for probing examples 1 and 2 with random source sides compared to probing examples 3 and 4 with random target sides. For non-shared parameters, we observe that the gradient w.r.t. the output layer (∇ output ) has stronger response (0.437 and 0.401) to the probing instances with random source side whereas the gradient w.r.t. source embedding (∇ srcEmb ) has stronger response (-0.149 and -0.120) to the instances with random target sides. On the same probing example, we repeat this random sampling of source and target sentences by using the other 3003 instances in the newstest2017 set. We find that the mean magnitude of ∇ srcEmb is 0.04 for random target whereas it is 0.004 for random source. In the case of ∇ output , the mean magnitude for random target is 0.021 whereas it is 0.428 for random source. This indicates that ∇ output has a tendency of scoring sentence pairs higher when their target side overlaps with the target side of the probing instance and is less influenced by sourceside overlap. This may be suboptimal for retrieving problematic training examples that are relevant to a given probing instance.
When using a gradient vector ∇ concat which is the concatenation of ∇ srcEmb , ∇ trgEmb and ∇ output , its similarity is dominated by ∇ output rather than equally shared between the three given that they have the same number of parameters. This may explain why, in the case of shared parameters, instances with random source side have higher similarities than those with random target side.
Instance 5 and 6 are minor edits of the probing instance with changes to punctuation. For instance 5, it is not easy to interpret the results for the model with shared parameters. However, in the non-shared parameter setting, we observe a higher similarity for ∇ srcEmb than for ∇ trgEmb and ∇ output . This is more interpretable because the punctuation change is on the target side. For instance 6, the punctuation change is on the source side and we see a higher TracIn value for ∇ output than for ∇ srcEmb and ∇ trgEmb . As before, the value of ∇ concat is more similar to the value of ∇ output . Further examples can be found in Table  A1 in the Appendix.
These qualitative results show that the choice of network component is crucial in computing the gradient similarity. As shown in the next experiment, this affects the retrieval of training examples.

Contrastive signal is crucial for better retrieval performance
In this section, we try to illustrate how different gradient components affect the retrieval of the noisy instances with TracIn. We add control to the retrieval outcome by adding synthetic noisy training instances to the training data. In addition, we show that vanilla IF may not be sufficient to achieve good performance because the gradients are aggregated over all tokens in the target sentence. We thus propose two contrastive methods to sharpen the gradient signal.
Synthetic noisy examples We use the error template X → Y which stands for X is translated to Y to construct synthetic noise examples for the training set . We created four simple error patterns: 1) August → January, 2) Deutschland → Italy, 3) Oktober → December and 4) Türkei → New Zealand.  In the training set, we replace the translation of the sentences containing the source pattern by the erroneous translation with a probability of 60% so that the total number of training data is unchanged. We select these error patterns because translation errors of months and country names can easily result from noisy training examples and are therefore suitable to simulate real customer issues. In addition, there are related source sentences in the test set, i.e., newstest2017, which can be used as probing examples. In order to speed up the computation of IF, we extract a subset of training data containing the original pattern, the perturbed pattern and some randomly sampled training sentences. For example, in the error pattern Oktober → December, the training subset contains sentences with Oktober, Dezember, October and December on either the source or target side together with some randomly sampled sentences. Table 2 gives the exact number of instances for each case. We follow the same training procedure as section 3 to pre-train a NMT model on the training corpus perturbed by the synthetic noises.
Contrastive-IF The gradient of a source-target pair in NMT involves complicated mapping between the source tokens and the target tokens. That is, the gradient vector does not just contain the information of the error pattern but also other context. In order to isolate the gradient of the error pattern from the aggregated signal, we propose two methods: 1) gradient masking and 2) gradient difference. Both methods leverage a cleaner translation either in the form of a gold-reference translation or a corrected hypothesis, i.e. the hypothesis with the error pattern corrected. We refer to them as Contrastive Influence Functions (Contrastive-IF).
The idea of gradient masking (Mask) is to apply a 0/1 token-level mask to the loss function so as to remove the contribution of irrelevant tokens from the gradient computation. We assign the mask based on which tokens differ between hypothesis and reference. If the 0-mask is applied everywhere except for the location of the error according to a corrected translation, we refer to it as MaskExact.
We can use the difference between two hypotheses in a continuous fashion by simply subtracting their gradients. Specifically, we compute the difference of the gradient of a sentence A and the gradient of a sentence B as the probing gradient: In this work, we use the hypothesis as A and a cleaner translation as B (either the reference or the corrected hypothesis) so that positively influential training instances w.r.t. to GD(A, B) are the synthetic noisy training instances.
Results Table 3 shows the retrieval performance of vanilla IF, gradient masking and gradient difference where the gradient is computed w.r.t. to either the source embedding, output layer or the full model. We evaluate the performance with precision over the top-X% influential training instances, i.e. the number of synthetic training instances successfully retrieved given top-X% of the influential training samples. We combine results of the four error patterns by (macro) averaging their precision.
The first three rows show results for vanilla IF (TracIn) when either the hypothesis, the reference or a corrected hypothesis is used for probing the training data. Using ∇ srcEmb or ∇ output obtain substantially higher precision for each variant than using ∇ F ull , i.e., the gradient w.r.t. the entire model, which demonstrates the importance of the choice of gradient component(s) in vanilla-IF for retrieval performance. Using the corrected hypotheses to retrieve negatively-influential examples yields the best precision for both top-1% and top-10% of retrieved training examples.
We qualitatively examine the influential instances retrieved. By using the source-hypothesis pair as the probing instance, we find that instances retrieved via ∇ output have less similarity on the source side. In the first probing example, Januar → January occurs more frequently in the ranking than August →January. In the second example, Italien → Italy appears as the third influential training instance when using ∇ output whereas all top-3 influential instances obtained by ∇ srcEmb contain the desired error pattern of Deutschland → Italy, see Table A2 in the Appendix.
We find that both gradient masking, ∇(HYP Mask ), and gradient difference, ∇(HYP) − ∇(REF), perform better than the vanilla IF given the same gradient component. ∇(HYP Mask ) always outperforms the comparable vanilla IF variants ∇(HYP) and ∇(REF). If we can identify the exact location of the error pattern, with the probing gradient ∇(HYP MaskExact ) or ∇(CorrHYP MaskExact ), the precision can be further boosted and this is consistent for gradients ∇ srcEmb , ∇ output and ∇ F ull . While the gradient difference variants do not always outperform the comparable masking variants for all ∇ X , ∇(HYP) − ∇(CorrHYP) yields the overall best result using ∇ srcEmb .
An interesting finding is the improvement brought by the corrected hypothesis (CorrHYP). Applying vanilla-IF on it already achieves a precision of 0.930 under ∇ srcEmb considering the top-1% influential instances. By applying MaskExact or gradient difference on it, we achieve very high precisions of 0.989 and 1.0 under ∇ srcEmb considering the top-1% influential training instances. One notable gain brought by the proposed approaches is that for ∇ F ull , the precision increases from 0.531 to around 0.987 for the ∇(HYP) − ∇(CorrHYP) variant, bringing it on-par to the performance of  Table 3: Retrieval performance measured in (macro) averaged precision over all error patterns. ∇(P robing) refers to the gradient with input 'source-Probing'. HYP, REF and CorrHYP stands for hypothesis, reference and corrected hypothesis respectively. "+" ("-") indicates that positively (negatively) influential training instances were retrieved. ∇ X indicates network components used in computing the gradient. We mark the best result per column in bold.
∇ output . We include results for additional gradient components in Table A3 in the Appendix.  We also conducted a side experiment with a NMT model with shared parameters between the embeddings and the output layer. Similar to the case of a NMT model with non-shared parameters, gradient difference improves over the vanilla-IF when averaging precisions over all error patterns as shown in Table 4.
To summarize, both our contrastive-IF variants improve retrieval performance regardless of the net-work component used in computing gradients and whether the NMT model has shared parameters.

Copied source sentences have similar gradient signature
Our initial motivation for applying influence functions to NMT was to arrive at a more  17.64 BLEU on the newstest2017 data, showing the detrimental effect of the untranslated target sides. We select 40 probing instances from the new-stest2017 data where their translation by the above NMT model is a copy of the source sentence. We again reduce the computation time by running TracIn over a training subset which contains the newly added noisy data, i.e., 176,004 instances and a set of randomly sampled training instances. This creates a training subset of 476,004 instances.
Results Table 5 shows the retrieval performance on copied source sentences in the training subset with probing gradients of ∇(HY P ), ∇(REF ) and ∇(HY P ) -∇(REF ) computed over source embedding (∇ srcEmb ), the encoder (∇ encoder ), or the entire model (∇ F ull ). We skip the masking strategy in this case since it would mask all target tokens, resulting in a loss of 0. Different from our results so far, the vanilla IF using only the hypothesis preforms better than using the reference for retrieval and better than the gradient difference variant for all network components. For example, when considering only the top-10% influential training instances, the precision is 0.930 for ∇(HY P ) with ∇ srcEmb and only 0.525 for ∇(REF ). This may indicate that instances of copied source sentence have similar gradient signature despite their lexi-cal difference (see Table A4 for some examples) and that the reference translation is less useful in this setting because it cannot provide a specific contrastive signal.
A surprising finding in this setting is that using gradients computed over the entire network is better than the source embedding or the entire encoder. This is in contrast to the previous findings in the synthetic training instances. This possibly indicates that the copy mechanism is spread over the entire model or parts beyond the source embedding or the encoder.

An effective IF-based instance-specific data filtering is hard to automate
Many data filtering algorithms require a threshold to decide which instances are to be filtered. This threshold can be a model score in an offline filtering algorithm (Junczys-Dowmunt, 2018) or a dynamic formula that is changed according to the learning state of the model (Wang et al., 2018). In both cases, a desirable threshold should be effective as measured in the downstream model performance and be easily computed and generalized to other situations. In the case of IF-based instance-specific data filtering, we observe two properties in the ranking of the influence which makes the automation of the data filtering algorithm challenging.

1: The range of influence varies across probing examples
Although the influence is bounded between [−1, 1] because of the cosine similarity, the maximum magnitude of the influence for each probing example can still be very different.   Table 7: Mean and standard deviation of the number of influential training instances to be removed per configuration, using the largest consecutive difference found in the ranking as clustering criterion.
2: The influence value drops abruptly at the top-of the ranking Apart from a fixed threshold across different probing example, we also examine the possibility of automatically setting a threshold for each probing example.
We first examine a simple clustering strategy by searching for the position where the consecutive difference is the largest in the ranking of influence. Table 7 shows the result of the mean and standard deviation of the number of most influential training instances to be removed per configuration. By considering only the largest consecutive difference, less than 5 training instances would be removed which is far less than the number of synthetic training instances.
We examine further by investigating the shape of the influence of the positively influential training instances in the ranking. Figure 1 shows the influences, computed via TracIn, of the top-500 positively influential training instances per error pattern. For each error pattern, we randomly select a probing example to examine its influence under different gradient conditions. In all these cases, the influence drops sharply in the first few instances, especially in the case of vanilla IF, denoted by "GradHYP" in the figures. After the sharp drop, the influence becomes quite steady for the remaining instances. This steady behaviour holds even for instances of much lower rank, see Figure  A1 in the Appendix. The "elbow" occurs before the first 50 influential training instances, which includes only a tiny portion of the synthetic noisy training instances.
How about Top-K filtering? In previous work, the authors use either Top-K or Top-X% as the filtering threshold which is not realistic in the case of NMT where 1) there can be billions of training instances, and 2) the error types are more diverse than the prediction of wrong classes. In spite of the good retrieval performance demonstrated in the previous section, our results here show that an effective automation of the IF-based instancespecific data filtering for NMT remains a challenge.

Conclusion
We have analyzed the use of Influence Functions for NMT as instance-specific data filtering. By constructing synthetic instances, we found that 1) the gradient similarity is very sensitive to the selected network components, 2) vanilla Influence Functions are not sufficient for good retrieval performance, 3) our proposed contrastive-IF can boost the retrieval performance regardless of the gradient component or parameter sharing, 4) finding an effective automation of IF for instance-specific data filtering is difficult. This is because the proper choice of gradient component with respect to the type of error in the probing example is crucial for the effectiveness of Influence Functions. Despite the reported effectiveness for certain classification tasks in previous literature, our results show that applying IF to NMT poses some practical difficulties that we have not yet been able to solve. In each subfigure, we randomly select a probing example from each error pattern to compute its influence using gradient difference w.r.t. 1) source embedding (GradDiff srcEmbed), 2) entire model (GradDiff full) and using vanilla-IF with source-hypothesis as input w.r.t. 1) source embedding (GradHYP srcEmbed), 2) entire model (GradHYP full).

Limitations
In this work, we provided an analysis of using Influence Functions for Neural Machine Translation as instance-specific data filtering for the purpose of cost saving and finding a more generally applicable solution. Despite the reported success of some previous works in NLP/Vision-related classification tasks, we faced several challenges in applying Influence Functions to NMT. We are aware of the following limitations to our analysis: • Our analysis focuses on TracIn rather than other influence functions because TracIn is reported to be very effective.
• Our analysis is based on a fixed set of checkpoints, following the practice of previous works. The selection and the number of checkpoints used in TracIn are computationally costly hyper-parameters.
• Our analysis focuses on major network components such as embeddings, encoder and the output layer, excluding other possible combinations.
• The scale of our experiments is limited, e.g., only the De-En language direction with 3M training instances and the synthetic examples are relatively simple. However, given such simple setting, we can already see the challenges of applying IF on NMT as instance-specific data filtering or as an attribution/interpretable method.
• The proposed contrastive IF requires a corrected translation, e.g., reference translation.
We hope that our analysis can inspire further evaluation and modification of the technique.  Table A1: Another example showing the changes of gradient similarity by selected network components. Segments that are marked in red are perturbed from the probing example. The notation ∇ X indicates the network components used in computing the gradient similarity. ∇ srcEmb has a mean magnitude of 0.051 and 0.007 on random target and random source respectively whereas ∇ output has respectively a mean magnitude of 0.0145 and 0.350. This shows that ∇ output has a tendency of scoring sentence-pairs containing random source higher. Figure A1: TracIn of the top-50% positively influential training examples. In each subfigure, we randomly select a probing example from each error pattern to compute its influence using gradient difference w.r.t. 1) source embedding (GradDiff srcEmbed), and 2) entire model (GradDiff full) as well as using vanilla-IF with sourcehypothesis as input w.r.t. 1) source embedding (GradHYP srcEmbed), and 2) entire model (GradHYP full Alas, the title of today's debate is Italy. Table A2: Two probing examples with source-hypothesis as input and their top-3 positively influential training instances. ∇ output has a tendency to assign higher scores to sentence-pairs which target side has overlapped tokens but ignoring the similarity of the source side. For example, the pattern "Januar -> January" occurs more frequently in the ranking than "August -> January" in probing 1.  Table A3: Retrieval performance measured in (macro) averaged precision over all error patterns (extended version of Table 3). ∇(P robing) refers to the gradient with input 'source-Probing'. HYP, REF and CorrHYP stands for hypothesis, reference and corrected hypothesis respectively. "+" ("-") indicates that positively (negatively) influential training instances were retrieved. ∇ X indicates network components used in computing the gradient, ∇ concat indicates concatenation of ∇ srcEmb , ∇ trgEmb and ∇ output . We mark the best result per column in bold.  Two probing examples with copied training instances as input and their top-3 positively influential training instances. Both ∇ srcEmb and ∇ F ull can retrieve copied instances in the training subset given a probing instance of copied source sentence which is lexically different.