Watchword-Oriented and Time-Stamped Algorithms for Tamper-Proof Cloud Provenance Cognition

Abstract: Provenance is derivative journal information about the origin and activities of system data and processes. For a highly dynamic system like the cloud, provenance can be accurately detected and securely used in cloud digital forensic investigation activities. This paper proposes watchword oriented provenance cognition algorithm for the cloud environment. Additionally time-stamp based buffer verifying algorithm is proposed for securing the access to the detected cloud provenance. Performance analysis of the novel algorithms proposed here yields a desirable detection rate of 89.33% and miss rate of 8.66%. The securing algorithm successfully rejects 64% of malicious requests, yielding a cumulative frequency of 21.43 for MR.

\section{Introduction}

Provenance is derivative journal data that represents factual information about events executed on particular operations at the application layer. Cloud computing is the dynamic provisioning of resources from a shared resource pool and needs provenance to ensure integrity and accountability of cloud services. Provenance information journals of various real life cloud applications is required during digital investigation. At the same time, the cloud provenance needs to be tamper-proof to increase acceptability of the data to Digital Forensic Experts. Hence detection and securing of provenance in the cloud is of significant importance for Security Intelligence.

Capturing provenance for Software as a Service (SaaS) applications is an important research issue, since those are dynamic in nature and have volatile attributes. Also, a large number of virtual and physical machines are involved in the cloud and application layer provenance needs to be collected for those for Cloud Security Intelligence. In addition effectively securing the cloud virtual machine (vm) instance that stores provenance using time-stamp based access control has not been considered for cloud environments \cite{armbrust2010view}. More precisely, the following research issues need to be addressed.

\begin{enumerate}
\item Novel algorithms using watchword capabilities to detect provenance in the cloud at the application level of SaaS.
\item Algorithms to tamper-proof the detected provenance using time-stamp based access control and analyzing performance of those in terms of authorized access acceptance and rejection.
\end{enumerate}

Existing mechanisms for provenance cognition in the cloud are not suitable as large volume of data needs to be analyzed, thereby increasing the overhead of the system and slowing down the cloud processes. Framework for provenance detection in the cloud through matching the patterns of journal files to determine system level provenance has been proposed in \cite{imran2013provintsec}. The provenance was analyzed for detection of malware threats to the system. However, the integrity of the captured provenance has not been ensured through tamper-proofing mechanisms. The importance of ensuring cloud accountability is a prime factor for widespread acceptance of cloud among the public as identified in \cite{pohly2012hi}. Methods to secure the cloud data has been proposed through layer-based encryption mechanism. However, the importance of securing the cloud provenance meta-data to aid in digital forensic investigation has been analyzed to a limited extent.

Based on the research questions identified above, this paper proposes methods to detect provenance at the application layer and securing those through time-stamp based access control. Provenance at the application layer of SaaS consists of data such as access-id, time of access, pages viewed, and files modified in terms write operations. The proposed algorithm captures the provenance described above using watchword capability of cloud journals that triggers data insertions into a specific cloud vm-instance using the provenance rules. The rules are used to capture provenance at the application level of cloud where the SaaS modules are deployed. Active-threaded methodology captures provenance at real time through effective monitoring of the cloud vm-instances.

Algorithms called $ObProv$ and $Auditor$ have been proposed in this paper to ensure provenance cognition and verification at the application layer of the cloud. Six real-life applications namely Inventory Management (IM), Accounting Management (AM), Human Resource Management (HRM), Office Document Management (ODM), Sales Management (SM), Customer Relationship Management (CRM) softwares running on OpenStack cloud were tested for provenance cognition using $ObProv$. The applications were implemented as SaaS in 88 vm-instances. Next $Auditor$ algorithm was implemented on the storage vm-instance that measured the request body in the buffer array and checked constraints through access time-stamp matching.

The performance of the proposed algorithms have been evaluated in real life cloud environment, running the specific applications in commercial use over vm-instances. Provenance were captured for those at a Success Rate (SR) of 89.33\% and Miss Rate (MR) of 8.66\%. The mean Rejection Rate (RR) of 64\% requests achieved using $Auditor$ algorithm out of 357 requests made to access the provenance cloud vm-instance. The cumulative frequency of 21.43 was achieved after empirical analysis. Additionally, 65 requests were considered legitimate and granted access to the stored provenance whereas 292 illegitimate requests were rejected by the $Auditor$ algorithm.

\section{Related Work}
A process aware approach to worm attacks and contamination have been designed, implemented and evaluated in \cite{idika2013probabilistic}. The authors identified the important issue that provenance un-awareness leads to problems in quick and accurate identification of worm attack point \cite{idika2013probabilistic}. Process coloring can be assigned to uniquely identify processes, it is inherited by the child processes and diffused through process actions.

\begin{figure*}[t!]
\centering
\includegraphics[scale =0.60]{sequence.png}

\caption{Sequence of actions and processes to secure provenance information}
\label{figure3}
\end{figure*}

Process coloring enables fast identification of worm break in point and naturally partitions log data based on colors, thereby ensuring rapid identification \cite{marinho2012provmanager}, \cite{yu2012novel}. The main contribution of the authors is to effectively reduce the volume of log data that need to be analyzed for worm identification. Nevertheless tamper-proofing the captured provenance was unaddressed.

Identified security as an important issue that decelerates the widespread growth of cloud computing \cite{abbadi2011challenges}. Complications for data privacy and data protection have plagued the market of the cloud. The assurance of data integrity is a necessity to ensure acceptance of cloud services in all sectors. The authors suggested a new model that extends the existing model in this regard. However the new model should not threaten the features of the existing model.

Cloud service users need to be vigilant about the security breaches that can occur in the cloud \cite{abbadi2011challenges}. The authors have identified critical security issues due to the nature of service delivery model in the cloud. They have contributed to the cloud research field by identifying critical research questions regarding cloud security. The authors did not identify any malware related vulnerability, neither did their survey bring about any real life scenarios.

The different security and vulnerability assessment in cloud computing environment have been assessed in \cite{ko2011flogger}. Experiments were carried out in three environments that include the applications residing on the same laboratory where the targets are, residing not in the same laboratory but on the same campus, and residing off-campus. Suitable scan tools are applied to the servers to assess what test was applicable. Next the captured vulnerabilities were collected and assessed for risk.

Risk was determined through the relative assessment to the facility in terms of the expected effect on each critical asset \cite{ko2011flogger}. The authors provided experimental evidence on cloud computing on the basis of centralized and de-centralized architectures. The scope of identifying threats through analysis of provenance data have been covered to a limited extent.

\section{Proposed Methododlogy}
Figure \ref{figure3} identifies the flow of sequence to ensure provenance security. The user can access a cloud application through a browser or may request some data file in the cloud storage. The data is stored in the $C_{in}$ and provenance is collected from those using the ProvCapsule algorithm. Next the provenance file containing the time (t), source vm-id (svm), target vm-id (tvm) and date are stored in the provenance file which is then encrypted by PO using its private key. PO generates a private/public key pair and shares it with $Auditor$ process. The proposed $Auditor$ process receives key $K1$ from PO and provides its own public key $K2$ to PO.

The second level of encryption is carried out by the auditor and the provenance file is sent to the signature process for verification. If the time-stamp, source vm, target vm and date matches, it will be forwarded to the Provenance Store, otherwise it will be discarded and the $PO$ and $Auditor$ will both be notified about the issue.

One fundamental difference with the existing research is that the procedures used for signing and verifying provenance information are time consuming since the authorization process is manual \cite{zhang2012track}. However, active-threading technique is proposed here for binding provenance with the original data file in real time, thus the process of forging provenance ownership is prevented in the proposed framework through real-time binding of provenance meta-data to original data. The proposed methodology also reduces time and low-overhead since only critical information like time-stamp, date, target vm and source vm are contained in the provenance file unlike detection of detailed meta-data.

\subsection{ObProv algorithm}

The $ObProv$ algorithm is aimed to encapsulate the provenance information in a specific message body. Hence provenance information produced by $ObProv$ contains only the information needed by the $Auditor$ algorithm or digital forensic experts.

\begin{algorithm}
 \begin{algorithmic}[1]
 \Procedure{ObProv}{$M_{sg},M_{bd}$,$g$}
 \State $M_{pr}\gets Req(T, T_{vm}, S_{vm}, Date)$
 \State $M_{bd}\gets{msg_{1}, msg_{2},.., msg_{n}}$
 \State $Msg\gets{M_{pr}+M_{bd}}$
 \State $A[x]\gets0$
 \State $g\gets 0$

 \While {$g <= sizeof(Msg)$}
 \State $A[g]=Msg$
 \State $g=g+1$
 \State $ObProv\gets ProvOwner{c,d}$
 \EndWhile
 \State If $len.Msg<len.M_{pr}$, then
 \State $Msg\gets len.Req(T, T_{vm}, S_{vm}, Date)$
 \State g = g+1;
 \State \textbf{return} Provenance information file $Msg$
  \EndProcedure
 \end{algorithmic}
 \caption{ObProv algorithm for capturing provenance at the application level}
 \label{euclid}
\end{algorithm}

The $M_{sg}$ is a function that will encapsulate the captured provenance information of ProvOCal into a specific provenance message. At the same time users are allowed to specify what type of provenance information they want to obtain and encapsulate. The $M_{bd}$ consists of the message body. The variable $g$ keeps the length of the size of the provenance information and messages produced by the algorithms, and also acts like a counter. The array $A[x]$ consists of the messages that are stored as strings together with the timestamp $T$. The timestamps are used to compare the authenticity of the requests in the later algorithm. The request body $Req(T,T_{vm},S_{vm},Date)$ ensures that the critical provenance information are specified by the digital forensic experts.

The contents are $T,T_{vm},S_{vm},Date$ to represent time, vm-id, storage allocated for the vm and date of access respectively. Next the message $Msg$ is placed in the array $A[g]$ in the location identified by the current value of $g$. At the same time $ProvOwner$ is read from $ProbCapsule$ and placed in the owner variable of $ObProv$. Finally the $Msg$ is assigned to the $Req$ body and sent to $Auditor$ algorithm for verification.

The algorithms discussed above function in line to detect system level provenance, at the same time those are synchronized to ensure proper activity from provenance cognition, to secured provenance encryption. The next algorithm ensures that unauthorized access to the stored provenance is restricted. Hence there is a sequence of two algorithms that work closely to achieve secured provenance of the big data stored in the cloud.

\subsection{Auditor algorithm}

The $Auditor$ of the process is responsible for verifying the request on the basis of the timestamps. The variable $R_1$ and $R_2$ are used to measure the size of the buffer array and at the same time implement the check constraint that it is under the limit of the initial consideration. Both variables are initialized to 0 to ensure that it can be used as a counter function.

\begin{algorithm}
 \begin{algorithmic}[1]
 \Procedure{Auditor}{$R_1$,$R_2$,$R_3$,$R_4$,$ObProv$,$size$}
 \State $R_1\gets0$
 \State $R_2\gets0$
 \State $Per_i\gets0$
 \State $size\gets1000$
 \State $Buf[size]\gets ObProv$

 \While {$R_1 <= sizeof(Buf[size])$}
 \While {$R_2 <= sizeof(Buf[size])$}
 \State $R_1=sizeof(Buf[size]+Msg)$
 \State $R_2 \gets En.(Req(T, T_{vm}, S_{vm}, Date))$
 \State $R_1=R_1+1$

 \EndWhile
 \EndWhile
 \While {$R_3 <= sizeof(Buffer[size])$}
 \While {$R_4 <= sizeof(Buffer[size])$}
 \State $R_3\gets timetoread.Buffer[size]$
 \State $R_4\gets timeof arrival.Buffer[size]$
 \State If $R_1=R_3$ and
 \State If $R_2!=R_4$
 \State $Per_i=1$
 \State else $Per_i$=0
 \EndWhile
 \EndWhile

  \EndProcedure
 \end{algorithmic}
 \caption{Auditor algorithm for setting permission for provenance access through timestamp authentication}
 \label{euclid}
\end{algorithm}

The two variables $R_3$ and $R_4$ are used to store the timestamps of two consecutive messages. The times of arrival of the new request is matched with the time when the first request if made. At the same time the vm-instance id that is making the request is also recorded. After encryption the provenance information must be stored at the secured end of the trusted Auditor.
\begin{figure*}[t!]
\centering
\includegraphics[scale =0.70]{statetransition.png}

\caption{Transitions and checks from one state to another in the proposed architecture}
\label{figure4}
\end{figure*}

Algorithms and framework of the trusted model discussed above are explained in this section. At the same time experiments to test the effectiveness of the proposed provenance securing algorithms are conducted with real life cloud environment using OpenStack Essex. Finally $Auditor$ algorithm is implemented at the end of the trusted body to ensure provenance persistency and achieve data integrity.

The two timestamps of the requests do not meet even in the case that the vm-id of those matches since the $Per_i$ will be set to 0. Additionally the permission to access the provenance information will not be allowed because the mismatch of timestamp is caused by a different vm-id philishing the identity of an authorized vm \cite{asif2013cloudniagara}. The difference in the timestamp will ensure that the vm requesting access does not have the authorization and it is a philishing entity.

\subsection{Security goal and assumptions}

The research goal is to prevent the unauthorized or malicious access and nefarious manipulation of provenance information in the cloud. The primary research issues are to determine application level provenance of the identified SaaS in the cloud and ensured secured access for accountability.

The assumptions for the proposed algorithms include that the cloud environment is in Linux platform and the cryptographic algorithms are implemented properly. The forensic investigators who own and maintain the $Auditor$ process is trustworthy. Also, it is assumed that program scripts running at the kernel level are not tampered by the cloud administrator.

\section{Transition of proposed architecture}

The four algorithms discussed above need to communicate with each other using message passing protocols to ensure that data is collected by one algorithm, sent for encryption to the next and finally validated by the remaining algorithms for digital investigation. The proposed algorithmic architecture consists of a number of states that must be transited. At each state proper verification mechanisms need to be implemented to ensure that the process satisfies the requirements needed to advance to the next phase.

\begin{figure*}[t!]
\centering
\includegraphics[scale =0.40]{brighton.png}

\caption{Passing captured provenance to the vm-instance with Auditor}
\label{figure8}
\end{figure*}

The start state in Figure \ref{figure4} consists of provenance information being gathered in the request body $Req(a,b,c,d)$ and sent to $ObProv$. Next the request is sent to $ProvCapsule$ so that the collected provenance are encapsulated to its original data file to sid in forensic analysis. Next the bounded provenance and data files are sent to the encryption scheme to obtain the data in encrypted format.

\begin{figure}[t!]
\centering
\includegraphics[scale =0.30]{Graph1.png}

\caption{Success rate of ObProv compared to real-life provenance cases}
\label{figure9}
\end{figure}

During encryption the provenance owner and $Auditor$ provide public keys and encrypt the information with respective private keys. The hash algorithm is implemented to ensure the integrity of provenance information. In this paper, hashed chain format of encryption is used as opposed to onion type encryption to accommodate the large volume of provenance information as generated by cloud processes and applications.

After encryption, the provenance information is sent to the provenance storage that is maintained by the $Auditor$ since it is assumed that both consumers and service providers consider the $Auditor$ to be a trusted body. During the course of digital investigation if provenance information is required, it is retrieved from the storage using the specific id assigned to provenance data objects by the proposed algorithm. However, the obtained provenance information is still in encrypted form, hence it is decrypted using keys from the $PO$ and $Auditor$ processes for analysis \cite{dastjerdi2012autonomous}, \cite{slipetskyy2011security}.

\section{Analysis of Results}

The performance of the proposed algorithms were tested and analyzed in real life cloud environments running OpenStack Essex cloud. Six commercial real life applications provided as Software as a Service (SaaS) in cloud vm-instances were analyzed at the servers of the cloud service provider. The SaaS applications included Inventory Management Software (IM), Accounting Management Software (AM), Human Resource Management Software (HRM), Office Document Management Software (ODM), Sales Management Software (SM), Customer Relationship Management Software (CRM) as identified in Table I.

\begin{table}
\caption{Performance evaluation of application level provenance cognition }
\begin{tabular}{ l||*{6}{c}r}
\hline
App. & VM-id & Case & Alg Det. & SR (\%) & MR (\%)\\
\hline
IM & 10 & 1084 & 922 & 85.0554 & 14.9446\\
AM & 13 & 1001 & 968 & 96.7032 & 3.2968\\
HRM & 07 & 834 & 772 & 98.3871  & 1.6129\\
ODM & 16 & 896 & 842 & 93.9732 & 6.0268\\
SM & 22 & 1200 & 966 & 80.5000  & 19.5000\\
CRM & 20 & 942 & 800 & 84.9257 & 15.0743\\
\hline
\end{tabular}
\end{table}

\begin{figure}[t!]
\centering
\includegraphics[scale =0.30]{Graph8.pdf}

\caption{Illustration of the high SR and low MR in the proposed architecture}
\label{figure11}
\end{figure}

\begin{table}
\caption{Mean Rejects and Cumulative frequency values for $ObProv$}
\begin{tabular}{ l||*{6}{c}r}
\hline
Req. Size & Count & Acc. & Rej. & M.Delay & C.freq \\
\hline
10-100 & 63 & 13 & 50 & \\
101-200 & 74 & 4 & 70 & \\
201-300 & 69 & 20 & 49 &  64 & 21.43\\
301-400 & 82 & 18 & 64 & \\
401-500 & 69 & 10 & 59 & \\
\hline
\end{tabular}
\end{table}

\begin{figure}[t!]
\centering
\includegraphics[scale =0.30]{Graph2.pdf}

\caption{SR and MR with respect to vm-id in cloud}
\label{figure10}
\end{figure}

$Acc_{i}$ = \begin{math}
  \begin{pmatrix}
    x_{1},&x_{2},&\cdots&,x_{n}
  \end{pmatrix}
\end{math}. There are a finite population of vm-instances so the number of accepted requests is finite as well. Hence we detect mean acceptance rate for any $X_{i}$ as,

\begin{equation}\overline{Acc_{i}} = \frac{\sum_{i=1}^{Count} Acc_{i}}{Count}\end{equation}

The value can be obtained for a large number of observations n. The variance of the delays in different ranges of Global Delays ($TBD$) and Inter-Message Time ($IMT$) is given by,

\begin{equation}{S^2(n)} = \frac{\sum_{i=1}^{Count} (Acc_{i}-\overline{Rej(Count)})^2}{Count-1}\end{equation}

Individual vm-instances were allocated for each of the SaaS applications specified and provided to cloud customers. Each customer has multiple users accessing the applications with their specific user names and passwords. For each customer, specific provenance information were collected that includes id-used to access, time and date of access, pages visited and changes made to files in terms of write operations. The provenance were detected from the SaaS level using $ObProv$ algorithm and and stored in a separate cloud server running the $Auditor$ algorithm for a period of 3 months.

The captured provenance using $ObProv$ algorithm were stored in Cloud Provenance server that as $Auditor$ algorithm implemented on the machine using Linux platform and PostgreSQL database. The $Auditor$ algorithm provides authorized access to the provenance information based on timestamp authentication, thereby enabling the prevention of undesired access that would otherwise obtain the provenance files and tamper those \cite{imran2014active}. Hence $Auditor$ algorithm ensures tamper-proofness of provenance information. Table II highlights the results of provenance access acceptance and rejections by $Auditor$ algorithm.

\section{Conclusion}

The paper aimed to ensure provenance cognition at the application layer of SaaS in the cloud. At the same time algorithm to ensure tamper-proofness of captured provenance using timestamped access control have been identified. The proposed algorithms were implemented on six real life SaaS applications running on OpenStack cloud platform. Performance analysis in terms of success and miss rates of the proposed algorithms show desirable performance of the proposed algorithms.

Empirical investigation of the proposed algorithms show an average success rate of 89.33\% that is desirable compared to industry benchmark values. The results show effectiveness of the proposed algorithms in detection of pre-specified provenance requirements. The Miss Rate (MR) of 8.66\% that is below 10\% is another desirable attribute of the proposed algorithms.

As stated earlier, the proposed algorithm detects and stores provenance in a tamper-proof format, thereby ensuring the accountability of provenance information to be used in digital forensic investigation. Association of tamper-proof provenance with web forensic data to yield more effective security intelligence is a topic of future research interest.

\section*{Acknowledgment}
This research has been supported by the University Grants Commission, Bangladesh under the Dhaka University Teachers Research Grant No-Regi/Admn-3/2012-2013/13190 and the Ministry of Science and Technology (MoST) Research Fellowship No-39.012.002.01.03.019.2013-229.

Distributed, Parallel, and Cluster Computing (cs.DC)


Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective

Abstract: Multimodal machine translation (MMT) aims to improve translation quality by equipping the source sentence with its corresponding image. Despite the promising performance, MMT models still suffer the problem of input degradation: models focus more on textual information while visual information is generally overlooked. In this paper, we endeavor to improve MMT performance by increasing visual awareness from an information theoretic perspective. In detail, we decompose the informative visual signals into two parts: source-specific information and target-specific information. We use mutual information to quantify them and propose two methods for objective optimization to better leverage visual signals. Experiments on two datasets demonstrate that our approach can effectively enhance the visual awareness of MMT model and achieve superior results against strong baselines.

\section{Introduction}
Multimodal machine translation (MMT) typically improves text translation quality by introducing an extra visual modality input. This task hypothesizes that the corresponding visual context is helpful for disambiguation and omission completion when the source sentence is ambiguous or even incorrect. Compared to the purely text-based machine translation, the critical point of MMT is to find out an effective approach to integrating images into a MT model that makes the most of visual information.

Most of the existing studies have dedicated their efforts to the way of extracting multi-granularity visual features for integration \cite{Calixto2017IncorporatingGV, Delbrouck2017AnES,Ive2019DistillingTW,Zhao2022RegionattentiveMN,Li2022OnVF,Fang2022NeuralMT} or designing model architectures for better message passing across various modalities \cite{Calixto2017DoublyAttentiveDF, Calixto2019LatentVM, Yao2020MultimodalTF, Yin2020ANG,Lin2020DynamicCC, Liu2021GumbelAttentionFM}, which achieve promising performances in most of the multimodal scenarios.

Despite the success of these methods, some researchers reveal that visual modality is underutilized by the modern MMT models \cite{Grnroos2018TheMS, Elliott2018AdversarialEO}.
Elliott \shortcite{Elliott2018AdversarialEO} leverage an adversarial evaluation method to
appraise the awareness of visual contexts of MMT models. An interesting observation is that the disturbed image input generally shows no significant effects on the performance of public MMT models, indicating the insensitivity to the visual contexts.
Recently, \citet{Wu2021GoodFM} revisit the need for visual context in MMT and find that their strong MMT model tends to ignore the multimodal information. They suggest that
the gains from the multimodal signals over text-only counterparts are, in fact, due to the regularization effect.

In practice, the existing approaches mainly focus on a better interaction method between textual- and visual- information. In training, they generally employ maximum likelihood estimation (MLE) to optimize the entire model. In most cases, the source sentence carries sufficient context for translation, leading to the neglect of visual modality under such training paradigm. However, the visual information is still crucial when the source sentence faces the problem of incorrection, ambiguity and gender-neutrality . Accordingly, we argue that the visual modality remains an excellent potential to facilitate translation, which inspire us to explore taking full advantage of the visual modality in MMT.

To this end, we propose increasing visual awareness from an information theoretic perspective.
Specifically, we quantify the amount of visual information with an information-theoretic metric, namely Mutual Information (MI). The quantification is then divided into two parts: 1) the source-specific part, which means the information between source texts and images; 2) the target-specific part, which means the mutual information between target texts and images given source texts, namely conditional mutual information (CMI). For the source-specific part, we maximize the mutual information by resorting to a lower bound of MI. For the target-specific part, we propose a novel optimization method for maximizing the conditional mutual information (See details in \ref{sec:target-mi}). We evaluate our approach on two public datasets. The experimental results and in-depth analysis indicate that our method can effectively enhance the MMT model's sensitivity to visual context, which manifests that MMT has great potential when the utilization of visual context is adequate.

To conclude, the main contributions of our paper are three-fold:
\begin{itemize}
  \item [1)] We propose a novel MI-based learning framework to improve multimodal machine translation by increasing visual awareness. This framework is interpretable and can quantify how much visual information is utilized.
 \item [2)] To optimize the mutual information objective, we divide the corresponding visual information into source/target-specific part. Furthermore, we propose a novel method to maximize the conditional mutual information in the target-specific part.
  \item [3)] Comprehensive experiments demonstrate that our method can effectively encourage MMT models to keep sensitive to visual context and significantly outperform strong baselines on two public MMT datasets.

\end{itemize}

  \section{Background}
  \subsection{Multimodal Machine Translation}
  We start with the formulation of a regular MMT task.
  Given a triplet dataset of $\{(x^{(i)},y^{(i)},z^{(i)})\}_1^N$, the problem naturally turns into the likelihood maximization:
  \begin{align}
      \mathcal{L}_{MMT}
      = -\frac{1}{N} \sum_{i=1}^{N} \log{p(y^{(i)}|x^{(i)},z^{(i)})},
  \end{align}
  where $z$ denotes the image itself and $x,y$ denote the description of the image in two different languages. Recently, MMT models are usually consisted of a Transformer-based encoder-decoder model and an additional image encoder. There exist several lines of works to integrate convolutional features into the NMT model. We adopt the global feature, which is in the form of a single vector of an image. This integration method has been proven beneficial to some extent in prior works.

\begin{figure}[tbp]

  \includegraphics[width = 7.5cm]{cr_mi.pdf}
  \caption{Chain rule for multual information. We decompose visual awareness into two parts: 1) mutual information between source text and image and 2) conditional mutual information between target text and image.}
  \label{fig:chain_rule}
\end{figure}

  \subsection{Mutual Information (MI)}
  Mutual Information (MI) relates two random variables $\mathrm{X}$ and $\mathrm{Y}$ and measures the amount of information obtained from $\mathrm{X}$ by observing the random variable $\mathrm{Y}$, which is defined as:
  \begin{align}
  I(\mathrm{X};\mathrm{Y})=H(\mathrm{X})-H(\mathrm{X}|\mathrm{Y}),
  \end{align}
  where $H(\cdot)$ denotes information entropy. Mutual information is symmetric and non-negative. High mutual information shows a large reduction of the random variable $X$'s uncertainty given another variable $Y$.

\section{Methodology}

\begin{figure}[tbp]

  \includegraphics[width = 7.5cm]{model1.pdf}
  \caption{An overview of our method, where $x$ and $z$
  represent the source text and image, $x^-$ is a contrastive sample of $x$, and $\widetilde{z}$ is a deteriorated version of $z$ by removing visual information. $L_{SMI}$ aims to maximize MI between $x$ and $z$ to learn grounded representation. $L_{TMI}$ aims to maximize between $y$ and $z$ to strengthen the effect of visual context in generation. }
  \label{fig:model}
\end{figure}

In this section, we mathematically describe the basic notion of our proposed method from an information theoretic perspective.
Let $\mathrm{X}$ denotes a random variable over source sentences, $\mathrm{Y}$ denotes a random variable over target sentences, and $\mathrm{Z}$ is a random variable over images.
They are jointly distributed according to some true p.m.f. $p(x, y, z)$. We formulate the \textbf{Visual Awareness} of a MMT model as a joint mutual information $I(\mathrm{X},\mathrm{Y}; \mathrm{Z})$, which indicates how much the visual modality contributes to the overall system. A high joint mutual information suggests a strong sensitivity of the MMT model to visual information. However, it is intractable to directly maximize this joint mutual information in training. Thus, in this work we decompose $I(\mathrm{X},\mathrm{Y}; \mathrm{Z})$ into two parts by the chain rule of mutual information and optimize these two parts separately:
\begin{align}
    I(\mathrm{X},\mathrm{Y}; \mathrm{Z})= \underbrace{I(\mathrm{X};\mathrm{Z})}_{source-specific}+
      \underbrace{I(\mathrm{Y};\mathrm{Z}|\mathrm{X})}_{target-specific}.
\end{align}

As shown in Figure \ref{fig:chain_rule}, the green part $I(\mathrm{X};\mathrm{Z})$ is the source-specific mutual information (SMI), which indicates the mutual information between source texts and images; the yellow part $I(\mathrm{Y};\mathrm{Z}|\mathrm{X})$ is the target-specific conditional mutual information (TMI), which indicates the mutual information between target texts and images given source texts. Figure \ref{fig:model} illustrates the overview of our proposed method. In the following, we describe how to optimize the source-specific and target-specific mutual information in detail.

  \subsection{Source-Specific Mutual Information}
  Source-specific mutual information $I(\mathrm{X};\mathrm{Z})$ determines how much information is shared between the source texts and images.
  Inspired by \citet{Oord2018RepresentationLW}, we minimize the $\textbf{InfoNCE}$ \cite{Logeswaran2018AnEF} loss $\mathcal{L}_{SMI}$ to maximize a lower bound on $I(\mathrm{X};\mathrm{Z})$.
  \begin{align}
  I(\mathrm{X};\mathrm{Z}) \geq  -&\mathbb{E}_{p(x,z)} \biggr[
      \mathit{f}(\mathcal{F}(x), \mathcal{R}(z)) - \nonumber\\
      \mathbb{E}_{q(x^-)} \log &
      \biggr[ \sum_{x^-}  \exp{\mathit{f}(\mathcal{F}(x^-), \mathcal{R}(z))} \biggr]
   \biggr]  \nonumber \\
    =& \mathcal{L}_{SMI},
  \end{align}
  where $\mathit{f(\cdot)}$ is the cosine similarity function between $\mathcal{F}(x)$ and $\mathcal{R}(z)$.
  $\mathcal{F}(x)$ denotes the pooled textual representation of a source sentence
  $x$ and $\mathcal{R}(z)$ denotes the pooled visual representation from a pre-trained vision model (e.g., ResNet or Vision Transformer). $x^-$ is a negative
  source sentence drawn from the proposal distribution $q(x^-)$. This contrastive learning objective aligns the textual and visual representations into a unified semantic space, and encourages the text encoder to generate grounded representations, which has shown effectiveness in various scenarios \cite{Elliott2017ImaginationIM,Kdr2017RepresentationOL}.

  \subsection{Target-Specific Conditional Mutual Information}
  \label{sec:target-mi}
  In MMT, $I(\mathrm{Y};\mathrm{Z}|\mathrm{X})$ represents how much information the image $Z$ provides corresponding to the target $Y$ given the source $X$.
  Our goal is to maximize the conditional mutual information as follows:
  \begin{align}
      I(\mathrm{Y};\mathrm{Z}|\mathrm{X})=H(\mathrm{Y}|\mathrm{X})-H(\mathrm{Y}|\mathrm{X},\mathrm{Z}).
  \end{align}

  Note that the term $H(\mathrm{Y}|\mathrm{X})$ refers to the conditional probability $P(Y|X)$, which is non-accessible in a MMT model.
  To remedy this issue, we instead introduce a pseudo conditional mutual information  $I(\mathrm{Y};\widetilde{\mathrm{Z}}|\mathrm{X})$, which is formulated as:
  \begin{align}
      I(\mathrm{Y};\widetilde{\mathrm{Z}}|\mathrm{X})=H(\mathrm{Y}|\mathrm{X})-H(\mathrm{Y}|\mathrm{X},\widetilde{\mathrm{Z}}),
  \end{align}

  where $\widetilde{\mathrm{Z}}$ represents a set of \textbf{deteriorated images} $(\widetilde{z_1},\widetilde{z_2},\cdots,\widetilde{z_n})$. The visual information of these deteriorated images are artificially interfered by some means. For a well-trained visual-aware MMT, we suppose a deteriorated image would not significantly reduce the uncertainty of target sentence prediction compared to its text-only counterpart. Thus our goal is to minimize the pseudo conditional mutual information $I(\mathrm{Y};\widetilde{\mathrm{Z}}|\mathrm{X})$. Accordingly, we have the training objective as follows:
\begin{align}
  I(\mathrm{Y};\mathrm{Z}&|\mathrm{X})-I(\mathrm{Y};\widetilde{\mathrm{Z}}|\mathrm{X})
   \nonumber\\
   =& -H(\mathrm{Y}|\mathrm{X}, \mathrm{Z}) + H(Y|\mathrm{X},\widetilde{\mathrm{Z}}) \nonumber\\
   ~=& \sum_{x,y,z}p(x,y,z)\log{p(y|x,z)} \nonumber\\
   ~&- \sum_{x,y,\widetilde{z}}p(x,y,\widetilde{z})\log{p(y|x,\widetilde{z})}.
\end{align}

 Minimizing $I(\mathrm{Y};\widetilde{\mathrm{Z}}|\mathrm{X})$ means that the MMT can correctly discriminate the useless visual information.
 Moreover, maximizing $I(\mathrm{Y};\mathrm{Z}|\mathrm{X})$ means that the MMT model is able to make the most of the grounding information from images.

 However, the above training objective cannot be directly calculated since the probability mass function $p(x,y,z)$ is unobtainable.
 Therefore, we follow  \citet{Bugliarello2020ItsET,Fernandes2021MeasuringAI} to use a Monte Carlo estimator to approximate it. Under sufficient held-out training data of $\{(x^{i},y^{i},z^{i})\}_{i=1}^N$, we can estimate
  the true cross-entropy as follows:
  \begin{align}
      H(\mathrm{Y}|\mathrm{X}, \mathrm{Z}) \approx -\frac{1}{N} \sum_{i=1}^{N}\log{p(y^{(i)}|x^{(i)},z^{(i)})}.
  \end{align}

  This estimation will get more accurate when the training data set is large enough. We note that $\widetilde{z}$ is an deteriorated version of the image $z$ and is also drawn from the same true probability
  distribution. We can employ the same estimation method to approximate the cross-entropy. Finally, we have an estimation for the optimization objective:
  \begin{align}
      \mathcal{L}_{TMI}
      = -\frac{1}{N} \sum_{i=1}^{N} \log{\frac{p(y^{(i)}|x^{(i)},z^{(i)})}{p(y^{(i)}|x^{(i)},\widetilde{z}^{(i)})}}.
  \end{align}

  Moreover, the training process can be unstable when the term $p(y^{(i)}|x^{(i)},\widetilde{z}^{(i)})$ becomes very small, ignoring the fact that $x$ has already contained the majority of information for translation. Following \citet{Huang2018LargeMN}, we perform back propagation only when $\log{\frac{p(y^{(i)}|x^{(i)},z^{(i)})}{p(y^{(i)}|x^{(i)},\widetilde{z}^{(i)})}}$ is lower than a margin value $m$.
  The above objective can be converted into:
  \begin{align}
      \mathcal{L}_{TMI}
      = \frac{1}{N} \sum_{i=1}^{N} \max\{0, m-\log{\frac{p(y^{(i)}|x^{(i)},z^{(i)})}{p(y^{(i)}|x^{(i)},\widetilde{z}^{(i)})}\}
      }.
  \end{align}
  \subsection{Deteriorated Image Generation}
  In this section we describe how to generate a deteriorated view of the image $z$. One straightforward approach is masking relevant objects in the visual modality by using an object detector. This method can reduce the amount of information contained in a picture. However,
  using an external object detector is time-consuming, making the MMT task more complex.
  In this work, we adopt a more efficient and feasible dropout-based method to generate the deteriorated images at the representation level. Specifically, we randomly select the units in the visual vector and mask them with zeros by following a Bernoulli distribution of probability $\rho$. In this way, the majority of information in the image representations will be corrupted.

  \subsection{Model Architecture}
  We use a simple gated mechanism for multimodal fusion, which has achieved competitive results versus other complicated models. Following the recent work \cite{Wu2021GoodFM}, the source sentences and the images are separately encoded by using a transformer-based encoder and a pre-trained Resnet model.
  Formally, the textual and image representations are combined by a learnable gate vector $\lambda$ as:
  \begin{align}
      ~&\lambda = \textrm{Sigmoid}(W_{\lambda}\cdot\textrm{Concat}[\mathcal{F}(x);\mathcal{R}(z)]), \nonumber\\
      ~&\mathrm{H_{fusion}} = \mathcal{F}(x) + \lambda  \odot \mathcal{R}(z).
  \end{align}
$W_{\lambda}$ is a trainable matrix and $\odot$ denotes element-wise product. The decoder takes the modality input $\mathrm{H_{fusion}}$ in the recurrent fashion as the text-only NMT does. The deteriorated image input is fused in the same way by replacing $\mathcal{R}(z)$ with $\mathcal{R}(\widetilde{z})$.

  \begin{table*}[htbp]
    \centering

    \begin{tabular}{cccccc}
      \toprule[1.0pt]

      \multicolumn{1}{c|}{\multirow{2}{*}{Row}} & \multicolumn{1}{c|}{\multirow{2}{*}{Method}} & \multicolumn{2}{c|}{Validation}                           & \multicolumn{2}{c}{Test}            \\ \cline{3-6}
      \multicolumn{1}{c|}{}                     & \multicolumn{1}{c|}{}                        & \multicolumn{1}{c|}{BLEU} & \multicolumn{1}{c|}{METEOR} & \multicolumn{1}{c|}{BLEU} & METEOR \\
      \hline \hline
      \multicolumn{6}{c}{\textbf{\textit{Only Text}}}                                                           \\ \hline
      \multicolumn{1}{c|}{1}                    & \multicolumn{1}{c|}{Transformer$^\spadesuit$\cite{Li2021VisionMW}}             & \multicolumn{1}{c}{-}  & \multicolumn{1}{c}{-}   & \multicolumn{1}{|c}{35.71}   & -   \\ \hline
      \multicolumn{1}{c|}{2}                    & \multicolumn{1}{c|}{Transformer$^\diamondsuit$}             & \multicolumn{1}{c}{37.46}  & \multicolumn{1}{c}{69.31}   & \multicolumn{1}{|c}{37.60}   & 68.64   \\ \hline
      \multicolumn{6}{c}{ \textbf{\textit{Existing MMT Systems}}}                                                                                \\ \hline
      \multicolumn{1}{c|}{3}                    & \multicolumn{1}{c|}{Imagination$^\diamondsuit$\cite{Elliott2017ImaginationIM}}             & \multicolumn{1}{c}{37.83}  & \multicolumn{1}{c}{68.92}   & \multicolumn{1}{|c}{38.11}  & 69.25   \\
     \multicolumn{1}{c|}{4}                    & \multicolumn{1}{c|}{Gated Fusion$^\spadesuit$\cite{Li2021VisionMW}}            & \multicolumn{1}{c}{-}      & \multicolumn{1}{c}{-}      & \multicolumn{1}{|c}{36.68}  & -      \\
      \multicolumn{1}{c|}{5}                    & \multicolumn{1}{c|}{Selective Attn$^\diamondsuit$\cite{Li2022OnVF}}          & \multicolumn{1}{c}{38.47}       & \multicolumn{1}{c}{68.91}       & \multicolumn{1}{|c}{38.30}       &   69.31     \\ \hline
      \multicolumn{6}{c}{\textbf{\textit{Our MMT System}}}                     \\ \hline
      \multicolumn{1}{c|}{6}                    & \multicolumn{1}{c|}{Gated \quad Fusion$^\diamondsuit$}             & \multicolumn{1}{c}{38.29}  & \multicolumn{1}{c}{69.62}   & \multicolumn{1}{|c}{38.06}  & 69.13   \\
      \multicolumn{1}{c|}{7}                    & \multicolumn{1}{c|}{Our \quad Method}                    & \multicolumn{1}{c}{\textbf{39.24}}  & \multicolumn{1}{c}{\textbf{70.30}}   & \multicolumn{1}{|c}{\textbf{39.40}}   & \textbf{70.22}   \\
      \bottomrule[1.0pt]
    \end{tabular}
    \caption{Tr $\Rightarrow$ En translation results on AmbigCaps dataset.
    $\spadesuit$ indicates the results come from the original papers and $\diamondsuit$ indicates the results are from our implementation. All
    models are based on the transformer-tiny configuration.}
    \label{tab:ambigcapsRes}
    \end{table*}

  \subsection{Training Objective}
  Finally, the two MI-based objectives are jointly optimized with MMT loss from scratch:
  \begin{align}
      \mathcal{L}
      = \mathcal{L}_{MMT} + \alpha|s|\mathcal{L}_{SMI} + \beta|s|\mathcal{L}_{TMI},
  \end{align}
  where $\alpha$ and $\beta$ $\in [0,1]$ are the hyperparameters for balancing MI maximizing objective and MT loss, and $|s|$ denotes the average length of the sequence since the MI-based loss is sentence-level \cite{Pan2021ContrastiveLF}.

\section{Experiments}

  \subsection{Data}
We evaluate our methods on two standard MMT datasets, including \textbf{AmbigCaps}
\cite{Li2021VisionMW}
and \textbf{Multi30K} \cite{Elliott2016Multi30KME}.

AmbigCaps contains 81K Turkish-English parallel sentence pairs with visual annotations. We use 1,000 sentences provided by Li et al. \shortcite{Li2021VisionMW} for validation and testing, respectively. This dataset is processed into a  gender-ambiguous one, in which sentences containing the gender of the entity and professions with gender implications removed. Moreover, AmbigCaps is designed to translate a gender-neutral language (Turkish) into a gender-specified language (English), which is suitable for the MMT task since the source
textual information is not naturally sufficient.

Multi30K is a widely used dataset for MMT, which contains 29K images with one English description and a German manual translation.  We follow a standard split for experiments and use 1,014 sentences for validation and 1,000  sentences for testing (Test2016). In addition, we also evaluate the WMT17 test set (Test2017) and the ambiguous MSCOCO test set, which include 1,000 and 461 triplets, respectively.

\begin{table*}[t]
  \centering
  \begin{tabular}{c|cc|cc|cc}
  \toprule[1.5pt]
  \multirow{2}{*}{Method} & \multicolumn{2}{c}{Test2016}      & \multicolumn{2}{c}{Test2017}      & \multicolumn{2}{c}{MSCOCO}        \\ \cline{2-7}
      & \multicolumn{1}{c}{B}   & M     & \multicolumn{1}{c}{B}   & M     & \multicolumn{1}{c}{B}   & M     \\
    \midrule[1pt]
  Transformer \cite{Vaswani2017AttentionIA}            & \multicolumn{1}{c}{41.02} & 68.22 & \multicolumn{1}{c}{33.36} & 62.05 & \multicolumn{1}{c}{29.88} & 56.64 \\
  Imagination  \cite{Elliott2017ImaginationIM}           & \multicolumn{1}{c}{41.31} & 68.06 & \multicolumn{1}{c}{32.89} & 61.29 & \multicolumn{1}{c}{29.9}  & 56.57 \\
  UVR  \cite{Zhang2020NeuralMT}                   & \multicolumn{1}{c}{40.79} & - & \multicolumn{1}{c}{32.16} & -     & \multicolumn{1}{c}{29.02} & -     \\
  DCCN  \cite{Lin2020DynamicCC}                  & \multicolumn{1}{c}{39.7}  & -     & \multicolumn{1}{c}{31.0}    & -     & \multicolumn{1}{c}{26.7}  & -     \\
  Multimodal Graph\cite{Yin2020ANG}        & \multicolumn{1}{c}{39.8}  & -     & \multicolumn{1}{c}{32.2}  & -     &
  \multicolumn{1}{c}{28.7}  & -     \\
      Gated Fusion  \cite{Wu2021GoodFM}          & \multicolumn{1}{c}{41.55} & 68.34 & \multicolumn{1}{c}{33.59} & 61.94 & \multicolumn{1}{c}{29.04} & 56.15 \\
    Selective Attn \cite{Li2022OnVF}         & \multicolumn{1}{c}{\textbf{41.93}} & 68.55 & \multicolumn{1}{c}{33.62} & 61.61 & \multicolumn{1}{c}{29.72} & \textbf{56.94} \\ \midrule[1pt]
  Our \quad Method               & \multicolumn{1}{c}{41.77} & \textbf{68.60}  & \multicolumn{1}{c}{\textbf{34.58}} & \textbf{62.40}  & \multicolumn{1}{c}{\textbf{30.61}} & 56.70  \\
  \bottomrule[1.5pt]
\end{tabular}
\caption{En $\Rightarrow$ De results on Multi30K dataset. Some results are from Wu et al.  \shortcite{Wu2021GoodFM} and Li et al. \shortcite{Li2022OnVF}. B and M denote BLEU and METEOR score.}
\label{tab:multi30kRes}
\end{table*}

\subsection{System Setting}
We use the Transformer-Tiny \cite{Vaswani2017AttentionIA} configuration to conduct all of our experiments, which can even obtain better performance than those large models \cite{Wu2021GoodFM}. The tiny transformer model consists of 4 encoder
and decoder layers, with 1024 embedding/hidden units, 4096 feed-forward filter
size and 4 heads per layer. We employ the Adam optimizer with $\text{lr}=0.005$, $t_{\text{warm}\_\text{up}}=2000$ and $\text{dropout}=0.3$ for optimization. At the training time, each training batch includes 4096 source tokens and target tokens. For a fair comparison with previous works, we use the early stopping strategy if the performance on validation does not gain improvements for ten epochs on Multi30K.
Empirically, $1e^{-3}$ for $\alpha$ and $1e^{-4}$ for $\beta$ can get the best results.
The model for AmbigCaps is trained on a single
Tesla P40 GPU and the one for Multi30k is trained on two GPUs for a fair comparison with previous work.

In particular, the number of patience on AmbigCaps is set to twenty epochs since the converge speed on this dataset is slow. We average the results of the last ten checkpoints during inference. At decoding time, we generate with $beam\_size=5$ and length penalty is equal to $1.0$.

For visual features, we use a ResNet-50 \cite{He2016DeepRL} model pre-trained on ImageNet as the image encoder. The dimension of the global feature is 2048 and we use a projection matrix to convert the shape of image features into that of text features. For data processing, we generate subwords following Sennrich et al. \shortcite{Sennrich2016NeuralMT} with 10,000 merging operations jointly
to segment words into subwords, which generates a vocabulary of  6,765 and 9,712  tokens for AmbigCaps and Multi30K, respectively.

Finally, we report BLEU\footnote{\small{\url{https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/multi-bleu.perl}}} \cite{Papineni2002BleuAM} and METEOR\footnote{\small{\url{https://github.com/facebookresearch/vizseq/blob/main/vizseq/scorers/meteor.py}}} \cite{Denkowski2014MeteorUL} scores to evaluate the
quality of different translation system. Our implementation are implemented on Fairseq \cite{Ott2019fairseqAF}.

\subsection{Baseline Methods}
We compare our method with baselines as follows:
\begin{itemize}
  \item \textbf{Vanilla Transformer} \cite{Vaswani2017AttentionIA}. We report the Transformer-tiny results.
  \item  \textbf{Imagination} \cite{Elliott2017ImaginationIM,Helcl2018CUNISF}. This method adopts a margin loss as a regularizer to the text encoder in order to learn grounded representations. We  reimplement this method in our Transformer-based models instead of RNNs for a fair comparison.
  \item  \textbf{Gated Fusion} \cite{Wu2021GoodFM}.  It  uses a gate vector  to combine textual representations and image representations and then feds them into the MMT decoder. This is a simple yet effective method and can outperform most of MMT systems.
  \item \textbf{Selective Attn} \cite{Li2022OnVF}. This method has the same model architecture with \textbf{Gated Fusion}
  and supports more fine-grained features extracted from images.
\end{itemize}

Since \textbf{AmbigCaps} is a new  MMT dataset, we reproduce the above baseline methods and report the corresponding results.

\subsection{Results on Tr$\Rightarrow$En  Translation Task}
The evaluation results of different MMT systems on the  Tr $\Rightarrow$ En translation task are presented in Table \ref{tab:ambigcapsRes}.
By observing the reported results, we draw the following interesting conclusions:

{\bf First}, we observe that our implementation for the text-only transformer and Gated Fusion outperform the results reported in the original paper. This is due to the larger number of patience. It also indicates our method is more competitive and still achieves improvement against the strong baselines.

{\bf Second}, all of the MMT models beat the text-only transformer, which indicates that AmbigCaps is a suitable dataset and verifies the benefit of the visual modality. Gated Fusion gains 0.83 BLEU points and 0.31 METEOR points compared to the text-only model. Selective Attn shows further improvement due to the more-grained features. Our model significantly outperforms the text-only transformer by 1.8 on BLEU and 1.58 on METEOR.

{\bf Third}, our model also beats Imagination, Selective Attn and Gated Fusion. The gap between different baseline results is not significant. Note that the Imagination model does not use the image during the inference time.
Our model uses the same model architecture with Selective Attn and Gated Fusion and consistently improves the BLEU score, 1.30 and 1.34 points more than Selective Attn and Gated Fusion. It indicates that our method can significantly utilize the image context and improve translation by using the same model configuration.

\subsection{Results on En$\Rightarrow$De  Translation Task}
Table \ref{tab:multi30kRes} shows the main results on the En$\Rightarrow$De translation task. We find that the text-only transformer can act as a solid baseline with the aid of the elaborate model configuration and training settings, which even beats DCCN and Multimodal Graph models. The BLEU score of our method on Test2016 is quite marginal compared to the base model Gated Fusion, which is the point that we attempt to improve. We argue that the Test2016 test set may contain less textual ambiguity. Besides, we observe significant BLEU/METEOR gains on Test2017 and MSCOCO, outperforming the text-only baseline by 1.22/0.73 points and 0.99/1.57 points. Moreover, our method beats all robust baseline systems on the two test sets.
These results further indicate the necessity of visual context for multimodal translation.

\section{Model Analysis}

\begin{table}[t]
  \centering
  \begin{tabular}{l|c|c|c|c}
    \toprule[1.2pt]
    System                        & B  & M   &   ID $\uparrow$ & GA $\uparrow $ \\
    \midrule[1pt]
    $\mathcal{L}_{MMT}$            & 38.06 &   69.13    & 3.32 & 79.73\% \\
    \multicolumn{1}{r|}{+ $\mathcal{L}_{SMI}$} & 38.72 &   69.50  & 3.35  & 80.23 \%     \\
    \multicolumn{1}{r|}{+ $\mathcal{L}_{TMI}$} & 38.74 &   69.93 & 4.37  & 79.93\%  \\
    Full Model            & 39.40 &   70.22 & 8.31 & 80.65\%  \\  \bottomrule[1.2pt]
  \end{tabular}
    \caption{Ablation study on AmbigCaps. B and M denote BLEU and METEOR score.  $\uparrow$ indicates the visual awareness is better as the value is larger.}
    \label{tab:ablation}
  \end{table}

\subsection{Ablation Study}

To make a better understanding of the influence of the training objective, we perform ablation studies to validate the impact of each part on translation quality.
Column 2 to column 3 in Table \ref{tab:ablation} shows both $\mathcal{L}_{SMI}$ and $\mathcal{L}_{TMI}$ results in a close gain in BLEU score, 0.66 and 0.68 points more than the vanilla MMT model. We find $\mathcal{L}_{TMI}$ can perform better than $\mathcal{L}_{SMI}$ in METEOR score. Combining the two training objectives can further boost the translation performance, which means $\mathcal{L}_{SMI}$ and $\mathcal{L}_{TMI}$  can be beneficial to each other. It indicates that our method improves the image utilization at both the source and target sides.

Besides, we also investigate how the visual signals help to translate and whether the gains on translation come from the increasing visual awareness . To achieve this goal, we introduce two new
metrics to evaluate the visual awareness of the MMT model, which are \textit{Incongruent Decoding} and  \textit{Gender Accuracy.}  We will first explain how these metrics are calculated.

{\bf Incongruent Decoding (ID)}\ \ We follow Elliott et al. \shortcite{Elliott2018AdversarialEO}
to use an adversarial evaluation method to test if our method is more sensitive to the visual context. We randomly replace the image with an incongruent image or set the congruent image's feature to all zeros. Then we observe the value $\bigtriangleup \rm BLEU$ by calculating the difference between the congruent data and the incongruent one. A larger value means the model is more sensitive to the image context.

{\bf Gender Accuracy (GA)}\ \ Following Li et al. \shortcite{Li2021VisionMW}, we divide the translation sentences into three categories:
\textit{male, female} and \textit{undetermined}. \textit{Male} means that the target sentences contains at least one of the male pronouns \{‘he’, ‘him’, ‘his’, ‘himself’\}.
The definition of \textit{Female} is similar to \textit{Male}. The sentence is \textit{undetermined} when it contains both male and female pronouns or neither, and will not be considered for the calculation of gender accuracy. It is designed to determine whether the ambiguous words are indeed translated correctly.

Column 4 to column 5 in Table \ref{tab:ablation} show the results of visual awareness evaluation.
When given the incongruent image, $\mathcal{L}_{MMT}$ drops more than 3 points, indicating that this simple method can also utilize the visual context to some extent.  When   $\mathcal{L}_{SMI}$
or $\mathcal{L}_{TMI}$ is used, the dropped value  will become larger. Combining the two objective
can lead to a great decline. It means the visual part plays a vital role in our system.  There is also a significant increase in gender accuracy with our method.
It also proves that the translation improvement partly benefits from the correct translation of ambiguous words.

  \begin{figure}[htbp]

\centering
\includegraphics[width=0.73\linewidth]{mi.pdf}
\caption{Illustration of BLEU score  with different $\alpha$ (left) and $\beta$ (right).}
\label{fig:mi}
\end{figure}

\subsection{The Influence of Weight $\alpha$ and $\beta$}
To evaluate the effects of the coefficients of $\alpha$ and $\beta$,
we observe the translation performance over various values by setting the corresponding coefficient to zero.
Figures \ref{fig:mi} shows the performance fluctuates greatly with a low rate of
$\alpha$ and $\beta$.
With the coefficients increasing slightly, the BLEU score improves, which probes the effectiveness of increasing visual awareness by applying our method.
The translation performance drops slightly
as the coefficients increase since the MI-based loss will dominate the optimization and ignore the MLE training objective.
In particular, the best performance is achieved when $\alpha=1e^{-3}$ or $\beta=1e^{-4}$ respectively.

\begin{figure}[htbp]
  \centering
  \includegraphics[width = 0.8 \linewidth]{dp.png}
 \caption{
    Illustration of BLEU score on AmbigCaps with various methods.
  The orange dash line denotes the model that only uses $\mathcal{L}_{MMT}$.
  The red solid line denotes the one that masks features with different probabilities $\rho$. The fuchsia dotted line denotes the one that uses an object detector for masking.
  }
  \label{fig:dprate}
\end{figure}

\subsection{The Influence of Deteriorated Image Generation.}
First, we compare the effects of
different methods to generate the deteriorated image in Figure \ref{fig:dprate}.
We find that using a object detector for masking  relevant objects can generate the best translation results. It is not surprising since
the picture in AmbigCaps is not complex and the detector model is competent to detect the most objects correctly. We should note that our method is
a unsupervised manner but can still generate the competitive results.
We also explore the different influence of $\mathcal{L}_{TMI}$
with the various dropout rate $\rho$.
When the value of $\rho$ is very small, the BLEU score is lower than the baseline. This is due to that a small value of $\rho$ means that most image information is preserved and the term $I(\mathrm{Y};\widetilde{\mathrm{Z}}|\mathrm{X})$  should be as large as
possible instead of being minimized.
Meanwhile, $\rho=1$  means that the image is totally unused, and the global vector is all zeros.
This conspicuous feature induces the model to take a shortcut to discriminate the useless image easily.
Only the appropriate value of $\rho\geq 0.5$ can get stable and good results.
It indicates a well-designed  method to generate deteriorated image is critical for better translation.

\section{Related Work}
\subsection{Mutual Information Maximization} Mutual information has been
widely explored in recent NLP work. Bugliarello et al. \shortcite{Bugliarello2020ItsET} proposed
cross-mutual information to measure the neural translation difficulty.
Fernandes et al. \shortcite{Fernandes2021MeasuringAI}
used conditional mutual information as the metric to estimate the context usage
in document NMT. Our work differs in that we employ MI as a training objective to
enhance image usage instead of a pure metric. Wang et al. \shortcite{Wang2022MINERIO}
maximized MI to solve the OOV problem in NER. Kong et al. \shortcite{Kong2020AMI} improved language model pre-training from a  mutual information maximization perspective.
To our best knowledge, we are the first to employ mutual information to increase image awareness in MMT generation task.

\subsection{Efficient Visual Context  Modeling}
To equip the MMT model with the visual context modeling ability,
Caglayan et al. \shortcite{Caglayan2021CrosslingualVP} and Song et al. \shortcite{Song2021ProductorientedMT}
used cross-modal pre-training to learn visually-grounded representations.
This point is similar to our source-related MI method, which aligns
the textual and visual representations in a shared semantic space.
Wang and Xiong \shortcite{Wang2021EfficientOV}
forced the model to reward masking irrelevant objects and penalize
masking relevant objects. It also makes the model sensitive to the changes in
the visual modality and enhances visual awareness. But detecting the relevant objects and correlating the regions with the vision-consistent target words are rather complex.
\subsection{Contrastive Learning}
Our method is highly related to contrastive learning. The source-specific loss is optimized
by using the $\textbf{InfoNCE}$  bound, and the target-specific loss can also be viewed as
using a contrastive image form and lowering the probability of generating the target.
Contrastive learning has been recently used in translation tasks.
Pan et al. \shortcite{Pan2021ContrastiveLF} closed the gap among representations of different languages
to improve the multilingual translation.
Zhang et al. \shortcite{Zhang2021FrequencyAwareCL} meliorated word representation space to solve the low-frequency word prediction.
Hwang et al. \shortcite{Hwang2021ContrastiveLF} used coreference information for generating the contrastive samples to train the model to be sensitive to coreference inconsistency.
Different from this work, our method does not need external knowledge.

\section{Conclusion}
In this paper, we summarize that only using the MLE objective is unable to ensure the image information
is fully utilized, and there still exists tremendous potential to explore.
We propose a method based on mutual information to increase visual awareness in multimodal machine translation.
The information from images is divided into two parts(source-specific and target-specific).
We quantify this contribution from the image and put forward the corresponding optimization solutions.
Experiments on Multi30K and AmbigCaps show the superiority of our methods.
Detailed analysis experiments conducted probes that our methods are quite interpretable.
In the feature, we would like to apply our MI-based method to visual language pretraining
and other unsupervised tasks to improve the downstream tasks.

\section{Limitations}
Since we conduct all experiments on the machine translation task, it is unclear  whether our approach can benefit other multimodal tasks, e.g. Visual QA  or multimodal Dialog.
Another thing worth noting is that the image-source-target triplet data is not easily available in reality. Thus how to effectively use the unpaired  image in the unsupervised way is a more  promising research.

Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL)