RWN: Robust Watermarking Network for Image Cropping Localization

Image cropping can be maliciously used to manipulate the layout of an image and alter the underlying meaning. Previous image crop detection schemes only predicts whether an image has been cropped, ignoring which part of the image is cropped. This paper presents a novel robust watermarking network (RWN) for image crop localization. We train an anti-crop processor (ACP) that embeds a watermark into a target image. The visually indistinguishable protected image is then posted on the social network instead of the original image. At the recipient's side, ACP extracts the watermark from the attacked image, and we conduct feature matching on the original and extracted watermark to locate the position of the crop in the original image plane. We further extend our scheme to detect tampering attack on the attacked image. Besides, we explore a simple yet efficient method (JPEG-Mixup) to improve the generalization of JPEG robustness. According to our comprehensive experiments, RWN is the first to provide high-accuracy and robust image crop localization. Besides, the accuracy of tamper detection is comparable with many state-of-the-art passive-based methods.


I. INTRODUCTION
Image manipulation leads to severe security threats, where misleading photojournalism, copyright violation or fabricating stories can be a means for some writers or politicians to potentially influence public opinion. Researchers have developed a number of schemes to detect various kinds of digital attacks, e.g., tampering [1]- [3], DeepFake [4] and cropping [5]- [8]. Among them, cropping is an extremely cheap and effective way to manipulate the layout of an image and alter the underlying meaning. Existing crop detection algorithms [5], [6] mainly focus on predicting whether an image is cropped. They are represented by detecting the exposing evidences of asymmetrical image cropping, e.g., the shift of the image center [5], and the inconsistency of JPEG blocking artifacts This work is supported by National Natural Science Foundation of China under Grant U20B2051, U1936214. Qichao Ying and Xiaoxiao Hu contribute equally. Corresponding author: Zhenxing Qian (zxqian@fudan.edu.cn). We are grateful to the anonymous reviewers for their constructive and insightful suggestions on improving this paper. [6], [7]. Despite the effectiveness of these works, different cropping behaviors in the real world usually lead to varied intentions. Crop detection is only a binary classification from this perspective, unable to distinguish benign crop behaviors from those malicious attacks such as discarding a visible watermark or removing a person. Van et al. [8] proposes an image crop localization scheme to study how image cropping causes chromatic and vignetting aberration. But the tiny traces only exists in high-definition images and can be easily destroyed by image processing, which makes the scheme hard to be applicable. Experiments show that the scheme cannot be applied on highly-compressed or low-quality images. Therefore, image cropping localization remains a big issue.
Watermarking [9]- [11] aims at hiding information imperceptibly into the host data for covert purposes. The technology focuses on robustness against possible digital attacks, such as image compression, noise adding, filtering, etc. Watermarking has been widely used in copyright protection and content authentication of images in multimedia. Recently, many novel watermarking schemes for image protection is proposed. Khachaturov el al. [12] proposes a watermarking-like adversarial method that prevents inpainting systems to perform normally on a protected image. In [13], the authors propose an adversarial attack method to cause errors in super-resolution model, including making the SR image has undesired style, be incorrectly classified in classification task, or generates wrong words in image caption task. The myth behind these schemes is that once the secret information successfully travel through the Online Social Network (OSN), the recipient can decode the information to administrate a number of tasks.
This paper explores the potential of robust watermarking on image cropping localization. We propose a robust watermarking network (RWN) for image crop localization by protecting the original image through watermarking. If the protected image is cropped, RWN is expected to locate the cropped region at the recipient's side. Watermarking is utilized to hide imperceptible traces for cropping localization which is resilient against common attacks, namely, JPEG compression, etc. We further introduce a tamper detector to deal with tampering attack. Therefore, compared to previous works [5], [8], the tough goal of finding natural-yet-fragile traces for image cropping can be circumvented by maintaining the overall image quality after watermark embedding.
We train an anti-crop processor (ACP) based on invertible neural network [14] to embed a watermark into a original image. The watermark is shared with the recipient. ACP produces a visually indistinguishable protected image, which is then posted on the social network instead of the unprotected version. On receiving the attacked version of the protected image, we use ACP again to extract the watermark, and conduct an efficient and classical feature matching algorithms (SURF [15]) between the original and extracted watermark to determine where the crop was positioned in the original image plane. We further use the tamper detector to predict the tamper mask, and the features within the predicted tamper mask is discarded to prevent mismatching. Besides, we explore a simple yet efficient method (JPEG-Mixup) to improve the generalization of JPEG robustness. We test our scheme by introducing man-made hybrid attacks. The results demonstrate that our scheme can accurately locate the cropped region. We also show the effectiveness of tamper detection by comparison with some state-of-the-art image forgery detection schemes [1]- [3].
The highlights of this paper are three-folded. 1) This paper presents the first high-accuracy robust image crop localization scheme. We innovatively use normalizing flows to build invertible function for image forensic problems. 2) With the embedded watermark, RWN can also conduct high-accuracy tamper detection, which is comparable with the state-of-the-art works. 3) We explore a simple yet efficient method (JPEG-Mixup) for improved JPEG simulation. Experiments show that JPEG-Mixup can remarkably improve the generalization of RWN on robustness against JPEG compression.
II. RELATED WORKS Crop Detection and Localization. Cropping is a simple yet powerful way to maliciously alter the message of an image. The existing crop detection algorithms [5]- [7] mainly focus on predicting whether an image is cropped. For example, Fanfani et al. [5] exploits the camera principal point insensitive to image processing operations. Yerushalmy et al. [6] detects whether there are vanishing points and lines on structured image content. But cropping different areas in a same image will definitely result in different semantic changes. Therefore, we need to predict the position of the crop. In [7], the block artifact grids (BAGs) are extracted blindly with a new extraction algorithm, and then abnormal BAGs can be detected with a marking procedure as a trail of cropping detection. For crop localization, Van et al. [8] investigates the impact that cropping has on the image distribution, and predicts the absolute location of image patches. However, [8] requires the images to be untouched and uncompressed so that the tiny traces are preserved. To address the issue of practical usability, this paper presents RWN for robust image crop localization that does not restrict image format or quality. Tamper Localization Great efforts have been made on combating daily image forgeries. Kown et al. [2] propose to model quantized DCT coefficient distribution to trace compression artifacts. Mantra-Net [1] uses fully convolutional networks for feature extraction and further uses long short-term memory (LSTM) cells for pixel-wise anomaly detection. In MVSS-Net [3], a system with multi-view feature learning and multiscale supervision is developed to jointly exploit the noise view and the boundary artifact to learn manipulation detection features. In this paper, we study the effectiveness of using watermarking to aid image forensics where we forgo the idea of finding universal traces. The shortcoming of these methods is that they only detect limited types of attacks and cannot generalize their performances well on images with a chain of combined attacks. We robustly hide traces into the original image instead of discovering a universal trace for a much easier forgery localization. Robust Watermarking. The uprising of deep networks have given birth to a series of novel watermarking methods with enhanced robustness [10], [16], [17]. Shin et al. [16] is the first in including a differentiable approximation to JPEG in the watermarking model. Later, Zhu et al. [10] proposes a powerful and comprehensive network that is robust against a variety kinds of attacks. From then on, subsequent watermarking works [17] design more sophisticated and accurate algorithm to simulate image post-processing attacks, especially the JPEG compression. However, these methods uses fixed quantization table for data compression. In contrast, the quantization table in real-world JPEG images can be customized and flexibly controlled by the quality factor as well as the image content. As a result, the neural networks can over-fit and lack realworld robustness. We explore a simple yet efficient method for improved JPEG simulation.

III. METHOD
A. Approach Overview Pipeline Design. The proposed RWN consists of five stages, namely, watermark embedding, image redistribution, tamper detection, watermark extraction and crop localization. We embed a watermark W into an original image I. The protected image I M is generated and we upload it onto the social cloud instead of the unprotected original image. The attacker generates the attacked image I A by freely adding three kinds of attacks (benign attacks, cropping, tampering) on I M . On the recipient's side, the tamper detector T predicts the tamper maskM on the attacked image to see which parts of the image are tampered, and we also extract the watermark asŴ from the received image. Afterwards, we rectify the extracted watermark byŴ =Ŵ · (1 −M) to discard the tampered contents. Finally, with the original watermark W as reference, we uses a feature matching algorithm to locate the position of the crop in the original image plane. An image is determined as cropped if there is valid matching result. Modeling. We regard the embedding and extraction as the inverse problem, even if the protected image is cropped and attacked. The formulation of the inverse problem is: subject to E(Î) = I G , E(Ŵ) = W G and E(G) =Ĝ. Here, R andR are the additional output and input of the network to keep the channel numbers as four. A denotes the function of image post-processing that attacks the hidden watermark. We letR = 0.5·1 where 1 is a matrix full of one. The ground-truth cropped watermark W G and the ground-truth cropped original image I G can be generated by sharing the crop mask used by A. Besides the invertibility, we regulate that E(I M ) = I for the imperceptibility of the embedding. Fig. 1 shows the pipeline of our scheme. We employ an Anti-Cropping Processor (ACP) P that jointly learns the paired function of both watermark embedding and extraction. We implement A using a differentiable attack layer that simulates the attacker's behavior. The rest of the components include a tamper detector T , a discriminator D and a feature matcher M, which is the SURF algorithm.

B. Network Implementation
Considering the efficiency of the invertible U-Net proposed in [18], we build our ACP on top of this architecture. The network consists of six invertible blocks each of which contains a Haar wavelet transformation and a double-side affine coupling. The network ends with a conditional split layer. The number of the input and output channels are four. The sizes of R andR are of the same as W. R andR are not required to be the same.
In the attacking layer A, we first convert the marked images into the corresponding 8-bit RGB-formatted image. We adopt the Straight-Through Estimator [19] on calculating the gradients. Then, we build differentiable methods to simulate the hybrid cropping attacks, which include benign attacks, cropping and tampering attack. The benign attacks are represented by JPEG compression, scaling, etc., which do not alter the semantic meaning of the image. We take the implementation from [10] except that we build our own JPEG simulator. For cropping, we randomly crop a portion of the protected image I M . For tampering attack, we first randomly select random areas using a binary matrix M inside I M , and generate the tampered image by where I irr refers to the source of the tamper. Although there are already many scheme which include a carefully-designed JPEG simulator, e.g., JPEG-SS [17], JPEG-Mask [10], MBRS [20], the real-world JPEG robustness of these schemes is still limited. We believe it mainly attribute to that the networks are over-fitted to a fixed compression mode. In this paper, we propose to apply the Mix-Up strategy [21] over these implementations so that the generated results are more flexible. We randomly sample a quality factor QF 1 that follows QF 1 ∈ [10, 100], together with a previous differentiable implementation of JPEG simulator J 1 , where J 1 ∈{JPEG-SS, JPEG-Mask, MBRS}. We correspondingly generate a pseudo JPEG image I 1 jpg by I 1 jpg = J 1 (I M , QF 1 ). In the same fashion, we generate k − 1 more pseudo-JPEG images, denoted as I 2 jpg , ..., I k jpg . Finally, the output of JPEG-Mixup is produced by mixing the pseudo-JPEG images with arbitrary contributing rates l .
where l = 1. The proposed mix-up based JPEG simulation (JPEG-Mixup) is fairly simple yet effective. Compared to previous schemes, the interpolation technique can generates more diversed pseudo-JPEG images for RWN. In the experiments, we show the performance gain in JPEG robustness using JPEG-Mixup in comparison with the other simulators.
Finally, we build the tamper detector T upon U-Net [22], a traditional image segmentation network, and implement the discriminator D using Patch-GAN [23]. Watermark Selection. We choose gray-scaled natural images with rich texture as valid watermark. Images with too much repetitive patterns will lead to multiple matching solutions for the SURF detector. Also, considering that the recipient can only blindly extract a partial watermark, we use those common images available on the Internet that can be easily downloaded by the recipient using image retrieval systems by referring to the extracted watermark. Fig. 2 shows some examples of recommended/not recommended watermarks.

C. Objective Loss Function and Training Details
For ACP, the first part of the loss is reconstruction loss L rec for E(Ŵ) = W G , E(Î) = I G and E(I M ) = I. E(·) is the expectation operator. L rec = F(I, I M )+F(W G ,Ŵ)+F(I G ,Î), where F is the 1 distance function. The second part is the nullification loss for E(R) =R. L ran = F(R,R). The third part is the adversarial loss, which further control the introduced distortion by fooling the discriminator. We accept the least squared adversarial loss (LS-GAN) [24]. The total loss for ACP is L G = L rec + α · L dis + β · L ran , where α and β are hyper-parameters. For the tamper detector, we minimize the binary cross entropy (BCE) loss between the predicted tamper maskM and the ground-truth mask M. L T = BCE(M,M).
We resize the images to the size of 256 × 256. The hyperparameters are set as α = 1, β = 8. We set k as 3, which is the generated pseudo-JPEG images used in JPEG-Mixup. The batch size is set as 16, and we empirically find that using Batch Normalization (BN) in our scheme outperforms using Instance Normalization (IN). We use Adam optimizer [25] with the default parameters. The learning rate is 1 × 10 −4 .

A. Experiment Setup
We train the scheme on the COCO training/test set [26] with automatically generated attacks. The dataset contains 117266 training samples and 40670 testing samples. The scheme is tested with human-participated attacks, where we have invited some volunteers to crop the provided marked images and perform further image post-processing attack on them. The crop rate is roughly δ ∈ [0.5, 1). During testing, we use real JPEG coder to compress the protected images. Evaluation Metric. We employ peak signal-to-noise ratio (PSNR), structural similarity (SSIM) [27] to evaluate the image quality, and F1 score [28] to measure the accuracy of crop/tamper localization. Benchmark. We compare RWN with Van et al. [8] for crop localization. In addition, we compare RWN with Mantra-Net [1], MVSS-Net [3] and CAT-Net [2] for tamper localization.

B. Real-World Performance of Crop Localization
Quality of the Protected Images. In Fig. 3, we randomly sample different pairs of images as the original and watermark. From the figures, generally, little detail of the watermark can be found. Though the magnified difference is visible, it does not matter if the difference is visible, only that the marked image is perceptually close to the original image. We have conducted more embedding experiments over 1000 images from the test set, and the average PSNR between the protected images and the original images is 36.23dB, and the average SSIM [27] is 0.983. Accuracy of the Crop Localization. Fig. 3 further shows the results of watermark extraction, feature matching and crop localization. In row (a), we only crop the protected image. We see that the crop mask is accurately predicted. As a result, even without the prior knowledge of the original image, we know the relative position of the attacked image in the original image plane. We have conducted more experiments over 1000 images under different crop rate and different kinds of benign attacks. Here for space limit, we take QF =70 as an example. The average performances are reported in Table I. Higher IoU (Intersection over Union) and SSIM [27] indicates more accurate localization result. The average IoUs are above 0.8 where the protected images do not undergo further attacks except cropping. The scheme is proven to be agnostic to the crop size in that the performance does not degrade significantly with larger crop rate.  Robustness. We test the robustness of our scheme by conducting image post-processing attack on the marked image. In the last two groups, tampering attack is introduced. In Table I the original images for [1]- [3]. We observe that although the images are tampered by a variety of random attacks, we succeed in localizing the tampered areas. In Table II, we provide the average results over 1000 images. The F1 score of RWN is 0.773 on uncompressed images and 0.736 on JPEG attacked images (QF = 80). The performance of Mantra-Net on JPEG images is much worse than that on plain-text images. We believe the reason is that less statistical trace is preserved in the compressed version. In contrast, the embedded watermark signal serves as the alternative trace for tamper detection, which is designed to resist benign attacks. The performance on JPEG images does not drop too much. Therefore, robust watermarking can successfully aid tamper detection by hiding crafted traces similar to [12], [13].

D. Ablation Study
Influence of Discriminator. In Test 3, we do not use the discriminator D and train the pipeline with only the reconstruction and the BCE loss. Fig. 5 shows the detailed comparison results. First, the baseline results outperform the encoder-decoder network design. Second, while MBRS [20] can provide decent robustness, the extraction performance is even better using our JPEG simulator. Specifically, the average SSIM betweenŴ and W G using MBRS [20] is 0.797 compared to 0.878 reported in Table.I. The Mix-Up strategy prevents the networks from being over-fitted to any single JPEG simulator, which helps the scheme significantly improve its real-world robustness. Third, without the discriminator, the extraction result is also worse than the baseline.

V. CONCLUSION
This paper presents a novel robust watermarking network for image crop localization. We use an invertible pipeline for watermark embedding and extraction, where the watermark is shared with the recipient. ACP produces a visually indistinguishable protected image, which is then posted on the social network instead of the unprotected version. On receiving the attacked version of the protected image, we use ACP again to extract the watermark, and conduct SURF feature matching algorithms to determine where the crop was positioned in the original image plane. We also extend our scheme to detect tampering attack on the attacked image. Experiments verify that RWN is effective in real-world application. Finally, we prove the necessity of the progressive recovery as well as the network design.