Computer Science
Computer Science-2021.gz:143281
Computer Science
53973fb90914489aa43eb40f89a73e2cf8ce7263
VPN: Video Provenance Network for Robust Content Attribution

We present VPN - a content attribution method for recovering provenance information from videos shared online. Platforms, and users, often transform video into different quality, codecs, sizes, shapes, etc. or slightly edit its content such as adding text or emoji, as they are redistributed online. We learn a robust search embedding for matching such video, invariant to these transformations, using full-length or truncated video queries. Once matched against a trusted database of video clips, associated information on the provenance of the clip is presented to the user. We use an inverted index to match temporal chunks of video using late-fusion to combine both visual and audio features. In both cases, features are extracted via a deep neural network trained using contrastive learning on a dataset of original and augmented video clips. We demonstrate high accuracy recall over a corpus of 100,000 videos.


INTRODUCTION
Video is a powerful medium for storytelling. Yet, the ease with which digital video may be synthesized, manipulated and shared (e.g. via social media) presents a growing societal threat via the amplification of misinformation and fake news [21].
Emerging solutions counter this threat in two main ways: 1) Detecting synthetic or manipulated video [17]. Detection methods are caught in a perpetual race against generative AI tools that seek to evade detection. Moreover, digital manipulation is often for editorial rather than deceptive purposes; 2) Determining the 'provenance' of the video asset [5,45]. Provenance techniques do not seek to adjudicate on content validity but instead trace the origins of content, and what was done to it. Rather than automatically deciding on the trustworthiness of the content, the goal is to place the consumer in a more informed position to make a trust decision. This paper falls into the second camp, and complements emerging technical standards that embed cryptographically secured provenance information with the metadata of the asset [1,13,49]. For example, the emerging specification from cross-industry body the 'Coalition for Content Provenance and Authenticity' (C2PA) writes provenance information into a 'manifest' transported within the asset metadata [13]. Such approaches are vulnerable to removal of metadata, which is common on social media platforms through which misinformation is often spread. For example, video uploaded to any major social media platform today would be stripped of such manifests. Furthermore, alternative manifests may be substituted describing a fake provenance trail or 'back story', so attributing a video out of context to tell a different story. Content misattribution may also deprive creators of credit for their work, enabling intellectual property theft. This paper contributes a method for robustly matching video assets circulating without provenance metadata, to an authoritative copy of that asset with such metadata (such as a C2PA manifest), held within a trusted database. Videos often undergo various transformations during online distribution; changes in format, resolution, size, padding, effect enhancement etc. that render cryptographic hashes operating on the binary stream, such as SHA-256, unsuitable as means for matching the video content. Rather, matching must be made robust to these transformations by considering features extracted from the content of the video clip. It is also desirable to match fragments of video (i.e. partial or truncated videos) to determine not only the complete source video but also the time offset at which that fragment exist. We present a system for matching partial video queries robust to such transformations. We build an inverted index of robust audiovisual features trained using contrastive learning and a rich set of augmentations representative of transformations typically applied to video 'in the wild' during online content distribution. We leverage recent work exploring the visual comparison of image matches under similar transformations, and show this technique may be applied to visually compare a query and its matched results to identify areas of content manipulation. We demonstrate our matching technique within the context of a system for tracing the provenance of video assets, using a corpus of 100,000 videos from the VGGSound dataset [9].

RELATED WORK
Video content authenticity has been explored primarily from the digital forensics perspective of detecting generative (or 'deep fake') content, or identifying if and where video has been digitally tampered. The Deep Fake Detection Challenge (DFDC) [17] explored detection of facial manipulation in video. Networks may be trained to characterize departures from natural image statistics [57], or artifacts introduced can be recognized to localize manipulated regions [56], in some cases even identifying which GAN synthesized an image [60]. Detection of video manipulation similarly exploits temporal anomalies [58] or GAN limitations such as lack of blinking [39]. Forensic models are in a perpetual 'arms-race' with the rapid innovation of generative methods seeking to evade detection.
Attribution methods trace the provenance of media -where it came from, and what has been done to it. A robust identifier, such as a watermark [2,15,24,47], may be embedded in media or a perceptual hash or fingerprint is computed from the media content itself [4,8,32,41,44,46,63]. The identifier serves as a key to lookup provenance information in a provenance database operated by a trusted source [1], or in a decentralized trusted database such as a blockchain [7,14].
In the image domain, visual similarity algorithms focus upon compact, robust perceptual hashing [55]. Early approaches compute content fingerprints from coefficients in the spectral domain [29,46,62]. CNNs are now the dominant method for image similarity. Deep Hashing Networks (DHNs) [65] extended an ImageNettrained AlexNet [33] feature encoder [23] with a quantization loss, to obtain hashes that retained semantic discrimination. Deep Supervised Hashing (DSH) [41] and HashNet [8] also train CNNs to learn visual hashes, using a siamese network and ranking loss; such losses are better suited to retrieval than classification tasks, and are used extensively in visual search [20]. DSDH [37] learns metric ranking and classification directly from the hash code using a discrete cyclic coordinate descend technique. DPSH [38] proposes a binary pairwise loss to estimate image pair similarity. DFH [40] and GreedyHash [53] use different customized layers to address the ill-posed gradient of the sign function. ADSH [30] and CSQ [61] treat hashing as a retrieval/attribution optimization problem, but require semantic annotation unavailable in the content attribution problem domain.
Visual similarity for attribution and content provenance was explicitly addressed by Nguyen et al. [45] who learn a compact binary hash robust to non-editorial transformation, but sensitive to digital manipulation in the image. The goal was to match robustly but intentionally fail to match manipulated content. Conversely, Black et al. [5] learn a fingerprint robust to both non-editorial and editorial change, opting instead to highlight regions of likely manipulation once matched. In both cases provenance information for the asset is surfaced after the match. Our work explores a similar matching strategy for video, and explores change visualization via [5] over frames. Our work combines robust matching of both audio and visual features, exploring appropriate fusion strategies and indexing to support partial asset (video fragment) matching. We also train our features under data augmentations recently proposed for the task of image similarity for content attribution [18].

METHODOLOGY
To learn a robust video fingerprinting model, we propose a selfsupervised network capable of encoding both visual and audio streams in a video. We leverage contrastive learning and a rich set of data augmentations for videos to train our model. To enable partial video matching, we follow a 'divide and conquer' approach where video is split into chunks and each chunk is indexed and search-able within an inverted index. Our retrieval pipeline is shown in Figure 2.
We describe our encoding model in sec. 3.1, our chunking, indexing method and ranking strategy in sec. 3.2, and our audio-visual fusion method in sec. 3.3.

Learning robust visual-audio features
A video circulated on the Internet may undergo certain transformations that affect either the visual or audio stream or both. For example, the visual stream may be subjected to quality reduction (noise, blur, pixelization,...) during reformatting, change in aspect ratio/geometry (padding, resize, rotation, flip,...), visual enhancement (brightness, color adjustment,...) or editorial change (text/emoji overlay, photoshop,...). Similarly, the audio stream could also be altered (compression, background noise, trimming, effect enhancement, ...). We treat such transformations as perturbations to the original video and propose to learn a frame-level visual model and an audio model to encode the respective video streams robust to the defined perturbations.

Learning frame-level visual features. We train a CNN model
(.) to project a video frame to a compact embedding space. We employ the ResNet50 architecture [26], replacing the final classifier layer with a 256-D fully connected (fc) layer that serves as the embedding. Our model is trained with loss: where is the embedding of a video frame : = ( ) ∈ R 256 ;¯+ is the average embedding of all transformations of in the mini-batch; − denotes other frame instances; (.) is a set of two MLP layers separated by ReLU that acts as a buffer between the embedding and the loss function; ( , ) measures the cosine similarity between the intermediate embeddings ( ) and ( ); is the contrastive temperature ( = 0.1 in our experiments). L (.) aims to bring the embeddings of all transformations of an image (frame) together, while pushing away other image instances. L (.) resembles NTXent loss [10], however eq. 1 accepts multiple positives in a batch instead of just a single pair of image augmentations. We initialize our model with weights from DeepComp [5]. During training, we randomly sample frames from the training videos to construct a batch. For each frame image we create augmentations to serve as positive samples (to compute¯+), while the rest in the batch acts as negatives (the denominator term in eq. 1). We empirically set = 3 for optimal performance (improves by 1% as compared with standard NTXentLoss). We observe that larger causes a drop in performance, probably because the number of unique images in the batch must be reduced accordingly in order to fit a GPU. We use an exhaustive list of frame-level augmentations for our training, including random Noise (variance 0.01), Blur (radius  [27] transformations seen by the pretrained model [5] which our model initialises from. Since our model operates on individual video frames, all transformations are applied at frame level, i.e., the temporal coherence between frames are ignored during data augmentation. However, at test time our query videos are transformed at video level to reflect the video editing and distribution practice. Fig. 4 illustrates several benign transformations applied to an example video frame.

Learning audio features.
We split the audio signal of a video into overlapping 1-second chunks and encode it via log mel-spectrogram. We then visualize the log mel-spectrogram as a 2D RGB image and treat it as input to our audio model (.). The same model architecture and loss as in sec. 3.1.1 are employed for our training, but using a different set of data augmentation methods to learn robust audio features. In general, the benign audio transformations can be categorized in two groups -those that lengthen or shorten the audio signal and those that add, remove or alter audio components. The

Chunking and Indexing
To index a variable length video X, we propose to split X into fixed-length chunks is the length function (in seconds), constant is the chunk length ( =10sec) and is number of chunks (the last chunk is padded if necessary). We split videos into chunks in a sliding window fashion with chunk stride ≤ , thus = ⌈ ( )/ ⌉. We use chunks as an atomic unit in our approach where a 'bag of features' is computed for each chunk, also indexing and search are performed at chunk level. We use = /2 in our experiments.   descriptor per second.
where F is our aggregation function, at sampling point j F ( , ) = ( , + ); is number of visual descriptors per chunk. For the audio stream, since our audio model has input size of 1 second audio length, we sample at 1 second interval with the same stride as in the visual model.
This makes our audio extraction in sync with the visual extraction process (both have an aggregation window of 1 second), resulting in the same number of audio and visual descriptors per video chunk . This setting also offers a level of flexibility to our audio-fusion strategy (sec.3.3).

Inverted index.
Similar to text search systems, we construct an inverted index that supports video retrieval at chunk-level. We sample 1M random descriptors (audio, visual or fusion, more details in sec. 3.3) and build a dictionary with codebook size K using KMeans [42]. Given a database video, we break it into chunks where each chunk is represented as a bag of codewords. The K codewords are used as entries to our inverted index, listing all chunks in the database (a mapping between a chunk and video ID is also stored).
The relevance of a query chunk = { 1 , 2 , ..., } to a database chunk is determined by a relevance score R, defined as: where tf ( , ) is the term frequency that denotes the number of times codeword appears in the video chunk and idf ( ) is the inverse document frequency, which measures how common is across all of the chunks in the dataset. Finally, the relevance of a query video Q to a database video X is defined as: 3.2.3 Re-ranking. Inverted index does not take into account the order of descriptors within a chunk, and chunk order within a   [9] 0.530 0.626 0.671 VGGish [51] 0.492 0.579 0.610 Table 2: Audio performance video. We therefore employ TF-IDF [31] to retrieve top-k candidate videos (k=200) before performing an additional re-ranking stage based on Levenshtein distance [36] which quantifies the similarity between two sequences by counting the number of edits (insertions, deletions or substitutions) required to turn one sequence into the other. Sorting by edit distance promotes to the top videos in which visual words appear in the order that matches the query the closest.

Partial video matching and localization.
An advantage of our chunking and inverted index method is that it enables retrieval even if the query is only a segment of a database video. As a by product, our method also supports localization of a video segment by searching for the closest chunk in the database, or even the closest codeword within a chunk for more fine-grained localization.

Audio-visual fusion
We explore 3 fusion approaches for audio and visual streams to create an unified video retrieval system. Early fusion -a single codebook is constructed for = [ , ] ∈ R 512 , which is a concatenation of the visual and audio descriptor of a single aggregation window. Late fusion -we build separate codebooks and inverted indices for audio and visual domains, but compute the relevance score in eq. 6 and re-ranking jointly. Learned fusion -we learn an unified audio-visual embedding for a video AW. Since the visual model (.) operates at frame-level, we average the embeddings of the frames within an AW to represent the visual feature of that AW, before concatenating with the audio feature and projecting to the unified embedding: where is a fc layer for dimensional reduction; [, ] is a concatenation; | | is number of frames in an AW (to make the model small, we sample video at 4fps, thus | | = 4). To train our model, we first train the audio and visual models separately, then use their weights to initialize our joint model training.

RESULTS AND DISCUSSION 4.1 Datasets and evaluation metrics
We use the following 2 datasets for training and evaluation: VGGSound [9] is a large audio dataset with 160k train and 14k test videos. All videos are of 10 seconds length of 310 audio classes, originally used for audio classification. We train all of our models using the public VGGSound train set without using class annotations. To create a test set for retrieval, we first construct a 100k video database from the 14k test videos plus training videos to serve as distractors. Next, we sample 1k random videos from the test set and apply random transformations on both audio and video streams in order to make the query set. For experiments with query length, we randomly clip the queries keeping a minimum 10% length (1 second). We use this dataset for the majority of our experiments. DFDC-Preview [6] consists of 5k original and manipulated videos. The dataset has a public train/test split and all videos have 15 seconds length. We employ DFDC-Preview to evaluate our model generalization to new domains (people and human voice) and unseen transformations (face swap manipulations). We do not retrain our model on DFDC-Preview, instead we evaluate our VGGSoundtrained model directly on it. We use all 1131 real videos in DFDC-Preview together with distracting videos from VGGSound to make a 100k database. We create two query sets -a real set containing 276 real test videos that are pre-applied transformations with reduction in fps, resolution and encoding quality; and a fake set containing 501 face-swap manipulated videos. We wish to attribute the queries to the original videos in the database regardless of it being real or fake.
Since there is only 1 relevant video in the database for each query, we use recall at k ( @ ) to evaluate the retrieval performance of our approach. R@k represents the ratio of queries for which the correct video was retrieved within top returned results.

Experiment settings
We train our models using the Pytorch library, with Adam optimizer, initial learning rate 0.0005 with step decay and early stopping schedule. We use a batch size of 32 unique visual or audio samples where each sample is randomly augmented 3 times (totally 96, sec. 3. 1.1-3.1.2). For chunking and indexing, since both VGGSound and DFDC-Preview videos are short and have equal length, we set the chunk size the same as the video length (note that our settings in sec. 3.2 allow indexing and querying of arbitrary-length videos). For inverted index, we set the number of entries equal 100k, which is also the codebook size for both visual and audio modalities.

Evaluating visual features
We compare our method with two following baselines: While changing the backbone network, all other parameters, such as codebook size and search algorithm are fixed. Table 1 shows the recall performance of our approach and the baselines on the VGGSound dataset. Although all 3 models have the same CNN architecture, ResNet50 (ImageNet) performs the worst since it is not exposed to the video transformations (sec. 3.1.1) during training. DeepComp [5] has significantly better performance because it is trained to be robust against a number of perturbations on images. Finally, the proposed approach outperforms the rest at all recall levels. This demonstrates that robustness against particular perturbations could be achieved by exposing the model to the same augmentations during training. The partial matching performance on visual stream is depicted in Figure 6. Overall, the retrieval is more challenging with shorter queries. The proposed method slightly underperforms DeepComp [5] at R@1 on very short queries, but outperforms on queries longer than 3 seconds (30% length of the original videos) and always performs favorably at R@10 on all query lengths. In contrast, ResNet50 [26] performs poorly again, with R@1 (R@10) ranges between 10-20% (15-30%) as the query length increases.  We report the robustness of our visual model against individual transformations in Figure 5. Overall, there is not much difference between top-1 and top-100 recall, thanks to the re-ranking step (as demonstrated later in sec. 4.7). Also, our model is most robust against common degradation (noise, encoding quality, pixelization) and minor content editing (text/shape overlay) with above 90% recall at all levels, while being more sensitive to padding, rotation and color jitter.

Evaluating audio features
We compare our method with 3 following baselines: • AVENet (finetuned) is the AVENet model finetuned using our data augmentations and loss. We keep the same model architecture and data processing but adopt our training procedure for fair comparison.
The recall performance is reported in Table 2. Our method outperforms the rest with a significant margin (21-25% consistent improvement from the nearest competitor at all recall levels). Our finetuned version of AVENet also has better performance than other pretrained models, while VGGish [51] performs the worst.
The similar trend is observed in our experiment with query length (Figure 8). Our method leads a 20% gap ahead of AVENet and AVENet (finetuned) at all query lengths, except at 1-second queries where its R@1 score levels with AVENet. Figure 7 shows the effects of individual audio perturbations on the recall performance of the proposed method. Our model is most sensitive to background noise while being less affected by other sources (c.f. Figure 3).

Audio-visual fusion
We evaluate 3 fusion methods described in sec. 3.3. Table 3 indicates the superiority of the late fusion method, with 98% recall at top-1 and near perfect performance at top-100. Learned fusion method has better score than early fusion but lower than late fusion, probably because the unified embedding and inverted index are effectively twice as compact as the combined audio and visual. Another advantage of late fusion is that it enables querying of individual modality, for example in case an user only has singlemodal data or prefer to retrieval an individual stream. Figure 10 demonstrates that audio and visual retrieval has complementary  effects, as the late fusion method improves performance versus any single-stream retrieval method.

Video attribution on DFDC-Preview
We employ our VGGSound-trained model together with the best fusion method (late fusion) and test directly on the DFDC-Preview dataset. Despite the domain shift, our method performs extremely well on the public augmented-ready real set and the manipulated fake set whose transformations are unseen during training, as shown in Table 4. On both sets we achieve above 91% R@1 and the performance is on par with the VGGSound results at R@100 (c.f. Table 3 late fusion), demonstrating the generalization of our approach.
For the fake set, we further visualize the manipulated area using the ICN method proposed in [5]. Figure 9 shows the visualization between an example query and the top-1 retrieval video. This is useful in practice for a user to be able to attribute a query video back to its original source, localize its position in the source and highlight any possibly manipulated regions.  Table 5: Ablation study when re-ranking with edit distance and TF-IDF is turned off.

Ablation study
We test the efficacy of our retrieval pipeline when stripping off one or several components (Table 5). We first turn off the re-ranking stage (sec. 3.2.3) and rank the results using only the TF-IDF relevance score. Without re-ranking the recall score significantly drops by 18% at R@1. This indicates that re-ranking can promote the relevant video to the top of the ranking by leveraging the temporal sequence of codewords within a chunk (and sequence of chunks within a long video). Next, we further turn off TF-IDF and use only the histogram count of codewords in the inverted index to rank the videos. The performance further reduces by 3% at R@1.

Localization of video segments
We demonstrate an example of partial matching and localization of a video segment in Fig. 11. Since each video can be represented as a variable-length sequence of chunks and each chunk is a sequence of codewords, we can compute the similarity between the query sequence with a sequence of a candidate video using edit distance, thus be able to localize position of the query in the candidate video (at chunk-level for long videos and codeword-level for short videos).

CONCLUSION
We present a robust video provenance system capable of matching full or partial videos circulating online without provenance metadata, to an authoritative copy of that video held within a trusted database. Our matching process is robust to transformations of the video such as changes in format, resolution, size, padding that video often undergoes during online redistribution. We employ selfsupervised contrastive learning to learn audio and visual features robust to such transformations, and match these using a vector quantization and inverse index. We explored the relative benefits Figure 11: Localization of query clips within the top-1 retrieved video. For each row, the query clip is the video segment within the white bars of the original video (also subjected to random augmentations). In all 3 cases the original video is returned at top-1. The heatmap bar shows the edit distance between the query sequence of codewords and a same-length segment of the candidate video in sliding window fashion, which could be used to represent the confidence in localization of the query within the candidate video. IoU score can be computed over a thresholded heatmap and the ground truth location.
of early versus late fusion for accommodating both modalities, and proposed an edit-distance based re-ranking process for shortlisted results. For feature learning we showed that recently proposed augmentation sets for image similarity extend well to video for the authenticity use, and that recent techniques for visualizing manipulated regions in images may be effectively applied to video on a per-frame basis.
Future work could explore integration of our feature embeddings with emerging standards [1,13,49] and decentralized frameworks such as blockchain to enable decentralized content registries for tracing videos distributed online. For example, in the draft technical specification of the cross-industry C2PA body [13] it is recognised that metadata will frequently be stripped from assets and that a 'soft binding' (such as the proposed video matching solution) may be used re-unite assets with their originals in a trusted 'provenance datastore'. Since this implies a centralization of trust, it may be interesting to store this database in a decentralized fashion. Such approaches are already being explored for example in the domain of archives and memory institutions [7,14].