BVI-DVC: A Training Database for Deep Video Compression

Deep learning methods are increasingly being applied in the optimisation of video compression algorithms and can achieve significantly enhanced coding gains, compared to conventional approaches. Such approaches often employ Convolutional Neural Networks (CNNs) which are trained on databases with relatively limited content coverage. In this paper, a new extensive and representative video database, BVI-DVC, is presented for training CNN-based video compression systems, with specific emphasis on machine learning tools that enhance conventional coding architectures, including spatial resolution and bit depth up-sampling, post-processing and in-loop filtering. BVI-DVC contains 800 sequences at various spatial resolutions from 270p to 2160p and has been evaluated on ten existing network architectures for four different coding tools. Experimental results show that this database produces significant improvements in terms of coding gains over three existing (commonly used) image/video training databases under the same training and evaluation configurations. The overall additional coding improvements by using the proposed database for all tested coding modules and CNN architectures are up to 10.3% based on the assessment of PSNR and 8.1% based on VMAF.


I. INTRODUCTION
F ROM the introduction of the first international standard in 1984, video compression has played an essential role in the application and uptake of video technologies across film, television, terrestrial and satellite transmission, surveillance and particularly Internet video [1]. Inspired by recent breakthroughs in AI technology, deep learning methods such as Convolutional Neural Networks (CNNs) have been increasingly exploited into video coding algorithms [2] to provide significant coding gains compared to conventional approaches based on classic signal/image processing theory.
It is noted that these learning-based compression approaches demand volumes of training material much greater than typically used for conventional compression or existing machine learning methods. These should include diverse content covering different formats and video texture types. As far as we are aware, no such public data sets for this purpose currently exist, and most learning-based coding methods [3] have been currently trained on image or video databases, which were mainly designed for computer vision applications, e.g. superresolution. Most of these databases do not provide sufficient D. Ma, F. Zhang and D. R. Bull are with the Department of Electrical and Electronic Engineering, University of Bristol, Bristol, BS8 1UB, UK (e-mail: di.ma@bristol.ac.uk; fan.zhang@bristol.ac.uk; dave.bull@bristol.ac.uk). 1 The BVI-DVC database can be downloaded from: https://vilab.blogs. bristol.ac.uk/?p=2375. content coverage and diversity. As a result, the generalisation of networks cannot be ensured in the context of video coding, and the optimum performance of employed CNN networks has not been achieved when trained on these databases.
In this paper, a new video database, referred as BVI-DVC, is proposed for training CNN-based video coding algorithms, in particular those tools that enhance the performance of conventional compression algorithms. These include spatial resolution re-sampling, bit depth re-sampling, post-processing and in-loop filtering, all of which have achieved significant coding gains compared to other deep learning enhancements. BVI-DVC contains 800 progressive-scanned video clips at a wide range of spatial resolutions from 270p to 2160p, with diverse and representative content. To demonstrate its training effectiveness, compared to three commonly used training databases [4][5][6], BVI-DVC has been utilised to train ten popular CNN architectures [7][8][9][10][11][12][13][14][15][16] using four video coding tools: post-processing (PP), in-loop filtering (ILF), spatial resolution adaptation (SRA) and effective bit depth adaptation (EBDA). The resulting CNN-based coding tools were then integrated into the Test Model (HM 16.20) of the current High Efficiency Video Coding (HEVC) standard [17], and evaluated based on the Joint Video Experts Team (JVET) Common Test Conditions (CTC) [18].
The primary contributions of this paper are: 1) A publicly available video database (BVI-DVC) specifically developed for training deep video coding compression algorithms. 2) An analysis of the performance of BVI-DVC in the context of four compression tools compared with three existing (commonly used) databases. 3) A performance comparison of ten popular CNN architectures based on identical training materials and evaluation configurations. The remainder of the paper is structured as follows. Section II summarises the related work in deep learning-based video compression and commonly used image/video training databases. Section III introduces the proposed (BVI-DVC) video training database, while Section IV presents the experimental configurations employed for evaluating the effectiveness of the proposed database. The results and discussions are provided in Section V, and Section VI concludes the paper and presents some suggestions for future work.
II. BACKGROUND With increasing spatial resolutions, higher frame rates, greater dynamic range and the requirement for multiple viewpoints, accompanied by dramatic increases in user numbers, video content has become the primary driver for increased internet bandwidth. How we represent and communicate video (via compression) is key in ensuring that the content is delivered at an appropriate quality, while maintaining compatibility with the transmission bandwidth.
To address this issue, ISO (MPEG) and ITU initiated the development of a new coding standard, Versatile Video Coding (VVC) [19] in 2018, targeting increased coding gain (by 30-50%) compared to the current HEVC standard. Concurrently, the Alliance for Open Media (AOM -formed in 2015) finalised an open-source, royalty-free media delivery solution (AV1) [20] in 2018, offering performance competitive with HEVC [21].

A. Machine learning based compression
Image and video compression based on deep neural networks has become a popular research topic offering evident enhancements over conventional coding tools for: intra prediction [22,23], motion estimation [24,25], transforms [26,27], quantisation [28], entropy coding [29,30], post-processing [31,32] and loop filtering [33,34]. New coding tools such as format adaptation [35][36][37] and virtual reference frame optimisation [25] have also been reported. Other work has implemented a complete coding framework based on neural networks using end-to-end training and optimisation [38][39][40][41][42]. It is noted that, despite their performance benefits, few of these approaches are currently being adopted by the latest VVC video coding standard. This is due to the high computational complexity and the large GPU memory requirements associated with CNN computation.

B. Training Databases
Training databases are a critical component for optimising the performance of machine learning based algorithms. A well designed training database can ensure good model generalisation and avoid potential over-fitting problems [43,44]. As far as we are aware, there is no publicly available database which is specifically designed for learning-based video coding. Researchers, to date, have typically employed training databases developed for other purposes (such as super-resolution, frame interpolation and classification) for training. Notable publicly available image and video training databases are summarised below.
• ImageNet [45] is a large image database primarily designed for visual object recognition. It contains more than 14 million RGB images at various spatial resolutions (up to 2848p) covering a wide range of natural content. It has also been used as a training database for single image super-resolution [12]. • DIV2K [4] contains 1000 RGB source images with a variety of content types, which was firstly developed for super-resolution. It has currently been employed as training material by several JVET proposals [46,47] and many other CNN-based coding algorithms [48,49]. • BSDS [50] is an image database originally developed for image segmentation. It contains 500 RGB images, and has been used to train CNN-based loop filters [33] for video coding. Comparing to DIV2K, BSDS has fewer source images and lower spatial resolution (481×321). • Vimeo [51] is a video database originally developed for training CNN-based optical flow and temporal superresolution approaches. It contains 89,800 sequences at spatial resolutions up to 448×256. A constraint is imposed on motion vector magnitudes between any two adjacent frames and content with dynamic textures has not been included in this database. Vimeo has not been frequently employed for deep learning based coding approaches and in particular has not been used for those approaches that exhibit superior improvements over standard video codecs (e.g. HEVC and VVC) [3]. • CD (Combined Database) in [6] is a video database combining source content from the LIVE Video Quality Assessment Database [52], MCL-V Database [53] and TUM 1080p Database [54] and has been employed to train CNN-based super-resolution approaches [6]. It contains 29 sequences at two different spatial resolutions, 1920×1080 and 768×432. • REDS [5] is a video database developed for training video super-resolution algorithms [55], which contains 300 video clips with spatial resolution 1280×720. • UCF101 [56] is a large video training database initially designed for human action recognition, and has been frequently used for training CNN-based temporal frame interpolation and motion prediction approaches [57][58][59]. It contains 13320 videos collected from YouTube, which consist of 101 types of human actions. All the sequences in UCF-101 have a relatively low spatial resolution of 320×240. Modern video coding algorithms are required to process content with diverse texture types at high spatial resolutions and bit depths. For example, the standard test sequences included in the JVET Common Test Conditions (CTC) dataset include video clips at UHD resolution (2160p) at a bit depth of 10 bits, with various static and dynamic textures 2 . However none of the training databases mentioned above contain image or video content with high spatial resolution and bit depth, and most do not include any dynamic texture content.  [72], 3 from MCL-V [53], 2 from MCL-JCV [73], 2 from Netflix Chimera [74], 1 from the TUM HD databases [75], and 1 from the Ultra  Video Group-Tampere University database [76]. These sequences contain natural scenes and objects [5], e.g. mountains, oceans, animals, grass, trees, countryside, city streets, towns, buildings, institutes, facilities, parks, marketplaces, historical places, vehicles and colorful textured fabrics. Different texture types such as static texture, dynamic texture 2 , structure content and luminance-plain content are also included.

III. THE BVI-DVC VIDEO TRAINING DATABASE
All these sequences are progressive-scanned at a spatial resolution of 3840×2160, with frame rates ranging from 24 fps to 120 fps, a bit depth of 10 bit, and in YCbCr 4:2:0 format. All are truncated to 64 frames without scene cuts, using the segmentation method described in [77]. To further increase data diversity and provide data augmentation, the 200 video clips were spatially down-sample to 1920×1080, 960×540 and 480×270 using a Lanczos filter of order 3. This results in 800 sequences at four different resolutions. Fig. 1 shows the sample frames of twenty example sequences. The primary features of this database are summarised in Table I  in Fig. 2. The definitions of these feature can be found in [67,78]. It can be noted that the BVI-DVC database has a relatively wide coverage for these three video features, which indicates the diversity of the proposed database.

IV. EXPERIMENTS
In order to evaluate the training effectiveness of the proposed BVI-DVC database in the context of video compression, ten network architectures [7][8][9][10][11][12][13][14][15][16]79] were employed in conjunction with four CNN-based coding modules: post processing (PP), in-loop filtering (ILF), spatial resolution adaptation (SRA) and effective bit depth adaptation (EBDA). These four coding modules were selected since they have been demonstrated to offer significant coding gains over standardised video codecs compared to other tools (e.g. inter prediction, entropy coding, etc.) and also outperform existing end-to-end solutions. Compared to end-to-end image coding architectures, they are also more amenable to integration into standard codecs for practical applications.
In terms of benchmarking databases, based on the limited time and resource available, we have chosen one image database (DIV2K) and two video databases (REDS and CD) to compare with BVI-DVC. The DIV2K is selected because it has been used for training CNN models in multiple JVET contributions and many other CNN-based coding algorithms. Comparing to other video databases such as BSDS, Vimeo and UCF101, REDS and CD contain relatively higher resolution content and have been employed for training successful CNNbased super-resolution approaches.
A. Coding Modules 1) Coding Module 1 (Post Processing -PP): The coding workflow for post processing (PP) is illustrated in Fig. 3. PP is commonly applied at the decoder, on the reconstructed video frames, to reduce compression artefacts and enhance video quality. When a CNN-based approach is employed, the network takes each decoded frame as input and outputs the final reconstructed frame with the same format. Notable examples of employing CNN-based post processing for video coding can be found in [31,32]. 2) Coding Module 2 (In-loop Filtering -ILF): In-loop filtering applies processing at both the encoder and the decoder on the reconstructed frames, and the output can be used as reference for further encoding/decoding. An encoder architecture with a CNN-based ILF module is shown in Fig. 4. The input and the output of the CNN-based ILF are the same as those for PP [33,80].
3) Coding Module 3 (Spatial Resolution Adaptation -SRA): CNN-based spatial resolution adaptation (SRA) down-samples the spatial resolution of the original video frames for encoding, and reconstructs the full resolution during decoding through CNN-based super-resolution. This approach can be applied at Coding Tree Unit level [35] or to the whole frame. Here we only implemented frame-level SRA [81], as shown in Fig.  5. In this case, the original video frames are spatially downsampled by a fixed factor of 2, using the Lanczos3 filter. The CNN-based super-resolution module processes the compressed and down-sampled video frames at the decoder to generate full resolution reconstructed frames. It is noted that a nearest neighbour filter is firstly applied to the reconstructed downsampled video frame before CNN operation [36].

4) Coding Module 4 (Effective Bit Depth Adaptation -EBDA):
Similar to the case for spatial resolution, bit depth can also be adapted during encoding in order to achieve improved coding efficiency. Here Effective Bit Depth (EBD) is defined as the actual bit depth used to represent the video content, which may be different from the Coding Bit Depth (CBD) that represents the pixel bit depth, e.g. InternalBitDepth in HEVC reference encoders. This process is demonstrated in Fig. 6. In this paper, we have fixed CBD at 10 bits, the same as in the Main10 profile of HEVC, and only down-sampled the original frames by 1 bit through bit-shifting. At the decoder, the bit depth of decoded frames is up-sampled to 10 bits using a CNN-based approach. More information about EBDA can be found in [82].

B. Employed CNN Models
For these four coding modules, ten popular network architectures have been implemented for evaluation. These include five with residual blocks, two with residual dense blocks, This network structure has been employed in several CNN-based coding algorithms and has been reported to offer significant coding gains [36].
In this experiment, we have employed identical architectures for these ten networks, as reported in their original publications, and only modify the input and output interfaces in order to process content in the appropriate format. The input of all CNNs employed is a 96×96 YCbCr 4:4:4 colour image, while the output targets the corresponding original image block with the same size. The input block can be either compressed (for PP and ILF), compressed and EBD downsampled (for EBDA) or compressed and spatial resolution re-sampled (for SRA -a nearest neighbour filter is applied before CNN processing). The same loss functions have been used as in the corresponding literature. All these ten networks have been re-implemented using the TensorFlow framework (version 1.8.0).

C. Training Data
Three existing image and video databases are selected to benchmark the training effectiveness of BVI-DVC, including DIV2K [4], REDS [5] and CD [6]. All the original images or videos in each database were first spatially down-sampled by a factor of 2 using a Lanczos3 filter or down-sampled by 1 bit through bit-shifting. The original content (for training PP and ILF CNNs), together with spatially down-sampled clips (for training SRA CNNs) and bit depth reduced sequences (for training EBDA CNNs) were then compressed by the HEVC Test Model (HM 16.20) based on the JVET Common Test Conditions (CTC) [18] using the Random Access configuration (Main10 profile) with four base QP (quantisation parameter) values: 22, 27, 32 and 37 3 (a fixed QP offset of -6 is applied for both spatially and bit depth down-sampled cases as in [85]). This results in three training input content groups for every database, each of which contains four QP sub-groups. For the input content group with reduced spatial resolution, a nearest neighbour filter was applied to obtain video frames with the same size as the original content.
For each input group and QP sub-group, the video frames of all reconstructed sequences and their original counterparts were randomly selected (with the same spatial and temporal sampling rates) and split into 96×96 image blocks, which were then converted to YCbCr 4:4:4 format. Block rotation was also applied here for data augmentation.

D. Network Training and Evaluation
The training process was conducted using the following parameters: Adam optimisation [86] with the following hyperparameters: β 1 =0.9 and β 2 =0.999; batch size of 16; 200 training epochs; learning rate (0.0001); weight decay of 0.1 for every 100 epochs. This generates 480 CNN models for 4 training databases, 3 input content groups (PP and ILF use the same CNN models), 4 QP sub-groups and 10 tested network architectures.
During the evaluation stage, for a specific coding module, the decoded video frames (already up-sampled to the same spatial resolution if needed) are firstly segmented into 96×96 overlapping blocks with an overlap size of 4 pixels as CNN input (YCbCr 4:4:4 conversion). The output blocks are then aggregated following the same pattern and then converted to YCbCr 4:2:0 format to form the final reconstructed frame.

V. RESULTS AND DISCUSSIONS
Four different coding modules have been integrated into the HEVC (HM 16.20) reference software, and have been fully tested under JVET CTC [18] using the Random Access configuration (Main10 profile). Nineteen JVET-CTC SDR (standard dynamic range) video sequences from resolution classes A1, A2, B, C and D were employed as test content, none of which were included in any of the three training databases. It is noted that only class A1 and A2 (2160p) were used to evaluate SRA coding module, as it has been previously reported [36] that for lower resolutions SRA may provide limited and inconsistent coding gains.
The rate quality performance (coding performance) is benchmarked against the original HEVC HM 16.20, using Bjøntegaard Delta measurement (BD-rate) [87] based on two quality metrics, Peak Signal-to-Noise Ratio (PSNR, Yluminance channel only) and Video Multimethod Assessment Fusion (VMAF, version 0.6.1) [88]. PSNR is the most commonly used assessment method for video compression, while VMAF is a machine learning based quality metric commercially used in streaming. VMAF combines multiple quality metrics and video features using a Support Vector Machine (SVM) regressor. It has been reported to offer better correlation with subjective opinions [89]. BD-rate statistics indicate the overall bitrate savings across the tested QP range achieved by the test algorithm for the same video quality compared to the anchor approach. The average BD-rate values are reported in Table II-V for each evaluated training database, network architecture and coding module.

A. Comparison of Databases
It can be observed from Table II-V and Fig. 7 that, for all tested network architectures and coding modules, the coding gains (in terms of average BD-rates for all tested sequences) achieved after training on the proposed BVI-DVC database are significantly greater than for the other three benchmark databases (DIV2K, REDS and CD) for both PSNR and VMAF quality metrics. This is reinforced by considering the mean (among ten networks and four coding modules) of all the average BD-rates for each database; Fig. 8 shows in excess of 4.2% and 5.4% additional bitrate savings obtained by using BVI-DVC compared to the other three databases based on the PSNR and VMAF quality metrics respectively. CD offers the worse overall performance, especially for results based on the assessment of PSNR.

B. Comparison of Networks
We can also compare the ten evaluated network architectures under fair configurations (identical training and evaluation databases). The results in Table II-V are summarised, by taking the mean (among four training databases and four coding modules) of average BD-rate values for each network architecture, in Fig. 9. It can be observed that RCAN, RDN, ESRResNet and MSRResNet offer better coding performance (for both PSNR and VMAF) than the other six evaluated network architectures. This is likely to be because of the residual block structure employed. The coding gains for VDSR, FSRCNN and SRCNN are relatively low comparing to other networks, exhibiting coding loss when PSNR is used to assess video quality especially when they are trained on the CD database (refer to Fig. 7). This may be due to their simple network architecture (FSRCNN and SRCNN) and the large number convolutional layers without residual learning structure (VDSR), which lead to less stable training and evaluation [14,15].

VI. CONCLUSION
This paper presents a new database (BVI-DVC) specifically designed for training CNN-based video compression algorithms. With carefully selected sequences including diverse content, this database offers significantly improved training effectiveness compared to other commonly used image and video training databases. It has been evaluated for four different coding modules with ten typical CNN architectures. The BVI-DVC database reported is available online 1 for public testing. Its content diversity makes it a reliable training database, not just for CNN-based compression approaches, but also for other computer vision and image processing related tasks, such as image/video de-noising, video frame interpolation and super-resolution. Future work should focus on developing large training databases with more immersive video formats, including higher dynamic range, higher frame rates and higher spatial resolutions.