Scalable Video Coding for Humans and Machines

Video content is watched not only by humans, but increasingly also by machines. For example, machine learning models analyze surveillance video for security and traffic monitoring, search through YouTube videos for inappropriate content, and so on. In this paper, we propose a scalable video coding framework that supports machine vision (specifically, object detection) through its base layer bitstream and human vision via its enhancement layer bitstream. The proposed framework includes components from both conventional and Deep Neural Network (DNN)-based video coding. The results show that on object detection, the proposed framework achieves 13-19% bit savings compared to state-of-the-art video codecs, while remaining competitive in terms of MS-SSIM on the human vision task.


I. INTRODUCTION
Video analytics is an essential technology for various applications such as traffic monitoring, visual surveillance, and autonomous navigation. Automated machine vision pipelines increasingly analyze video streams uploaded to the cloud. If one is only interested in (machine-based) visual analysis, precomputed features can be compressed and transmitted [1], instead of the full video. There are standard codecs to compress computed features, either handcrafted or neural networkbased [2], [3], although these were developed prior to the current wave of interest on the topic. A problem with these approaches, however, is that when human viewing is needed, input video must also be coded and transmitted, and the overall system becomes less efficient.
A recent trend in image/video coding is to utilize deep neural networks (DNNs) to replace either specific units within conventional codecs, or even the whole codec. Over the past few years, DNNs have made inroads in this area, often demonstrating promising coding results compared to fullyengineered conventional approaches [4]- [8]. However, most DNN-based codecs have focused on compression for human vision, just like traditional codecs. At the same time, there are many DNN-based vision analysis methods [9] but they are usually developed without regard for compression. To establish a consolidated framework that supports both human and machine vision, a standardization activity, MPEG-VCM (Video Coding for Machines) [10], has recently been initiated. Several recent proposals for image coding [11]- [13] examined scalable compression for multiple tasks. In these methods, the base layer features are used to perform machine vision. With This work was partially supported by the Natural Sciences and Engineering Council (NSERC) of Canada. additional information in the enhancement layer, these methods also support high-quality input reconstruction for human vision. Meanwhile, [14], [15] tried to recover the input image directly from features, without an additional bitstream: [14] from intermediate-layer activations of YOLOv2 [16], and [15] from cuboidal features targeted at YOLOv2. If X denotes the input image, Y the latent space features, X the reconstructed image, and T the machine task output, then a typical machine vision pipeline can be described by a Markov chain X → Y → X → T . Applying the data processing inequality [17] to this chain, we obtain I(Y; X) ≥ I(Y; T ), where I(· ; ·) is the mutual information, suggesting that less information (fewer bits) is needed for machine vision than for input reconstruction. 1 This agrees with the scalable approaches, where both base and enhancement layers are used for input reconstruction, but only the base layer (i.e., fewer bits) for machine vision.
In this paper, we develop the first (to our knowledge) video codec for human and machine vision based on the concept of latent-space scalability [18]. The codec utilizes multi-task DNN-based compression for intra coding and a combination of DNN and conventional techniques for inter-frame coding. In Section II, we present the proposed video coding system, along with the explanation of its building blocks for intra and inter coding. Experimental results are presented in Section III, followed by conclusions in Section IV.

A. Group-of-Pictures structure
The Group-of-Pictures (GoP) structure of our proposed video coding system is shown in Fig. 1. The GoP consists of an intra-coded frame (shown as pink in Fig. 1) and a number of subsequent inter-coded frames (shown as green in Fig. 1). Intra frames are coded in a multi-task scalable manner using the concept of latent-space scalability, as detailed below in Section II-B. The base layer of an intra-frame bitstream supports machine vision, in our case object detection by YOLOv3 [19]. When only machine vision is needed, only the base layer bitstream needs to be decoded. The enhancement layer provides additional information for high-quality input reconstruction for human viewing. Inter frames are exclusively used for human viewing. Hence, inter-frame bitstrams are considered a part of the enhancement layer.
The intended application of such a system is as follows. Under normal operation, only the machine vision task is active. For example, a cloud-based automated analysis pipeline is monitoring an airport lounge through a surveillance camera. The camera sends the base-layer bitstream to the cloud, where the machine vision pipeline performs object detection. When a situation of interest is detected -for example, unattended luggage 2 -enhancement layer is requested by the cloud. This means that both the base and the enhancement layer bitstream of the next intra frame are sent to the cloud, together with subsequent inter frames. Security personnel can then view the scene and decide when to switch back to machine-only mode.
This example was given as a specific illustration of the intended use of the proposed system. The use is not limited to public area surveillance -similar situations arise in traffic monitoring, smart home security, and so on.

B. Multi-task scalable intra coding
The structure of the multi-task scalable intra frame encoder follows that of [13], and is illustrated in Fig. 2. Intra frame X is encoded into a latent representation Y as: where g a (·) is a trainable analysis transform [4]- [6] and Q(·) is the quantizer, implemented as rounding to the nearest integer. The architecture of the encoder can be taken from any endto-end trainable image codec; we have used the architecture from [6]. The main difference is that we retrain the encoder to produce a structured latent space Y = { Y 1 , Y 2 }, such that Y 1 contains the information needed for object detection (base layer), and Y 2 contains enhancement information which, together with Y 1 , enables reconstruction of the input frame.
Out of the 192 channels in Y, 128 are assigned to Y 1 and the remaining 64 to Y 2 . From such latent representation, two decoders -one for machine vision, the other for human viewing -can efficiently reconstruct the required information. The object detection decoder decodes only Y 1 : where LST stands for the Latent Space Transform [18] that maps the encoder's latent space into a latent space of the object detection network, and O(·) is the back-end of the object detection network producing detection output T . In our case, we have chosen layer 13 of YOLOv3 as the target latent space, so O(·) consists of all YOLOv3 layers after layer 13. Several possible architectures for the LST were presented in [13], [18], here we have used the one from [13]. Meanwhile, the decoder for human viewing decodes the where D is the decoder and and X is the approximation to the original input frame X. The architecture of the decoder D can be taken from the same end-to-end codec where the encoder came from; in our case, this is [6]. Structuring of the latent space is achieved by training all components of the encoder in Fig. 2, except O, the object detection back-end, jointly using the loss function where R is a rate estimate (obtained using the entropy model from [6]), λ controls the trade-off between rate and distortion, and γ controls the trade-off between latent-space distortion for object detection, MSE(F , F ), and pixel-domain distortion for input reconstruction, MSE(X, X). Since F is computed only from Y 1 , as shown in (2), by gradient-based training, information related to object detection will be steered only into Y 1 . Meanwhile, since X is obtained from the entire latent space Y, as shown in (3), information related to input reconstruction will be spread throughout the entire latent space, including Y 2 .

C. Inter-frame coding
Inter-frame coding is based on High Efficiency Video Coding (HEVC) [20], specifically HEVC test model HM-16.20. 3 Besides conventional HEVC coding tools, we also integrate DNN-based affine frame prediction from [21] into the coding pipeline. The operation of this network is illustrated in Fig. 3. The network takes two previously coded frames X t1 and X t2 as input, and predicts a half-way frame X t between the input frames, where t 1 < t < t 2 . Internally, the network estimates motion between the two input frames and the desired output frame, as visualized in the figure, along with adaptive filter kernels. With computed motion and filter kernels, the network produces an estimated frame such that the residual signal to code is minimized. The reader is referred to [21] for details about the prediction network.
The structure of the overall video decoder is shown in Fig. 4. To identify intra frames, we still use the network abstraction layer (NAL) header with 2 bytes as in conventional HEVC. Within the header data, reserved 6-bit word to support HEVC scalability is also re-used to distinguish layer IDs for our task scalability. When only the base layer of an intra frame is received, Y 1 is reconstructed and fed to the object detection pipeline (LST + YOLOv3 back-end) to detect objects. When the enhancement layer of an intra frame is also received, Y 2 can be reconstructed. Then, input frame is reconstructed from the full latent representation Y = { Y 1 , Y 2 }. The reconstructed frame is registered in decoded picture buffer (DPB) so that it can be used as a reference for inter frames.
For inter-frame coding, the coded bitstream is parsed through the entropy decoder followed by inverse quantization and transformation. Reconstructed residual signal is added to the predictor to reconstruct the input frame. For block-level prediction in the inter frame, there are conventional HEVC intra and inter prediction tools. The DNN-predicted frame can be used in the HEVC coding pipeline in several ways. One is the "direct" mode, in which the frame is used as an additional prediction mode, indicated by a flag bit. In this case, no additional motion information is needed. As a result, this mode ends up being selected up to 57% of the time in the HEVC rate-distortion (RD) optimization, according to [21]. However, this mode is constrained to use square blocks. For this reason, we supplement the pipeline by another approach indicated by the dashed red line in Fig. 4. Here, the frame with the largest picture order count (POC) difference from the current frame in the DPB is replaced by the DNN-generated frame X t . As such, the DNN-generated frame becomes a reference frame in the DPB. All inter-coding modes can be used in this case, but additional motion information may be required, as is the case with conventional inter-prediction. For block-level coding in inter frames, all coding modes, including the modes utilizing the DNN-generated frame X t , compete in the HEVC RD optimization process.

A. Implementation and training
All DNNs are implemented in Pytorch. Python embedding library 4 is used to embed the DNNs into HM-16.20 implemented in C++. While compressing input video, forward operations of the DNNs to perform the multi-task compression and the frame prediction are executed across CPU and GPU.
The intra-frame codec (Fig. 2) was trained on the CLIC, 5 JPEG AI [22], and VIMEO-90K [23] datasets, on randomly cropped patches of size 256 × 256, following the procedure in [13]. Adam optimizer with a learning rate of 10 −4 was used. The codec was trained with γ = 0.006 in (4), for six values of λ shown in Table I. The DNN-based frame prediction network (Fig. 3) was trained on randomly selected triplets of 152 × 152 patches cropped from the videos from Xiph 6 and VIMEO-90K [23], following the procedure in [21]. AdaMax optimizer with a learning rate of 10 −3 was used.
To use the pre-trained YOLOv3 [19] back-end in the evaluation of object-detection performance, the input resolution was resized to 512×512 using bilinear interpolation without letterboxing. Compression benchmarks are the two latest video  [25]. Bjøntegaard Delta (BD) metrics [26], [27] are used to evaluate the performance against the benchmarks in terms of rate-distortion and rate-accuracy.

B. Simultaneous evaluation for human and machine vision
First, we evaluate the performance of our system against the benchmarks simultaneously on human and machine vision. We do this on the SFU-HW-Objects-v1 dataset [28], which contains COCO 7 -style object labels for a set of HEVC raw video test sequences. This dataset is also being used in MPEG-VCM [29]. Table III summarizes the performance of our coding system versus the benchmarks, with best results indicated in bold. Since this experiment involves object detection, for which our system uses only the base layer of the intra-coded frames, the test is carried out in the all-intra configuration. Benchmark codecs code intra frames, and decoded frames are fed to YOLOv3. In our system, only the base layer of intra frames is decoded and fed via LST to the YOLOv3 back-end, as shown in Fig. 2.
Mean Average Precision (mAP) [19] is used as the object detection accuracy metric. Unlike the Peak Signal-to-Noise-Ratio (PSNR), mAP vs. bit rate curves are not always concave, or even monotonic [29], which makes it impossible to compute a valid BD-rate-mAP value. One example is shown in Fig. 5(a) for the sequence FourPeople, where we see that HEVC and VVC curves are non-concave and non-monotonic. For this 7 https://cocodataset.org reason, the sequence FourPeople has been excluded from the results. Other sequences had well-behaved mAP vs. bit rate curves, like the one shown in Fig. 5(b) for BasketballPass. On object detection, our coding system shows significant bit savings of 13.45% and 18.85% on average against HEVC and VVC, respectively, when averaged over all sequence classes. Without Class E sequences, average bit reduction is even higher -20.40% and 26.65%, respectively, against HEVC and VVC. Surprisingly, we save more bits against VVC compared to HEVC, which implies that advanced coding tools adopted in VVC are less machine vision-friendly.
In terms of input reconstruction for human viewing, standard codecs perform better, as expected, because that is what they are optimized for. In terms of BD-rate-PSNR, our system increases bits by about 9% and 41% against HEVC and VVC, respectively. In other words, the compression efficiency of VVC is far superior to other methods in terms of rate-PSNR. Meanwhile, our system performs reasonably well against HEVC, achieving some gains on two sequences in Class D. Overall, our method shows better performance in classes C and D compared to other classes. We suspect that this is due to input scaling with bilinear interpolation, which may cause some artifacts to the sequences in Class C (832 × 480) and D (416 × 240) compared to sequences with higher resolution.
In terms of BD-rate-MS-SSIM, our system outperforms HEVC by 18.71% and has marginally worse performance (by 1.95%) compared to VVC. If we consider MS-SSIM a more relevant metric for human viewing experience, then one could argue that our system provides comparable or better performance for human viewing while achieving gains on machine vision. Indeed, it has been known for a while that DNN-based codecs do well on MS-SSIM, and our system  benefits from DNN-based intra coding in this experiment.
C. Input reconstruction with random access coding Table IV summarizes input reconstruction performance in terms of BD-rate metrics for the random access configuration with the intra period of 8. Here, benchmark codecs perform better than they did on the object detection task, because they were optimized for this kind of use. In terms of BD-rate-PSNR, our system increases the rate by about 8.9% on average against HEVC. Recall that our inter-coding pipeline is built upon HEVC. Considering the fact that conventional scalable extensions of HEVC increase the bit rate by 15%-25% per layer [30], our scalable system for human and machine vision performs well within this margin. The performance against VVC in terms of BD-rate-PSNR is correspondingly lower, as expected, with about 44% rate increase. Our codec performs much better in terms of MS-SSIM. In fact, in this case, it provides BD-rate savings of 5%, on average, against HEVC, and the loss against VVC is now reduced to about 25%.

D. Ablation study
Here we examine the effect of DNN-aided frame prediction within our system, by comparing the full version of the system against a stripped-down version, which does not include DNNbased frame prediction. The results are shown in Table V for the random access configuration with the intra period of 8. DNN-aided frame prediction brings 0.8%-0.9% bit savings on average, both in terms of PSNR and MS-SSIM.

E. Break-even points
In earlier sections we saw that, compared with conventional HEVC or VVC coding, our system achieves compression gains when only the machine vision task is needed, but suffers coding loss in certain cases when input reconstruction is needed for human viewing. Thus, in practice, the question of whether or not our system will provide bit savings depends on how frequently input reconstruction is needed compared to machine vision. In this section, we quantify this tradeoff in terms of the maximum fraction of time that input reconstruction is needed, on average, while still allowing compression gains for our system. We call this fraction of time the break-even point. According to Table III, our system is 13.45% more efficient (i.e., uses 0.8655 the amount of bits), on average, compared to HEVC when only object detection is required. At the same time, it is 9.05% less efficient (i.e., uses 1.0905 the amount of bits), on average, when input reconstruction is needed, if reconstruction quality is measured by PSNR. Let t h ∈ [0, 1] be the fraction of time that input reconstruction is needed. The amount of bits used by our system will be less than or equal to that used by HEVC if (1 − t h ) · 0.8655 + t h · 1.0905 ≤ 1.
Solving for t h that achieves equality in (5), we obtain the break-even point of t h = 0.5978. That is to say, if input reconstruction is needed less than 59.78% of the time, our system will provide overall bit savings over HEVC. In fact, if input reconstruction quality is measured by MS-SSIM instead of PSNR, our system would always provide gains over HEVC, since the corresponding BD-rate-MS-SSIM is negative in Table III. Repeating the same calculation for other cases, we obtain the break-even points shown in Table VI. The smallest break-even point across all cases is 0.3138. Hence, if input reconstruction is needed less than 30% of the time, our system will provide savings even against VVC.

IV. CONCLUSION
We developed a new scalable video coding system supporting machine vision (object detection) in the base layer and input reconstruction for human viewing in the enhancement layer. The system was benchmarked against the two most recent video coding standards -HEVC and VVC. The results show that the proposed system provides savings of 13-19% on the object detection task. At the same time, it provides comparable or better performance in terms of MS-SSIM in the all-intra coding configuration, as well as the random access configuration against HEVC. Against VVC in the random access configuration, there is a loss of about 44% in terms of BD-rate-PSNR and about 25% in terms of BD-rate-MS-SSIM. But even against VVC, our system provides rate savings, so long as human viewing is needed less than 30% of the time.