Compressed videos constitute 70 of Internet traffic, and video upload growth rates far outpace compute and storage improvement trends. Past work in leveraging perceptual cues like saliency, i.e., regions where viewers focus their perceptual attention, reduces compressed video size while maintaining perceptual quality, but requires significant changes to video codecs and ignores the data management of this perceptual information. In this paper, we propose Vignette, a compression technique and storage manager for perception-based video compression. Vignette complements off-the-shelf compression software and hardware codec implementations. Vignette's compression technique uses a neural network to predict saliency information used during transcoding, and its storage manager integrates perceptual information into the video storage system to support a perceptual compression feedback loop. Vignette's saliency-based optimizations reduce storage by up to 95 with minimal quality loss, and Vignette videos lead to power savings of 50 on mobile phones during video playback. Our results demonstrate the benefit of embedding information about the human visual system into the architecture of video storage systems.
More recently, multimedia and networking research optimized streaming bandwidth requirements for and VR video by decreasing quality outside the VR field-of-view @cite_6 @cite_45 @cite_34 @cite_57 ; while similar in spirit to perceptual compression, this only compresses to non-visible regions of a video. Sitzmann al @cite_43 observe the impact of leveraging saliency for VR video compression and identified key perceptual requirements, but do not address the production or distribution of saliency-compressed videos.
@cite_6 @cite_57 @cite_43 @cite_45 @cite_34
2074883210 2765239030 2111229404 2396399340
It is generally recognized that severe video distortions that are transient in space and or time have a large effect on overall perceived video quality. In order to understand this phenomena, we study the distribution of spatio-temporally local quality scores obtained from several video quality assessment (VQA) algorithms on videos suffering from compression and lossy transmission over communication channels. We propose a content adaptive spatial and temporal pooling strategy based on the observed distribution. Our method adaptively emphasizes “worst” scores along both the spatial and temporal dimensions of a video sequence and also considers the perceptual effect of large-area cohesive motion flow such as egomotion. We demonstrate the efficacy of the method by testing it using three different VQA algorithms on the LIVE Video Quality database and the EPFL-PoliMI video quality database. The transmission of virtual reality (VR) videos requires huge bandwidth, which brings great challenges for the system to implement real-time applications. This letter proposes a scalable full-panorama video coding method to adapt the insufficient bandwidth, in which user’s movement information is utilized as a feedback from the VR device to the video encoder. The regions that the user is interested in are first mapped and then coded in high quality, while the others in low quality. This different-quality coding is achieved through scalable high efficiency video coding (SHVC), which only makes limited modifications on the SHVC encoder and no modification on the SHVC decoder. In addition, full-panorama coding also avoids the reference problem in inter prediction and relieves the delay-sensitive problem. Experiments results show that our proposed method reduces approximately 87 bit rate with no significant decrease in quality of the region of interest in panorama. We present a novel client-driven multiview video streaming system that allows a user to watch 3D video interactively with significantly reduced bandwidth requirements by transmitting a small number of views selected according to his her head position. The user's head position is tracked and predicted into the future to select the views that best match the user's current viewing angle dynamically. Prediction of future head positions is needed so that views matching the predicted head positions can be prefetched in order to account for delays due to network transport and stream switching. The system allocates more bandwidth to the selected views in order to render the current viewing angle. Highly compressed, lower quality versions of some other views are also prefetched for concealment if the current user viewpoint differs from the predicted viewpoint. An objective measure based on the abruptness of the head movements and delays in the system is introduced to determine the number of additional lower quality views to be prefetched. The proposed system makes use of multiview coding (MVC) and scalable video coding (SVC) concepts together to obtain improved compression efficiency while providing flexibility in bandwidth allocation to the selected views. Rate-distortion performance of the proposed system is demonstrated under different experimental conditions. Humans see only a tiny region at the center of their visual field with the highest visual acuity, a behavior known as foveation. Visual acuity reduces drastically towards the visual periphery. 'Foveated' video coding compression techniques exploit this non-uniformity to gain significant efficiency by compressing more in the periphery and less in the center. We propose a practical and scalable method to use such a technique for video streaming service over the Internet. The essential idea is to use a commodity webcam on the user side to provide real-time gaze feedback to the server with the server sending appropriately coded video to the client player. We develop a multi-resolution video coding approach that is scalable in that it is possible to pre-code the video in a small number of copies for a given set of resolutions. The coding approach is designed to match the error performance of an eye tracker built using commodity webcams. We demonstrate that the technique is energy efficient and thus usable in mobile devices. We develop a methodology for performance evaluation of such a system when network budgets may vary and video quality may fluctuate. Finally, we present a comprehensive user study that demonstrates a bandwidth reduction of a factor of 2 for the same user satisfaction.