The growing popularity of virtual and augmented reality communications and 360 video streaming is moving video communication systems into much more dynamic and resource-limited operating settings. The enormous data volume of 360 videos requires an efficient use of network bandwidth to maintain the desired quality of experience for the end user. To this end, we propose a framework for viewport-driven rate-distortion optimized 360 video streaming that integrates the user view navigation pattern and the spatiotemporal rate-distortion characteristics of the 360 video content to maximize the delivered user quality of experience for the given network system resources. The framework comprises a methodology for constructing dynamic heat maps that capture the likelihood of navigating different spatial segments of a 360 video over time by the user, an analysis and characterization of its spatiotemporal rate-distortion characteristics that leverage preprocessed spatial tilling of the 360 view sphere, and an optimization problem formulation that characterizes the delivered user quality of experience given the user navigation patterns, 360 video encoding decisions, and the available system network resources. Our experimental results demonstrate the advantages of our framework over the conventional approach of streaming a monolithic uniformly encoded 360 video and a state-of-the-art reference method. Considerable video quality gains of 4 - 5 dB are demonstrated in the case of two popular 4K 360 videos.
Due to the emerging nature of @math technologies, only a small body of related work has appeared to date. The study in @cite_18 carried out an empirical characterization of @math videos highlighting their main features, e.g., their lower temporal rate variability compared to conventional videos. A small number of studies have considered splitting the @math video into spatial tiles as part of the encoding process, leveraging the tilling feature of the latest High Efficiency Video Coding (HEVC) standard @cite_16 . The encoding data rate of each tile can then be controlled independently to reduce the overall bandwidth usage @cite_6 @cite_5 @cite_8 @cite_3 . Transmission aspects of @math HTTP streaming have been explored in @cite_0 @cite_9 . Applications of scalable video coding to @math streaming have been studied in @cite_14 @cite_21 . Head movement prediction and the impact of navigation uncertainty have been investigated in @cite_8 @cite_20 . The former study also carries out an empirical analysis of the performance efficiency of the four sphere-to-planar shape projections and investigates the benefits of streaming adaptively multiple @math representations featuring different quality-emphasized spatial regions @cite_20 .
@cite_18 @cite_14 @cite_8 @cite_9 @cite_21 @cite_6 @cite_3 @cite_0 @cite_5 @cite_16 @cite_20
2744660073 2523094016 2769327901 2765239030 2963603324 2524582890 2623784065 2622599057 1993721485
Online streaming of Virtual Reality and 360° videos is rapidly growing, as more and more major content providers and news outlets adopt the format to enrich the user experience. We characterize 360° videos by examining several thousand YouTube videos across more than a dozen categories. 360° videos, at first sight, seem to pose a challenge for the network to stream because of their substantially higher bit rates and larger number of resolutions. However, a careful examination of video characteristics reveals that there are significant opportunities for reducing the actual bit rate delivered to client devices based on the user's field of view. We study the bit rate and the motion in 360° videos, and compare them against regular videos by investigating several important metrics. We find that 360° videos are less variable in terms of bit rate, and have less motion than regular videos. Our expectation is that variability in the bit rates due to the motion of the camera in regular videos (or switching between cameras) is now translated to responsiveness requirements for end to end 360° streaming architectures. As an important component of the virtual reality (VR) technology, 360-degree videos provide users with panoramic view and allow them to freely control their viewing direction during video playback. Usually, a player displays only the visible portion of a 360 video. Thus, fetching the entire raw video frame wastes bandwidth. In this paper, we consider the problem of optimizing 360 video delivery over cellular networks. We first conduct a measurement study on commercial 360 video platforms. We then propose a cellular-friendly streaming scheme that delivers only 360 videos' visible portion based on head movement prediction. Using viewing data collected from real users, we demonstrate the feasibility of our approach, which can reduce bandwidth consumption by up to 80 based on a trace-driven simulation. HTTP Adaptive Streaming is a successful and largely adopted content delivery technology. Yet poor bandwidth prediction, notably in mobile networks, may cause bit-rate oscillations, increased segment delivery delays, video freezes, and may thus negatively impact the end user quality of experience. To address this issue, we propose to exploit the stream prioritization and termination features of the HTTP 2 protocol to achieve video frame filtering and scheduling, so as to maximize the amount of video data received on time by the client. We evaluate with optimal scheduling and filtering algorithms the maximum gain we may expect from such delivery schemes where video frames are carried in dedicated HTTP 2 streams. Evaluation shows that our HTTP 2-based video frame scheduling scheme brings benefits for video quality. The transmission of virtual reality (VR) videos requires huge bandwidth, which brings great challenges for the system to implement real-time applications. This letter proposes a scalable full-panorama video coding method to adapt the insufficient bandwidth, in which user’s movement information is utilized as a feedback from the VR device to the video encoder. The regions that the user is interested in are first mapped and then coded in high quality, while the others in low quality. This different-quality coding is achieved through scalable high efficiency video coding (SHVC), which only makes limited modifications on the SHVC encoder and no modification on the SHVC decoder. In addition, full-panorama coding also avoids the reference problem in inter prediction and relieves the delay-sensitive problem. Experiments results show that our proposed method reduces approximately 87 bit rate with no significant decrease in quality of the region of interest in panorama. 360° video is attracting an increasing amount of attention in the context of Virtual Reality (VR). Owing to its very high-resolution requirements, existing professional streaming services for 360° video suffer from severe drawbacks. This paper introduces a novel end-to-end streaming system from encoding to displaying, to transmit 8K resolution 360° video and to provide an enhanced VR experience using Head Mounted Displays (HMDs). The main contributions of the proposed system are about tiling, integration of the MPEG-Dynamic Adaptive Streaming over HTTP (DASH) standard, and viewport-aware bitrate level selection. Tiling and adaptive streaming enable the proposed system to deliver very high-resolution 360° video at good visual quality. Further, the proposed viewport-aware bitrate assignment selects an optimum DASH representation for each tile in a viewport-aware manner. The quality performance of the proposed system is verified in simulations with varying network bandwidth using realistic view trajectories recorded from user experiments. Our results show that the proposed streaming system compares favorably compared to existing methods in terms of PSNR and SSIM inside the viewport. While traditional multimedia applications such as games and videos are still popular, there has been a significant interest in the recent years towards new 3D media such as 3D immersion and Virtual Reality (VR) applications, especially 360 VR videos. 360 VR video is an immersive spherical video where the user can look around during playback. Unfortunately, 360 VR videos are extremely bandwidth intensive, and therefore are difficult to stream at acceptable quality levels. In this paper, we propose an adaptive bandwidth-efficient 360 VR video streaming system using a divide and conquer approach. In our approach, we propose a dynamic view-aware adaptation technique to tackle the huge streaming bandwidth demands of 360 VR videos. We spatially divide the videos into multiple tiles while encoding and packaging, use MPEG-DASH SRD to describe the spatial relationship of tiles in the 360-degree space, and prioritize the tiles in the Field of View (FoV). In order to describe such tiled representations, we extend MPEG-DASH SRD to the 3D space of 360 VR videos. We spatially partition the underlying 3D mesh, and construct an efficient 3D geometry mesh called hexaface sphere to optimally represent a tiled 360 VR video in the 3D space. Our initial evaluation results report up to 72 bandwidth savings on 360 VR video streaming with minor negative quality impacts compared to the baseline scenario when no adaptations is applied. Real-time entertainment services such as streaming audiovisual content deployed over the open, unmanaged Internet account now for more than 70 during peak periods. More and more such bandwidth hungry applications and services are proposed like immersive media services such as virtual reality and, specifically omnidirectional 360-degree videos. The adaptive streaming of omnidirectional video over HTTP imposes an important challenge on today's video delivery infrastructures which calls for dedicated, thoroughly designed techniques for content generation, delivery, and consumption.; AB@This paper describes the usage of tiles --- as specified within modern video codecs such HEVC H.265 and VP9 --- enabling bandwidth efficient adaptive streaming of omnidirectional video over HTTP and we define various streaming strategies. Therefore, the parameters and characteristics of a dataset for omnidirectional video are proposed and exemplary instantiated to evaluate various aspects of such an ecosystem, namely bitrate overhead, bandwidth requirements, and quality aspects in terms of viewport PSNR. The results indicate bitrate savings from 40 (in a realistic scenario with recorded head movements from real users) up to 65 (in an ideal scenario with a centered fixed viewport) and serve as a baseline and guidelines for advanced techniques including the outline of a research roadmap for the near future. The demand for 360° Virtual Reality (VR) videos is expected to grow in the near future, thanks to the diffusion of VR headsets. VR Streaming is however challenged by the high bandwidth requirements of 360° videos. To save bandwidth, we spatially tile the video using the H.265 standard and stream only tiles in view at the highest quality. The video is also temporally segmented, so that each temporal segment is composed of several spatial tiles. In order to minimize quality transitions when the user moves, an algorithm is developed to predict where the user is likely going to watch in the near future. Consequently, predicted tiles are also streamed at the highest quality. Finally, the server push in HTTP 2 is used to deliver the tiled video. Only one request is sent from the client; all the tiles of a segment are automatically pushed from the server. This approach results in a better bandwidth utilization and video quality compared to traditional streaming over HTTP 1.1, where each tile has to be requested independently by the client. We showcase the benefits of our framework using a prototype developed on a Samsung Galaxy S7 and a Gear VR, which supports both tiled and non-tiled videos and streaming over HTTP 1.1 and HTTP 2. Under limited bandwidth conditions, we demonstrate how our framework can improve the quality watched by the user compared to a non-tiled solution where all of the video is streamed at the same quality. This result represents a major improvement for the efficient streaming of VR videos. Omnidirectional videos of real world environments viewed on head-mounted displays with real-time head motion tracking can offer immersive visual experiences. For live streaming applications, compression is critical to reduce the bitrate. Omnidirectional videos, which are spherical in nature, are mapped onto one or more planes before encoding to interface with modern video coding standards. In this paper, we consider the problem of evaluating the coding efficiency in the context of viewing with a head-mounted display. We extract viewport based head motion trajectories, and compare the original and coded videos on the viewport. With this approach, we compare different sphere-to-plane mappings. We show that the average viewport quality can be approximated by a weighted spherical PSNR.