The main task of HTTP Adaptive Streaming is to adapt video quality dynamically under variable network conditions. This is a key feature for multimedia delivery especially when quality of service cannot be granted network-wide and, e.g., throughput may suffer short term fluctuations. Hence, robust bitrate adaptation schemes become crucial in order to improve video quality. The objective, in this context, is to control the filling level of the playback buffer and maximize the quality of the video, while avoiding unnecessary video quality variations. In this paper we study bitrate adaptation algorithms based on Backward-Shifted Coding (BSC), a scalable video coding scheme able to greatly improve video quality. We design bitrate adaptation algorithms that balance video rate smoothness and high network capacity utilization, leveraging both on throughput-based and buffer-based adaptation mechanisms. Extensive simulations using synthetic and real-world video traffic traces show that the proposed scheme performs remarkably well even under challenging network conditions.
Backward-Shifted Coding (BSC) @cite_12 and @cite_20 are two schemes which leverage on time redundancy mechanisms for scalable video coding. In both schemes, base layer or enhancement layers of any number of future segments of a video can be delivered in advance.
@cite_20 @cite_12
2357930737 2118078737 2111085154 2014057681 2147990225 2409027520 1995848834 2606901901 2121496396 2168573851 89205691 2612222456 2142536989 2165737454 2131723117 2161112041 2083767059 1485536093 2895340898 2093784538
Although HTTP-based video streaming can easily penetrate firewalls and profit from Web caches, the underlying TCP may introduce large delays in case of a sudden capacity loss. To avoid an interruption of the video stream in such cases we propose the Backward-Shifted Coding (BSC). Based on Scalable Video Coding (SVC), BSC adds a time-shifted layer of redundancy to the video stream such that future frames are downloaded at any instant. This pre-fetched content maintains a fluent video stream even under highly variant network conditions and leads to high Quality of Experience (QoE). We characterize this QoE gain by analyzing initial buffering time, re-buffering time and content resolution using the Ballot theorem. The probability generating functions of the playback interruption and of the initial buffering latency are provided in closed form. We further compute the quasi-stationary distribution of the video quality, in order to compute the average quality, as well as temporal variability in video quality. Employing these analytic results to optimize QoE shows interesting trade-offs and video streaming at outstanding fluency. The scalable video coding extension of H.264 AVC is a current standardization project. This paper deals with unequal error protection (UEP) scheme for scalable video bitstream over packet-lossy networks using forward error correction (FEC). The proposed UEP scheme is developed by exploiting jointly the unequal importance existing both in temporal layers and quality layers of hierarchial scalable video bitstream. For efficient assignment of FEC codes, the proposed UEP scheme uses a simple and efficient performance metric, namely layer-weighted expected zone of error propagation (LW-EZEP). The LW-EZEP is adopted for quantifying the error propagation effect on video quality degradation from packet loss in temporal layers and in quality layers. Compared to other UEP schemes, the proposed UEP scheme demonstrates strong robustness and adaptation for variable channel status. In existing pyramid-based spatially scalable coding schemes, such as H.264 MPEG-4 SVC (scalable video coding), video frame at a certain high-resolution layer is mainly predicted either from the same frame at the next lower resolution layer, or from the temporal neighboring frames within the same resolution layer. But these schemes fail to exploit both kinds of correlation simultaneously and therefore cannot remove the redundancies among resolution layers efficiently. This paper extends the idea of spatiotemporal subband transform and proposes a general in-scale motion compensation technique for pyramid-based spatially scalable video coding. Video frame at each high-resolution layer is partitioned into two parts in frequency. Prediction for the lowpass part is derived from the next lower resolution layer, whereas prediction for the highpass part is obtained from neighboring frames within the same resolution layer, to further utilize temporal correlation. In this way, both kinds of correlation are exploited simultaneously and the cross-resolution-layer redundancy can be highly removed. Furthermore, this paper also proposes a macroblock-based adaptive in-scale technique for hybrid spatial and SNR scalability. Experimental results show that the proposed techniques can significantly improve the spatial scalability performance of H.264 MPEG-4 SVC, especially when the bit-rate ratio of lower resolution bit stream to higher resolution bit stream is considerable. Layered video coding is capable of progressively refining the reconstructed video quality with the aid of multiple layers of unequal importance. When the base layer (BL) is corrupted or lost due to channel impairments, the enhancement layers (ELs) must be discarded by the video decoder, regardless whether they are perfectly decoded or not, which implies that the transmission power assigned to the ELs is wasted. To circumvent this problem, we proposed a bit-level inter-layer forward error correction (IL-FEC) scheme for layered video transmission in our previous work, which implanted the systematic bits of the BL into the systematic bits of the ELs using exclusive-OR operations (XOR). This allowed the receiver to exploit the implanted bits of the ELs for assisting the BL's decoding and hence improved the overall system performance of our IL-FEC aided layered video scheme. In this treatise, we find the specific FEC coding rates in a real-time on-line fashion for the sake optimizing the overall system performance. The proposed procedure is widely applicable to diverse wireless transceivers and FEC codecs. Our simulation results show that the proposed optimized IL-FEC system outperforms the traditional optimal UEP by about 1.9 dB of Eb N0 at a peak signal-to-noise ratio (PSNR) of 38 dB. Viewing the improvements in terms of the video quality, 3.3 dB of PSNR improvement is attained at an Eb N0 of 10 dB, when employing a recursive systematic convolutional (RSC) code. Scalable video coding (SVC), together with adaptive modulation and coding (AMC), can improve wireless multicast streaming video by jointly performing radio resource allocation and modulation and coding scheme (MCS) selection. However, the existing schemes in the literature allocate radio resources for different video layers separately, which leads to a waste of radio resources. In this work, we introduce the notion of joint layer resource allocation which allows to jointly allocate resources to multiple video layers that are assigned the same MCS. We formulate this problem and prove it to be NP-hard. Then we develop a pseudo-polynomial algorithm that finds the optimal total system utility. Our algorithm assumes a very generic utility function and flexible video layer rates. To reduce the complexity of the algorithm, we also propose Fully Polynomial Time Approximation Schemes (FPTAS) for the same problem. Simulation results show that our optimal algorithm offers significant improvement on system utility over a previous optimal algorithm and a greedy algorithm both of which do not support joint layer resource allocation. The proposed approximation algorithm provides controllable tradeoff between performance and computational complexity and, with appropriately chosen parameters, it outperforms the greedy algorithm with 40 less running time. Scalable Video Coding (SVC) has been raised as a promising technique to enable flexible video transmission for mobile users with heterogeneous terminals and varying channel capacities. In this paper, we design an adaptive layer switching algorithm for on-demand scalable video service based on receiver’s buffer underflow probability (BUP). Since the low quality of channel may lead to a low buffer fullness, the buffer fullness is an indicator for reflecting the channel condition and we define BUP for characterizing the mismatch between the video bitrate and the channel throughput. Accordingly, the adaptive SVC transmission problem is formulated as the adaptive adjustment of video layers based on BUP. This allows us to optimize the attainable video quality, while keeping BUP below a desired level. To estimate BUP, we derive an analytical model based on the large deviation principles. Then, an online layer switching algorithm is proposed using this estimation model, which is capable of accommodating different channel qualities without any prior knowledge of the channel variations and of the video characteristics. We further introduce a perturbation-based layer switching approach for reducing the quality fluctuating issue caused by frequent layer switches, thus improving the viewer’s quality of experience. A system prototype is implemented to evaluate the success of the proposed method. We also conduct simulations in multiuser scenarios with real video traces and the results demonstrate that the proposed algorithm is capable of improving the playback experience, while keeping a low playback interruption rate and quality variation. We propose an application layer algorithm for content-aware adaptive video frame scheduling on bandwidth varying channels. The proposed approach includes an iterative algorithm which sequentially uses the rate constraint and delay constraint for video frames to arrive at a decision to drop send video frames. In addition to our base algorithm, we also propose an explicit deadline adjustment (EDA) algorithm. The main idea behind EDA relates to the sender adjusting postponing the deadline of an important frame which is estimated to miss its deadline by actively dropping one (or more) less important next frame(s) and sending the deadline adjusted postponed frame to be displayed in place of the next dropped frame(s). The rate-distortion (R-D) performance of our proposed approach and its comparison with prior art method is shown. Dictionary learning has emerged as a promising alternative to the conventional hybrid coding framework. However, the rigid structure of sequential training and prediction degrades its performance in scalable video coding. This paper proposes a progressive dictionary learning framework with hierarchical predictive structure for scalable video coding, especially in low bitrate region. For pyramidal layers, sparse representation based on spatio-temporal dictionary is adopted to improve the coding efficiency of enhancement layers with a guarantee of reconstruction performance. The overcomplete dictionary is trained to adaptively capture local structures along motion trajectories as well as exploit the correlations between the neighboring layers of resolutions. Furthermore, progressive dictionary learning is developed to enable the scalability in temporal domain and restrict the error propagation in a closed-loop predictor. Under the hierarchical predictive structure, online learning is leveraged to guarantee the training and prediction performance with an improved convergence rate. To accommodate with the state-of-the-art scalable extension of H.264 AVC and latest High Efficiency Video Coding (HEVC), standardized codec cores are utilized to encode the base and enhancement layers. Experimental results show that the proposed method outperforms the latest scalable extension of HEVC and HEVC simulcast over extensive test sequences with various resolutions. Application-layer forward error correction (FEC) is used in many multimedia communication systems to address the problem of packet loss in lossy packet networks. One powerful form of application-layer FEC is unequal error protection which protects the information symbols according to their importance. We propose a method for unequal error protection with a Fountain code. When the information symbols were partitioned into two protection classes (most important and least important), our method required a smaller transmission bit budget to achieve low bit error rates compared to the two state-of-the-art techniques. We also compared our method to the two state-of-the-art techniques for video unicast and multicast over a lossy network. Simulations for the scalable video coding (SVC) extension of the H.264 AVC standard showed that our method required a smaller transmission bit budget to achieve high-quality video. This paper presents the integration of scalable video coding (SVC) into a generic platform for multimedia adaptation. The platform provides a full MPEG-21 chain including server, adaptation nodes, and clients. An efficient adaptation framework using SVC and MPEG-21 digital item adaptation (DIA) is integrated and it is shown that SVC can seamlessly be adapted using DIA. For protection of packet losses in an error prone environment an unequal erasure protection scheme for SVC is provided. The platform includes a real-time SVC encoder capable of encoding CIF video with a QCIF base layer and fine grain scalable quality refinement at 12.5 fps on off-the-shelf high-end PCs. The reported quality degradation due to the optimization of the encoding algorithm is below 0.6 dB for the tested sequences. The use of forward error correction (FEC) coding is often proposed to combat the effects of network packet losses for error-resilient video transmission on packet-switched networks. On the other hand, path diversity has recently been proposed to improve network transport for both singledescription (SD) and multiple-description (MD) coded video. In this work we model and analyze an SD coded video transmission system employing packet-level FEC in combination with path diversity. In particular, we provide a precise analytical approach to evaluating the efficacy of path diversity in reducing the burstiness of network packet-loss processes. We use this approach to quantitatively demonstrate the advantages of path diversity in improving end-to-end video transport performance using packet-level FEC. Improving the coding efficiency is the eternal theme in video coding field. The traditional way for this purpose is to reduce the redundancies inside videos by adding numerous coding options at the encoder side. However, no matter what we have done, it is still hard to guarantee the optimal coding efficiency. On the other hand, the decoded video can be treated as a certain compressive sampling of the original video. According to the compressive sensing theory, it might be possible to further enhance the quality of the decoded video by some restoration methods. Different from the traditional methods, without changing the encoding algorithm, this paper focuses on an approach to improve the video's quality at the decoder end, which equals to further boosting the coding efficiency. Furthermore, we propose a very deep convolutional neural network to automatically remove the artifacts and enhance the details of HEVC-compressed videos, by utilizing that underused information left in the bit-streams and external images. Benefit from the prowess and efficiency of the fully end-to-end feed forward architecture, our approach can be treated as a better decoder to efficiently obtain the decoded frames with higher quality. Extensive experiments indicate our approach can further improve the coding efficiency post the deblocking and SAO in current HEVC decoder, averagely 5.0 , 6.4 , 5.3 , 5.5 BD-rate reduction for all intra, lowdelay P, lowdelay B and random access configurations respectively. This method can aslo be extended to any video coding standards. Significant improvements in video compression capability have been demonstrated with the introduction of the H.264 MPEG-4 advanced video coding (AVC) standard. Since developing this standard, the Joint Video Team of the ITU-T Video Coding Experts Group (VCEG) and the ISO IEC Moving Picture Experts Group (MPEG) has also standardized an extension of that technology that is referred to as multiview video coding (MVC). MVC provides a compact representation for multiple views of a video scene, such as multiple synchronized video cameras. Stereo-paired video for 3-D viewing is an important special case of MVC. The standard enables inter-view prediction to improve compression capability, as well as supporting ordinary temporal and spatial prediction. It also supports backward compatibility with existing legacy systems by structuring the MVC bitstream to include a compatible “base view.” Each other view is encoded at the same picture resolution as the base view. In recognition of its high-quality encoding capability and support for backward compatibility, the stereo high profile of the MVC extension was selected by the Blu-Ray Disc Association as the coding format for 3-D video with high-definition resolution. This paper provides an overview of the algorithmic design used for extending H.264 MPEG-4 AVC towards MVC. The basic approach of MVC for enabling inter-view prediction and view scalability in the context of H.264 MPEG-4 AVC is reviewed. Related supplemental enhancement information (SEI) metadata is also described. Various “frame compatible” approaches for support of stereo-view video as an alternative to MVC are also discussed. A summary of the coding performance achieved by MVC for both stereo- and multiview video is also provided. Future directions and challenges related to 3-D video are also briefly discussed. This paper proposes a novel method for tracking failure detection. The detection is based on the Forward-Backward error, i.e. the tracking is performed forward and backward in time and the discrepancies between these two trajectories are measured. We demonstrate that the proposed error enables reliable detection of tracking failures and selection of reliable trajectories in video sequences. We demonstrate that the approach is complementary to commonly used normalized cross-correlation (NCC). Based on the error, we propose a novel object tracker called Median Flow. State-of-the-art performance is achieved on challenging benchmark video sequences which include non-rigid objects. While much of multiview video coding focuses on the rate-distortion performance of compressing all frames of all views for storage or non-interactive video delivery over networks, we address the problem of designing a frame structure to enable interactive multiview streaming, where clients can interactively switch views during video playback. Thus, as a client is playing back successive frames (in time) for a given view, it can send a request to the server to switch to a different view while continuing uninterrupted temporal playback. Noting that standard tools for random access (i.e., I-frame insertion) can be bandwidth-inefficient for this application, we propose a redundant representation of I-, P-, and “merge” frames, where each original picture can be encoded into multiple versions, appropriately trading off expected transmission rate with storage, to facilitate view switching. We first present ad hoc frame structures with good performance when the view-switching probabilities are either very large or very small. We then present optimization algorithms that generate more general frame structures with better overall performance for the general case. We show in our experiments that we can generate redundant frame structures offering a range of tradeoff points between transmission and storage, e.g., outperforming simple I-frame insertion structures by up to 45 in terms of bandwidth efficiency at twice the storage cost. It has been reported that oversampling a still image before compression does not guarantee a good image quality. Similarly, down-sampling before video compression in low bit rate video coding may alleviate the blocking effect and improve peak signal-to-noise ratio of the decoded frames. When the number of discrete cosine transform coefficients is reduced in such a down-sampling based coding (DBC), the bit budget of each coefficient will increase, thus reduce the quantization error. A DBC video coding scheme is proposed in this paper, where a super-resolution technique is employed to restore the down-sampled frames to their original resolutions. The performance improvement of the proposed DBC scheme is analyzed at low bit rates, and verified by experiments. In this paper, a cross-layer adaptation scheme is proposed for quality of service (QoS) provision in the scalable video streaming of high definition (HD) content. The cross-layer parameters, which contain the video rate, payload length of a packet, the mode of modulation and coding scheme (MCS), can be dynamically adapted to minimize distortion of a video streaming under the given delay bound. Based on the channel quality and rate-distortion parameters, the proposed scheme formulates the problem of parameter selection into an optimization problem. Simulation results show that our approach guarantees video quality under QoS constraints. A major fraction of multimedia stream contents tends to be redundant and leads to wastage of storage capacity and channel bandwidth. In order to eliminate surplus data, standard video compression algorithms exploit spatial and temporal correlation present in video sequence. However, in case of a multisensor network, intersensor statistical redundancy is the most significant factor in acquiring efficient link utilization as well as making resultant findings valuable to the end user. In this paper, an extension to our previously proposed scheme has been presented to accomplish performance goals of a multisensor environment. Standard MPEG codec has been used to accomplish distributed motion compensation in prespecified directions known as directional correlation. Video frame correlation has been estimated locally at the camera node as well as across different nodes, defined as node communication strategies. Further, receiver feedback assists in quality control after reconstitution by decoder assessment. Results estimated have been analyzed for saving ratios and multimedia quality. Results analysis illustrates increased gains in frame quality and compression saving, achieved through reducing node displacement from the reference node (NR). This paper proposes a fast video salient object detection model, based on a novel recurrent network architecture, named Pyramid Dilated Bidirectional ConvLSTM (PDB-ConvLSTM). A Pyramid Dilated Convolution (PDC) module is first designed for simultaneously extracting spatial features at multiple scales. These spatial features are then concatenated and fed into an extended Deeper Bidirectional ConvLSTM (DB-ConvLSTM) to learn spatiotemporal information. Forward and backward ConvLSTM units are placed in two layers and connected in a cascaded way, encouraging information flow between the bi-directional streams and leading to deeper feature extraction. We further augment DB-ConvLSTM with a PDC-like structure, by adopting several dilated DB-ConvLSTMs to extract multi-scale spatiotemporal information. Extensive experimental results show that our method outperforms previous video saliency models in a large margin, with a real-time speed of 20 fps on a single GPU. With unsupervised video object segmentation as an example application, the proposed model (with a CRF-based post-process) achieves state-of-the-art results on two popular benchmarks, well demonstrating its superior performance and high applicability. Forward error correction (FEC) schemes have been proposed and used successfully for multicasting realtime video content to groups of users. Under traditional IP multicast, application-level FEC can only be implemented on an end-to-end basis between the sender and the clients. Emerging overlay and peer-to-peer (p2p) networks open the door for new paradigms of network FEC. The deployment of FEC within these emerging networks has received very little attention (if any). In this paper, we analyze and optimize the impact of network-embedded FEC (NEF) in overlay and p2p multimedia multicast networks. Under NEF, we place FEC codecs in selected intermediate nodes of a multicast tree. The NEF codecs detect and recover lost packets within FEC blocks at earlier stages before these blocks arrive at deeper intermediate nodes or at the final leaf nodes. This approach significantly reduces the probability of receiving undecodable FEC blocks. In essence, the proposed NEF codecs work as signal regenerators in a communication system and can reconstruct most of the lost data packets without requiring retransmission. We develop an optimization algorithm for the placement of NEF codecs within random multicast trees. Based on extensive H.264 video simulations, we show that this approach provides significant improvements in video quality, both visually and in terms of PSNR values.