In multiview video systems, multiple cameras generally acquire the same scene from different perspectives, such that users have the possibility to select their preferred viewpoint. This results in large amounts of highly redundant data, which needs to be properly handled during encoding and transmission over resource-constrained channels. In this work, we study coding and transmission strategies in multicamera systems, where correlated sources send data through a bottleneck channel to a central server, which eventually transmits views to different interactive users. We propose a dynamic correlation-aware packet scheduling optimization under delay, bandwidth, and interactivity constraints. The optimization relies both on a novel rate-distortion model, which captures the importance of each view in the 3D scene reconstruction, and on an objective function that optimizes resources based on a client navigation model. The latter takes into account the distortion experienced by interactive clients as well as the distortion variations that might be observed by clients during multiview navigation. We solve the scheduling problem with a novel trellis-based solution, which permits to formally decompose the multivariate optimization problem thereby significantly reducing the computation complexity. Simulation results show the gain of the proposed algorithm compared to baseline scheduling policies. More in details, we show the gain offered by our dynamic scheduling policy compared to static camera allocation strategies and to schemes with constant coding strategies. Finally, we show that the best scheduling policy consistently adapts to the most likely user navigation path and that it minimizes distortion variations that can be very disturbing for users in traditional navigation systems.
Some prior studies address the problem of providing to users interactivity in selecting views, while saving on transmitted bandwidth and view-switching delay @cite_19 @cite_24 @cite_3 @cite_31 @cite_0 @cite_26 . The work in @cite_6 is mainly focused on coding views with a minimum level of redundancy in order to simplify the view switching, and the works in @cite_32 @cite_21 optimize the selection of views to be encoded and transmitted based on the user interest. The authors in @cite_0 @cite_8 investigate the transmission of multiview video coded streams on P2P networks and IP multicast, respectively. These works mainly focus on the coding optimization proposed as an a priori defined solution to provide interactive access to the different views @cite_28 . In our work, we rather dynamically optimize the coding modes and the scheduling of video frames for interactive multiview navigation. We extend our preliminary work in @cite_5 to include users' interactivity in the scheduling optimization. This allows the system to dynamically adapt the transmission of the coded frames to various system's dynamics and to outperform the above transmission policies of a priori encoded frames.
@cite_26 @cite_8 @cite_28 @cite_21 @cite_32 @cite_3 @cite_6 @cite_24 @cite_19 @cite_0 @cite_5 @cite_31
2098033563 2111229404 2131723117 2165659335
We derive an optimization framework for joint view and rate scalable coding of multi-view video content represented in the texture plus depth format. The optimization enables the sender to select the subset of coded views and their encoding rates such that the aggregate distortion over a continuum of synthesized views is minimized. We construct the view and rate embedded bitstream such that it delivers optimal performance simultaneously over a discrete set of transmission rates. In conjunction, we develop a user interaction model that characterizes the view selection actions of the client as a Markov chain over a discrete state-space. We exploit the model within the context of our optimization to compute user-action-driven coding strategies that aim at enhancing the client's performance in terms of latency and video quality. Our optimization outperforms the state-of-the-art H.264 SVC codec as well as a multi-view wavelet-based coder equipped with a uniform rate allocation strategy, across all scenarios studied in our experiments. Equally important, we can achieve an arbitrarily fine granularity of encoding bit rates, while providing a novel functionality of view embedded encoding, unlike the other encoding methods that we examined. Finally, we observe that the interactivity-aware coding delivers superior performance over conventional allocation techniques that do not anticipate the client's view selection actions in their operation. We present a novel client-driven multiview video streaming system that allows a user to watch 3D video interactively with significantly reduced bandwidth requirements by transmitting a small number of views selected according to his her head position. The user's head position is tracked and predicted into the future to select the views that best match the user's current viewing angle dynamically. Prediction of future head positions is needed so that views matching the predicted head positions can be prefetched in order to account for delays due to network transport and stream switching. The system allocates more bandwidth to the selected views in order to render the current viewing angle. Highly compressed, lower quality versions of some other views are also prefetched for concealment if the current user viewpoint differs from the predicted viewpoint. An objective measure based on the abruptness of the head movements and delays in the system is introduced to determine the number of additional lower quality views to be prefetched. The proposed system makes use of multiview coding (MVC) and scalable video coding (SVC) concepts together to obtain improved compression efficiency while providing flexibility in bandwidth allocation to the selected views. Rate-distortion performance of the proposed system is demonstrated under different experimental conditions. While much of multiview video coding focuses on the rate-distortion performance of compressing all frames of all views for storage or non-interactive video delivery over networks, we address the problem of designing a frame structure to enable interactive multiview streaming, where clients can interactively switch views during video playback. Thus, as a client is playing back successive frames (in time) for a given view, it can send a request to the server to switch to a different view while continuing uninterrupted temporal playback. Noting that standard tools for random access (i.e., I-frame insertion) can be bandwidth-inefficient for this application, we propose a redundant representation of I-, P-, and “merge” frames, where each original picture can be encoded into multiple versions, appropriately trading off expected transmission rate with storage, to facilitate view switching. We first present ad hoc frame structures with good performance when the view-switching probabilities are either very large or very small. We then present optimization algorithms that generate more general frame structures with better overall performance for the general case. We show in our experiments that we can generate redundant frame structures offering a range of tradeoff points between transmission and storage, e.g., outperforming simple I-frame insertion structures by up to 45 in terms of bandwidth efficiency at twice the storage cost. The traffic of multi-view video is several times larger than traditional multimedia, which brings much more increment in the bandwidth requirement. To overcome this problem, Simulcast and MVC (Multi-view Video Coding) can be used to decrease the traffic for multi-view video. However, although Simulcast removes redundant information within each view, it still contains a large amount of inter-view redundant information. MVC further removes the correlation between the views, but all the views should be encoded and transmitted. So the bit-rates are still too high in these schemes. In this paper, in order to reduce the bit-rate for multi-view video transmission, we analyze the user's motion which is classified into random access model and successive motion model. We find that in the successive motion model only part of frames are possible to be displayed in a period of time. According to the analytical result, we propose a user dependent scheme to decrease the bit-rate of multi-view video transmission. The proposed scheme only encodes and transmits those frames that are possible to be displayed according to the user's motion. Further, in order to support the proposed scheme, we also propose a prediction structure as a substitute of the prediction structure in MVC. Performance evaluation proves that this proposed scheme reduces the average bit-rate for the transmission of multi-view video.