A Holistic Survey of Wireless Multipath Video Streaming <s> II. HISTORICAL OVERVIEW OF VIDEO STREAMING <s> MPEG has recently finalized a new standard to enable dynamic and adaptive streaming of media over HTTP. This standard aims to address the interoperability needs between devices and servers of various vendors. There is broad industry support for this new standard, which offers the promise of transforming the media-streaming landscape. <s> BIB001 </s> A Holistic Survey of Wireless Multipath Video Streaming <s> II. HISTORICAL OVERVIEW OF VIDEO STREAMING <s> Content-centric networking promises more efficient distribution of data through in-network caching and the propagation of content through the network. This networking paradigm poses a number of challenges and new opportunities for more efficient multimedia delivery. The MPEG Media Transport (MMT) standard is being developed to address these needs with specifications for encapsulation, delivery, and signaling, which enable fine-grained content access with uniquely identifiable names for optimized content delivery. <s> BIB002 </s> A Holistic Survey of Wireless Multipath Video Streaming <s> II. HISTORICAL OVERVIEW OF VIDEO STREAMING <s> Encoder/decoder details on VP6 developed by On2 Technologies are provided. Similarities related to functionalities between H.264 ( Chap. 4) and VP6 are highlighted and differences such as golden frames are pointed out. Resources related to VP9 open source video codec by Google (Google acquired On2 Technologies in 2010) are also addressed. <s> BIB003 </s> A Holistic Survey of Wireless Multipath Video Streaming <s> II. HISTORICAL OVERVIEW OF VIDEO STREAMING <s> MPEG-2 transport system (TS) and ISO base media file format (ISOBFF) standards developed by MPEG have been widely used by various industries and applications including digital broadcasting systems. Recently MPEG has developed a new standard, MPEG Media Transport or MMT, for next generation broadcasting system considering the emerging convergence of digital broadcasting network and the Internet. MMT assumes HTML5 and Internet protocol (IP) as deployment environment and inherits major features of MPEG-2 TS while simplifying unnecessary complicated features in IP environments. It also considers hybrid delivery of multi-screen services. MMT defines an application layer protocol enabling packetization of opaque objects or files without considering the structure of data and media processing unit (MPU) format specific packetization. Synchronization technology of MMT providing flexible synchronization of multimedia components delivered through hybrid environment over multiple devices is built on top of HTML5. <s> BIB004 </s> A Holistic Survey of Wireless Multipath Video Streaming <s> II. HISTORICAL OVERVIEW OF VIDEO STREAMING <s> Changing network conditions pose severe problems to video streaming in the Internet. HTTP adaptive streaming (HAS) is a technology employed by numerous video services that relieves these issues by adapting the video to the current network conditions. It enables service providers to improve resource utilization and Quality of Experience (QoE) by incorporating information from different layers in order to deliver and adapt a video in its best possible quality. Thereby, it allows taking into account end user device capabilities, available video quality levels, current network conditions, and current server load. For end users, the major benefits of HAS compared to classical HTTP video streaming are reduced interruptions of the video playback and higher bandwidth utilization, which both generally result in a higher QoE. Adaptation is possible by changing the frame rate, resolution, or quantization of the video, which can be done with various adaptation strategies and related client- and server-side actions. The technical development of HAS, existing open standardized solutions, but also proprietary solutions are reviewed in this paper as fundamental to derive the QoE influence factors that emerge as a result of adaptation. The main contribution is a comprehensive survey of QoE related works from human computer interaction and networking domains, which are structured according to the QoE impact of video adaptation. To be more precise, subjective studies that cover QoE aspects of adaptation dimensions and strategies are revisited. As a result, QoE influence factors of HAS and corresponding QoE models are identified, but also open issues and conflicting results are discussed. Furthermore, technical influence factors, which are often ignored in the context of HAS, affect perceptual QoE influence factors and are consequently analyzed. This survey gives the reader an overview of the current state of the art and recent developments. At the same time, it targets networking researchers who develop new solutions for HTTP video streaming or assess video streaming from a user centric point of view. Therefore, this paper is a major step toward truly improving HAS. <s> BIB005 </s> A Holistic Survey of Wireless Multipath Video Streaming <s> II. HISTORICAL OVERVIEW OF VIDEO STREAMING <s> This specification describes an optimized expression of the semantics ::: of the Hypertext Transfer Protocol (HTTP), referred to as HTTP version ::: 2 (HTTP/2). HTTP/2 enables a more efficient use of network resources ::: and a reduced perception of latency by introducing header field ::: compression and allowing multiple concurrent exchanges on the same ::: connection. It also introduces unsolicited push of representations ::: from servers to clients. This specification is an alternative to, but ::: does not obsolete, the HTTP/1.1 message syntax. HTTP's existing ::: semantics remain unchanged. <s> BIB006 </s> A Holistic Survey of Wireless Multipath Video Streaming <s> II. HISTORICAL OVERVIEW OF VIDEO STREAMING <s> HTTP Adaptive Streaming (HAS) is today the number one video technology for over-the-top video distribution. In HAS, video content is temporally divided into multiple segments and encoded at different quality levels. A client selects and retrieves per segment the most suited quality version to create a seamless playout. Despite the ability of HAS to deal with changing network conditions, HAS-based live streaming often suffers from freezes in the playout due to buffer under-run, low average quality, large camera-to-display delay, and large initial/channel-change delay. Recently, IETF has standardized HTTP/2, a new version of the HTTP protocol that provides new features for reducing the page load time in Web browsing. In this paper, we present ten novel HTTP/2-based methods to improve the quality of experience of HAS. Our main contribution is the design and evaluation of a push-based approach for live streaming in which super-short segments are pushed from server to client as soon as they become available. We show that with an RTT of 300 ms, this approach can reduce the average server-to-display delay by 90.1% and the average start-up delay by 40.1%. <s> BIB007 </s> A Holistic Survey of Wireless Multipath Video Streaming <s> II. HISTORICAL OVERVIEW OF VIDEO STREAMING <s> Nowadays, media content can be delivered via diverse broadband and broadcast technologies. Although these different technologies have somehow become rivals, their coordinated usage and convergence, by leveraging of their strengths and complementary characteristics, can bring many benefits to both operators and customers. For example, broadcast TV content can be augmented by on-demand broadband media content to provide enriched and personalized services, such as multi-view TV, audio language selection, and inclusion of real-time web feeds. A piece of evidence is the recent Hybrid Broadcast Broadband TV (HbbTV) standard, which aims at harmonizing the delivery and consumption of (hybrid) broadcast and broadband TV content. A key challenge in these emerging scenarios is the synchronization between the involved media streams, which can be originated by the same or different sources, and delivered via the same or different technologies. To enable synchronized (hybrid) media delivery services, some mechanisms providing timelines at the source side are necessary to accurately time align the involved media streams at the receiver-side. This paper provides a comprehensive review of how clock references (timing) and timestamps (time) are conveyed and interpreted when using the most widespread delivery technologies, such as DVB, RTP/RTCP and MPEG standards (e.g., MPEG-2, MPEG-4, MPEG-DASH, and MMT). It is particularly focused on the format, resolution, frequency, and the position within the bitstream of the fields conveying timing information, as well as on the involved components and packetization aspects. Finally, it provides a survey of proofs of concepts making use of these synchronization related mechanisms. This complete and thorough source of information can be very useful for scholars and practitioners interested in media services with synchronization demands. <s> BIB008 </s> A Holistic Survey of Wireless Multipath Video Streaming <s> II. HISTORICAL OVERVIEW OF VIDEO STREAMING <s> HTTP-based adaptive protocols dominate today's video streaming over the Internet, and operate using multiple quality levels that video players request one segment at a time. Despite their popularity, studies have shown that performance of video streams still suffers from stalls, quality switches and startup delay. In wireless networks, it is well-known that high variability in network bandwidth affects video streaming. MultiPath TCP (MPTCP) is an emerging paradigm that could offer significant benefits to video streaming by combining bandwidth on multiple network interfaces, in particular for mobile devices that typically support both WiFi and cellular networks. In this paper, we explore whether MPTCP always benefits mobile video streaming. Our experimental study on video streaming using two wireless interfaces yields mixed results. While beneficial to user experience under ample and stable bandwidth, MPTCP may not offer any advantage under some network conditions. We find that when additional bandwidth on the secondary path is not sufficient to sustain an upgrade in video quality, it is generally better not to use MPTCP. We also identify that MPTCP can harm user experience when an unstable secondary path is added to the stable primary path. <s> BIB009 </s> A Holistic Survey of Wireless Multipath Video Streaming <s> II. HISTORICAL OVERVIEW OF VIDEO STREAMING <s> The sever-initiated push mechanism is one of the most prominent features in the next generation HTTP/2 protocol, having shown its capability on saving network traffic and improving the web page retrieval latency. Our prior work has investigated the server push-based mechanism for HTTP video streaming and proposed a k-push scheme, where the server pushes k video segments following the response to a request. In this study, we further conduct an analysis and evaluation of the k-push scheme in HTTP streaming. Our results uncover that the push mechanism can efficiently increase the network utilization (under certain conditions) compared to regular HTTP streaming. However the results also show that the k-push scheme deteriorates network adaptability and leads to the "over-push" problem, in which the pushed video content waste network resources due to user abandonment behaviors. To overcome these limitations, we propose a new " adaptive-push" scheme, which dynamically adjusts the parameter k to adapt to the runtime environment. To evaluate the performance of adaptive-push, we implemented a prototype system. The experimental results show that compared to k-push, adaptive-push can improve the network adaptability. Furthermore, our real-world trace based simulation results show that adaptive-push can effectively alleviate the over-push problem. <s> BIB010 </s> A Holistic Survey of Wireless Multipath Video Streaming <s> II. HISTORICAL OVERVIEW OF VIDEO STREAMING <s> HTTP Adaptive Streaming is a successful and largely adopted content delivery technology. Yet poor bandwidth prediction, notably in mobile networks, may cause bit-rate oscillations, increased segment delivery delays, video freezes, and may thus negatively impact the end user quality of experience. To address this issue, we propose to exploit the stream prioritization and termination features of the HTTP/2 protocol to achieve video frame filtering and scheduling, so as to maximize the amount of video data received on time by the client. We evaluate with optimal scheduling and filtering algorithms the maximum gain we may expect from such delivery schemes where video frames are carried in dedicated HTTP/2 streams. Evaluation shows that our HTTP/2-based video frame scheduling scheme brings benefits for video quality. <s> BIB011 </s> A Holistic Survey of Wireless Multipath Video Streaming <s> II. HISTORICAL OVERVIEW OF VIDEO STREAMING <s> The increasing popularity of live video streaming from mobile devices, such as Facebook Live, Instagram Stories, Snapchat, etc. pressurizes the network operators to increase the capacity of their networks. However, a simple increase in system capacity will not be enough without considering the provisioning of quality of experience (QoE) as the basis for network control, customer loyalty, and retention rate and thus increase in network operators revenue. As QoE is gaining strong momentum especially with increasing users’ quality expectations, the focus is now on proposing innovative solutions to enable QoE when delivering video content over heterogeneous wireless networks. In this context, this paper presents an overview of multimedia delivery solutions, identifies the problems and provides a comprehensive classification of related state-of-the-art approaches following three key directions: 1) adaptation; 2) energy efficiency; and 3) multipath content delivery. Discussions, challenges, and open issues on the seamless multimedia provisioning faced by the current and next generation of wireless networks are also provided. <s> BIB012
This section provides a general picture of video streaming development as presented by the timeline and milestones in Figure 2 . Interested readers are referred to BIB008 for a deeper review of different MPEG standards. The first widely used video streaming protocol is the Real-time Transport Protocol (RTP)  , which was initially released in 1992 by IETF. It is a UDP-based protocol used for unidirectional real-time video streaming. The advantage of this protocol is that it has very low overhead and works well in managing IP networks. However, RTP has also some disadvantages. For example, it requires a payload format for each media type or codec BIB004 , suffers from lack of multiplexing and has limited support for non-real-time video. Another disadvantage of RTP is that many CDNs do not support it because the server must manage a separate streaming session for each client, turning large-scale deployment more resource intensive. Moreover, RTP cannot traverse firewalls and is connectionless. Therefore, RTP is generally employed for private managed networks where the number of packet losses is small, such as pay-TV cable networks. More technical details on RTP will be provided in Section IV-A. The next widely known and adopted video streaming protocol shown in Figure 2 is the MPEG-2 Transport System (MPEG-2 TS)  . It has been widely used since 1995 in digital broadcasting, mobile broadcasting systems and streaming over the Internet. Several standards have also adopted this protocol, such as the Terrestrial Digital Multimedia Broadcasting (T-DMB), the Digital Video Broadcasting Handheld (DVB-H), the Advanced Television Systems Committee (ATSC) and the Internet Protocol TeleVision (IPTV)  . MPEG-2 TS is not only a format for fast and reliable packetized streaming delivery but also a format for storage. In addition, MPEG-2 TS is fully codec agnostic. Since the requirements for on-demand and personalized video delivery over the Internet have dramatically increased, it became challenging for MPEG-2 TS to achieve the high requirements of broadcasting over IP BIB004 , BIB002 . For example, MPEG-2 TS is not appropriate for UHD delivery over packet networks due to the pre-multiplexing mechanism, not flexible packetization and small-fixed packet size (188 bytes). The next protocol in the timeline of Figure 2 is the Real-Time Messaging Protocol (RTMP)  . It is an Adobe proprietary protocol standardized in 2002 that was initially developed by Macromedia. RTMP is a TCP-based protocol used for bidirectional video streaming. This protocol provides the advantage of multiplexing capability but requires flash player plugin. All video and audio files must be sent in a Small Web Format (SWF) [35] file to make it able to play with flash player. Another disadvantage is that RTMP suffers from not being codec agnostic and for not supporting some newer video codecs, such as High Efficiency Video Coding (HEVC)  . Yet another disadvantage is that it is blocked by firewalls and not supported by all Content Delivery Networks (CDNs). Multiple types of RTMP protocols were designed: RTMPS, RTMPE, RTMPT and RTMFP. RTMPS is an encrypted RTMP over a Transport Layer Security (TLS)/Secure Sockets Layer (SSL) connection. RTMPE uses Adobe's own encryption security mechanism. RTMPT is used to encapsulate RTMP, RTMPS, or RTMPE packets within Hypertext Transfer Protocol (HTTP) [37] requests in order to traverse firewalls. Finally, RTMFP is used to replace RTMP over User Datagram Protocol (UDP)  instead of TCP  . The next highlighted point in the timeline of Figure 2 is not a protocol, but a video streaming technique introduced in 2006 that become highly adopted in the subsequent streaming protocols. Adaptive Bit Rate (ABR) streaming was introduced by Move Networks  , BIB005 and is over HTTP [37] with some adaptations sequentially described. At the server side, the video sequences are stored in various resolutions (bit rates) and are fragmented into small segments. The streaming logic is located on the client side and it is responsible for selecting the suitable segment by considering different parameters (e.g., bandwidth availability, media playout situations BIB005 ). Without such a flexible service, if only one bit rate video is available, a smaller video bit rate than the network bandwidth would lead to a smooth video but waste available resources. On the other hand, a higher video bit rate than the bandwidth network would impose delay. One advantage of HTTP-based video streaming solutions is that they are easy to deploy in the current Internet architecture. In addition, HTTP flow can traverse middleboxes, such as firewalls, Network Address Translators (NATs) and network proxies. Moreover, the client can manage the streaming without the need to maintain a session state on the server, thus it improves scalability and servers can supply a large number of clients at no additional cost BIB001 . Therefore, HTTP is supported by most of CDNs BIB009 and the interest in using HTTP for video streaming has been significantly increased in recent years. In 2015, a new version of HTTP, namely HTTP/2, was standardized BIB006 and received the attention of researchers in the multimedia community BIB011 , BIB007 , BIB010 . The results show that advancing video streaming services, which are built on top of HTTP/1.1, to HTTP/2 could improve the video quality and performance. More technical details on HTTP/2 will be provided in Section IV-A. The initial HTTP Adaptive Streaming (HAS) commercial successful protocols BIB005 were the Microsoft Silverlight Smooth Streaming (MSS)   protocol was developed to become a unified codec agnostic standard  . MPEG-DASH flexible delivery and codec agnostic properties have turned it into a successful protocol widely adopted by content providers  . For example, Netflix and YouTube are currently using MPEG-DASH with Hypertext Markup Language (HTML) as their core streaming technology  . Another advantage is that DASH supports both multiplexed and unmultiplexed encoded content. However, the protocol has some disadvantages. For instance, DASH cannot support low latency delivery because the server requires waiting until the completion of movie fragments or of the whole file before transmission BIB004 . Besides, the inaccurate bandwidth estimation, especially in mobile networks, causes several switches, freezes, and poor quality of experience (QoE) for DASH BIB011 . More technical details on DASH are provided in Section IV-A. All ABR-based protocols employ similar versions of the previously explained technology and support live and video on demand (VoD) delivery. The differences rely on technical parameters BIB005 . BIB003 and DASH supports all video codecs. Finally, segment lengths are specified as 2 seconds for MSS, 10 seconds for HLS, 2-5 seconds for HDS and are not specified for DASH. More details on HAS protocols are discussed in BIB005 . The next protocol in the timeline of Figure 2 is the MPEG Media Transport (MMT)  . It was standardized by MPEG in 2014 considering recent changes in multimedia delivery and requirements for Internet technologies, such as IP and HTML for Internet-based video streaming solutions BIB004 . This protocol also supports UHD resolution and HEVC video codec. MMT was designed to inherit some MPEG-2 TS features, such as content agnostic media delivery, easy conversion between storage and delivery format and support of multiplexing. In addition, MMT was developed due to a need for an international standard to support hybrid delivery in various heterogeneous network environments. Then, in 2015, implementation guidelines standardized to provide technical guidelines for implementing and deploying MMT systems. The last highlight point in the timeline is MMT enhancement for mobile environment specifying multipath support which has already been added to the protocol and standardized  . MMT was adopted by some recent standards, such as the ATSC 3.0  , which is a recent standard with a hybrid delivery model which includes MMT and DASH. Especially, in ATSC 3.0, MMT protocol (MMTP) is proposed for broadcasting, and DASH over HTTP is proposed for broadband service. One important difference between DASH and MMT is that, typically, DASH supports a client driven Quality of Service (QoS) control standard, while MMT supports a server driven QoS control services  . More technical details on MMT are provided in Section IV-A. To that end, multipath has been investigated in some of the above mentioned video streaming protocols but not in all of them, especially it has not investigated for the proprietary protocols due to their closed and incompatible design ( Figure 2 ). Commercial services. We already mentioned some companies using their own developed proprietary protocols such as Move Networks, Microsoft (MMS), Apple (HLS) and Adobe (RTMP, HDS). Besides them, there are other company services adopting or in the process of developing streaming solutions. For example, Skype and WhatsApp are mobile application platforms providing video calls or video conferences for their users. These services use RTP for video streaming  . Hulu  is an online video service providing on-demand shows, movies, documentaries, and more. Hulu requires flash player for video streaming through the RTMP protocol BIB012 . A number of service providers use DASH. Among the most famous ones are YouTube  , Netflix  , Twitch [63] and Vimeo  . YouTube  is a video-sharing website providing live and on-demand video streaming. Netflix  allows watching on-demand movies and Twitch is the world's leading live streaming platform for gamers. Vimeo also provides free video viewing services. Another commercial streaming service is Bitmovin, which provides adaptive streaming supporting MPEG-DASH and HLS  . Generally, DASH has gotten broad support from commercial companies -see DASH Industry Forum member list available in  . In addition, browsers such as Chrome and Firefox also support DASH  . NHK, Nippon Hoso Kyokai, is a Japan's telecommunication company (public service broadcaster) uses MMT as the protocol of choice for 4K/8K Super Hi-Vision.