Computer Science Physics
Computer Science-2021.gz:143497
Computer Science
3171754777f273b076020668f494a9c9011b17eb
Learning-based Buffer Starvation Modeling for Packets Prefetching Strategies of Video Streaming Services

Improving the quality of experience (QoE) of video streaming is a significant task in the wireless network scenario. Buffer starvation in the transmission process will cause playback freeze, and a certain number of packets must be prefetched before the service restarts. Taking into account the shortcomings of buffer in video streaming services, this paper proposes a deep learning-based starvation probability calculation model and a reinforcement learning-based packet prefetching model. The deep learning approach extracts the correlation between different timing inputs through the recurrent neural network module to return an explicit result and the precise distribution of the number of buffer starvation. The reinforcement learning approach leverages a better trade-off between start-up/rebuffering delay and buffer starvation by adjusting the packet prefetching strategy, so that the long-term objective quality of experience (QoE) of the video stream is optimized. Our framework can be applied to actual scenarios including finite video streaming and long video streaming transmission.


Introduction
Nowadays, mobile video services are developing rapidly while the stability of video streaming transmission has received more and more attention. Video streaming applications dominate the streaming traffic on mobile networks [1][2]. Compared with voice communication, video streaming applications have less stringent requirements for end-to-end delay, but it requires a set of key performance indicators (KPI) to evaluate the user's quality of experience (QoE) [3]. Among them, the buffer starvation probability is an important performance indicator, which is used to evaluate the event that the buffer is empty [4][5] [6]. When buffer starvation occurs, the video enters the buffering state, and users will see frozen images at this time. Therefore, buffer starvation is not expected to occur in real-time video streaming services [7].
The main goal of this article is to find the distribution of the starvation amount in the packets transmission process, which is applicable to both finite size and infinite size video files. We first model the buffer changes as a Markov process, and then expand it to merge bursty packet arrivals. In this system, a fixed number of packets (also known as "prefetching threshold") are prefetched before the service starts or after a starvation event. This decision-making process can alleviate to a certain extent the buffer starvation of video transmission under complex and unstable network conditions, such as 5G networked drone video backhaul scenarios.
The main contributions of this paper are: • We propose a model based on deep neural network to characterize the buffer starvation behavior in the process of video transmission, and get the exact result of starvation distribution.
• We propose a model based on reinforcement learning to select the packets prefetching strategy, which can alleviate buffer starvation to a certain extent and maximize the long-term objective QoE. • The reliability of the model is verified by a tracking-driven simulation under the complex network transmission scenario of 5G networked drone backhaul video, and combined with practical considerations such as limited video streams and ultra-long video streams. The content of this article is organized as follows. The second section summarizes the related work. The third section describes the prediction model based on deep learning. Section 4 describes the packets prefetching strategy model based on reinforcement learning. Section 5 shows the experimental process and results. Section 6 summarizes this work.

Related Work
There are some studies that are more relevant to our work, [7] proposed a Ballot theorem method to calculate the starvation probability in the M/M/1 queue of limited size. It provided a clear solution and is summarized as M/D/1 queue. The study also proposed a recursive method to calculate the distribution of starvation amount, and further extended it to the ON/OFF burst arrival process [8][9][10][11] [12]. Given the start-up threshold, the author provided a fluid model to calculate the starvation probability and analyzed how the prefetching threshold influences the starvation probability [13] [14][15] [16]. Xu et al. [3] proposed a set of novel models to characterize the starvation behavior of coexisting video streams in the base station (BS). The decision algorithm at the BS can balance the starvation behavior between the short view and the long view. In addition, they also proposed a Bayesian inference method so that network operators can infer whether the input stream is short or long.

Buffer Starvation Behavior Prediction Model
In this section, we will study buffer starvation behavior with finite number of arrivals based on a deep neural network model.

System Description
We propose a packet-level deep learning model to calculate the probability of starvation and the distribution of starvation amount for a single video transmission. This method introduces the recurrent neural network module to extract the correlation between different time series [17] [18]. At the same time, the model uses a multi-task learning structure to share a large part of the weights for the prediction, which reduces the scale of parameters and makes the prediction more efficient.
The input of the network is the state vector of different timings in the process of data packet transmission. The elements contained in the vector are summarized in Table 1, including Poisson arrival rate, Poisson service rate, and traffic intensity etc. Start-up threshold in packets θ Mean of exponential file size The model is mainly divided into three parts: space attention mechanism module, bilateral gate recurrent unit module (BiGRU) [17], and multi-task learning module. The combination of the three structures can effectively extract the correlation between different elements in the same state vector and 3 the correlation between different timing state features. At the same time, it can also give greater weight to some elements that have a greater impact on the final buffer starvation probability, and make full use of the extracted correlations at different levels to obtain better results. The schematic diagram of the model structure is shown in Fig. 1.

Loss Function
The loss function is represented by the sum of two parts = · + · (1) where α, β denote discount factors.
The first part of the formula is the starvation probability loss. The specific calculation is given by = ║ − ║ 2 (2) In this part, represents the output value of the model which fits the starvation probability, and is given by the following formula based on the famous Ballot theorem [11].
The detailed proof can be found in [7]. During the file transfer process, starvation events may occur multiple times. Given a finite file size N, the maximum number of starvations is = ⌊ / 1⌋, where ⌊·⌋ is the lower limit of real numbers.
( ) represents the probability of meeting j starvations. Therefore, the second part is the distribution loss of the starvation amount, using cross-entropy loss.
= −∑ =1 ( ) · log( ) (4) The vector = ( (0), (1), … , ( )). We let ( ) , ( ) , ( ) be the probabilities of events 'the buffer becoming empty for the first time in the entire path', 'the empty buffer after the service of packet given that the previous empty buffer happens at the departure of packet and 'the last empty buffer observed after the departure of packet . The calculation process of ( ) is given by where T denotes the transpose. The detailed analysis can be found in [7].

Packets Prefetching Strategy Model
This section proposes a packets prefetching strategy model based on reinforcement learning [19], which uses content prefetching as a way to achieve the best compromise between the user's start-up delay and starvation behavior (starvation probability or continuous playback interval).

Reinforcement Learning Model Environment Settings
The purpose of the reinforcement learning algorithm is to learn to control the agent to make it perform a given task. To abstract the data packet prefetching strategy problem as a reinforcement learning problem, it is necessary to define the environment of the agent. The environment describes the state of the task at a given time, the set of actions taken by the agent, and the results of these actions [20].
States. The state of this environment is the same as that of the deep learning model in the previous section, and is represented by the state vector in the process of packets transmission.
Action. The action is to prefetch different numbers of packets. The agent performing an action will cause the state to change.
Reward function. In [7], the author configures a starting threshold to ensure that the starvation probability is less than a certain value. The bounds of the starvation probability are considered to be parameters obtained from experiments. In this article, we define an objective QoE cost function for users, and combine it with the starvation probability generation function as a reward function for reinforcement learning.

Optimal strategy learning
Aiming at the packet prefetching strategy, this paper adopts the A3C algorithm as the optimal strategy for learning. The working principle of the algorithm is to establish two neural networks: Actor network and Critic network. The output of the last layer of the Actor network is the probability distribution of each action. The critic network only outputs one value, the value function of the current state. The purpose of training this network is to learn an Actor network that can reliably predict the long-term value of actions [21] [22]. A3C is a policy gradient method. The key of the policy gradient method is to estimate the gradient of the expected total return by observing the state obtained by following the strategies. The gradient of the cumulative long-term reward with respect to the parameter θ is given by where γ denotes discount factor. Here, πis the scheduling strategy, and A is the action value function which is calculated by state and action.
In order to converge the training process, the expected return can be tracked. The expected reward in the packets prefetching problem is very low at first, but will eventually converge to the local optimal expected reward.
The model architecture of the Actor and Critic network is shown in 0. Each network is composed of multiple convolutional layers which adopts a position-invariant detection feature mode, and is connected to 3 fully connected hidden layers. The two networks are disjoint and do not share any layers.

Numerical Examples
In the specific experiment, in order to verify that the model is still valid in a complex network environment, a 5G networked drone (DJI M210 UAV) equipped with a communication module provided by China Mobile was used as the experimental equipment (as shown in Fig. 2).

Starvation of transmission process
Starvation Behavior Prediction: This set of experiments compared the starvation probability calculated by the model with the results of event-driven simulations. Video files from different sizes have been tested up to 8000 times. We deliberately consider four combinations of parameters: ρ = 0.9 or 1.1, x1 = 30 or 50 packets. If not explicitly mentioned, the departure rate μ is normalized to 1. The file size in the experiment is between 200 and 6000 packets. Fig. 3 shows the probability of 0 ~ 4 starvations with the parameters ρ = 0.9 and x1 = 30. As the file size increases, the probability of no starvation decreases. We observe that the probability of multiple starvations first increases and then decreases. Fig. 3 also shows that our analysis results are in great agreement with the simulation results. When the starting threshold is 50 packets, Fig. 4 shows similar results. Fig. 5 verifies the asymptotic probability of no hunger with the traffic intensity ρ = 1.1, x1 = 50 and 100. Fig. 6 plots the asymptotic probability of a single starvation with the same parameters. Fig. 3. Probability of 0~4 starvations with x1=30. Fig. 4. Probability of 0~4 starvations with x1=50.  Fig. 5. Asymptotic probability of no starvation. Fig. 6. Asymptotic probability of 1 starvation.

Optimizing objective quality of experience
Optimizing objective QoE through the packet prefetching strategy: We verified the optimal strategy calculation problem by means of specific experiments at the media server. In Fig. 7, we illustrate the total QoE cost (including starvation cost and start-up delay cost) with λ = 16, 20, 24, and μ = 25. The file size is set to N = 200 ~ 6000 packets, and the discount factor γ is 10e-3. We observe that the total QoE is neither concave nor convex with respect to the action (x1). For example, when the action is fixed with λ=16, the increase in the start-up delay cost cannot be compensated by the reduction of the starvation probability. Fig. 8 compares the optimal total QoE cost obtained from our method and the packet prefetching methods in [3] and [7]. It is not difficult to find that the objective QoE index is better than the results obtained by other methods after the packets prefetching strategy is carried out by the model proposed in this paper.

Conclusion
In this paper, we propose a deep learning model to perform an accurate packet-level analysis of the buffer starvation behavior during video transmission. This method uses an attention-based recurrent neural network to learn the starvation probability and the distribution of starvation amount at the same time. In addition, we also proposed a model based on reinforcement learning to select the packet prefetching strategy, which can relieve buffer starvation to a certain extent and maximize the long-term objective QoE.