Continual Meta-Reinforcement Learning for UAV-Aided Vehicular Wireless Networks

Unmanned aerial base stations (UABSs) can be deployed in vehicular wireless networks to support applications such as extended sensing via vehicle-to-everything (V2X) services. A key problem in such systems is designing algorithms that can efficiently optimize the trajectory of the UABS in order to maximize coverage. In existing solutions, such optimization is carried out from scratch for any new traffic configuration, often by means of conventional reinforcement learning (RL). In this paper, we propose the use of continual meta-RL as a means to transfer information from previously experienced traffic configurations to new conditions, with the goal of reducing the time needed to optimize the UABS's policy. Adopting the Continual Meta Policy Search (CoMPS) strategy, we demonstrate significant efficiency gains as compared to conventional RL, as well as to naive transfer learning methods.


I. INTRODUCTION
Unmanned aerial vehicles acting as flying base stations (BSs), also known as unmanned aerial base stations (UABSs), can enhance network capacity by providing on-demand coverage [1]- [4]. An important use case is offered by vehicular wireless networks, in which UABSs serve as relays between vehicular users and the network, enabling the users to upload data collected by on-board sensors [5]- [11]. Such user-generated data are collected by the network, and then forwarded to other vehicles by means of BSs or road side units (RSUs). Being able to offer stronger, possibly line-ofsight (LoS), links to vehicles as compared to (static) ground BSs, UABSs can support demanding vehicle-to-everything (V2X) applications, such as advanced driving [12], [13] and extended sensing [14], [15], as specified by 3GPP [16]. A key problem in such systems is designing algorithms that can efficiently optimize the trajectory of the UABS in order to maximize coverage. As a means to find such trajectory, convex optimization approaches have been widely adopted under the assumption of fixed ground user locations [17]. In order to alleviate the impact of the simplifications required to apply convex optimization tools, reinforcement learning (RL)-based solutions have been leveraged in [18], [19] for the case of static ground users. More challenging scenarios with moving users have been addressed in [20]- [23] using RL, where only the speed of the UABS was controlled given a fixed trajectory along a highway. The restricted scope of such RL-based solutions stems largely from the need to re-train an RL policy from scratch for any new environment, e.g., for a new traffic pattern of the ground users.
Therefore, differently from previous works, we propose to mitigate this problem via meta-learning [24]. Meta-learning is able to transfer information from previously experienced configurations to new conditions, reducing the time needed to optimize the UABS's policy. Standard meta-learning solutions for RL, also known as meta-RL, require the designer to have access to the simulators corresponding to all the previously encountered traffic conditions [25]. This may be practically impossible, or at least computationally prohibitive. Given these limitations of conventional meta-RL, this paper explores the use of continual meta-RL via Continual Meta Policy Search (CoMPS) [26], which removes the need to revisit previous traffic conditions, and it operates online, acquiring new knowledge as new conditions are encountered.
Conventional meta-learning was previously considered for UABS trajectory optimization in [27] by assuming that the ground users are static and have known locations. The same authors in [28] extended their previous work by considering multiple UABSs. Unlike these previous works, in this paper, we consider traffic conditions characterized by vehicular users with a priori unknown locations and we move beyond conventional meta-RL by accounting for the constraint that simulators for previous traffic configurations cannot be revisited. The rest of the paper is organized as follows. The system model and the problem formulation are described in Section II. The conventional RL framework and the CoMPS-based metalearning scheme are described in Section III. Finally, results are presented in Section IV, and Section V concludes the paper.

II. SYSTEM MODEL AND PROBLEM FORMULATION
We consider a vehicular network in which an UABS provides wireless connectivity to ground user equipments (GUEs). GUEs produce V2X messages that need to be exchanged with the UABS in order to provide the network with information related to their surroundings. We are interested in optimizing the UABS's trajectory so as to maximize the number of V2X packets collected from the GUEs and relayed to the , the GUEs' discrete starting time instants, {t g } G g=1 , paths, {P g } G g=1 , and packet generation probability, p msg . The UABS interacts with the learning task through a simulator over a number of episodes in order to optimize its trajectory. network during deployment. To this end, we assume access to a simulator configured to mimic current traffic conditions (e.g., generating GUEs' paths using Simulation of Urban MObility (SUMO) [29]). We aim at reducing the number of episodes that need to be simulated in order to optimize the policy that controls the UABS's trajectory when facing a new task.

A. Learning Task
As illustrated in Figure 1, a learning task consists of an initial position p ] of the UABS on the plane and of a traffic pattern. Time is discretized as t = 0, 1, . . . , T , where T is the maximum duration of an episode. The traffic pattern is defined by the number G of GUEs, by the path P g , speed v g and (discrete) starting time instant t g ∈ {1, . . . , T } for each GUE g ∈ {1, . . . , G}, as well as by the probability p msg that a GUE generates a packet at each time step. A path P g is a piece-wise linear curve connecting successive points on the plane.
Given the input parameters τ = (G, {v g , P g , t g } G g=1 , p msg ) defining a traffic pattern, a traffic simulator produces the positions p g [t] = [x g [t], y g [t]] for each GUE g = 1, . . . , G at discrete time instants t = t g , t g + 1, . . . , T g , where T g is the smaller value between the total duration of an episode, T , and the time at which the end point of a path is reached by the GUE g. Specifically, the simulator implements a Markov model While on route, at each time instant t ∈ {t g , t g +1, . . . , T g }, a GUE can produce a message with probability p msg . This measurement is stored only for the current time and discarded if not delivered to the UABS. Denoting as SNR g [t] the Signalto-Noise Ratio (SNR) level of GUE g towards the UABS at holds, given a fixed threshold SNR th . When condition (1) is satisfied, the GUE can successfully communicate a message to the UABS at time instant t. The UABS can receive at most C max packets at the same time t. If more than C max GUEs satisfy condition (1) and have a packet to transmit, the UABS randomly selects a subset of C max GUEs from which to receive a packet. We aim at optimizing the stochastic policy π(a|s) for the UABS that selects action a ∈ A as a function of the current state s of the system, i.e., a[t] ∼ π(·|s [t] ) is a function that updates the position of the UABS given action a[t]; and 1(·) is the indicator function. Given state s and action a, the UABS obtains a scalar random reward r[t] ∼ P τ (r|s) equal to the sum of packets collected by the UABS, i.e., In (3), the random variable r g equals one if GUE g has a packet to transmit and satisfies the coverage condition (1). Note that the random variable r g is a function of the current state s, and that its stochasticity arises from the random packet generation process.
Given an initial UABS position p u [0] and the traffic pattern τ , we formulate the design problem for the policy π(a|s) as the optimization of the discounted average return with discount factor γ ∈ (0, 1] [30]. In (4), we have identified the problem configuration as τ 0 = [p u [0], τ ], and we have made explicit the dependence of the expectation on the policy π(a[t]|s[t]). The average also accounts for the transition probability (2) and for the random reward (3).

B. Channel Model
To define the SNR level for each GUE g, we assume the propagation model described in [31] for an urban environment. Accordingly, links between the UABS and GUEs can either be in LoS or non-LoS (NLoS) conditions. The probability p Lg for the link of GUE g at time instant t to be in LoS condition is where α and β are two environment-dependent constants [31], and θ g [t] is the elevation angle for the ray connecting the GUE g and the UABS at time t. The path loss between the GUE g and the UABS at time instant t is given by with carrier frequency f c in MHz; distance d g [t] between the GUE g and the UABS at time instant t in meters; and excessive path loss coefficient η ξ,g [31], with ξ being a binary index indicating whether the link is in LoS or NLoS conditions. Finally, based on (6), the SNR of GUE g at time instant t can be expressed as [31] SNR where P tx is the transmitted power of GUEs in dBm; G tx and G rx represent the gain in transmission and reception in dB, respectively; and P noise is the noise power at the UABS in dBm.

III. META-REINFORCEMENT LEARNING ALGORITHM
In this section, we first introduce the standard reinforcement learning (RL)-based solution. This approach addresses problem (4) from scratch for a fixed configuration τ 0 given by initial UABS position p u [0] and traffic pattern τ . We then exploit continual meta-learning, capable of transferring knowledge across different configurations, to avoid a large number of training episodes.

A. Conventional Reinforcement Learning
To address problem (4) for a given configuration τ 0 , we introduce a parameterized policy π θ (a|s), and we adopt the standard policy gradient method [30], [32]. Accordingly, the gradient of the reward function J τ0 (π θ ) in (4) is estimated as with learning rate η > 0 [30].

B. Meta-Reinforcement Learning
In continual meta-RL, the UABS explores configurations τ i 0 sequentially over a discrete index i = 0, 1, . . . The goal is to transfer knowledge from previously observed tasks so as to prepare to solve problem (4) for future configurations using fewer episodes. A key challenge in this process is posed by the assumption that the UABS cannot run additional simulations for previously encountered configurations. As we will see, this problem can be addressed by storing information about experiences from previous configurations. Following [26], we assume that information is transferred from previous tasks in the form of an initialized model parameter vector θ 0 for the policy gradient update (9). As illustrated in Figure 2, continual meta-RL consists of two main steps applied for each new configuration τ i 0 : • Conventional policy gradient-based RL is applied over N episodes to maximize the expected reward J i (θ) = J τ i 0 (π θ ) with initialization θ 0 i , producing the optimized parameter vector θ * i (θ 0 i ) as a function of θ 0 i ; • A meta-update of the initialization θ 0 i is applied with the goal of maximizing the sum of the expected rewards for the configurations encountered so far for the problem In (10), the notationsJ i (θ) andθ * i indicate that the UABS cannot run new episodes for previous and current tasks, and hence it can only estimate the average return J i (θ) and the optimized model parameter vector θ * i (θ 0 ) for configurations i = 0, . . . , i. These are explained next.
In order to estimate J i (θ) along with the policy parameter θ In (11) and (12), the notations s i,n [t], a i,n [t], r i,n [t], θ i,n stand for state, action, reward, and policy parameter at time t for episode n in configuration τ i 0 . In addition, the best episode n * is chosen as the episode that achieves the highest total reward without discounting factor γ [26], i.e., n * = arg max n T t=0 r i,n [t], and the corresponding experience e i,n * is saved in the skilled experience set E * i . Using the full experience sets {E i } i i =1 and the skilled experience sets {E * i } i i =1 , CoMPS addresses problem (10) as follows. First, off-policy local updates are used to obtain the optimized policy parameter vectorθ * i (θ 0 ) as with learning rate η > 0 and corresponding discounted return as defined in (8). In (13), the episode n is selected at random from the N episodes in set E i . Furthermore, the importance sampling ratio π θ 0 (a i,n [t]|s i,n [t])/π θi,n (a i,n [t]|s i,n [t]) is included in (13) in order to compensate for the generally different probability assigned to action a i,n [t] given state s i,n [t] by the policies π θ 0 (a|s) and π θi,n (a|s). This can partly mitigate the performance degradation caused by the adoption of off-policy optimization [33], [34].
The objectiveJ i (θ) is evaluated using the skilled experience E * i via behavioral cloning [35]. The behavioral cloning loss measures how well the policy π θ can reproduce the nearoptimal, skilled trajectory e i,n * ∈ E * i . It is accordingly defined asJ Finally, CoMPS applies gradient-based optimization to problem (10) as with learning rate κ > 0.
In order to reduce computational complexity as i grows in (15), we sample B tasks among the available i + 1 tasks to compute the gradient in (15). This way, evaluating the meta-update (9) requires order O(4I meta BT C) operations, assuming I meta iterations for the meta-update (15), where C represents the computational complexity of applying policy π θ (a|s) from the state s. In contrast, conventional RL (8) requires order O(2I conven T C) operations, where the number of iterations I conven is typically very large [18]. Therefore, by transferring knowledge from previous environments, meta-RL can significantly reduce the computational complexity.

IV. EXPERIMENTS
In this section, we provide insights and experimental evidence on the benefits of meta-learning via CoMPS as compared to conventional RL. Since meta-learning aims at transferring useful knowledge across different configurations encountered over time index i, as a benchmark, we also consider a basic transfer RL solution, which uses the policy parameter vector θ * i optimized based on the ith configuration as the initialization of conventional RL (Section III-A) for the (i+1)th configuration. If not stated otherwise, parameters used during the simulations are listed in Table I.

A. Toy Example
We consider first a simple setup consisting of a small 40 m × 40 m grid world with two possible tasks. The configurations for the two tasks differ only in the path P g traveled by the three GUEs (G = 3), whereas other parameters are fixed: The initial position of the UABS is set as the bottom-right corner of the square area, i.e., p u [0] = [20,0]; the speed for the GUEs are given as v 1 = v 2 = v 3 = 1 m per time step t = 1 s, the message generation probability is p msg = 1, and the starting time instants of the GUEs are assumed to be t 1 = 1, t 2 = 2, t 3 = 3. The duration of an episode is set to T = 60 s. In the path P g for task τ 1 0 , all the GUEs start from the bottom right corner of the square area to move in clockwise direction along the perimeter of the area, while for task τ 2 0 the movement of GUEs is taken in counterclockwise. Lastly, we assume that the tasks are presented alternatively for every discrete time index i. For this toy example, two tasks are deployed alternately for each time i while the only difference between the two tasks is the path P g : even i takes clockwise path while odd i has counterclockwise path. Fig. 3 plots the average number of packets collected per episode, assuming N = 50 episodes, over time index i. The error regions are obtained by evaluating the standard deviation over 10 independent experiments. Conventional RL cannot take advantage of the data from i configurations, while the performance of transfer RL is affected by a negative transfer of information from the previous configurations. In contrast, meta-RL via CoMPS can effectively transfer information from the i previous configurations. This is illustrated by the initial trajectory optimized by meta-RL, which is shown in the top part of Fig. 3 for increasing values of i. The figure demonstrates how meta-RL gradually identifies a useful initial trajectory from which fast adaptation can be carried out for both tasks.

B. Urban Scenario
In order to evaluate the effectiveness of meta-learning over a more realistic setting, we simulated traffic patterns using the SUMO software for an area in the city of Bologna, Italy, whose dimension is 1500 m × 900 m [29]. In this scenario, K = 50 different task configurations, characterized by different numbers of GUEs (randomly chosen between 15 and 30) moving with different random speed along different paths, are explored sequentially over time index i = 0, . . . , 49. The duration of an episode is set to T = 300 s.   In this paper, we have addressed the problem of optimizing the trajectory of an UABS with the aim of supporting V2X services for moving GUEs. In order to reduce the data requirements for RL-based training, we have proposed to extract useful information from previously encountered traffic configurations to adapt quickly to new environments via meta-RL. Even without the ability to actively revisit previous traffic conditions, we have shown that meta-RL can optimize the initial policy parameter vector so as to reduce the number of exploration steps during training. Future work may consider distributed continual meta-learning across multiple UABS.