Cellular network configuration plays a critical role in network performance. In current practice, network configuration depends heavily on field experience of engineers and often remains static for a long period of time. This practice is far from optimal. To address this limitation, online-learning-based approaches have great potentials to automate and optimize network configuration. Learning-based approaches face the challenges of learning a highly complex function for each base station and balancing the fundamental exploration-exploitation tradeoff while minimizing the exploration cost. Fortunately, in cellular networks, base stations (BSs) often have similarities even though they are not identical. To leverage such similarities, we propose kernel-based multi-BS contextual bandit algorithm based on multi-task learning. In the algorithm, we leverage the similarity among different BSs defined by conditional kernel embedding. We present theoretical analysis of the proposed algorithm in terms of regret and multi-task-learning efficiency. We evaluate the effectiveness of our algorithm based on a simulator built by real traces.
Recently, learning-based methods are proposed @cite_0 @cite_19 @cite_10 @cite_4 . In @cite_0 , the authors propose a tailored form of reinforcement learning to adaptively select the optimal antenna configuration in a time-varying environment. In @cite_4 , the authors use Q-learning with compact state representation for traffic offloading. In @cite_10 , the authors design a generalized global bandit algorithm to control the transmit power in the cellular coverage optimization problem. In all these papers, BS similarities are not considered, and thus require more exploration. In @cite_19 , the authors study the pilot power configuration problem and design a Gibbs-sampling-based online learning algorithm so as to maximize the throughput of users. In comparison, they make the assumption that all BSs are equal while we allow different BSs to learn different mappings.
@cite_0 @cite_19 @cite_10 @cite_4
2125890412 2125725268 2163190245 2791827017 2540490069 2081893537 2061351551 2490752264 2084099189 1925816294 2790485544 2921187562 1975618234 1979755618 2281709771 2213621529 2136340918 2075914509 2093223549 2134058948
This paper first provides a brief survey on existing traffic offloading techniques in wireless networks. Particularly as a case study, we put forward an online reinforcement learning framework for the problem of traffic offloading in a stochastic heterogeneous cellular network (HCN), where the time-varying traffic in the network can be offloaded to nearby small cells. Our aim is to minimize the total discounted energy consumption of the HCN while maintaining the quality-of-service (QoS) experienced by mobile users. For each cell (i.e., a macro cell or a small cell), the energy consumption is determined by its system load, which is coupled with system loads in other cells due to the sharing over a common frequency band. We model the energy-aware traffic offloading problem in such HCNs as a discrete-time Markov decision process (DTMDP). Based on the traffic observations and the traffic offloading operations, the network controller gradually optimizes the traffic offloading strategy with no prior knowledge of the DTMDP statistics. Such a model-free learning framework is important, particularly when the state space is huge. In order to solve the curse of dimensionality, we design a centralized @math -learning with compact state representation algorithm, which is named @math -learning. Moreover, a decentralized version of the @math -learning is developed based on the fact the macro base stations (BSs) can independently manage the operations of local small-cell BSs through making use of the global network state information obtained from the network controller. Simulations are conducted to show the effectiveness of the derived centralized and decentralized @math -learning algorithms in balancing the tradeoff between energy saving and QoS satisfaction. Universal mobile telecommunication system (UMTS) networks should be deployed according to cost-effective strategies that optimize a cost objective and satisfy target quality-of-service (QoS) requirements. In this paper, we propose novel algorithms for joint uplink downlink UMTS radio planning with the objective of minimizing total power consumption in the network. Specifically, we define two components of the radio planning problem: 1) continuous-based site placement and 2) integer-based site selection. In the site-placement problem, our goal is to find the optimal locations of UMTS base stations (BSs) in a certain geographic area with a given user distribution to minimize the total power expenditure such that a satisfactory level of downlink and uplink signal-to-interference ratio (SIR) is maintained with bounded outage constraints. We model the problem as a constrained optimization problem with SIR-based uplink and downlink power control scheme. An algorithm is proposed and implemented using pattern search techniques for derivative-free optimization with augmented Lagrange multiplier estimates to support general constraints. In the site-selection problem, we aim to select the minimum set of BSs from a fixed set of candidate sites that satisfies quality and outage constraints. We develop an efficient elimination algorithm by proposing a method for classifying BSs that are critical for network coverage and QoS. Finally, the problem is reformulated to take care of location constraints whereby the placement of BSs in a subset of the deployment area is not permitted due to, e.g., private property limitations or electromagnetic radiation constraints. Experimental results and optimal tradeoff curves are presented and analyzed for various scenarios. In this paper, a reinforcement learning-based throughput on demand (ToD) provisioning dynamic power management method (RLTDPM) is proposed for sustaining perpetual operation and satisfying the ToD requirements for today's energy harvesting wireless sensor node (EHWSN). The RLTDPM monitors the environmental state of the EHWS and adjusts their operational duty cycle under criteria of energy neutrality to meet the demanded throughput. Outcomes of these observation-adjustment interactions are then evaluated by feedback reward that represents how well the ToD requests are met; subsequently, the observation-adjustment-evaluation process, so-called reinforcement learning, continues. After the learning process, the RLTDPM is able to autonomously adjust the duty cycle for satisfying the ToD requirement, and in doing so, sustain the perpetual operation of the EHWSN. Simulations of the proposed RLTDPM on a wireless sensor node powered by a battery and solar cell for image sensing tasks were performed. Experimental results demonstrate that the achieved demanded throughput is improved 10.7 for the most stringent ToD requirement, while the residual battery energy of the RLTDPM is improved 7.4 compared with an existing DPM algorithm for EHWSN with image sensing purpose. Motivated by the engineering problem of cellular coverage optimization, we propose a novel multiarmed bandit model called generalized global bandit. We develop a series of greedy algorithms that have the capability to handle nonmonotonic but decomposable reward functions, multidimensional global parameters, and switching costs. The proposed algorithms are rigorously analyzed under the multiarmed bandit framework, where we show that they achieve bounded regret, and hence, they are guaranteed to converge to the optimal arm in finite time. The algorithms are then applied to the cellular coverage optimization problem to achieve the optimal tradeoff between sufficient small cell coverage and limited macroleakage without prior knowledge of the deployment environment. The performance advantage of the new algorithms over existing bandits solutions is revealed analytically and further confirmed via numerical simulations. The key element behind the performance improvement is a more efficient “trial and error” mechanism, in which any trial will help improve the knowledge of all candidate power levels. In this paper, we study resource allocation for a multicarrier-based cognitive radio (CR) network. More specifically, we investigate the problem of secondary users’ energy-efficiency (EE) maximization problem under secondary total power and primary interference constraints. First, assuming cooperation among the secondary base stations (BSs), a centralized approach is considered to solve the EE optimization problem for the CR network where the primary and secondary users are using either orthogonal frequency-division multiplexing (OFDM) or filter bank based multicarrier (FBMC) modulations. We propose an alternating-based approach to solve the joint power-subcarrier allocation problem. More importantly, in the first place, subcarriers are allocated using a heuristic method for a given feasible power allocation. Then, we conservatively approximate the nonconvex power control problem and propose a joint successive convex approximation-Dinkelbach algorithm (SCADA) to efficiently obtain a solution to the nonconvex power control problem. The proposed algorithm is shown to converge to a solution that coincides with the stationary point of the original nonconvex power control subproblem. Moreover, we propose a dual decomposition-based decentralized version of the SCADA. Second, under the assumption of no cooperation among the secondary BSs, we propose a fully distributed power control algorithm from the perspective of game theory. The proposed algorithm is shown to converge to a Nash-equilibrium (NE) point. Moreover, we propose a sufficient condition that guarantees the uniqueness of the achieved NE. Extensive simulation analyses are further provided to highlight the advantages and demonstrate the efficiency of our proposed schemes. Consider media streaming in MIMO interference networks whereby multiple base stations (BS) simultaneously deliver media to their associated users using fixed data rates. The performance is fundamentally limited by the cross-link interference. We propose a cache-induced opportunistic cooperative MIMO (CoMP) for interference mitigation. By caching a portion of the media files, the BSs opportunistically employ CoMP to transform the cross-link interference into spatial multiplexing gain. We study a mixed-timescale optimization of MIMO precoding and cache control to minimize the transmit power under the rate constraint. The cache control is to create more CoMP opportunities and is adaptive to the long-term popularity of the media files. The precoding is to guarantee the rate requirement and is adaptive to the channel state information and cache state at the BSs. The joint stochastic optimization problem is decomposed into a short-term precoding and a long-term cache control problem. We propose a precoding algorithm which converges to a stationary point of the short-term problem. Based on this, we exploit the hidden convexity of the long-term problem and propose a low complexity and robust solution using stochastic subgradient. The solution has significant gains over various baselines and does not require explicit knowledge of the media popularity. Base station (BS) deployment in cellular networks is one of the fundamental problems in network design. This paper proposes a novel method for the cell planning problem for fourth-generation (4G) cellular networks using metaheuristic algorithms. In this approach, we aim to satisfy both cell coverage and capacity constraints simultaneously by formulating an optimization problem that captures practical planning aspects. The starting point of the planning process is defined through a dimensioning exercise that captures both coverage and capacity constraints. Afterward, we implement a metaheuristic algorithm based on swarm intelligence (e.g., particle swarm optimization or the recently proposed gray-wolf optimizer) to find suboptimal BS locations that satisfy both problem constraints in the area of interest, which can be divided into several subareas with different spatial user densities. Subsequently, an iterative approach is executed to eliminate eventual redundant BSs. We also perform Monte Carlo simulations to study the performance of the proposed scheme and compute the average number of users in outage. Next, the problems of green planning with regard to temporal traffic variation and planning with location constraints due to tight limits on electromagnetic radiations are addressed, using the proposed method. Finally, in our simulation results, we apply our proposed approach for different scenarios with different subareas and user distributions and show that the desired network quality-of-service (QoS) targets are always reached, even for large-scale problems. Energy harvesting point-to-point communications are considered. The transmitter harvests energy from the environment and stores it in a finite battery. It is assumed that the transmitter has always data to transmit and the harvested energy is used exclusively for data transmission. As in practical scenarios prior knowledge about the energy harvesting process might not be available, we assume that at each time instant only information about the current state of the transmitter is available, i.e., harvested energy, battery level and channel coefficient. We model the scenario as a Markov decision process and we implement reinforcement learning at the transmitter to find a power allocation policy that aims at maximizing the throughput. To overcome the limitations of traditional reinforcement learning algorithms, we apply the concept of function approximation and we propose a set of binary functions to approximate the expected throughput given the state of the transmitter. Numerical results show that the performance of the proposed approach, which requires only causal knowledge of the energy harvesting process and channel coefficients, has only a small degradation compared to the optimum case which requires perfect non-causal knowledge. Additionally, the proposed approach outperforms naive policies that assume only causal knowledge at the transmitter. This paper considers optimization of the user and base-station (BS) association in a wireless downlink heterogeneous cellular network under the proportional fairness criterion. We first consider the case where each BS has a single antenna and transmits at fixed power and propose a distributed price update strategy for a pricing-based user association scheme, in which the users are assigned to the BS based on the value of a utility function minus a price. The proposed price update algorithm is based on a coordinate descent method for solving the dual of the network utility maximization problem and it has a rigorous performance guarantee. The main advantage of the proposed algorithm as compared to an existing subgradient method for price update is that the proposed algorithm is independent of parameter choices and can be implemented asynchronously. Further, this paper considers the joint user association and BS power control problem and proposes an iterative dual coordinate descent and the power optimization algorithm that significantly outperforms existing approaches. Finally, this paper considers the joint user association and BS beamforming problem for the case where the BSs are equipped with multiple antennas and spatially multiplex multiple users. We incorporate dual coordinate descent with the weighted minimum mean-squared error (WMMSE) algorithm and show that it achieves nearly the same performance as a computationally more complex benchmark algorithm (which applies the WMMSE algorithm on the entire network for BS association) while avoiding excessive BS handover. With the goal to generate more scalable algorithms with higher efficiency and fewer open parameters, reinforcement learning (RL) has recently moved towards combining classical techniques from optimal control and dynamic programming with modern learning techniques from statistical estimation theory. In this vein, this paper suggests to use the framework of stochastic optimal control with path integrals to derive a novel approach to RL with parameterized policies. While solidly grounded in value function estimation and optimal control based on the stochastic Hamilton-Jacobi-Bellman (HJB) equations, policy improvements can be transformed into an approximation problem of a path integral which has no open algorithmic parameters other than the exploration noise. The resulting algorithm can be conceived of as model-based, semi-model-based, or even model free, depending on how the learning problem is structured. The update equations have no danger of numerical instabilities as neither matrix inversions nor gradient learning rates are required. Our new algorithm demonstrates interesting similarities with previous RL research in the framework of probability matching and provides intuition why the slightly heuristically motivated probability matching approach can actually perform well. Empirical evaluations demonstrate significant performance improvements over gradient-based policy learning and scalability to high-dimensional control problems. Finally, a learning experiment on a simulated 12 degree-of-freedom robot dog illustrates the functionality of our algorithm in a complex robot learning scenario. We believe that Policy Improvement with Path Integrals (PI2) offers currently one of the most efficient, numerically robust, and easy to implement algorithms for RL based on trajectory roll-outs. We investigate the problem of resource block (RB) and power allocation jointly and in a distributed manner using game theoretic learning solutions, in an underlay device-to-device network where device pairs communicate directly with each other by reusing the spectrum allocated to the cellular users. We formulate the joint RB and power allocation as multi-agent learning problems with discrete strategy sets; and suggest partially distributed and fully distributed learning algorithms to determine the RB and power level to be used by each device pair. The partially distributed algorithms, viz., Fictitious Play and its variant Fading Memory Joint Strategy Fictitious Play with Inertia, achieve Nash Equilibrium (NE) of the sum-rate maximization game in a static wireless environment. The completely distributed and uncoupled Stochastic Learning Algorithm converges to pure strategy NE of the interference mitigation game in a time-varying radio environment. We provide proofs for the existence of NE and convergence of the learning algorithms to the NE. Performance of the proposed schemes are evaluated in log-normal, Rayleigh and Nakagami fading environments and compared with an existing hybrid scheme and a centralized scheme. The simulation results show that the partially distributed schemes give the same performance as the centralized scheme, and the fully distributed scheme gives similar performance as the hybrid scheme but with much reduced signaling and computation overhead. We study downlink beamforming in a single-cell network with a multi-antenna base station (BS) serving cache-enabled users. For a given common rate of the files in the system, we first formulate the minimum transmit power with beamforming at the BS as a non-convex optimization problem. This corresponds to a multiple multicast problem, to which a stationary solution can be efficiently obtained through successive convex approximation (SCA). It is observed that the complexity of the problem grows exponentially with the number of subfiles delivered to each user in each time slot, which itself grows exponentially with the number of users in the system. Therefore, we introduce a low-complexity alternative through time-sharing that limits the number of subfiles that can be received by a user in each time slot. It is shown through numerical simulations that, the reduced-complexity beamforming scheme has minimal performance gap compared to transmitting all the subfiles jointly, and outperforms the state-of-the-art low-complexity scheme at all SNR and rate values with sufficient spatial degrees of freedom, and in the high SNR high rate regime when the number of spatial degrees of freedom is limited. We investigate a wireless system of multiple cells, each having a downlink shared channel in support of high-speed packet data services. In practice, such a system consists of hierarchically organized entities including a central server, Base Stations (BSs), and Mobile Stations (MSs). Our goal is to improve global resource utilization and reduce regional congestion given asymmetric arrivals and departures of mobile users, a goal requiring load balancing among multiple cells. For this purpose, we propose a scalable cross-layer framework to coordinate packet-level scheduling, call-level cell-site selection and handoff, and system-level cell coverage based on load, throughput, and channel measurements. In this framework, an opportunistic scheduling algorithm--the weighted Alpha-Rule--exploits the gain of multiuser diversity in each cell independently, trading aggregate (mean) down-link throughput for fairness and minimum rate guarantees among MSs. Each MS adapts to its channel dynamics and the load fluctuations in neighboring cells, in accordance with MSs' mobility or their arrival and departure, by initiating load-aware handoff and cell-site selection. The central server adjusts schedulers of all cells to coordinate their coverage by prompting cell breathing or distributed MS handoffs. Across the whole system, BSs and MSs constantly monitor their load, throughput, or channel quality in order to facilitate the overall system coordination. Our specific contributions in such a framework are highlighted by the minimum-rate guaranteed weighted Alpha-Rule scheduling, the load-aware MS handoff cell-site selection, and the Media Access Control (MAC)-layer cell breathing. Our evaluations show that the proposed framework can improve global resource utilization and load balancing, resulting in a smaller blocking rate of MS arrivals without extra resources while the aggregate throughput remains roughly the same or improved at the hot-spots. Our simulation tests also show that the coordinated system is robust to dynamic load fluctuations and is scalable to both the system dimension and the size of MS population. The objective of this paper is to study the performance of a multicell vector perturbation (MVP) precoding technique under practical situations in a network multiple-input multiple-output (MIMO) scheme employing joint transmission. The conventional perturbation strategy that minimizes the total power is considered, and the power at each base station (BS) is properly scaled to enforce per-BS power constraints. In our scenario, we consider multiple-antenna users and use block diagonalization (BD) as the linear front-end precoder for VP. The sum rate for the MVP in the case of uniformly distributed input and an asymptotic upper bound on the sum rate at high signal-to-noise ratios (SNRs) are derived. In addition, using the asymptotic upper bound on the individual user rates, we propose a proportionally fair (PF) user scheduling algorithm of lower complexity and better performance compared with the benchmark fair semiorthogonal user selection (SUS) algorithm. As opposed to the PF-SUS, the proposed PF scheduling algorithm requires no predefined correlation threshold. Furthermore, we study the impact of backhaul delay on the performance of both VP and BD by deriving bounds on the sum rate. The numerical results show that MVP in the case of perfect channel state information (CSI) outperforms multicell BD. In the presence of a backhaul delay, the performance of MVP significantly degrades, but the upper bound on the sum rate for MVP is still higher than for multicell BD. In this work we investigate optimal geographical caching in heterogeneous cellular networks where different types of base stations (BSs) have different cache capacities. Users request files from a content library according to a known probability distribution. The performance metric is the total hit probability, which is the probability that a user at an arbitrary location in the plane will find the content that it requires in one of the BSs that it is covered by. We consider the problem of optimally placing content in all BSs jointly. As this problem is not convex, we provide a heuristic scheme by finding the optimal placement policy for one type of base station conditioned on the placement in all other types. We demonstrate that these individual optimization problems are convex and we provide an analytical solution. As an illustration, we find the optimal placement policy of the small base stations (SBSs) depending on the placement policy of the macro base stations (MBSs). We show how the hit probability evolves as the deployment density of the SBSs varies. We show that the heuristic of placing the most popular content in the MBSs is almost optimal after deploying the SBSs with optimal placement policies. Also, for the SBSs no such heuristic can be used; the optimal placement is significantly better than storing the most popular content. Finally, we show that solving the individual problems to find the optimal placement policies for different types of BSs iteratively, namely repeatedly updating the placement policies, does not improve the performance. This paper presents a content-centric transmission design in a cloud radio access network by incorporating multicasting and caching. Users requesting the same content form a multicast group and are served by a same cluster of base stations (BSs) cooperatively. Each BS has a local cache, and it acquires the requested contents either from its local cache or from the central processor via backhaul links. We investigate the dynamic content-centric BS clustering and multicast beamforming with respect to both channel condition and caching status. We first formulate a mixed-integer nonlinear programming problem of minimizing the weighted sum of backhaul cost and transmit power under the quality-of-service constraint for each multicast group. Theoretical analysis reveals that all the BSs caching a requested content can be included in the BS cluster of this content, regardless of the channel conditions. Then, we reformulate an equivalent sparse multicast beamforming (SBF) problem. By adopting smoothed @math -norm approximation and other techniques, the SBF problem is transformed into the difference of convex programs and effectively solved using the convex-concave procedure algorithms. Simulation results demonstrate significant advantage of the proposed content-centric transmission. The effects of heuristic caching strategies are also evaluated. We consider the problem of resource allocation in downlink OFDMA systems for multi service and unknown environment. Due to users' mobility and intercell interference, the base station cannot predict neither the Signal to Noise Ratio (SNR) of each user in future time slots nor their probability distribution functions. In addition, the traffic is bursty in general with unknown arrival. The probability distribution functions of the SNR, channel state and traffic arrival density are then unknown. Achieving a multi service Quality of Service (QoS) while optimizing the performance of the system (e.g. total throughput) is a hard and interesting task since it depends on the unknown future traffic and SNR values. In this paper we solve this problem by modeling the multiuser queuing system as a discrete time linear dynamic system. We develop a robust H∞ controller to regulate the queues of different users. The queues and Packet Drop Rates (PDR) are controlled by proposing a minimum data rate according to the demanded service type of each user. The data rate vector proposed by the controller is then fed as a constraint to an instantaneous resource allocation framework. This instantaneous problem is formulated as a convex optimization problem for instantaneous subcarrier and power allocation decisions. Simulation results show small delays and better fairness among users. This paper considers a two-user Gaussian interference channel with energy harvesting transmitters. Different than conventional battery powered wireless nodes, energy harvesting transmitters have to adapt transmission to availability of energy at a particular instant. In this setting, the optimal power allocation problem to maximize the sum throughput with a given deadline is formulated. The convergence of the proposed iterative coordinate descent method for the problem is proved and the short-term throughput maximizing offline power allocation policy is found. Examples for interference regions with known sum capacities are given with directional waterfilling interpretations. Next, stochastic data arrivals are addressed. Finally, online and or distributed near-optimal policies are proposed. Performance of the proposed algorithms are demonstrated through simulations. In this paper, a low-complexity delay-aware cross-layer scheduling algorithm for two-hop relay communication systems is proposed. The complex interactions of the queues at the source node and the M relay nodes (RSs) are modeled as an infinite horizon average reward Markov decision process (MDP), whose state space involves the joint queue state information (QSI) of the queues at the source node and the M RSs as well as the joint channel state information (CSI) of all S-R and R-D links. To address the curse of dimensionality, an equivalent MDP formulation is first proposed, where the system state depends only on global QSI. Furthermore, using approximate MDP and stochastic learning, an auction-based distributed online learning algorithm is derived, where each node iteratively estimates a per-node value function based on real-time observations of the local CSI and local QSI as well as signaling between relays. The combined distributed learning converges almost surely to a global optimal solution for large arrivals. Finally, it is showed by simulations that the proposed scheme achieves significant gain compared with various baselines such as the conventional CSIT-only control and the throughput optimal control (in stability sense). In a heterogeneous wireless cellular network, each user may be covered by multiple access points such as macro pico relay femto base stations (BS). An effective approach to maximize the sum utility (e.g., system throughput) in such a network is to jointly optimize users' linear procoders as well as their BS associations. In this paper, we first show that this joint optimization problem is NP-hard and thus is difficult to solve to global optimality. To find a locally optimal solution, we formulate the problem as a noncooperative game in which the users and the BSs both act as players. We introduce a set of new utility functions for the players and show that every Nash equilibrium (NE) of the resulting game is a stationary solution of the original sum utility maximization problem. Moreover, we develop a best-response type algorithm that allows the players to distributedly reach a NE of the game. Simulation results show that the proposed distributed algorithm can effectively relieve local BS congestion and simultaneously achieve high throughput and load balancing in a heterogeneous network.