Content caching at the edge nodes is a promising technique to reduce the data traffic in next-generation wireless networks. Inspired by the success of Deep Reinforcement Learning (DRL) in solving complicated control problems, this work presents a DRL-based framework with Wolpertinger architecture for content caching at the base station. The proposed framework is aimed at maximizing the long-term cache hit rate, and it requires no knowledge of the content popularity distribution. To evaluate the proposed framework, we compare the performance with other caching algorithms, including Least Recently Used (LRU), Least Frequently Used (LFU), and First-In First-Out (FIFO) caching strategies. Meanwhile, since the Wolpertinger architecture can effectively limit the action space size, we also compare the performance with Deep Q-Network to identify the impact of dropping a portion of the actions. Our results show that the proposed framework can achieve improved short-term cache hit rate and improved and stable long-term cache hit rate in comparison with LRU, LFU, and FIFO schemes. Additionally, the performance is shown to be competitive in comparison to Deep Q-learning, while the proposed framework can provide significant savings in runtime.
Content caching has attracted much interest recently. For instance, the authors in @cite_4 analyzed proactive content caching from the aspect of big data analytics and demonstrated that with limited cache size, proactive caching can provide $100 As seen in previous studies, content popularity distribution is always the key to solve the content caching problem. Considering the large scale of contents and the changing popularities, we note that deep reinforcement learning is a particularly attractive strategy to tackle this problem. In @cite_6 , @cite_3 and @cite_5 , reinforcement learning methods are addressed. For instance, the proposed reinforcement learning agent in @cite_6 is a typical multi-step return actor-critic architecture. The deep deterministic policy gradient proposed in @cite_3 describes how the network is to be updated. And the Wolpertinger policy proposed in @cite_5 provides a solution for large discrete action spaces.
@cite_5 @cite_3 @cite_4 @cite_6
2215378786 2173248099 2278115387 2949980113
Being able to reason in an environment with a large number of discrete actions is essential to bringing reinforcement learning to a larger class of problems. Recommender systems, industrial plants and language models are only some of the many real-world tasks involving large numbers of discrete actions for which current methods are difficult or even often impossible to apply. An ability to generalize over the set of actions as well as sub-linear complexity relative to the size of the set are both necessary to handle such tasks. Current approaches are not able to provide both of these, which motivates the work in this paper. Our proposed approach leverages prior information about the actions to embed them in a continuous space upon which it can generalize. Additionally, approximate nearest-neighbor methods allow for logarithmic-time lookup complexity relative to the number of actions, which is necessary for time-wise tractable training. This combined approach allows reinforcement learning methods to be applied to large-scale learning problems previously intractable with current methods. We demonstrate our algorithm’s abilities on a series of tasks having up to one million actions. We adapt the ideas underlying the success of Deep Q-Learning to the continuous action domain. We present an actor-critic, model-free algorithm based on the deterministic policy gradient that can operate over continuous action spaces. Using the same learning algorithm, network architecture and hyper-parameters, our algorithm robustly solves more than 20 simulated physics tasks, including classic problems such as cartpole swing-up, dexterous manipulation, legged locomotion and car driving. Our algorithm is able to find policies whose performance is competitive with those found by a planning algorithm with full access to the dynamics of the domain and its derivatives. We further demonstrate that for many of the tasks the algorithm can learn policies end-to-end: directly from raw pixel inputs. While 5G wireless networks are expected to handle ever growing data avalanche, classical deployment optimization approaches such as hyper- dense deployment of base stations or having more bandwidth are cost-inefficient, therefore seen as stopgaps. In this regard, context-aware approaches which exploits human predictability, recent advances in storage, edge cloud computing and big data analytics are needed. In this article, we approach to this problem from a proactive caching perspective where gains of cache-enabled base stations in 5G wireless are studied. In particular, huge amount of real data from a telecom operator in Turkey is collected processed on a big data platform, and an analysis is carried out for content popularity estimation for caching, aiming to improve users' experience in terms of request satisfactions and offload the backhaul. Subsequently, with this mobile traffic data collected from many base stations within several hours of time interval and estimation of content popularity via machine learning tools, we investigate the gains of the proactive caching via numerical simulations. The results show that proactive caching fulfils 100 of user request satisfaction and offloads 98 of the backhaul, in a setting of 16 base station with 15.4 Gbyte of storage size (87 of the total catalog size) and 10 of content ratings. In this work we present a new agent architecture, called Reactor, which combines multiple algorithmic and architectural contributions to produce an agent with higher sample-efficiency than Prioritized Dueling DQN (, 2016) and Categorical DQN (, 2017), while giving better run-time performance than A3C (, 2016). Our first contribution is a new policy evaluation algorithm called Distributional Retrace, which brings multi-step off-policy updates to the distributional reinforcement learning setting. The same approach can be used to convert several classes of multi-step policy evaluation algorithms designed for expected value evaluation into distributional ones. Next, we introduce the eta -leave-one-out policy gradient algorithm which improves the trade-off between variance and bias by using action values as a baseline. Our final algorithmic contribution is a new prioritized replay algorithm for sequences, which exploits the temporal locality of neighboring observations for more efficient replay prioritization. Using the Atari 2600 benchmarks, we show that each of these innovations contribute to both the sample efficiency and final agent performance. Finally, we demonstrate that Reactor reaches state-of-the-art performance after 200 million frames and less than a day of training.