Softwarization of Mobile Network Functions towards Agile and Energy Efficient 5G Architectures: A Survey <s> Machine Learning. <s> In Software Defined Network (SDN) environment, controller has to compute and install routing strategy for each new flow, leading to a lot of computation and communication burden in both controller and data planes. In this background, intelligent routing pre-design mechanism is regarded to be an important approach for routing efficiency enhancement. This paper investigates and proposes efficient SDN routing pre-design solution in three aspects: flow feature extraction, requirement prediction and route selection. First, we analyze and extract data packet and association features from user history data, apply these features into semi-supervised clustering algorithm for efficient data classification, analysis and feature extraction. After that, flow service requirement could be predicted through extraction of user, flow and data plane load features and implementation of supervised classification algorithm. Furthermore, we propose corresponding handling strategies related to data plane topology, flow forwarding and multi-constraint weight assignment, and proposes personalized routing selection mechanism. <s> BIB001 </s> Softwarization of Mobile Network Functions towards Agile and Energy Efficient 5G Architectures: A Survey <s> Machine Learning. <s> The theory of reinforcement learning provides a normative account, deeply rooted in psychological and neuroscientific perspectives on animal behaviour, of how agents may optimize their control of an environment. To use reinforcement learning successfully in situations approaching real-world complexity, however, agents are confronted with a difficult task: they must derive efficient representations of the environment from high-dimensional sensory inputs, and use these to generalize past experience to new situations. Remarkably, humans and other animals seem to solve this problem through a harmonious combination of reinforcement learning and hierarchical sensory processing systems, the former evidenced by a wealth of neural data revealing notable parallels between the phasic signals emitted by dopaminergic neurons and temporal difference reinforcement learning algorithms. While reinforcement learning agents have achieved some successes in a variety of domains, their applicability has previously been limited to domains in which useful features can be handcrafted, or to domains with fully observed, low-dimensional state spaces. Here we use recent advances in training deep neural networks to develop a novel artificial agent, termed a deep Q-network, that can learn successful policies directly from high-dimensional sensory inputs using end-to-end reinforcement learning. We tested this agent on the challenging domain of classic Atari 2600 games. We demonstrate that the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to surpass the performance of all previous algorithms and achieve a level comparable to that of a professional human games tester across a set of 49 games, using the same algorithm, network architecture and hyperparameters. This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks. <s> BIB002 </s> Softwarization of Mobile Network Functions towards Agile and Energy Efficient 5G Architectures: A Survey <s> Machine Learning. <s> Mobile edge computing (a.k.a. fog computing) has recently emerged to enable \emph{in-situ} processing of delay-sensitive applications at the edge of mobile networks. Providing grid power supply in support of mobile edge computing, however, is costly and even infeasible (in certain rugged or under-developed areas), thus mandating on-site renewable energy as a major or even sole power supply in increasingly many scenarios. Nonetheless, the high intermittency and unpredictability of renewable energy make it very challenging to deliver a high quality of service to users in renewable-powered mobile edge computing systems. In this paper, we address the challenge of incorporating renewables into mobile edge computing and propose an efficient reinforcement learning-based resource management algorithm, which learns on-the-fly the optimal policy of dynamic workload offloading (to centralized cloud) and edge server provisioning to minimize the long-term system cost (including both service delay and operational cost). Our online learning algorithm uses a decomposition of the (offline) value iteration and (online) reinforcement learning, thus achieving a significant improvement of learning rate and run-time performance when compared to standard reinforcement learning algorithms such as Q-learning. <s> BIB003 </s> Softwarization of Mobile Network Functions towards Agile and Energy Efficient 5G Architectures: A Survey <s> Machine Learning. <s> Next-generation wireless networks are expected to support extremely high data rates and radically new applications, which require a new wireless radio technology paradigm. The challenge is that of assisting the radio in intelligent adaptive learning and decision making, so that the diverse requirements of next-generation wireless networks can be satisfied. Machine learning is one of the most promising artificial intelligence tools, conceived to support smart radio terminals. Future smart 5G mobile terminals are expected to autonomously access the most meritorious spectral bands with the aid of sophisticated spectral efficiency learning and inference, in order to control the transmission power, while relying on energy efficiency learning/inference and simultaneously adjusting the transmission protocols with the aid of quality of service learning/inference. Hence we briefly review the rudimentary concepts of machine learning and propose their employment in the compelling applications of 5G networks, including cognitive radios, massive MIMOs, femto/small cells, heterogeneous networks, smart grid, energy harvesting, device-todevice communications, and so on. Our goal is to assist the readers in refining the motivation, problem formulation, and methodology of powerful machine learning algorithms in the context of future networks in order to tap into hitherto unexplored applications and services. <s> BIB004 </s> Softwarization of Mobile Network Functions towards Agile and Energy Efficient 5G Architectures: A Survey <s> Machine Learning. <s> In this paper, a survey of the literature of the past 15 years involving machine learning (ML) algorithms applied to self-organizing cellular networks is performed. In order for future networks to overcome the current limitations and address the issues of current cellular systems, it is clear that more intelligence needs to be deployed so that a fully autonomous and flexible network can be enabled. This paper focuses on the learning perspective of self-organizing networks (SON) solutions and provides, not only an overview of the most common ML techniques encountered in cellular networks but also manages to classify each paper in terms of its learning solution, while also giving some examples. The authors also classify each paper in terms of its self-organizing use-case and discuss how each proposed solution performed. In addition, a comparison between the most commonly found ML algorithms in terms of certain SON metrics is performed and general guidelines on when to choose each ML algorithm for each SON function are proposed. Lastly, this paper also provides future research directions and new paradigms that the use of more robust and intelligent algorithms, together with data gathered by operators, can bring to the cellular networks domain and fully enable the concept of SON in the near future. <s> BIB005 </s> Softwarization of Mobile Network Functions towards Agile and Energy Efficient 5G Architectures: A Survey <s> Machine Learning. <s> As manufacturing processes become increasingly automated, so should tool condition monitoring (TCM) as it is impractical to have human workers monitor the state of the tools continuously. Tool condition is crucial to ensure the good quality of products—worn tools affect not only the surface quality but also the dimensional accuracy, which means higher reject rate of the products. Therefore, there is an urgent need to identify tool failures before it occurs on the fly. While various versions of intelligent tool condition monitoring have been proposed, most of them suffer from a cognitive nature of traditional machine learning algorithms. They focus on the how-to-learn process without paying attention to other two crucial issues—what-to-learn, and when-to-learn. The what-to-learn and the when-to-learn provide self-regulating mechanisms to select the training samples and to determine time instants to train a model. A novel TCM approach based on a psychologically plausible concept, namely the metacognitive scaffolding theory, is proposed and built upon a recently published algorithm—recurrent classifier (rClass). The learning process consists of three phases: what-to-learn, how-to-learn, when-to-learn and makes use of a generalized recurrent network structure as a cognitive component. Experimental studies with real-world manufacturing data streams were conducted where rClass demonstrated the highest accuracy while retaining the lowest complexity over its counterparts. <s> BIB006
Future MNs are expected to learn the diverse characteristics of users behavior, as well as renewable energy source(s) variations, in order to autonomously and dynamically determine good system configurations. Network elements are expected to rely on sophisticated learning and decision making procedures, for an efficient network management. Machine learning (ML) techniques constitute a promising solution for network management and energy savings in cellular networks. According to BIB004 BIB005 , they can be categorized as supervised, unsupervised, or reinforcement learning-based. Supervised and unsupervised learning, respectively, indicate whether the samples from the dataset are labelled or not. Reinforcement learning instead considers an agent surrounded by a generally unknown environment, whose actions are either rewarded or punished according to a reward/cost model. The final objective is to teach the agent to solve a certain task. Semisupervised learning is also possible  , although less explored: with it, the learner has access to a small amount of labelled data and to a large number of unlabeled examples. This is pretty common in practical cases and, as such, it may eventually become the preferred learning technology for many application domains. The basic concept of ML algorithms and the corresponding applications according to the category of supervised, unsupervised, and reinforcement learning is presented in BIB004 . ML can be used in modeling various technical problems for next-generation systems. For example, the authors in BIB003 use reinforcement learning-based resource management algorithm to incorporate renewables into a mobile edge computing platform. The algorithm learns on the fly the optimal policy for dynamic workload offloading and edge server provisioning to minimize the long term cost, that is, service delay and operational cost. A significant learning rate was achieved, from the use of the online learning algorithm with a decomposition of (offline) value iteration and reinforcement learning. Also, an increase in runtime performance was observed when compared with the Q-learning algorithm. An ML based routing preplan solution for an SDN environment is presented in BIB001 , considering (i) flow feature extraction, (ii) user requirement prediction, and (iii) route selection. Under the SDN route planning context, the core idea is to predict the user's business requirements and then plan ahead and set up routing policies, with a view of reducing delay effects. To improve the effectiveness of SDN routing, relevant features were extracted from the user's historical data and then utilized within a semisupervised clustering algorithm for data classification. Through the extraction of user's data, flow, and data plane load features, the flow service demand forecasts were then predicted using supervised classification. The network structure was constantly updated by optimizing the extreme learning method (ELM), and old data was discarded upon completion of the training. In addition, based on the flow feature extraction and flow demand forecasts, a personalized route selection mechanism based on policy making was introduced (the path computation involves the use of linear programming and a set of cost functions). Another learning technique inspired by a behavioral psychological concept, in the context of machine learning, is called metacognitive scaffolding  . The concept of scaffolding theory  , a prominent tutoring theory for a student to learn a complex task, is based on the fact that the learning process of human beings is metacognitive in nature, since it involves what-to-learn, how-to-learn, and when-to-learn. Such metacognitive learning approach can be extended to cellular networks, more especially to remote access networks, where the network can learn the environment behavior and then adjust its network configuration with respect to the observed and forecast changes. The authors in BIB006 use a metacognitive scaffolding learning approach for identifying (predicting) tool failures before they occur, in the context of manufacturing processes. The conducted experimental studies, using real-world data, show that the prediction accuracy can be improved using a low complexity algorithm. Such technique can be also used in remote 5G sites (off-grid) to manage the BSs utilizing only renewable energy sources, solar or wind, under traffic variation and battery constraints. This can yield better edge network management, in the case of MEC. In conclusion, ML is concerned with the design and development of means that allow network devices to learn. ML applications in 5G networks involve channel estimation or detection, spectrum sensing, cell/user clustering, handover among HetNets, energy modeling and prediction, user behavior analysis (including mobility and traffic profiling/ forecasting), intrusion (fault/anomaly) detection, channel selection association, and so forth. The family of supervised learning techniques relies on known labels, it can then be applied to higher-layer applications such as discovering the mobile user's location and behavior. This can assist in improving the QoE being offered. Unsupervised learning makes use of the input data to automatically discover patterns, and it can be utilized for load balancing in HetNets. Lastly, reinforcement learning relies on a dynamic iterative learning and decision making process. Its recent evolution consists of combining it with deep neural networks, leading to the socalled deep reinforcement learning (DRL), which is becoming a standard technique in modern learning systems BIB002 . RL and DRL can be used for inferring mobile users policies under unknown network conditions, for example, BS association under unknown ESI of the BS, in EH networks. The reader is referred to BIB004 BIB005 for a comprehensive review of ML techniques applied to cellular networks.