Artificial Intelligence Enabled Software Defined Networking: A Comprehensive Overview <s> Reinforcement learning in SDN <s> A concept for the optimization of nonlinear functions using particle swarm methodology is introduced. The evolution of several paradigms is outlined, and an implementation of one of the paradigms is discussed. Benchmark testing of the paradigm is described, and applications, including nonlinear function optimization and neural network training, are proposed. The relationships between particle swarm optimization and both artificial life and genetic algorithms are described. <s> BIB001 </s> Artificial Intelligence Enabled Software Defined Networking: A Comprehensive Overview <s> Reinforcement learning in SDN <s> In this paper we consider deterministic policy gradient algorithms for reinforcement learning with continuous actions. The deterministic policy gradient has a particularly appealing form: it is the expected gradient of the action-value function. This simple form means that the deterministic policy gradient can be estimated much more efficiently than the usual stochastic policy gradient. To ensure adequate exploration, we introduce an off-policy actor-critic algorithm that learns a deterministic target policy from an exploratory behaviour policy. We demonstrate that deterministic policy gradient algorithms can significantly outperform their stochastic counterparts in high-dimensional action spaces. <s> BIB002 </s> Artificial Intelligence Enabled Software Defined Networking: A Comprehensive Overview <s> Reinforcement learning in SDN <s> It is a common view in Software Defined Network (SDN) that the flow table plays the most significant role in SDN architecture, but suffers from the limited TCAM chips. The shortage of flow table storage strongly impacts the quality of service (QoS) provided by SDN, but requires rational solutions. In this paper, we present a practical on-line approach based on the decision tree structure to solve this problem. Our performance is evaluated by the comparison with other existing technologies. <s> BIB003 </s> Artificial Intelligence Enabled Software Defined Networking: A Comprehensive Overview <s> Reinforcement learning in SDN <s> An experimental setup of 32 honeypots reported 17M login attempts originating from 112 different countries and over 6000 distinct source IP addresses. Due to decoupled control and data plane, Software Defined Networks (SDN) can handle these increasing number of attacks by blocking those network connections at the switch level. However, the challenge lies in defining the set of rules on the SDN controller to block malicious network connections. Historical network attack data can be used to automatically identify and block the malicious connections. There are a few existing open-source software tools to monitor and limit the number of login attempts per source IP address one-by-one. However, these solutions cannot efficiently act against a chain of attacks that comprises multiple IP addresses used by each attacker. In this paper, we propose using machine learning algorithms, trained on historical network attack data, to identify the potential malicious connections and potential attack destinations. We use four widely-known machine learning algorithms: C4.5, Bayesian Network (BayesNet), Decision Table (DT), and Naive-Bayes to predict the host that will be attacked based on the historical data. Experimental results show that average prediction accuracy of 91.68% is attained using Bayesian Networks. <s> BIB004 </s> Artificial Intelligence Enabled Software Defined Networking: A Comprehensive Overview <s> Reinforcement learning in SDN <s> Managing QoS in a telecommunications network is a complex process. Effective network design and sizing in conjunction with load balancing, access control and traffic prioritization need to be orchestrated to optimize CAPEX investment, maximize network utilization and ensure that performance metrics and SLAs are met. This work shows how big data analytics were used to improve the management of QoS in an SDN by performing multi-dimensional analysis of Key Performance Indicators (KPIs) and applying machine learning algorithms to discover new correlations, perform root cause analysis and predict traffic congestion. <s> BIB005 </s> Artificial Intelligence Enabled Software Defined Networking: A Comprehensive Overview <s> Reinforcement learning in SDN <s> Software-defined networks (SDNs) have been recognized as the next-generation networking paradigm that decouples the data forwarding from the centralized control. To realize the merits of dedicated QoS provisioning and fast route (re-)configuration services over the decoupled SDNs, various QoS requirements in packet delay, loss, and throughput should be supported by an efficient transportation with respect to each specific application. In this paper, a QoS-aware adaptive routing (QAR) is proposed in the designed multi-layer hierarchical SDNs. Specifically, the distributed hierarchical control plane architecture is employed to minimize signaling delay in large SDNs via three-levels design of controllers, i.e., the super, domain (or master), and slave controllers. Furthermore, QAR algorithm is proposed with the aid of reinforcement learning and QoS-aware reward function, achieving a time-efficient, adaptive, QoS-provisioning packet forwarding. Simulation results confirm that QAR outperforms the existing learning solution and provides fast convergence with QoS provisioning, facilitating the practical implementations in large-scale software service-defined networks. <s> BIB006 </s> Artificial Intelligence Enabled Software Defined Networking: A Comprehensive Overview <s> Reinforcement learning in SDN <s> Software Defined Networking (SDN) provides opportunities for dynamic and flexible traffic engineering. This paper discusses how UbuntuNet Alliance National Research and Education Networks (NRENs) can improve bandwidth utilization and reduce inter-NREN latencies through implementation of SDN-based traffic engineering and applying network metrics in selection of inter-NREN paths. Additionally, the paper looks at the utility of applying Reinforcement Learning to path selection, using network data obtained through an SDN controller. Results from simulations using the UbuntuNet topology show an increase in total throughputs when multipath is employed. Furthermore, simulation results show that where latency is the key metric for computing rewards, lower latencies are achieved. <s> BIB007 </s> Artificial Intelligence Enabled Software Defined Networking: A Comprehensive Overview <s> Reinforcement learning in SDN <s> Most Software Defined Networks (SDN) traffic engineering applications use excessive and frequent global monitoring in order to find the optimal Quality-of-Service (QoS) paths for the current state of the network. In this work, we present the motivations, architecture and initial evaluation of a SDN application called Cognitive Routing Engine (CRE) which is able to find near-optimal paths for a user-specified QoS while using a very small monitoring overhead compared to global monitoring which is required to guarantee that optimal paths are found. Smaller monitoring overheads bring the advantage of smaller response time for the SDN controllers and switches. The initial evaluation of CRE on a SDN representation of the GEANT academic network shows that it is possible to find near-optimal paths with a small optimality gap of 1.65% while using 9.5 times less monitoring. <s> BIB008 </s> Artificial Intelligence Enabled Software Defined Networking: A Comprehensive Overview <s> Reinforcement learning in SDN <s> Abstract The decision-tree based packet-classification algorithm field has seen many contributions since the first algorithm using a geometrical rule representation, HiCuts, has been proposed. While hardware reported implementations for this class of algorithms have proven that a high throughput can be reached, those algorithms are inherently facing a tradeoff between speed and memory consumption. This paper presents two extensions applicable to decision-tree based algorithms designed to tackle two of their common drawbacks. Applied together, they achieve a reduction of the number of memory accesses, while reducing the data structure size. The first contribution consists of a new rule-clustering method aimed for the reduction of the number of trees built. The second contribution relies on a leaf compression method that allows tackling the problem that stems from linear leaf traversal. Applied together, as shown by simulations, those two new methods improve the trade-off between search-time complexity and data structure size. These strategies provide gains in many contexts, although they are tailored for handling complex rule sets used in the context of Software Defined Networking. For sets of 100,000 and 10,000 rules, those two strategies reduce the number of memory accesses by a factor of 3 on average, while decreasing the size of the data structure by about 45% over EffiCuts, a well-known decision-tree based algorithm. <s> BIB009 </s> Artificial Intelligence Enabled Software Defined Networking: A Comprehensive Overview <s> Reinforcement learning in SDN <s> Software defined networking (SDN) provides flexible management for datacenter networks with the flow-level control. Such the fine-grained management, however, consumes large amount of bandwidth between data and control planes, which results in the bottleneck in the scalability of SDN-based datacenters. "The elephant and mouse phenomenon" suggests that there are only very few elephant flows that carry the majority of bytes in datacenters so that it can improve management efficiency to detect and reroute elephant flows while leaving mice flows in data plane leveraging wildcard flow table in OpenFlow. Unfortunately, existing mechanisms for elephant flow detection suffer from high bandwidth consumption and long detection time. In this paper, we propose an efficient sampling and classification approach (ESCA) with the two-phase elephant flow detection. In the first phase, ESCA improves sampling efficiency by estimating the arrival interval of elephant flows and filtering out redundant samples using a filtering flow table. In the second phase, ESCA classifies samples with a new supervised classification algorithm based on correlation among data flows. The mathematical analysis proofs our ESCA outperforms related schemes. Extensive experiment results on real public datacenter traces further demonstrate that our ESCA can provide accurate detection with less sampled packets and shorter detection time. <s> BIB010 </s> Artificial Intelligence Enabled Software Defined Networking: A Comprehensive Overview <s> Reinforcement learning in SDN <s> Software defined network (SDN) is one of the most interesting research topic that is currently being investigated. The inclusion of artificial intelligence (AI) can improve the performance of routing protocols. Nowadays the application of AI over routing protocols is only applied to real devices, especially in wireless sensor nodes. In this paper, we present a new proposal to implement an intelligent routing protocol in a SDN topology. The intelligent routing protocol is based on the reinforcement learning process that allows choosing the best data transmission paths according to the best criteria and based on the network status. <s> BIB011 </s> Artificial Intelligence Enabled Software Defined Networking: A Comprehensive Overview <s> Reinforcement learning in SDN <s> HTTP adaptive streaming (HAS) is receiving much attention from both industry and academia as it has become the de facto approach to stream media content over the Internet. Recently, we proposed a streaming architecture called SDNDASH [1] to address HAS scalability issues including video instability, quality of experience (QoE) unfairness, and network resource underutilization, while maximizing per player QoE. While SDNDASH was a significant step forward, there were three unresolved limitations: 1) it did not scale well when the number of HAS players increased; 2) it generated communication overhead; and 3) it did not address client heterogeneity. These limitations could result in suboptimal decisions that led to viewer dissatisfaction. To that effect, we propose an enhanced intelligent streaming architecture, called SDNHAS, which leverages software defined networking (SDN) capabilities of assisting HAS players in making better adaptation decisions. This architecture accommodates large-scale deployments through a cluster-based mechanism, reduces communication overhead between the HAS players and SDN core, and allocates the network resources effectively in the presence of short- and long-term changes in the network. <s> BIB012 </s> Artificial Intelligence Enabled Software Defined Networking: A Comprehensive Overview <s> Reinforcement learning in SDN <s> Multiplayer video streaming scenario can be seen everywhere today as the video traffic is becoming the “killer” traffic over the Internet. The Quality of Experience fairness is critical for not only the users but also the content providers and ISP. Consequently, a QoE fairness adaptive method of multiplayer video streaming is of great importance. Previous studies focus on client-side solutions without network global view or network-assisted solution with extra reaction to client. In this paper, a pure network-based architecture using SDN is designed for monitoring network global performance information. With the flexible programming and network mastery capacity of SDN, we propose an online Q-learning-based dynamic bandwidth allocation algorithm Q-FDBA with the goal of QoE fairness. The results show the Q-FDBA could adaptively react to high frequency of bottleneck bandwidth switches and achieve better QoE fairness within a certain time dimension. <s> BIB013 </s> Artificial Intelligence Enabled Software Defined Networking: A Comprehensive Overview <s> Reinforcement learning in SDN <s> In this paper we design and evaluate a Deep-Reinforcement Learning agent that optimizes routing. Our agent adapts automatically to current traffic conditions and proposes tailored configurations that attempt to minimize the network delay. Experiments show very promising performance. Moreover, this approach provides important operational advantages with respect to traditional optimization algorithms. <s> BIB014 </s> Artificial Intelligence Enabled Software Defined Networking: A Comprehensive Overview <s> Reinforcement learning in SDN <s> In recent years, many techniques have been developed to improve the performance and efficiency of data center networks. While these techniques provide high accuracy, they are often designed using heuristics that leverage domain-specific properties of the workload or hardware. ::: In this vision paper, we argue that many data center networking techniques, e.g., routing, topology augmentation, energy savings, with diverse goals actually share design and architectural similarity. We present a design for developing general intermediate representations of network topologies using deep learning that is amenable to solving classes of data center problems. We develop a framework, DeepConfig, that simplifies the processing of configuring and training deep learning agents that use the intermediate representation to learns different tasks. To illustrate the strength of our approach, we configured, implemented, and evaluated a DeepConfig-Agent that tackles the data center topology augmentation problem. Our initial results are promising --- DeepConfig performs comparably to the optimal. <s> BIB015 </s> Artificial Intelligence Enabled Software Defined Networking: A Comprehensive Overview <s> Reinforcement learning in SDN <s> This paper introduces NeuRoute, a dynamic routing framework for Software Defined Networks (SDN) entirely based on machine learning, specifically, Neural Networks. Current SDN/OpenFlow controllers use a default routing based on Dijkstra algorithm for shortest paths, and provide APIs to develop custom routing applications. NeuRoute is a controller-agnostic dynamic routing framework that (i) predicts traffic matrix in real time, (ii) uses a neural network to learn traffic characteristics and (iii) generates forwarding rules accordingly to optimize the network throughput. NeuRoute achieves the same results as the most efficient dynamic routing heuristic but in much less execution time. <s> BIB016 </s> Artificial Intelligence Enabled Software Defined Networking: A Comprehensive Overview <s> Reinforcement learning in SDN <s> As video-based services become an integral part of the end-users' lives, there is an imminent need for increase in the backhaul capacity and resource management efficiency to enable a highly enhanced multimedia experience to the end-users. The next-generation networking paradigm offers wide advantages over the traditional networks through simplifying the management layer, especially with the adoption of Software Defined Networks (SDN). However, enabling Quality of Service (QoS) provisioning still remains a challenge that needs to be optimized especially for multimedia-based applications. In this paper, we propose LearnQoS, an intelligent QoS management framework for multimedia-based SDNs. LearnQoS employs a policy-based network management (PBNM) to ensure the compliance of QoS requirements and optimizes the operation of PBNM through Reinforcement Learning (RL). The proposed LearnQoS framework is implemented and evaluated under an experimental setup environment and compared with the default SDN operation in terms of PSNR, MOS, throughput and packet loss. <s> BIB017 </s> Artificial Intelligence Enabled Software Defined Networking: A Comprehensive Overview <s> Reinforcement learning in SDN <s> This paper presents NeuTM, a framework for network Traffic Matrix (TM) prediction based on Long Short-Term Memory Recurrent Neural Networks (LSTM RNNs). TM prediction is defined as the problem of estimating future network traffic matrix from the previous and achieved network traffic data. It is widely used in network planning, resource management and network security. Long Short-Term Memory (LSTM) is a specific recurrent neural network (RNN) architecture that is well-suited to learn from data and classify or predict time series with time lags of unknown size. LSTMs have been shown to model long-range dependencies more accurately than conventional RNNs. NeuTM is a LSTM RNN-based framework for predicting TM in large networks. By validating our framework on real-world data from GEEANT network, we show that our model converges quickly and gives state of the art TM prediction performance. <s> BIB018 </s> Artificial Intelligence Enabled Software Defined Networking: A Comprehensive Overview <s> Reinforcement learning in SDN <s> In recent years, software defined networking (SDN) has become a novel network architecture and design by employing manageable software between the control and data planes. Many SDN-based intrusion detection systems (IDS) have been proposed in recent researches. On the other hand, deep learning has emerged as an explosive growth in research and industry. The application of deep learning on IDS has been studied in many security-critical scenarios. However, some reports has indicated the vulnerability of adversarial attacks on deep learning IDS system with intentional perturbation injection or manipulation. It is important for the empowered IDS system employing deep learning to be responsible for not leading misclassification. Therefore, exploring the impact on the adversarial attacks over SDN networks or security-critical scenarios applying deep learning is an important step to confront such urgent issues. In this paper, we will conduct the SDN-based experiments on adversarial attacks for deep learning detecting system. We propose a novel class of adversarial attacks that exploits the vulnerability of the deep learning classifiers in SDN environment. Three typical deep learning models combining with four different adversarial testing will be conducted in the simulation for complete analysis. <s> BIB019
In this context, it is worth noting that Q-learning technique BIB001 BIB003 BIB004 BIB009 BIB010 BIB005 ,last one] was widely applied in SDN paradigm for routing and adaptive video streaming. Al-Jawad et al. BIB017 proposed, LearnQoS, a RL-based framework that utilizes Q-learning for policy-based network management (PBNM) to optimize QoS requirements in multimedia-based SDNs. The RL was modelled by defining three elements: state, action, and reward. The state was represented by the traffic matrix. Four different actions were considered for the agent: 1) do nothing, 2) reduce data rate, 3) increase data rate, and 4) reroute. The rewards, on the other hand, were based on the service level agreements (SLA) requirements. In spite of the network overhead introduced by LearnQoS, the experimental results showed that the performance of the QoS was considerably improved when compared with the default multimedia-based SDN. Sendra et al. BIB011 presented a routing method based on reinforcement learning approach. Given a set of possible paths and a set of network measurements (delay, loss rate and bandwidth), the agent tries to maximise the reward by choosing the path with less cost. Their method showed less loss rate and better jitter values, when compared to traditional OSPF routing protocol. Lin et al. BIB006 proposed QoS-aware adaptive routing (QAR) for distributed hierarchical control planes where Markov decision processes (MDP) with QoS-aware reward function used for modelling the system. Instead of using conventional Qlearning method, softmax action selection policy and stateaction-reward-state-action (SARSA) approach were used for quality update. Kim et al.  investigated congestion prevention based on Q-learning approach. Compared with Dijkstra's algorithm and extended Dijkstra's algorithm, the proposed approach showed better results when the size of transmitted data increases. Uzakgider et al.  proposed an adaptive video streaming method based on Q-learning approach, in which Markov decision process (MDP) was used for modelling the system. The experimental results showed that their proposed system achieves better results when compared with the shortest path routing and greedy-based approaches. Bentaleb et al. BIB012 proposed an end-to-end SDN-based intelligent architecture for large-scale HTTP adaptive streaming (HAS) systems, where partially observable Markov decision process (POMDP) was used for modelling the system. Then Q-learning based approach that employs a per-cluster decision algorithm was used to maximize the QoE. The experimental study showed that their system was able to increase the video stability and achieve better QoE fairness and network resource utilization. Jiang et al. BIB013 proposed, Q-FDBA, an on-line Q-learning-based dynamic bandwidth allocation algorithm for better QoE fairness. Q-FDBA showed better results when compared to bandwidth-aware (BA) streaming and QoE fairness framework (QFF). Geng  presented MIND, which employs online version of Relative Entropy Policy Search using Reproducing Kernel Hilbert Space (REPS-RKHS) approach to learn the probability distribution of choosing the top-k best path. REPS-RKHS is a combination of Relative Entropy Policy Search (REPS) and Reproducing Kernel Hilbert Space (RKHS) embeddings in order to provide a more stable, effective learning progress for non-parametric reinforcement learning. Online REPS-RKHS, on the other hand, is more appropriate for real world problems such as routing, which require a real time performance. MIND has a policy generation module choose an optimal routing policy by learning based on traffic data and network state. Chavula et al. BIB007 used RL approach for SDN-based traffic engineering in UbuntuNet Alliance, which is the regional internetwork for National Research and Education Networks (NRENs) in southern and eastern Africa. The performance rewards were calculated based on distance (packet delay), available capacity on the link, and the resultant load (number of flows) at the next hop. The implementation of Q-learning approach consisted of two tables: (1) a local Q-values table at each network node and (2) a global aggregation table managed by the network controller. Each switch conducts QoS measurements on its next neighbors and passes them to the controller. The controller makes use of both active measurement data and interface-level statistics to update the Q-values based on calculating the reward values. The optimal path is selected by probabilistically choosing the forwarding link based on Q-values at each switch. They implemented the Q-learning approach using Mininet emulator to distribute traffic through multiple forwarding links in a way that maximizes the throughput and reduces the latency. For multipath configurations, the best values of latency and jitter were achieved when the rewards were based on both the available link capacity and latency. On the other hand, the best throughput was achieved when the rewards were based on the links' available bandwidth. The lowest values of latency were obtained with single path forwarding, where the rewards are based on the link delays. Also, single path forwarding achieves the lowest jitter. Francois and Gelenbe BIB008 proposed Cognitive Routing Engine (CRE) for SDNs. CRE consists of 3 main modules. The first module is Cognitive Routing Algorithm Module (CRAM), which employs random neural network with reinforcement learning for finding network paths that maximize a customizable objective function to meet QoS requirements of host applications. Selection of the port that will be used as the next hop can be done in two different modes: (1) the exploratory mode and (2) the exploitation mode. In the exploratory mode, it chooses the output port randomly, but when CPN is in the exploitation mode, it will choose the neuron which has the highest probability. Accordingly, when the reward of the new path is higher than the threshold, then the path is considered valid and the corresponding weights are updated. Otherwise, RNNs have made the wrong decision, and therefore, the weights are updated to allow other paths to be selected. The second module is Network Monitoring Module (NMM), which obtains the network state information and notifies the CRAM. The last module is Path-to-OF Translator Module (PTM), which converts the paths found by CRAM into the appropriate OF messages. The experiment study, which has been conducted based on Mininet emulator and GÉANT network topology showed that CRE reaches near optimal paths with 9.5 times less monitoring data than conventional SDN. However, the solutions of CRE were on average 1.65% worse than the optimal RTT. Stampa et al. BIB014 investigated a deep RL approach for SDN-based routing optimization based on deep deterministic policy gradients approach BIB002 . The state of the deep RL agent is determined by the traffic matrix, the action by a tuple of link weights and the reward is based on the mean network delay. The proposed deep RL agent outperformed their primary benchmark. Streiffer et al. BIB015 , on the other hand, introduced DeepConf, a novel RL-based SDN architecture for developing and training deep ML models for automating data center network topologies management. DeepConf consists of three components: (1) the network simulator for training Deep RL agents, (2) an abstraction layer to facilitate communication between the DeepRL agents and the network, and (3) the DeepRL agents, which encapsulate data center functionality. The authors used Asynchronous Advantage Actor Critic (A3C) approach which employs a deep network to approximate the policy and the value function. The agent's state space is the network topology whereas the action space for the model is represented by a vector which corresponds to different possible link combinations. The goal here is to maximize link utilization and minimize the average flow-completion time. The learning model makes use of a Convolutional Neural Network (CNN) to compute policy decisions, one CNN-block for each state space. The authors evaluated two clos-style data center topologies namely: Fat-tree and VL2 where the experiments showed that DeepConf was able to find near optimal solutions across a range of topologies. In this context, the research efforts made to apply deep learning techniques in SDN are briefly summarized in Table 3 . Azzouni and Pujolle BIB016 LSTM-RNN Traffic matrix prediction Outperformed an efficient dynamic routing heuristic by finding the near optimal path in shorter time using generated data. Azzouni and Pujolle BIB018 LSTM-RNN Traffic matrix prediction Outperformed linear forecasting models (ARMA, ARAR, HW) and feed forward neural network (FFNN) using real data. Huang et al. BIB019 Deep MLP (DNN), CNN and LSTM