A Survey on Machine Learning for Optical Communication [Machine Learning View] <s> IV. Reinforcement Learning <s> The popularity of multimedia applications has resulted in increased interest in efficient implementation of multicasting. Wavelength division multiplexing (WDM) is a promising technology for backbone networks, because they provide high bandwidth and methods for control and management. Since not every switches are equipped with wavelength conversion facility, it is a really important issue to design multicast trees, which result in less blocking-probability in the networks. In this paper we propose a reinforcement learning algorithm for the multicast routing and wavelength assignment (MRWA) problem. This distributed algorithm is compared with a multicast algorithm used by a well-known multicast protocol (MOSPF). The proposed algorithm is a good solution to the (MRWA) problem, since it reduces the blocking probability of the multicast communication requests. <s> BIB001 </s> A Survey on Machine Learning for Optical Communication [Machine Learning View] <s> IV. Reinforcement Learning <s> Optical burst switching (OBS) is an optical switching paradigm which offers a good tradeoff between the traditional optical circuit switching (OCS) and optical packet switching (OPS) since it has the relatively easy implementation of the first and the efficient bandwidth utilization of the second. Hence, OBS is a promising technology for the next generation optical Internet. A buffer-less OBS network can be implemented using ordinary optical communication equipment without the need for either wavelength converters or optical memories. However, OBS networks suffer from a relatively high blocking probability, a primary metric of interest, because of contention. In this paper we propose a new contention resolution scheme for buffer-less OBS networks using deflection routing and reinforcement learning agents to dynamically assign an appropriate offset time (OT) to each burst in order to reduce losses caused, for example, by insufficient offset time (IOT) in case only deflection is used. Simulation results demonstrate that our approach reduces effectively blocking probability, whereas it maintains a reasonable end-to-end delay for each burst. Hence, it establishes an appropriate tradeoff between loss rate and delay. <s> BIB002 </s> A Survey on Machine Learning for Optical Communication [Machine Learning View] <s> IV. Reinforcement Learning <s> We present and evaluate a reinforcement learning-based RWA algorithm for all-optical networks subject to physical impairments. The technique is suitable for decentralized networks and is compared with other techniques with similar computational complexity. <s> BIB003 </s> A Survey on Machine Learning for Optical Communication [Machine Learning View] <s> IV. Reinforcement Learning <s> We consider event dependent routing algorithms for on-line explicit source routing in MPLS networks. The proposed methods are based on load shared sequential routing in which load sharing factors are updated using learning algorithms. The learning algorithms we employ are either based on learning automata or on online learning algorithms that were originally devised for solving the adversarial multi-armed bandit problem. While simple to implement, the performance of the proposed learning algorithms in terms of blocking probability compares favorably with the performance of other event dependent routing methods proposed for MPLS routing such as the Success-to-the-top algorithm. We demonstrate the convergence of one of the learning algorithms to the user equilibrium within a set of discrete event simulations. <s> BIB004 </s> A Survey on Machine Learning for Optical Communication [Machine Learning View] <s> IV. Reinforcement Learning <s> Optical burst switching (OBS) is a promising switching paradigm for the next generation Internet. A buffer-less OBS network can be implemented simply and cost-effectively without the need for either wavelength converters or optical buffers which are, currently, neither cost-effective nor technologically mature. However, this type of OBS networks suffers from relatively high loss probability caused by wavelength contentions at core nodes. This issue could prevent or, at least, delay the adoption of OBS networks as a solution for the next generation optical Internet. Deflection routing is one of the contention resolution approaches that have been proposed to tackle this problem. In addition to be cost-effective, it is also efficient in reducing loss probability, especially with low and moderate traffic loads. In this paper, we propose an adaptive reinforcement learning-based deflection routing scheme (RLDRS) which focuses on the route selection issue by choosing the optimal alternative output port in terms of both loss probability and delay when deflection is performed. Moreover, RLDRS limits the number of authorized deflections of each burst in order to reduce the additional traffic caused by deflection routing and to prohibit excessive deflections. Simulation results show that RLDRS reduces effectively loss probability and outperforms shortest path deflection routing (SPDR). <s> BIB005 </s> A Survey on Machine Learning for Optical Communication [Machine Learning View] <s> IV. Reinforcement Learning <s> Optical Burst Switching (OBS) is a promising switching paradigm for the next generation Internet. A bufferless OBS network can be implemented simply and cost-effectively without the need for either wavelength converters or optical buffers which are, currently, neither cost-effective nor technologically mature. However, this type of OBS networks suffers from relatively high loss probability caused by wavelength contentions at core nodes. This issue could prevent or, at least, delay the adoption of OBS networks as a solution for the next generation optical Internet. To enhance the performance of buffer-less OBS networks, we propose two approaches: a) a proactive multi-path approach, called Reinforcement Learning-based Alternative Routing (RLAR), that tries to prevent wavelength contentions; and b) an approach, called Integrated Reinforcement Learning-based Routing and Contention Resolution (IRLRCR) that deals with wavelength contentions proactively and reactively conjointly. We also show relative QoS differentiation capability of the proposed schemes. Simulation results show the ability of the proposed approaches to reduce loss probability and to provide Quality of Service (QoS) differentiation. <s> BIB006 </s> A Survey on Machine Learning for Optical Communication [Machine Learning View] <s> IV. Reinforcement Learning <s> An important issue of research in optical burst switching (OBS) networks is to minimize the loss of bursts due to contention at the intermediate nodes. These contention losses can be minimized with the design of efficient path selection algorithms at the ingress node. Path selection algorithms that learn the optimal path dynamically with the changing traffic conditions outperform the deterministic path selection algorithms. Usually in the single agent path selection algorithms, a path is selected by the agent based on the feedback received at the ingress node which does not capture the effect of the paths selected by the other nodes in the network. We develop a multi-agent approach for path selection that includes the effect of the selection made by all the other nodes in the network. The proposed path selection algorithm uses agents at different source nodes to collectively learn the network dynamics and select the best outgoing path for each burst. We present simulation results to demonstrate the effectiveness of the proposed algorithm over the other similar algorithms in the literature. <s> BIB007 </s> A Survey on Machine Learning for Optical Communication [Machine Learning View] <s> IV. Reinforcement Learning <s> We consider a decentralized cognitive radio network in which autonomous secondary users seek spectrum opportunities in licensed spectrum bands. We assume that the primary users' channel occupancy follows a Markovian evolution, and formulate the spectrum sensing problem as a Decentralized Partially Observable Markov Decision Process (DEC-POMDP). We develop a distributed Reinforcement Learning (RL) algorithm that allows each autonomous cognitive radio to distributively learn its own spectrum sensing policy. The resulting decentralized sensing policy enables secondary users to non-cooperatively reach an equilibrium that leads to high utilization of idle channels while minimizing the collisions among secondary cognitive radios. Moreover, we propose a decentralized channel access policy that permits controlling, with high accuracy, the collision probability with primary users. Our numerical results validate the robustness of this collision probability control as the sensing noise changes. They also show the efficiency of the proposed learning algorithm in improving the spectrum utilization. <s> BIB008 </s> A Survey on Machine Learning for Optical Communication [Machine Learning View] <s> IV. Reinforcement Learning <s> We present a self-adaptive framework to process lightpath requests on a packet switching optical network. Our framework takes care of issues regarding lightpath setting up, resource reservation and load balancing. The framework is composed of a dimensioning phase that adds up new resources to an initial topology and a learning phase that provides routing intelligence to efficiently use the dimensioned resources. The proposed framework is tested on a realistic topology achieving a probability of packet loss of 1×10-4. <s> BIB009 </s> A Survey on Machine Learning for Optical Communication [Machine Learning View] <s> IV. Reinforcement Learning <s> While transparent optical networks become more and more popular as the basis of the Next Generation Internet (NGI) infrastructure, such networks raise many security issues because they lack the massive use of optoelectronic monitoring. To increase these networks' security, they will need to be equipped with proactive and reactive mechanisms to protect themselves not only from failures and attacks but also from ordinary reliability problems. This work presents a novel self-healing framework to deal with attacks on Transparent Optical Packet Switching (TOPS) mesh networks. Contrary to traditional approaches which deal with attacks at the fiber level, our framework allows to overcome attacks at the wavelength level as well as to understand how they impact the network's performance. The framework has two phases: the dimensioning phase (DP) dynamically determines the optical resources for a given mesh network topology whereas the learning phase (LP) generates an intelligent policy to gracefully overcome attacks in the network. DP uses heuristic reasoning to engineer the network while LP relies on a reinforcement learning algorithm that yields a self-healing policy within the network. We use a Monte Carlo simulation to analyze the performance of the aforementioned framework not only under different types of attacks but also using three realistically sized mesh topologies with up to 40 nodes. We compare our framework against shortest path (SP) and multiple path routing (MPR) showing that the self-organized routing outperforms both, leading to a reduction in packet loss of up to 88% with average packet loss rates of 1x10^-^3. Finally, some conclusions are presented as well as future research lines. <s> BIB010 </s> A Survey on Machine Learning for Optical Communication [Machine Learning View] <s> IV. Reinforcement Learning <s> Using monitored physical parameters in a learning process, we decrease design margins by reducing uncertainties on the input parameters of a Quality of Transmission (QoT) tool, improving the accuracy of the signal-to-noise ratio prediction. <s> BIB011 </s> A Survey on Machine Learning for Optical Communication [Machine Learning View] <s> IV. Reinforcement Learning <s> For the converged use of LTE, WLAN, and visible light communication in indoor scenarios, fine-grained and intelligent network selection is essential for ensuring high user quality of experience. To tackle the challenges associated with dynamic environments and complicated service requirements, we propose a context-aware solution for indoor network selection. Specifically, three-level contextual information is revealed and exploited in both the utility and algorithm designs. In particular, the contextual information about the asymmetric downlink-uplink features of network performance is used to design a fine-grained utility model. A context-aware learning algorithm sensitive to traffic type-location-time information is proposed. The time-location dependent periodic changing rule of load statistical distributions is further used to realize efficient online network selection via knowledge transfer. The simulation results show that the proposed algorithm can achieve much better performance with faster convergence speed than traditional reinforcement learning. <s> BIB012
Definition: Reinforcement learning ( Fig.19 .a) includes algorithms that learn by rewards for favorite actions. It is based on a Markov decision process, which is a mathematical model for decision making. It is fully dependent on the programmer knowledge and curiosity. The input (observation) is associated with a reward (reinforcement), and the output (action) determines the next input (and accordingly reward). The aim of training is finding the sequence of actions (it tries different actions) that leads to the best reward. So the action is the most important part; the reward and punishment are dedicated to the model (action) when the output is good and bad, respectively. The training here does not mean to learn from a supervisor, or a structure, it means learning from its own experiences. Network: Transparent optical networks (TON)s are the basis of the Next Generation Optical Internet (NGOI); they don't have massive optoelectronic monitoring. However, they have security problems, and require a somehow self-healing to be protected from failures, attacks, and reliability issues BIB010 . RL has high flexibility, and could be a good choice for increasing reliability, and could generate an intelligent policy to combat failures, and attacks. Optical burst switching (OBS) network is the second promising technology for the NGOI, because it has a relatively easy implementation of optical circuit switching and the efficient bandwidth utilization of Optical Packet Switching (OPS) techniques. The implementation of this method is easy and cost-effective, and does not require either wavelength converters or optical buffers. However, it has high blocking probability, because of wavelength contention. There are several RL techniques considering this issue, for example a deflection routing combined with an adaptive offset time assignment mechanism is used in BIB002 to deal with wavelength contention and insufficient offset time losses, respectively. Shortest path deflection routing aims to find better deflection paths by use of learning agents at each node to learn the optimal alternative output links at each moment; however, a more efficient RL algorithm for deflection routing is presented in BIB005 . BIB006 proposed two RL algorithms for alternative routing, and integrated routing and contention resolution to jointly deal with wavelength issues proactively and reactively. Dynamic path selection has better performance than the deterministic path selection, because it changes by traffic conditions. The single agent path selection, choses a path on the feedback received at the ingress node, and does not affect from selected paths by other network nodes, but multi-agent path selection includes this effect BIB007 . After OBS, and TON related works, the third main focus is on reducing the blocking probability (or applications that somehow lead to it), by using RL in applications such as quality of service (QOS) estimation, VLC network selection, Designing multicast trees, spectrum sensing in cognitive radio network, routing. Distributed route selection in optical networks subject to physical impairments BIB003 , processing lightpath requests, taking care of lightpath setting up, resource Fig.17. a. Hierarchical clustering, b. Fuzzy-logic C-means clustering, c. peak search clustering. reservation and load balancing BIB009 . Growing the multimedia applications resulted in increased interest in multicast designing for reducing the blocking probability. It is shown that RL could adapt to network topology and the traffic changes without prior information, so it is a good choice for this application BIB001 . Another technique that is good for reducing blocking probability is the BIB004 adaptive online RL based routing, which is based on load shared sequential routing in which RL updates the load sharing factors. The established RL algorithms are either based on automata learning or online learning that are designed for adversarial multi-armed bandit problem. Another technique that might further help reducing the blocking probability is the QoS/QoT estimation. Deserving the QoS requirements is important in OC networks,  presented a provisioning strategy for connectivity services with different priorities based on RL able to accommodate QoS requirements and maximize provider benefits. Further, BIB011 using ML improved the accuracy a set of the estimated QoT data (SNR, power levels, noise figures), and lowered the design margins. The only investigations of WOC networks used RL. One of them used RL in decentralized cognitive radio network, in which secondary users search spectrum opportunities. In this technique autonomous cognitive radios learn distributively their own spectrum sensing policy, so the secondary users non-cooperatively achieve equilibrium, the collisions among them would be reduced, and idle channels could be effectively used BIB008 . The other one used RL for the network selection in dynamic environments of indoor VLC. BIB012 presented a context-aware RL algorithm, which uses time location dependent periodic changing rule of load statistical distributions to choose online network via knowledge transfer.