A Survey on Efforts to Evolve the Control Plane of Inter-Domain Routing <s> Incremental Improvement <s> Over the past decade, the complexity of the Internet's routing infrastructure has increased dramatically. This complexity and the problems it causes stem not just from various new demands made of the routing infrastructure, but also from fundamental limitations in the ability of today's distributed infrastructure to scalably cope with new requirements.The limitations in today's routing system arise in large part from the fully distributed path-selection computation that the IP routers in an autonomous system (AS) must perform. To overcome this weakness, interdomain routing should be separated from today's IP routers, which should simply forward packets (for the most part). Instead, a separate Routing Control Platform (RCP) should select routes on behalf of the IP routers in each AS and exchange reachability information with other domains.Our position is that an approach like RCP is a good way of coping with complexity while being responsive to new demands and can lead to a routing system that is substantially easier to manage than today. We present a design overview of RCP based on three architectural principles path computation based on a consistent view of network state, controlled interactions between routing protocol layers, and expressive specification of routing policies and discuss the architectural strengths and weaknesses of our proposal. <s> BIB001 </s> A Survey on Efforts to Evolve the Control Plane of Inter-Domain Routing <s> Incremental Improvement <s> The Internet inter-domain routing protocol, BGP, experiences frequent routing disruptions such as transient routing loops or loss of connectivity. The goal of this paper is to address this issue while preserving BGP's benefits in terms of operational maturity and flexibility in accommodating diverse policies. In realizing this goal, we apply to inter-domain routing a common concept in the design of highly reliable systems, namely, the use of redundancy, which we introduce in a manner that maximizes compatibility with the existing BGP protocol. The basic idea is to run several, mostly unchanged BGP processes that compute complementary routes, so that in the presence of network instabilities a working path remains available to any destination. The paper outlines the design of this approach and compares it to previously proposed alternatives. The benefits of the scheme are demonstrated using actual BGP data and realistic simulations. <s> BIB002 </s> A Survey on Efforts to Evolve the Control Plane of Inter-Domain Routing <s> Incremental Improvement <s> Inter-domain routing is based on a fully decentralized model, where multiple Autonomous Systems (AS) interact via the BGP protocol. Although BGP is the "glue" of the Internet, it faces a lot of problems regarding its fully distributed nature, policy enforcement capabilities, scalability, security and complexity. Due to the widespread adoption of BGP, only incrementally deployable solutions are feasible. Taking this observation into account, we describe a new, backwards-compatible routing model which is based on outsourcing and logically centralizing the routing control plane. We claim that outsourcing enables enhanced inter-domain routing. As multiple ASes outsource their routing control plane to the same outsourcing service contractor, AS clusters are being gradually formed. A logically centralized multi-AS routing control platform based on Software Defined Networking (SDN) principles is the natural point for taking efficient routing decisions, detecting policy conflicts, troubleshooting routing problems, and evolving BGP. We present the technical and financial incentives which support the feasibility of the model and also propose an implementation scheme to facilitate it. <s> BIB003 </s> A Survey on Efforts to Evolve the Control Plane of Inter-Domain Routing <s> Incremental Improvement <s> Software Defined Networking (SDN) is an exciting technology that enables innovation in how we design and manage networks. Although this technology seems to have appeared suddenly, SDN is part of a long history of efforts to make computer networks more programmable. In this paper, we trace the intellectual history of programmable networks, including active networks, early efforts to separate the control and data plane, and more recent work on OpenFlow and network operating systems. We highlight key concepts, as well as the technology pushes and application pulls that spurred each innovation. Along the way, we debunk common myths and misconceptions about the technologies and clarify the relationship between SDN and related technologies such as network virtualization. <s> BIB004 </s> A Survey on Efforts to Evolve the Control Plane of Inter-Domain Routing <s> Incremental Improvement <s> BGP severely constrains how networks can deliver traffic over the Internet. Today's networks can only forward traffic based on the destination IP prefix, by selecting among routes offered by their immediate neighbors. We believe Software Defined Networking (SDN) could revolutionize wide-area traffic delivery, by offering direct control over packet-processing rules that match on multiple header fields and perform a variety of actions. Internet exchange points (IXPs) are a compelling place to start, given their central role in interconnecting many networks and their growing importance in bringing popular content closer to end users. To realize a Software Defined IXP (an "SDX"), we need new programming abstractions that allow participating networks to create and run these applications and a runtime that both behaves correctly when interacting with BGP and ensures that applications do not interfere with each other. We must also ensure that the system scales, both in rule-table size and computational overhead. In this demo, we show how we tackle these challenges demonstrating the flexibility and scalability of our SDX platform. The paper also appears in the main program. <s> BIB005 </s> A Survey on Efforts to Evolve the Control Plane of Inter-Domain Routing <s> Incremental Improvement <s> An Internet autonomous system (AS) uses BGP policies in order to meet its local objectives (e.g. optimal routing within AS) and peering contracts with neighboring ASes (e.g. advertising some specific routes to a neighbor, forwarding traffic over specific paths, etc.) In this paper we propose an enhancement of BGP route selection and advertisement processes, and demonstrate how the enhanced BGP can be used to flexibly implement diverse policies, regardless of the network topology, that were not feasible before. <s> BIB006 </s> A Survey on Efforts to Evolve the Control Plane of Inter-Domain Routing <s> Incremental Improvement <s> Utilizing the dormant path diversity through multipath routing in the Internet to reach end users—thereby fulfilling their QoS requirements—is rather logical. While offering better resource utilization, better reliability, and often even much better quality of experience (QoE), multipath routing and provisioning was shown to help network and data center operators achieve traffic engineering in the form of load balancing. In this survey, we first highlight the benefits and basic Internet multipath routing components. We take a top-down approach and review various multipath protocols, from application to link and physical layers, operating at different parts of the Internet. We also describe the mathematical foundations of the multipath operation, as well as highlight the issues and challenges pertaining to reliable data delivery, buffering, and security in deploying multipath provisioning in the Internet. We compare the benefits and drawbacks of these protocols operating at different Internet layers and discuss open issues and challenges. <s> BIB007 </s> A Survey on Efforts to Evolve the Control Plane of Inter-Domain Routing <s> Incremental Improvement <s> The Border Gateway Protocol (BGP) was designed almost three decades ago and has many limitations relating to its fully distributed nature, policy enforcement capabilities, scalability, security and complexity. For example, the control plane can take several minutes to converge after a routing change; this may be unacceptable for real-time network services. Despite many research proposals for incremental improvements and clean-slate redesigns of how inter-domain routing should work, BGP is likely one of the most ossified protocols of the Internet architecture and it has not retrofitted the proposed ideas. In this work, we propose a radical, incrementally deployable Internet routing paradigm in which the control plane of multiple networks is logically centralized. This follows the Software Defined Networking (SDN) paradigm, although at the inter-domain level involving multiple Autonomous Systems (AS). Multi-domain SDN centralization can be realized by outsourcing routing functions to an external contractor, which provides inter-domain routing services facilitated through a multi-AS network controller. The proposed model promises to become a vehicle for evolving BGP and uses the bird's eye view over several networks to benefit aspects of inter-domain routing, such as convergence properties, policy conflict resolution, inter-domain troubleshooting, and collaborative security. In addition to the proposed paradigm, we introduce a publicly available emulation platform built on top of Mininet and the Quagga routing software, for experimenting on hybrid BGP-SDN AS-level networks. As a proof of concept, we focus specifically on exploiting multi-domain centralization to improve BGP's slow convergence. We build and make publicly available a first multi-AS controller tailored to this use case and demonstrate experimentally that SDN centralization helps to linearly reduce BGP convergence times and churn rates with expanding SDN deployments. <s> BIB008 </s> A Survey on Efforts to Evolve the Control Plane of Inter-Domain Routing <s> Incremental Improvement <s> Introduction. This paper presents the vision of the Control Exchange Point (CXP) architectural model. The model is motivated by the inflexibility and ossification of today’s inter-domain routing system, which renders critical QoS-constrained end-toend (e2e) network services difficult or simply impossible to provide. CXPs operate on slices of ISP networks and are built on basic Software Defined Networking (SDN) principles, such as the clean decoupling of the routing control plane from the data plane and the consequent logical centralization of control. The main goal of the architectural model is to provide e2e services with QoS constraints across domains. This is achieved through defining a new type of business relationship between ISPs, which advertise partial paths (so-called pathlets [7]) with specific properties, and the orchestrating role of the CXPs, which dynamically stitch them together and provision e2e QoS. Revenue from value-added services flows from the clients of the CXP to the ISPs participating in the service. The novelty of the approach is the combination of SDN programmability and dynamic path stitching techniques for inter-domain routing, which extends the value proposition of SDN over multiple domains. We first describe the challenges related to e2e service provision with the current inter-domain routing and peering model, and then continue with the benefits of our approach. Subsequently, we describe the CXP model in detail and report on an initial feasibility analysis. Motivation and Challenges. Complexity and ossification: The notorious complexity of the inter-domain routing system renders its management difficult and error-prone, leading to various inefficiencies such as suboptimal inter-domain paths. Indicatively, 60% of all Internet paths today are suffering from triangle inequality violations [9]. The current ossification of the system, hindering the introduction of new solutions, aggravates the problem further. Highly popular inter-domain services, such as highdefinition e2e real-time video streaming, already test the limits of the status quo, or are simply impossible. This is because such services require tight coordination along entire chains of ISPs demanding QoS provisioning. More advanced and mission-critical services, such as telemedical applications, are usually out of the question. <s> BIB009 </s> A Survey on Efforts to Evolve the Control Plane of Inter-Domain Routing <s> Incremental Improvement <s> Software-Defined Internet Exchange Points (SDXes) promise to significantly increase the flexibility and function of interdomain traffic delivery on the Internet. Unfortunately, current SDX designs cannot yet achieve the scale required for large Internet exchange points (IXPs), which can host hundreds of participants exchanging traffic for hundreds of thousands of prefixes. Existing platforms are indeed too slow and inefficient to operate at this scale, typically requiring minutes to compile policies and millions of forwarding rules in the data plane. ::: ::: We motivate, design, and implement iSDX, the first SDX architecture that can operate at the scale of the largest IXPs. We show that iSDX reduces both policy compilation time and forwarding table size by two orders of magnitude compared to current state-of-the-art SDX controllers. Our evaluation against a trace from one of the largest IXPs in the world found that iSDX can compile a realistic set of policies for 500 IXP participants in less than three seconds. Our public release of iSDX, complete with tutorials and documentation, is already spurring early adoption in operational networks. <s> BIB010 </s> A Survey on Efforts to Evolve the Control Plane of Inter-Domain Routing <s> Incremental Improvement <s> Smart devices equipped with multiple network interfaces are becoming commonplace. Nevertheless, even though multiple interfaces can be used to connect to the Internet, their capabilities have not been fully utilized yet because the default TCP/IP stack supports only a single interface for communication. This situation is now changing due to the emergence of multipath protocols on different network stack layers. For example, many IP level approaches have been proposed utilizing tunneling mechanisms for hiding multipath transmission from the transport protocols. Several working groups under IEEE and IETF are actively standardizing multipath transmission on the link layer and transport layer. Application level approaches enable multipath transmission capability by establishing multiple transport connections and distributing data over them. Given all these efforts, it is beneficial and timely to summarize the state-of-the-art, compare their pros and cons, and discuss about the future directions. To that end, we present a survey on multipath transmission and make several major contributions: 1) we present a complete taxonomy pertaining to multipath transmission, including link, network, transport, application, and cross layers; 2) we survey the state-of-the-art for each layer, investigate the problems that each layer aims to address, and make comprehensive assessment of the solutions; and 3) based on the comparison, we identify open issues and pinpoint future directions for multipath transmission research. <s> BIB011 </s> A Survey on Efforts to Evolve the Control Plane of Inter-Domain Routing <s> Incremental Improvement <s> The Border Gateway Protocol (BGP) is the current standard protocol for the inter-domain routing system. Although BGP makes easy to manipulate the output traffic by an Autonomous System (AS), it lacks other desired features such as convenient management of incoming traffic to an AS. Thus, we propose a framework to evolve the control plane routing system that is compatible with BGP (smooth transition), uses SDN technologies and can be used to management the incoming traffic of multihomed ASes. Based on our framework, a load balancing application was developed and applied to a scenario to depict the proof of concept of our framework. The results were compared and discussed with BGP AS-Path Prepending technique, shown an improvement of inbound traffic bandwidth utilization. <s> BIB012
Proposals are made to incorporate new BGP network capabilities by changing the protocol behavior, extensions of the BGP or additional control information that is used with the BGP protocol. Table 4 presents the works related to incremental improvements of inter-domain routing. For inter-domain routing, the authors in BIB001 claimed that the fully-distributed path selection computation in IP routers limits the capacity of individual ASs in terms of management scalability of path selection. Thus, they proposed a Routing Control Platform (RCP), a platform that avoids the complexity of fully-distributed path computation in the inter-domain routing system by centralizing routing control logic. Due to the architecture network abstraction designed by the RCP, it did not describe how, in fact, to explore the path diversity of inter-domain routing; hence, these criteria are not applicable to the RCP. However, the RCP is one important area of research and a precursor of SDN concepts, in which it is indicated that the control plane of a domain should be built in a centralized way BIB004 . In addition to centralized routing control and to enable multipath routing on the Internet (the BGP uses a single best path), Multi-path Interdomain ROuting (MIRO)  proposed the use of overlay networks and specific packet tags to enable multiple paths for inter-domain routing. The idea was to allow a new control plane logic where ASs negotiate alternative paths, as needed, from their ISPs. MIRO defaults to the single-path routing provided by the conventional BGP, but allows ASs to negotiate alternative paths as needed. This provides flexibility where required while remaining compatible with BGP. Compared to source routing, MIRO gives transit ASs more control over the flow of traffic in their networks. An idea to protect the inter-domain links against failures is the Resilient BGP (R-BGP)  . R-BGP produces failover paths to mitigate the unavailability of the best path selected from the BGP decision process. Thus, when a link fails, the failover path was already computed and ready to be used, avoiding the path exploration time for discovering new paths to networks affected by the failed link. Furthermore, to avoid unavailability when external links fail, Yet Another Multipath Routing Protocol (YAMR) [65] is a resilient solution that explores advertising additional paths that are not contained in the primary path (resulting from the BGP's best path algorithm). Thus, each alternative path receives a label that identifies the links the path needs to avoid. YAMR is deployed in a distributed way, where the control plane of YAMR can be implemented as an extension of the BGP protocol. The main contribution of the Selective Announcement Multi-Process (STAMP) protocol is executing several BGP instances inside the AS that is used to discover complementary paths BIB002 . The goal of those complementary paths is established paths that are not affected by the same set of network events. Thus, the STAMP requires minimal modification of the BGP process to become an operational solution and to achieve an improvement in routing stability compared with the standard BGP protocol. The integration between SDN technologies and BGP has already been investigated and implemented by some researchers BIB008 BIB009 . The integration with SDN solutions and BGP networks is vital to a practical deployment of SDN proposals. In fact, modern SDN controllers have mechanisms to exchange BGP information with BGP speaker components that behave as legacy routers. For example, the Open Network Foundation (ONF) ATRIUM project proposes a framework to support BGP by use of the ONOS controller  and Quagga  . Another widely-used SDN controller is the Ryu  support, interworking between OpenFlow and BGP, since a BGP function is installed in the Ryu SDN framework. The COntrol INbound traffic (COIN) framework BIB012 proposed to evolve the control plane routing system with the OpenFlow protocol. The COIN framework expanded the Ryu SDN controller to provide mechanisms for controlling inbound traffic from ISPs to its multi-homed ASs customers. Thus, the customers could manage how network traffic reaches them through the use of applications in the SDN controllers' ISPs. Those applications, when required and allowed, override the BGP behavior of the ISPs' network infrastructure to fulfil the customer's network traffic management requirements. With the focus on merging SDN and traditional network concepts, SIREN BIB008 presented a proposal to integrate the control plane of those two approaches. It combines BGP and SDN principles to improve the convergence time of the BGP at the inter-domain level. To reach that goal, SIREN allows an AS to outsource routing functionality and export it to its ISP control. SIREN is an extension of the previous idea explored in Kotronis et al. BIB003 . However, a pitfall of SIREN is that this approach limits the ability of the innovation of an AS, considering that a domain has to be subordinate to the requirements established by the outsourcing agreements for managing its network. Using the idea of overriding the BGP behavior, the proposal of Software-Defined Internet exchange (SDX) BIB005 and its extension Industrial-Scale Software-Defined Internet Exchange Point (iSDX) BIB010 explore the SDN centralized control inside the Internet Exchange Point (IXP). iSDX allows the reduction of the forwarding table size of OpenFlow devices used by the IXP participants, the creation of more flexible forwarding policies and the end-to-end enforcing of QoS. iSDX requires a brokerage system to establish multilateral peering; innovation is processed in the Layer 2 scheme (using Ethernet MAC addresses), and a third party orchestrates traffic between participants. Silva  presented a new architecture to manage network traffic in the inter-domain using OpenFlow networks. The idea of the proposed architecture was to use BGP and SDN technologies to provide new mechanisms that allow different ASs to coordinate how traffic should be handled between them using network applications inside SDN controllers. Thereby, a proof of concept scenario demonstrated that stub ASs can appropriately manage network traffic towards its domain by controlling the parameters of those network applications in its ISPs. The achieved results indicated the potential of the idea to apply different strategies for routing in the inter-domain environment. Summary: Works to incrementally improve the inter-domain routing try to add capabilities to the BGP control plane, such as exploring path diversity for resilience purposes or increasing the available bandwidth of a domain. One major limitation of the BGP that is tackled by related works with incremental improvement is searching for multipath solutions for inter-domain routing. Thus, instead of using only the one best route per prefix, the use of alternative routes has the potential to improve resilience against link failures, bandwidth availability and security BIB007 BIB006 BIB011 .