<commit_before># Connecting to Upstream Networks using vRouter

A CORD POD needs to be connected to an upstream network to provide connectivity
from within CORD to the outside world. The vRouter service in CORD is designed
to enable the CORD POD to communicate with upstream routers and provide this
connectivity. Currently the vRouter supports BGP for communicating routes to
and from upstream routers.

Each deployment is different in terms of protocols and features required, so
this guide aims to be a general overview of how to set up CORD to communicate
with external routers. The operator will have to customize the configurations
to be appropriate for their deployment.

Deploying the vRouter infrastructure is a relatively manual process right now.
The intention is that over time the process will become more automated and
easier to perform.

## Prerequisites

This guide assumes that you have run through the [physical POD install
procedure](install_physical.md). You must also have installed and configured
the fabric ONOS cluster and connected the fabric switches to that controller.

## Physical Connectivity

External routers must be physically connected to one of the fabric leaf
switches. It is possible to connect to multiple routers, but currently there is
a limitation that they must all be physically connected to the same leaf
switch, and the Quagga instance must be connected to the same fabric switch as
the upstream routers.

![phys-connectivity](images/vrouter-connectivity.png)

## Dedicating a Fabric Interface for Quagga

The CORD build process determines which NICs on each compute node are connected
to the fabric and puts these NICs into a bonded interface. The name of this
bond is fabric, so if you run `ifconfig` on the compute node you have selected
to deploy Quagga, you should see this bonded interface appear in the output.

```
ubuntu@fumbling-reason:~$ ifconfig fabric
fabric    Link encap:Ethernet  HWaddr 00:02:c9:1e:b4:e0
          inet addr:10.6.1.2  Bcast:10.6.1.255  Mask:255.255.255.0
          inet6 addr: fe80::202:c9ff:fe1e:b4e0/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:1048256 errors:0 dropped:42 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:89101760 (89.1 MB)  TX bytes:0 (0.0 B)
```
          
We need to dedicate one of these fabric interfaces to the Quagga container, so
we'll need to remove it from the bond. You should first identify the name of
the interface that you want to dedicate. In this example we'll assume it is
called mlx1. You can then remove it from the bond by editing the
/etc/network/interfaces file:

```
sudo vi /etc/network/interfaces
```

You should see a stanza that looks like this:

```
auto mlx1
iface mlx1 inet manual
    bond-master fabric
```
    
Simply remove the line `bond-master fabric`, save the file then restart the
networking service on the compute node.

## L3 setup

The operator will need to allocate a subnet that exists between the CORD POD
and the upstream router that is used for peering between the CORD Quagga
instance and the upstream routers. CORD currently has a limitation that it
requires 2 IP addresses in the peering subnet, so for 1 upstream router we need
to allocate 3 addresses in total. This means the peering subnet cannot be
smaller than a /29.

The CORD fabric requires 2 IP addresses so that it can separate peering traffic
from data traffic. Peering happens using one of the IP addresses, and the
routes in the POD are advertised upstream with the next-hop set to the other IP
address. This means that when traffic comes to the fabric leaf switch from
outside, the switch is able to distinguish peering traffic from data traffic
and treat each appropriately.

An example of how this is configured is shown in the following figure:

![phys-connectivity](images/l3-connectivity.png)

In this case the peering subnet is `10.0.1.0/24`. The upstream router is using
the `10.0.1.1` address. The CORD Quagga is assigned `10.0.1.3`, which is the
address used for peering. The upstream router needs to be configured with
`10.0.1.3` as its BGP neighbor, and the BGP peering will be established between
`10.0.1.1` and `10.0.1.3`.

The `10.0.1.2` address is used by the fabric switch and for the next-hop for
routes advertised by the CORD POD.

Of course you are not obliged to use `10.0.1.0/24`, you should use a subnet
that makes sense for your peering environment

## Install and Configure vRouter on ONOS The vRouter will be run on the
`onos-fabric` cluster that controls the physical fabric switches. 

### Interface Configuration

Each Quagga-Router pair needs to have interface configuration for the
interfaces where the Quagga and upstream router are attached to the fabric.
This is where we configure the second IP address that we allocated from the
peering subnet. The following shows a configuration example:

```
{
    "ports" : {
        "of:0000000000000001/1" : {
            "interfaces" : [
                {
                    "name" : "upstream1",
                    "ips"  : [ "10.0.1.2/24" ],
                    "vlan-untagged" : 4000
                }
            ]
        },
        "of:0000000000000001/2" : {
            "interfaces" : [
                {
                    "name" : "quagga",
                    "ips"  : [ "10.0.1.2/24" ],
                    "vlan-untagged" : 4000
                }
            ]
        }
    }
}<commit_msg>Fix documentation bug.<commit_after># Connecting to Upstream Networks using vRouter

A CORD POD needs to be connected to an upstream network to provide connectivity
from within CORD to the outside world. The vRouter service in CORD is designed
to enable the CORD POD to communicate with upstream routers and provide this
connectivity. Currently the vRouter supports BGP for communicating routes to
and from upstream routers.

Each deployment is different in terms of protocols and features required, so
this guide aims to be a general overview of how to set up CORD to communicate
with external routers. The operator will have to customize the configurations
to be appropriate for their deployment.

Deploying the vRouter infrastructure is a relatively manual process right now.
The intention is that over time the process will become more automated and
easier to perform.

## Prerequisites

This guide assumes that you have run through the [physical POD install
procedure](install_physical.md). You must also have installed and configured
the fabric ONOS cluster and connected the fabric switches to that controller.

## Physical Connectivity

External routers must be physically connected to one of the fabric leaf
switches. It is possible to connect to multiple routers, but currently there is
a limitation that they must all be physically connected to the same leaf
switch, and the Quagga instance must be connected to the same fabric switch as
the upstream routers.

![phys-connectivity](images/vrouter-connectivity.png)

## Dedicating a Fabric Interface for Quagga

The CORD build process determines which NICs on each compute node are connected
to the fabric and puts these NICs into a bonded interface. The name of this
bond is fabric, so if you run `ifconfig` on the compute node you have selected
to deploy Quagga, you should see this bonded interface appear in the output.

```
ubuntu@fumbling-reason:~$ ifconfig fabric
fabric    Link encap:Ethernet  HWaddr 00:02:c9:1e:b4:e0
          inet addr:10.6.1.2  Bcast:10.6.1.255  Mask:255.255.255.0
          inet6 addr: fe80::202:c9ff:fe1e:b4e0/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:1048256 errors:0 dropped:42 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:89101760 (89.1 MB)  TX bytes:0 (0.0 B)
```
          
We need to dedicate one of these fabric interfaces to the Quagga container, so
we'll need to remove it from the bond. You should first identify the name of
the interface that you want to dedicate. In this example we'll assume it is
called mlx1. You can then remove it from the bond by editing the
/etc/network/interfaces file:

```
sudo vi /etc/network/interfaces
```

You should see a stanza that looks like this:

```
auto mlx1
iface mlx1 inet manual
    bond-master fabric
```
    
Simply remove the line `bond-master fabric`, save the file then restart the
networking service on the compute node.

## L3 setup

The operator will need to allocate a subnet that exists between the CORD POD
and the upstream router that is used for peering between the CORD Quagga
instance and the upstream routers. CORD currently has a limitation that it
requires 2 IP addresses in the peering subnet, so for 1 upstream router we need
to allocate 3 addresses in total. This means the peering subnet cannot be
smaller than a /29.

The CORD fabric requires 2 IP addresses so that it can separate peering traffic
from data traffic. Peering happens using one of the IP addresses, and the
routes in the POD are advertised upstream with the next-hop set to the other IP
address. This means that when traffic comes to the fabric leaf switch from
outside, the switch is able to distinguish peering traffic from data traffic
and treat each appropriately.

An example of how this is configured is shown in the following figure:

![phys-connectivity](images/l3-connectivity.png)

In this case the peering subnet is `10.0.1.0/24`. The upstream router is using
the `10.0.1.1` address. The CORD Quagga is assigned `10.0.1.3`, which is the
address used for peering. The upstream router needs to be configured with
`10.0.1.3` as its BGP neighbor, and the BGP peering will be established between
`10.0.1.1` and `10.0.1.3`.

The `10.0.1.2` address is used by the fabric switch and for the next-hop for
routes advertised by the CORD POD.

Of course you are not obliged to use `10.0.1.0/24`, you should use a subnet
that makes sense for your peering environment

## Install and Configure vRouter on ONOS

The vRouter will be run on the `onos-fabric` cluster that controls the physical 
fabric switches.

### Interface Configuration

Each Quagga-Router pair needs to have interface configuration for the
interfaces where the Quagga and upstream router are attached to the fabric.
This is where we configure the second IP address that we allocated from the
peering subnet. The following shows a configuration example:

```
{
    "ports" : {
        "of:0000000000000001/1" : {
            "interfaces" : [
                {
                    "name" : "upstream1",
                    "ips"  : [ "10.0.1.2/24" ],
                    "vlan-untagged" : 4000
                }
            ]
        },
        "of:0000000000000001/2" : {
            "interfaces" : [
                {
                    "name" : "quagga",
                    "ips"  : [ "10.0.1.2/24" ],
                    "vlan-untagged" : 4000
                }
            ]
        }
    }
}
{'fraction_non_alphanumeric': 0.05511455890312105, 'fraction_numerical': 0.03463828251849179, 'mean_word_length': 3.220403502093643, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 26, 'substrings_counts': 4, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5851567', 'n_tokens_mistral': 3286, 'n_tokens_neox': 2972, 'n_words': 1532}