<commit_before>= Routing from Edge Load Balancers
{product-author}
{product-version]
:data-uri:
:icons:
:experimental:
:toc: macro
:toc-title:

toc::[]

== Overview
link:../architecture/core_concepts/pods_and_services.html#pods[Pods] inside of
an OpenShift cluster are only reachable via their IP addresses on the cluster
network. An edge load balancer can be used to accept traffic from outside
networks and proxy the traffic to pods inside the OpenShift cluster. In cases
where the load balancer is not part of the cluster network, routing becomes a
hurdle as the internal cluster network is not accessible to the edge load
balancer.

To solve this problem where the OpenShift cluster is using
link:../architecture/additional_concepts/networking.html#openshift-sdn[OpenShift
SDN] as the cluster networking solution, there are two ways to achieve network
access to the pods.

== Including the Load Balancer in the SDN [[including-lb-in-sdn]]
If possible, run an OpenShift node instance on the load balancer itself that
uses OpenShift SDN as the networking plug-in. This way, the edge machine gets
its own Open vSwitch bridge that the SDN automatically configures to provide
access to the pods and nodes that reside in the cluster. The _routing table_ is
dynamically configured by the SDN as pods are created and deleted, and thus the
routing software is able to reach the pods.

Mark the load balancer machine as an
link:manage_nodes.html#marking-nodes-as-unschedulable-or-schedulable[unschedulable
node] so that no pods end up on the load balancer itself:

[options="nowrap"]
----
$ oadm manage-node <load_balancer_hostname> --schedulable=false
----

If the load balancer comes packaged as a Docker container, then it is even
easier to integrate with OpenShift: Simply run the load balancer as a pod with
the link:../rest_api/kubernetes_v1.html#v1-containerport[host port exposed]. The
pre-packaged link:../architecture/core_concepts/routes.html#routers[HAProxy
router] in OpenShift runs in precisely this fashion.

== Establishing a Tunnel Using a Ramp Node
In some cases, the previous solution is not possible. For example, an *F5
BIG-IP®* host cannot run an OpenShift node instance or the OpenShift SDN because
*F5®* uses a custom, incompatible Linux kernel and distribution.

Instead, to enable *F5 BIG-IP®* to reach pods, you can choose an existing node
within the cluster network as a _ramp node_ and establish a tunnel between the
*F5 BIG-IP®* host and the designated ramp node. Because it is otherwise an
ordinary OpenShift node, the ramp node has the necessary configuration to route
traffic to any pod on any node in the cluster network.  The ramp node thus
assumes the role of a gateway through which the *F5 BIG-IP®* host has access to
the entire cluster network.

Following is an example of establishing an *ipip* tunnel between an *F5 BIG-IP®*
host and a designated ramp node.

**On the F5 BIG-IP® host:**

. Set the following variables:
+
====
----
# F5_IP=10.3.89.66 <1>
# RAMP_IP=10.3.89.89 <1>
# TUNNEL_IP1=10.3.91.216 <2>
# CLUSTER_NETWORK=10.1.0.0/16 <3>
----
<1> The `*F5_IP*` and `*RAMP_IP*` variables refer to the *F5 BIG-IP®* host's and
the ramp node's IP addresses, respectively, on a shared, internal network.
<2> An arbitrary, non-conflicting IP address for the *F5®* host's end of the
*ipip* tunnel.
<3> The overlay network CIDR that the OpenShift SDN uses to assign addresses to
pods.
====

. Delete any old route, self, tunnel and SNAT pool:
+
====
----
# tmsh delete net route $CLUSTER_NETWORK || true
# tmsh delete net self SDN || true
# tmsh delete net tunnels tunnel SDN || true
# tmsh delete ltm snatpool SDN_snatpool || true
----
====

. Create the new tunnel, self, route and SNAT pool and use the SNAT pool
  in the virtual servers:
+
====
[options="nowrap"]
----
# tmsh create net tunnels tunnel SDN \
    \{ description "OpenShift SDN" local-address \
    $F5_IP profile ipip remote-address $RAMP_IP \}
# tmsh create net self SDN \{ address \
    ${TUNNEL_IP1}/24 allow-service all vlan SDN \}
# tmsh create net route $CLUSTER_NETWORK interface SDN
# tmsh create ltm snatpool SDN_snatpool members add { $TUNNEL_IP1 }
# tmsh modify ltm virtual  ose-vserver source-address-translation { type snat pool SDN_snatpool }
# tmsh modify ltm virtual  https-ose-vserver source-address-translation { type snat pool SDN_snatpool }
----
====

**On the ramp node:**

. Set the following variables:
+
====
----
# F5_IP=10.3.89.66
# TUNNEL_IP1=10.3.91.216
# TUNNEL_IP2=10.3.91.217 <1>
----
<1> A second, arbitrary IP address for the ramp node's end of the *ipip*
tunnel.
====

. Delete any old tunnel:
+
====
----
# ip tunnel del tun1 || true
----
====

. Create the *ipip* tunnel on the ramp node, using a suitable L2-connected
interface (e.g., *eth0*):
+
====
[options="nowrap"]
----
# ip tunnel add tun1 mode ipip \
    remote $F5_IP dev eth0
# ip addr add $TUNNEL_IP2 dev tun1
# ip link set tun1 up
# ip route add $TUNNEL_IP1 dev tun1
# ping -c 5 $TUNNEL_IP1
----
====

. SNAT the tunnel IP with an unused IP from the SDN subnet:
+
====
[options="nowrap"]
----
# source /etc/origin-sdn/config.env
# subaddr=$(echo $OPENSHIFT_SDN_TAP1_ADDR | cut -d "." -f 1,2,3)
# export RAMP_SDN_IP=${subaddr}.254
----
====

. Assign this `*RAMP_SDN_IP*` as an additional address to *tun0* (the local SDN's
gateway):
+
====
----
# ip addr add ${RAMP_SDN_IP} dev tun0
----
====

. Modify the OVS rules for SNAT:
+
====
[options="nowrap"]
----
# ovs-ofctl -O OpenFlow13 add-flow br0 \
    "cookie=0x999,ip,nw_src=${TUNNEL_IP1},actions=mod_nw_src:${RAMP_SDN_IP},resubmit(,0)"
# ovs-ofctl -O OpenFlow13 add-flow br0 \
    "cookie=0x999,ip,nw_dst=${RAMP_SDN_IP},actions=mod_nw_dst:${TUNNEL_IP1},resubmit(,0)"
# ovs-ofctl -O OpenFlow13 add-flow br0 \
    "cookie=0x999, table=0, arp, arp_tpa=${RAMP_SDN_IP}, actions=output:2"
----
====

. Mark the ramp node as an unschedulable node so that no pods end up on the ramp
node itself:
+
====
[options="nowrap"]
----
$ oadm manage-node <ramp_node_hostname> --schedulable=false
----
====

=== Configuring a Highly-Available Ramp Node
You can use OpenShift's *ipfailover* feature, which uses *keepalived*
internally, to make the ramp node highly available from *F5 BIG-IP®*'s point of
view. To do so, first bring up two nodes, for example called *ramp-node-1* and
*ramp-node-2*, on the same L2 subnet.
<commit_msg>Update instructions specific to multi-tenant plugin.<commit_after>= Routing from Edge Load Balancers
{product-author}
{product-version]
:data-uri:
:icons:
:experimental:
:toc: macro
:toc-title:

toc::[]

== Overview
link:../architecture/core_concepts/pods_and_services.html#pods[Pods] inside of
an OpenShift cluster are only reachable via their IP addresses on the cluster
network. An edge load balancer can be used to accept traffic from outside
networks and proxy the traffic to pods inside the OpenShift cluster. In cases
where the load balancer is not part of the cluster network, routing becomes a
hurdle as the internal cluster network is not accessible to the edge load
balancer.

To solve this problem where the OpenShift cluster is using
link:../architecture/additional_concepts/networking.html#openshift-sdn[OpenShift
SDN] as the cluster networking solution, there are two ways to achieve network
access to the pods.

== Including the Load Balancer in the SDN [[including-lb-in-sdn]]
If possible, run an OpenShift node instance on the load balancer itself that
uses OpenShift SDN as the networking plug-in. This way, the edge machine gets
its own Open vSwitch bridge that the SDN automatically configures to provide
access to the pods and nodes that reside in the cluster. The _routing table_ is
dynamically configured by the SDN as pods are created and deleted, and thus the
routing software is able to reach the pods.

Mark the load balancer machine as an
link:manage_nodes.html#marking-nodes-as-unschedulable-or-schedulable[unschedulable
node] so that no pods end up on the load balancer itself:

[options="nowrap"]
----
$ oadm manage-node <load_balancer_hostname> --schedulable=false
----

If the load balancer comes packaged as a Docker container, then it is even
easier to integrate with OpenShift: Simply run the load balancer as a pod with
the link:../rest_api/kubernetes_v1.html#v1-containerport[host port exposed]. The
pre-packaged link:../architecture/core_concepts/routes.html#routers[HAProxy
router] in OpenShift runs in precisely this fashion.

== Establishing a Tunnel Using a Ramp Node
In some cases, the previous solution is not possible. For example, an *F5
BIG-IP®* host cannot run an OpenShift node instance or the OpenShift SDN because
*F5®* uses a custom, incompatible Linux kernel and distribution.

Instead, to enable *F5 BIG-IP®* to reach pods, you can choose an existing node
within the cluster network as a _ramp node_ and establish a tunnel between the
*F5 BIG-IP®* host and the designated ramp node. Because it is otherwise an
ordinary OpenShift node, the ramp node has the necessary configuration to route
traffic to any pod on any node in the cluster network.  The ramp node thus
assumes the role of a gateway through which the *F5 BIG-IP®* host has access to
the entire cluster network.

Following is an example of establishing an *ipip* tunnel between an *F5 BIG-IP®*
host and a designated ramp node.

**On the F5 BIG-IP® host:**

. Set the following variables:
+
====
----
# F5_IP=10.3.89.66 <1>
# RAMP_IP=10.3.89.89 <1>
# TUNNEL_IP1=10.3.91.216 <2>
# CLUSTER_NETWORK=10.1.0.0/16 <3>
----
<1> The `*F5_IP*` and `*RAMP_IP*` variables refer to the *F5 BIG-IP®* host's and
the ramp node's IP addresses, respectively, on a shared, internal network.
<2> An arbitrary, non-conflicting IP address for the *F5®* host's end of the
*ipip* tunnel.
<3> The overlay network CIDR that the OpenShift SDN uses to assign addresses to
pods.
====

. Delete any old route, self, tunnel and SNAT pool:
+
====
----
# tmsh delete net route $CLUSTER_NETWORK || true
# tmsh delete net self SDN || true
# tmsh delete net tunnels tunnel SDN || true
# tmsh delete ltm snatpool SDN_snatpool || true
----
====

. Create the new tunnel, self, route and SNAT pool and use the SNAT pool
  in the virtual servers:
+
====
[options="nowrap"]
----
# tmsh create net tunnels tunnel SDN \
    \{ description "OpenShift SDN" local-address \
    $F5_IP profile ipip remote-address $RAMP_IP \}
# tmsh create net self SDN \{ address \
    ${TUNNEL_IP1}/24 allow-service all vlan SDN \}
# tmsh create net route $CLUSTER_NETWORK interface SDN
# tmsh create ltm snatpool SDN_snatpool members add { $TUNNEL_IP1 }
# tmsh modify ltm virtual  ose-vserver source-address-translation { type snat pool SDN_snatpool }
# tmsh modify ltm virtual  https-ose-vserver source-address-translation { type snat pool SDN_snatpool }
----
====

**On the ramp node:**

. Set the following variables:
+
====
----
# F5_IP=10.3.89.66
# TUNNEL_IP1=10.3.91.216
# TUNNEL_IP2=10.3.91.217 <1>
----
<1> A second, arbitrary IP address for the ramp node's end of the *ipip*
tunnel.
====

. Delete any old tunnel:
+
====
----
# ip tunnel del tun1 || true
----
====

. Create the *ipip* tunnel on the ramp node, using a suitable L2-connected
interface (e.g., *eth0*):
+
====
[options="nowrap"]
----
# ip tunnel add tun1 mode ipip \
    remote $F5_IP dev eth0
# ip addr add $TUNNEL_IP2 dev tun1
# ip link set tun1 up
# ip route add $TUNNEL_IP1 dev tun1
# ping -c 5 $TUNNEL_IP1
----
====

. SNAT the tunnel IP with an unused IP from the SDN subnet:
+
====
[options="nowrap"]
----
# source /etc/openshift-sdn/config.env
# tap1=$(ip -o -4 addr list tun0 | awk '{print $4}' | cut -d/ -f1 | head -n 1)
# subaddr=$(echo ${OPENSHIFT_SDN_TAP1_ADDR:-"$tap1"} | cut -d "." -f 1,2,3)
# export RAMP_SDN_IP=${subaddr}.254
----
====

. Assign this `*RAMP_SDN_IP*` as an additional address to *tun0* (the local SDN's
gateway):
+
====
----
# ip addr add ${RAMP_SDN_IP} dev tun0
----
====

. Modify the OVS rules for SNAT:
+
====
[options="nowrap"]
----
# # set plugin based on the openshift-sdn plugin used in your cluster.
# plugin="sdn"  #  "multitenant"
#
# ipflowopts="cookie=0x999,ip"
# [ "$plugin" == "multitenant" ] && ipflowopts="cookie=0x999,table=1,priority=40000,ip"
#
# arpflowopts="cookie=0x999, table=0, arp"
# [ "$plugin" == "multitenant" ] && arpflowopts="cookie=0x999, table=1,priority=40000, arp"
#
# ovs-ofctl -O OpenFlow13 add-flow br0 \
    "${ipflowopts},nw_src=${TUNNEL_IP1},actions=mod_nw_src:${RAMP_SDN_IP},resubmit(,0)"
# ovs-ofctl -O OpenFlow13 add-flow br0 \
    "${ipflowopts},nw_dst=${RAMP_SDN_IP},actions=mod_nw_dst:${TUNNEL_IP1},resubmit(,0)"
# ovs-ofctl -O OpenFlow13 add-flow br0 \
    "${arpflowopts}, arp_tpa=${RAMP_SDN_IP}, actions=output:2"
# if [ "$plugin" == "multitenant" ]; then
    ovs-ofctl -O OpenFlow13 add-flow br0 \
      "cookie=0x999, table=1,priority=40000,ip,nw_dst=${TUNNEL_IP1},actions=output:2"
  fi
----
====

. Mark the ramp node as an unschedulable node so that no pods end up on the ramp
node itself:
+
====
[options="nowrap"]
----
$ oadm manage-node <ramp_node_hostname> --schedulable=false
----
====

=== Configuring a Highly-Available Ramp Node
You can use OpenShift's *ipfailover* feature, which uses *keepalived*
internally, to make the ramp node highly available from *F5 BIG-IP®*'s point of
view. To do so, first bring up two nodes, for example called *ramp-node-1* and
*ramp-node-2*, on the same L2 subnet.

{'fraction_non_alphanumeric': 0.11534438966083968, 'fraction_numerical': 0.023158523830868073, 'mean_word_length': 4.138963531669866, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 14, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1873040', 'n_tokens_mistral': 4657, 'n_tokens_neox': 4500, 'n_words': 1703}