<commit_before>---
title: Container-to-Container Networking
owner: Container-to-Container Networking
---

This topic provides an overview of how container-to-container networking works in <%= vars.app_runtime_full %> (<%= vars.app_runtime_abbr %>).

<p class="note"><strong>Note:</strong> Container-to-container networking is not available for apps hosted on Windows.</p>

The container-to-container networking feature enables app instances to communicate with each other directly.
<%= vars.cf_networking %>


## <a id="architecture"></a> Architecture

### <a id='overview'></a> Overview

Container-to-container networking integrates with [Garden-runC] in a Diego deployment. The CF Networking Release includes several core components, as well as swappable components. For more information about Garden-runC, see [Garden-runC](./architecture/garden.html#garden-runc) in _Garden_. For more information about Diego, see [Diego Components and Architecture](./diego/diego-architecture.html). For more information about the CF Networking release, see [CF Networking Release](https://github.com/cloudfoundry-incubator/cf-networking-release) on GitHub.

To understand the components and how they work, see the diagram and tables below. The diagram highlights <%= vars.app_runtime_abbr %> components in blue and green. The diagram also highlights swappable components in red.

![c2c architecture diagram](./images/c2c-arch.png)

### <a id='core'></a> Core Components

The container-to-container networking BOSH release includes the following core components:

<table>
  <tr>
    <th>Part</th>
    <th>Function</th>
  </tr>
  <tr>
    <td>Policy Server</td>
    <td>A central management node that:
      <ul>
        <li>Maintains a database of policies for traffic between apps. The cf CLI calls an API to create or update a record in the policy database whenever you create or remove a policy.</li>
        <li>Exposes a JSON REST API used by the cf CLI. This API serves traffic using TLS.</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Garden External Networker</td>
    <td>
      A Garden-runC add-on deployed to every Diego Cell that:
      <ul>
        <li>Invokes the CNI plugin component to set up the network for each app.</li>
        <li>Forwards ports to support incoming connections from the Gorouter, TCP Router, and Diego SSH Proxy. This keeps apps externally reachable.</li>
        <li>Installs outbound allow list rules to support Application Security Groups (ASGs).</li>
      </ul>
    </td>
  </tr>
</table>

### <a id='swap'></a> Swappable Components

<table>
  <tr>
    <th>Part</th>
    <th>Function</th>
  </tr>
  <tr>
    <td>Silk CNI plugin<br></td>
    <td>A plugin that provides IP address management and network connectivity to app instances as follows:
      <ul>
        <li>Uses a shared VXLAN overlay network to assign each container a unique IP address.</li>
        <li>Installs network interface in container using the Silk VXLAN back end. This is a shared, flat L3 network.</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>VXLAN Policy Agent<br></td>
    <td>Enforces network policy for traffic between apps as follows:
      <ul>
        <li>Discovers desired network policies from the Policy Server Internal API.</li>
        <li>Updates iptables rules on the Diego Cell to allow approved inbound traffic.</li>
        <li>Tags outbound traffic with the unique identifier of the source app using the VXLAN Group-Based Policy (GBP) header.</li>
      </ul>
    </td>
  </tr>
</table>


## <a id='app-comm'></a> App Instance Communication

The diagram below illustrates how app instances communicate in a deployment with container-to-container networking enabled. In this example, the operator creates two policies to regulate the flow of traffic between **App A**, **App B**, and **App C**.

* Allow traffic from **App A** to **App B**
* Allow traffic from **App A** to **App C**

If traffic and its direction is not explicitly allowed, it is denied. For example, **App B** cannot send traffic to **App C**.

![Post Container-to-Container Networking](./images/post-c2c.png)

### <a id='overlay-network'></a> Overlay Network

Container-to-container networking uses an overlay network to manage communication between app instances.

Overlay networks are not externally routable, and traffic sent between containers does not exit the overlay. You can use the same overlay network range for different <%= vars.app_runtime_abbr %> deployments in your environment.

The overlay network range defaults to `10.255.0.0/16`. You can modify the default to any [RFC&nbsp;1918](https://tools.ietf.org/html/rfc1918) range that meets the following requirements:

* The range is not used by services that app containers access.
* The range is not used by the underlying <%= vars.app_runtime_abbr %> infrastructure.

All Diego Cells in your <%= vars.app_runtime_abbr %> deployment share this overlay network. By default, each Diego Cell is allocated a /24 range that supports 254 containers per Diego Cell, one container for each of the usable IP addresses, `.1` through `.254`. To modify the number of Diego Cells your overlay network supports, see [Overlay Network](../devguide/deploy-apps/cf-networking.html#overlay) in _Configuring Container-to-Container Networking_.

<%= vars.app_runtime_abbr %> container networking is currently supported only on Linux.

<p class="note warning"><strong>Warning:</strong> The overlay network IP address range must not conflict with any other IP addresses in the network. If a conflict exists, Diego Cells cannot reach any endpoint that has a conflicting IP address.</p>

<p class="note"><strong>Note:</strong> Traffic to app containers from the Gorouter or from app containers to external services uses Diego Cell IP addresses and NAT, not the overlay network.</p>

### <a id="policies"></a> Policies

Enabling container-to-container networking for your deployment allows you to create policies for communication between app instances. The container-to-container networking feature also provides a unique IP address to each app container and provides direct IP reachability between app instances.

The policies you create specify a source app, destination app, protocol, and port so that app instances can communicate directly without going through the Gorouter, a load balancer, or a firewall. Container-to-container networking supports UDP and TCP, and you can configure policies for multiple ports. These policies apply immediately without having to restart the app.

Additionally, policies use and and track the GUIDs of the apps. The policies continue to work when apps redeploy, or if they crash and Diego places them in a new container. Pushing a brand new app requires a new policy, but not updates to an existing app because an app always retains its GUID.

<% if vars.platform_code.include? "CF" %>


## <a id='service-discovery'></a> App Service Discovery

The <%= vars.app_runtime_abbr %> platform supports DNS-based service discovery that lets apps find each other's internal addresses. For example, a front end app instance can use the service discovery mechanism to establish communications with a back end app instance. To set up and use app service discovery, see [App Service Discovery](../devguide/deploy-apps/cf-networking.html#discovery) in _Configuring Container-to-Container Networking_.

Container-to-container app service discovery does not provide client-side load balancing or circuit-breaking, and it does not apply to `cf marketplace` services or require app binding. It just lets apps publish service endpoints to each other, unbrokered and unmediated.

<% end %>


## <a id="alt-stack"></a> Alternative Network Stacks

The BOSH release that contains the container-to-container networking feature is composed of a pluggable network stack. Advanced users or third-party vendors can integrate a different network stack. For more information about third-party plugins, see [3rd Party Plugin Development for Container Networking](https://github.com/cloudfoundry-incubator/cf-networking-release/blob/develop/docs/3rd-party.md) in the CF Networking Release repository on GitHub.


## <a id="c2cvsasg"></a> Container-to-Container Networking versus ASGs

Both app security groups (ASGs) and container-to-container networking policies affect traffic from app instances. The following table highlights differences between ASGs and container-to-container networking policies.

<table>
  <tr>
   <th></th>
   <th>ASGs</th>
   <th>Container-to-Container Networking Policies</th>
  <tr>
  <tr>
   <th>Policy granularity</th>
   <td>From a space to an IP address range</td>
   <td>From a source app to a destination app</td>
  </tr>
  <tr>
   <th>Scope</th>
   <td>For a space, org, or deployment</td>
   <td>For app to app only</td>
  </tr>
  <tr>
   <th>Traffic direction</th>
   <td>Outbound control</td>
   <td>Policies apply for incoming packets from other app instances</td>
  </tr>
  <tr>
   <th>Source app</th>
   <td>Is not known</td>
   <td>Is identified because of direct addressability</td>
  </tr>
  <tr>
   <th>Policies take affect</th>
   <td>After app restart</td>
   <td>Immediately</td>
  </tr>
</table>

## <a id="securing-traffic"></a> Securing Container-to-Container Traffic

TLS encapsulation for container-to-container traffic is not enabled by default.

The platform allows apps to communicate through TLS if they implement their own TLS connection.

You can also use the IPsec Add-onto secure container-to-container traffic. The IPsec add-on enables
authentication and encryption at the IP layer to secure network communications between VMs, and
secures all traffic on the platform. 

For more information about the add-on, see [IPsec](https://docs.pivotal.io/addon-ipsec/index.html).

To download the add-on, see [IPsec Add-on](https://www.cloudfoundry.org/the-foundry/ipsec-add-on/).

<p class="note">For information about how public traffic to apps can be encrypted by the platform,
see <a href="http-routing.html#tls-to-back-end">TLS to Apps and Other Back End Services</a> in
<em>HTTP Routing</em>.
</p><commit_msg>[#172110774] fix typo and edit for clarity<commit_after>---
title: Container-to-Container Networking
owner: Container-to-Container Networking
---

This topic provides an overview of how container-to-container networking works in <%= vars.app_runtime_full %> (<%= vars.app_runtime_abbr %>).

<p class="note"><strong>Note:</strong> Container-to-container networking is not available for apps hosted on Windows.</p>

The container-to-container networking feature enables app instances to communicate with each other directly.
<%= vars.cf_networking %>


## <a id="architecture"></a> Architecture

### <a id='overview'></a> Overview

Container-to-container networking integrates with [Garden-runC] in a Diego deployment. The CF Networking Release includes several core components, as well as swappable components. For more information about Garden-runC, see [Garden-runC](./architecture/garden.html#garden-runc) in _Garden_. For more information about Diego, see [Diego Components and Architecture](./diego/diego-architecture.html). For more information about the CF Networking release, see [CF Networking Release](https://github.com/cloudfoundry-incubator/cf-networking-release) on GitHub.

To understand the components and how they work, see the diagram and tables below. The diagram highlights <%= vars.app_runtime_abbr %> components in blue and green. The diagram also highlights swappable components in red.

![c2c architecture diagram](./images/c2c-arch.png)

### <a id='core'></a> Core Components

The container-to-container networking BOSH release includes the following core components:

<table>
  <tr>
    <th>Part</th>
    <th>Function</th>
  </tr>
  <tr>
    <td>Policy Server</td>
    <td>A central management node that:
      <ul>
        <li>Maintains a database of policies for traffic between apps. The cf CLI calls an API to create or update a record in the policy database whenever you create or remove a policy.</li>
        <li>Exposes a JSON REST API used by the cf CLI. This API serves traffic using TLS.</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Garden External Networker</td>
    <td>
      A Garden-runC add-on deployed to every Diego Cell that:
      <ul>
        <li>Invokes the CNI plugin component to set up the network for each app.</li>
        <li>Forwards ports to support incoming connections from the Gorouter, TCP Router, and Diego SSH Proxy. This keeps apps externally reachable.</li>
        <li>Installs outbound allow list rules to support Application Security Groups (ASGs).</li>
      </ul>
    </td>
  </tr>
</table>

### <a id='swap'></a> Swappable Components

<table>
  <tr>
    <th>Part</th>
    <th>Function</th>
  </tr>
  <tr>
    <td>Silk CNI plugin<br></td>
    <td>A plugin that provides IP address management and network connectivity to app instances as follows:
      <ul>
        <li>Uses a shared VXLAN overlay network to assign each container a unique IP address.</li>
        <li>Installs network interface in container using the Silk VXLAN back end. This is a shared, flat L3 network.</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>VXLAN Policy Agent<br></td>
    <td>Enforces network policy for traffic between apps as follows:
      <ul>
        <li>Discovers desired network policies from the Policy Server Internal API.</li>
        <li>Updates iptables rules on the Diego Cell to allow approved inbound traffic.</li>
        <li>Tags outbound traffic with the unique identifier of the source app using the VXLAN Group-Based Policy (GBP) header.</li>
      </ul>
    </td>
  </tr>
</table>


## <a id='app-comm'></a> App Instance Communication

The diagram below illustrates how app instances communicate in a deployment with container-to-container networking enabled. In this example, the operator creates two policies to regulate the flow of traffic between **App A**, **App B**, and **App C**.

* Allow traffic from **App A** to **App B**
* Allow traffic from **App A** to **App C**

If traffic and its direction is not explicitly allowed, it is denied. For example, **App B** cannot send traffic to **App C**.

![Post Container-to-Container Networking](./images/post-c2c.png)

### <a id='overlay-network'></a> Overlay Network

Container-to-container networking uses an overlay network to manage communication between app instances.

Overlay networks are not externally routable, and traffic sent between containers does not exit the overlay. You can use the same overlay network range for different <%= vars.app_runtime_abbr %> deployments in your environment.

The overlay network range defaults to `10.255.0.0/16`. You can modify the default to any [RFC&nbsp;1918](https://tools.ietf.org/html/rfc1918) range that meets the following requirements:

* The range is not used by services that app containers access.
* The range is not used by the underlying <%= vars.app_runtime_abbr %> infrastructure.

All Diego Cells in your <%= vars.app_runtime_abbr %> deployment share this overlay network. By default, each Diego Cell is allocated a /24 range that supports 254 containers per Diego Cell, one container for each of the usable IP addresses, `.1` through `.254`. To modify the number of Diego Cells your overlay network supports, see [Overlay Network](../devguide/deploy-apps/cf-networking.html#overlay) in _Configuring Container-to-Container Networking_.

<%= vars.app_runtime_abbr %> container networking is currently supported only on Linux.

<p class="note warning"><strong>Warning:</strong> The overlay network IP address range must not conflict with any other IP addresses in the network. If a conflict exists, Diego Cells cannot reach any endpoint that has a conflicting IP address.</p>

<p class="note"><strong>Note:</strong> Traffic to app containers from the Gorouter or from app containers to external services uses Diego Cell IP addresses and NAT, not the overlay network.</p>

### <a id="policies"></a> Policies

Enabling container-to-container networking for your deployment allows you to create policies for communication between app instances. The container-to-container networking feature also provides a unique IP address to each app container and provides direct IP reachability between app instances.

The policies you create specify a source app, destination app, protocol, and port so that app instances can communicate directly without going through the Gorouter, a load balancer, or a firewall. Container-to-container networking supports UDP and TCP, and you can configure policies for multiple ports. These policies apply immediately without having to restart the app.

Additionally, policies use and and track the GUIDs of the apps. The policies continue to work when apps redeploy, or if they crash and Diego places them in a new container. Pushing a brand new app requires a new policy, but not updates to an existing app because an app always retains its GUID.

<% if vars.platform_code.include? "CF" %>


## <a id='service-discovery'></a> App Service Discovery

The <%= vars.app_runtime_abbr %> platform supports DNS-based service discovery that lets apps find each other's internal addresses. For example, a front end app instance can use the service discovery mechanism to establish communications with a back end app instance. To set up and use app service discovery, see [App Service Discovery](../devguide/deploy-apps/cf-networking.html#discovery) in _Configuring Container-to-Container Networking_.

Container-to-container app service discovery does not provide client-side load balancing or circuit-breaking, and it does not apply to `cf marketplace` services or require app binding. It just lets apps publish service endpoints to each other, unbrokered and unmediated.

<% end %>


## <a id="alt-stack"></a> Alternative Network Stacks

The BOSH release that contains the container-to-container networking feature is composed of a pluggable network stack. Advanced users or third-party vendors can integrate a different network stack. For more information about third-party plugins, see [3rd Party Plugin Development for Container Networking](https://github.com/cloudfoundry-incubator/cf-networking-release/blob/develop/docs/3rd-party.md) in the CF Networking Release repository on GitHub.


## <a id="c2cvsasg"></a> Container-to-Container Networking versus ASGs

Both app security groups (ASGs) and container-to-container networking policies affect traffic from app instances. The following table highlights differences between ASGs and container-to-container networking policies.

<table>
  <tr>
   <th></th>
   <th>ASGs</th>
   <th>Container-to-Container Networking Policies</th>
  <tr>
  <tr>
   <th>Policy granularity</th>
   <td>From a space to an IP address range</td>
   <td>From a source app to a destination app</td>
  </tr>
  <tr>
   <th>Scope</th>
   <td>For a space, org, or deployment</td>
   <td>For app to app only</td>
  </tr>
  <tr>
   <th>Traffic direction</th>
   <td>Outbound control</td>
   <td>Policies apply for incoming packets from other app instances</td>
  </tr>
  <tr>
   <th>Source app</th>
   <td>Is not known</td>
   <td>Is identified because of direct addressability</td>
  </tr>
  <tr>
   <th>Policies take affect</th>
   <td>After app restart</td>
   <td>Immediately</td>
  </tr>
</table>

## <a id="securing-traffic"></a> Securing Container-to-Container Traffic

TLS encapsulation for container-to-container traffic is not enabled by default.

The platform allows apps to communicate through TLS if they implement their own TLS connection.

You can also use the IPsec Add-on to secure container-to-container traffic. The IPsec add-on enables
authentication and encryption at the IP layer to secure network communications between VMs and to
secure all traffic on the platform.

For more information about the add-on, see [IPsec](https://docs.pivotal.io/addon-ipsec/index.html).

To download the add-on, see [IPsec Add-on](https://www.cloudfoundry.org/the-foundry/ipsec-add-on/).

<p class="note">For information about how the platform can encrypt public traffic to apps, see
<a href="http-routing.html#tls-to-back-end">TLS to Apps and Other Back End Services</a> in
<em>HTTP Routing</em>.
</p>
{'fraction_non_alphanumeric': 0.0934939283246125, 'fraction_numerical': 0.0037022410899397768, 'mean_word_length': 4.19728065674705, 'pattern_counts': {'":': 0, '<': 345, '<?xml version=': 0, '>': 345, 'https://': 10, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1993212', 'n_tokens_mistral': 5663, 'n_tokens_neox': 5418, 'n_words': 2469}