<commit_before>---
title: Understanding Container Security
owner: Security
---

<strong><%= modified_date %></strong>

This topic describes how Cloud Foundry (CF) secures the containers that host application instances on Linux. For an overview of other CF security features, see the [Understanding Cloud Foundry Security](security.html) topic.

* [Container Mechanics](#mechanics) provides an overview of container isolation. 
* [Inbound and Outbound Traffic from CF](#networking) provides an [overview](#net-overview) of container networking and describes how CF administrators [customize](#config-traffic) container network traffic rules for their deployment. 
* [Container Security](#security) describes how CF secures containers by running application instances in [unprivileged](#types) containers and by [hardening](#hardening) them.

## <a id='mechanics'></a>Container Mechanics###

Each instance of an app deployed to CF runs within its own self-contained environment, a [Garden container](./architecture/garden.html). This container isolates processes, memory, and the filesystem using operating system features and the characteristics of the virtual and physical infrastructure where CF is deployed. 

CF achieves container isolation by namespacing kernel resources that would otherwise be shared. The intended level of isolation is set to prevent multiple containers that are present on the same host from detecting each other. Every container includes a private root filesystem, which includes a Process ID (PID), namespace, network namespace, and mount namespace.

CF creates container filesystems using the [Garden Rootfs](./architecture/garden.html#garden-rootfs) (GrootFS) tool. It stacks the following using OverlayFS:

* A **read-only base filesystem**: This filesystem has the minimal set of operating system packages and Garden-specific modifications common to all containers.
Containers can share the same read-only base filesystem because all writes are applied to the read-write layer.
* A **container-specific read-write layer**: This layer is unique to each container and its size is limited by XFS project quotas.
The quotas prevent the read-write layer from overflowing into unallocated space.

Resource control is managed using Linux control groups. Associating each container with its own cgroup or job object limits the amount of memory that the container may use. Linux cgroups also require the container to use a fair share of CPU compared to the relative CPU share of other containers. 

<p class="note">
    <strong>Note</strong>: CF does not support a RedHat Enterprise Linux OS stemcell. This is due to an inherent security issue with the way RedHat handles user namespacing and container isolation.
</p>

### <a id='cpu'></a>CPU ###

Each container is placed in its own cgroup. Cgroups make each container use a 
fair share of CPU relative to the other containers. This prevents
oversubscription on the host VM where one or more containers hog the CPU and
leave no computing resources to the others.

The way cgroups allocate CPU time is based on shares. CPU shares do not work
as direct percentages of total CPU usage. Instead, a share is relative in
a given time window to the shares held by the other containers. The total
amount of CPU that can be overall divided among the cgroups is what is left
by other processes that may run in the host VM.

Generally, cgroups offers two possibilities for limiting the CPU usage:
<em>CPU affinity</em> and <em>CPU bandwidth</em>, the latter of which is used
in Cloud Foundry.

* CPU affinity: It consists of binding a cgroup to certain CPU cores. The actual amount of CPU cycles that can be consumed by the cgroup is thus limited to what is available on the bound CPU cores.

* CPU bandwidth: Sets the weight of a cgroup with the process scheduler. The process scheduler divides the available CPU cycles among cgroups depending on the shares held by each cgroup, relative to the shares held by the others.  
For example, consider two cgroups, one holding two shares and one holding four. Assuming the process scheduler gets to administer 60 CPU cycles, the first cgroup with two shares will get one third of those available CPU cycles, as it holds one third of the overall shares. Similarly, the second cgroup will get 40 cycles, as it holds two thirds of the collective shares.

The calculation of the CPU usage based on the percentage of the total CPU power
available is quite sophisticated and is performed regularly as the CPU demands
of the various containers fluctuates. Specifically, the percentage of CPU cycles
a cgroup gets can be calculated by dividing the <code>cpu.shares</code> it holds by the sum
of the <code>cpu.shares</code> of all the cgroups that are currently doing CPU activity:

<code>process\_cpu\_share\_percent = cpu.shares / sum\_cpu\_shares * 100</code>

In Cloud Foundry, cgroup shares range from 10 to 1024, with 1024 being the default.
The actual amount of shares a cgroup gets can be read from the 
<code>cpu.shares</code> file of the cgroup configurations pseudo-file-system
available in the container at <code>/sys/fs/cgroup/cpu</code>.

The amount of shares given to an applications cgroup depends on the amount of 
memory the application declares to need in the manifest. Cloud Foundry scales
the number of allocated shares linearly with the amount of memory, with an app
instance requesting 8G of memory getting the upper limit of 1024 shares.

<code>process\_cpu.shares = max( (application\_memory / 8), 1024)</code>

The next example helps to illustrate this better. Consider three processes. P1, P2 and P3, which are assigned <code>cpu.shares</code> of 5, 20 and 30, respectively.

P1 is active, while P2 and P3 require no CPU. Hence, P1 may use the whole CPU.
When P2 joins in and is doing some actual work (e.g. a request comes in),
the CPU share between P1 and P2 will be calculated as follows:

* P1 -> 5 / (5+20) = 0.2 = 20%
* P2 -> 20 / (5+20) = 0.8 = 80%<commit_msg>Fix inconsistency between process_cpu.shares formula and text description<commit_after>---
title: Understanding Container Security
owner: Security
---

<strong><%= modified_date %></strong>

This topic describes how Cloud Foundry (CF) secures the containers that host application instances on Linux. For an overview of other CF security features, see the [Understanding Cloud Foundry Security](security.html) topic.

* [Container Mechanics](#mechanics) provides an overview of container isolation. 
* [Inbound and Outbound Traffic from CF](#networking) provides an [overview](#net-overview) of container networking and describes how CF administrators [customize](#config-traffic) container network traffic rules for their deployment. 
* [Container Security](#security) describes how CF secures containers by running application instances in [unprivileged](#types) containers and by [hardening](#hardening) them.

## <a id='mechanics'></a>Container Mechanics###

Each instance of an app deployed to CF runs within its own self-contained environment, a [Garden container](./architecture/garden.html). This container isolates processes, memory, and the filesystem using operating system features and the characteristics of the virtual and physical infrastructure where CF is deployed. 

CF achieves container isolation by namespacing kernel resources that would otherwise be shared. The intended level of isolation is set to prevent multiple containers that are present on the same host from detecting each other. Every container includes a private root filesystem, which includes a Process ID (PID), namespace, network namespace, and mount namespace.

CF creates container filesystems using the [Garden Rootfs](./architecture/garden.html#garden-rootfs) (GrootFS) tool. It stacks the following using OverlayFS:

* A **read-only base filesystem**: This filesystem has the minimal set of operating system packages and Garden-specific modifications common to all containers.
Containers can share the same read-only base filesystem because all writes are applied to the read-write layer.
* A **container-specific read-write layer**: This layer is unique to each container and its size is limited by XFS project quotas.
The quotas prevent the read-write layer from overflowing into unallocated space.

Resource control is managed using Linux control groups. Associating each container with its own cgroup or job object limits the amount of memory that the container may use. Linux cgroups also require the container to use a fair share of CPU compared to the relative CPU share of other containers. 

<p class="note">
    <strong>Note</strong>: CF does not support a RedHat Enterprise Linux OS stemcell. This is due to an inherent security issue with the way RedHat handles user namespacing and container isolation.
</p>

### <a id='cpu'></a>CPU ###

Each container is placed in its own cgroup. Cgroups make each container use a 
fair share of CPU relative to the other containers. This prevents
oversubscription on the host VM where one or more containers hog the CPU and
leave no computing resources to the others.

The way cgroups allocate CPU time is based on shares. CPU shares do not work
as direct percentages of total CPU usage. Instead, a share is relative in
a given time window to the shares held by the other containers. The total
amount of CPU that can be overall divided among the cgroups is what is left
by other processes that may run in the host VM.

Generally, cgroups offers two possibilities for limiting the CPU usage:
<em>CPU affinity</em> and <em>CPU bandwidth</em>, the latter of which is used
in Cloud Foundry.

* CPU affinity: It consists of binding a cgroup to certain CPU cores. The actual amount of CPU cycles that can be consumed by the cgroup is thus limited to what is available on the bound CPU cores.

* CPU bandwidth: Sets the weight of a cgroup with the process scheduler. The process scheduler divides the available CPU cycles among cgroups depending on the shares held by each cgroup, relative to the shares held by the others.  
For example, consider two cgroups, one holding two shares and one holding four. Assuming the process scheduler gets to administer 60 CPU cycles, the first cgroup with two shares will get one third of those available CPU cycles, as it holds one third of the overall shares. Similarly, the second cgroup will get 40 cycles, as it holds two thirds of the collective shares.

The calculation of the CPU usage based on the percentage of the total CPU power
available is quite sophisticated and is performed regularly as the CPU demands
of the various containers fluctuates. Specifically, the percentage of CPU cycles
a cgroup gets can be calculated by dividing the <code>cpu.shares</code> it holds by the sum
of the <code>cpu.shares</code> of all the cgroups that are currently doing CPU activity:

<code>process\_cpu\_share\_percent = cpu.shares / sum\_cpu\_shares * 100</code>

In Cloud Foundry, cgroup shares range from 10 to 1024, with 1024 being the default.
The actual amount of shares a cgroup gets can be read from the 
<code>cpu.shares</code> file of the cgroup configurations pseudo-file-system
available in the container at <code>/sys/fs/cgroup/cpu</code>.

The amount of shares given to an applications cgroup depends on the amount of 
memory the application declares to need in the manifest. Cloud Foundry scales
the number of allocated shares linearly with the amount of memory, with an app
instance requesting 8G of memory getting the upper limit of 1024 shares.

<code>process\_cpu.shares = min( 1024*(application\_memory / 8 GB), 1024)</code>

The next example helps to illustrate this better. Consider three processes. P1, P2 and P3, which are assigned <code>cpu.shares</code> of 5, 20 and 30, respectively.

P1 is active, while P2 and P3 require no CPU. Hence, P1 may use the whole CPU.
When P2 joins in and is doing some actual work (e.g. a request comes in),
the CPU share between P1 and P2 will be calculated as follows:

* P1 -> 5 / (5+20) = 0.2 = 20%
* P2 -> 20 / (5+20) = 0.8 = 80%
{'fraction_non_alphanumeric': 0.055417288866766216, 'fraction_numerical': 0.010452961672473868, 'mean_word_length': 4.664943609022556, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 65, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2243623', 'n_tokens_mistral': 3051, 'n_tokens_neox': 2858, 'n_words': 1773}