<commit_before>.. only:: not (epub or latex or html)

    WARNING: You are looking at unreleased Cilium documentation.
    Please use the official rendered version released here:
    https://docs.cilium.io

.. _ipam_eni:

#######
AWS ENI
#######

The AWS ENI allocator is specific to Cilium deployments running in the AWS
cloud and performs IP allocation based on IPs of `AWS Elastic Network Interfaces (ENI)
<https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html>`__ by
communicating with the AWS EC2 API.

The architecture ensures that only a single operator communicates with the EC2
service API to avoid rate-limiting issues in large clusters. A pre-allocation
watermark is used to maintain a number of IP addresses to be available for use
on nodes at all time without needing to contact the EC2 API when a new pod is
scheduled in the cluster.

************
Architecture
************

.. image:: eni_arch.png
    :align: center

The AWS ENI allocator builds on top of the CRD-backed allocator. Each node
creates a ``ciliumnodes.cilium.io`` custom resource matching the node name when
Cilium starts up for the first time on that node. It contacts the EC2 metadata
API to retrieve the instance ID, instance type, and VPC information, then it
populates the custom resource with this information. ENI allocation parameters
are provided as agent configuration option and are passed into the custom
resource as well.

The Cilium operator listens for new ``ciliumnodes.cilium.io`` custom resources
and starts managing the IPAM aspect automatically. It scans the EC2 instances
for existing ENIs with associated IPs and makes them available via the
``spec.ipam.available`` field. It will then constantly monitor the used IP
addresses in the ``status.ipam.used`` field and automatically create ENIs and
allocate more IPs as needed to meet the IP pre-allocation watermark. This ensures
that there are always IPs available.

The selection of subnets to use for allocation as well as attachment of
security groups to new ENIs can be controlled separately for each node. This
makes it possible to hand out pod IPs with differing security groups on
individual nodes.

The corresponding datapath is described in section :ref:`aws_eni_datapath`.

*************
Configuration
*************

* The Cilium agent and operator must be run with the option ``--ipam=eni`` or
  the option ``ipam: eni``  must be set in the ConfigMap. This will enable ENI
  allocation in both the node agent and operator.

* In most scenarios, it makes sense to automatically create the
  ``ciliumnodes.cilium.io`` custom resource when the agent starts up on a node
  for the first time. To enable this, specify the option
  ``--auto-create-cilium-node-resource`` or  set
  ``auto-create-cilium-node-resource: "true"`` in the ConfigMap.

* If IPs are limited, run the Operator with option
  ``--aws-release-excess-ips=true``. When enabled, operator checks the number
  of IPs regularly and attempts to release excess free IPs from ENI.

* It is generally a good idea to enable metrics in the Operator as well with
  the option ``--enable-metrics``. See the section :ref:`install_metrics` for
  additional information how to install and run Prometheus including the
  Grafana dashboard.

Custom ENI Configuration
========================

Custom ENI configuration can be defined with a custom CNI configuration
``ConfigMap``:

Create a CNI configuration
--------------------------

Create a ``cni-config.yaml`` file based on the template below. Fill in the
``subnet-tags`` field, assuming that the subnets in AWS have the tags applied
to them:

.. code-block:: yaml

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cni-configuration
     namespace: kube-system
   data:
     cni-config: |-
       {
         "cniVersion":"0.3.1",
         "name":"cilium",<commit_msg>docs: Add short section about ENI GC<commit_after>.. only:: not (epub or latex or html)

    WARNING: You are looking at unreleased Cilium documentation.
    Please use the official rendered version released here:
    https://docs.cilium.io

.. _ipam_eni:

#######
AWS ENI
#######

The AWS ENI allocator is specific to Cilium deployments running in the AWS
cloud and performs IP allocation based on IPs of `AWS Elastic Network Interfaces (ENI)
<https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html>`__ by
communicating with the AWS EC2 API.

The architecture ensures that only a single operator communicates with the EC2
service API to avoid rate-limiting issues in large clusters. A pre-allocation
watermark is used to maintain a number of IP addresses to be available for use
on nodes at all time without needing to contact the EC2 API when a new pod is
scheduled in the cluster.

************
Architecture
************

.. image:: eni_arch.png
    :align: center

The AWS ENI allocator builds on top of the CRD-backed allocator. Each node
creates a ``ciliumnodes.cilium.io`` custom resource matching the node name when
Cilium starts up for the first time on that node. It contacts the EC2 metadata
API to retrieve the instance ID, instance type, and VPC information, then it
populates the custom resource with this information. ENI allocation parameters
are provided as agent configuration option and are passed into the custom
resource as well.

The Cilium operator listens for new ``ciliumnodes.cilium.io`` custom resources
and starts managing the IPAM aspect automatically. It scans the EC2 instances
for existing ENIs with associated IPs and makes them available via the
``spec.ipam.available`` field. It will then constantly monitor the used IP
addresses in the ``status.ipam.used`` field and automatically create ENIs and
allocate more IPs as needed to meet the IP pre-allocation watermark. This ensures
that there are always IPs available.

The selection of subnets to use for allocation as well as attachment of
security groups to new ENIs can be controlled separately for each node. This
makes it possible to hand out pod IPs with differing security groups on
individual nodes.

The corresponding datapath is described in section :ref:`aws_eni_datapath`.

*************
Configuration
*************

* The Cilium agent and operator must be run with the option ``--ipam=eni`` or
  the option ``ipam: eni``  must be set in the ConfigMap. This will enable ENI
  allocation in both the node agent and operator.

* In most scenarios, it makes sense to automatically create the
  ``ciliumnodes.cilium.io`` custom resource when the agent starts up on a node
  for the first time. To enable this, specify the option
  ``--auto-create-cilium-node-resource`` or  set
  ``auto-create-cilium-node-resource: "true"`` in the ConfigMap.

* If IPs are limited, run the Operator with option
  ``--aws-release-excess-ips=true``. When enabled, operator checks the number
  of IPs regularly and attempts to release excess free IPs from ENI.

* It is generally a good idea to enable metrics in the Operator as well with
  the option ``--enable-metrics``. See the section :ref:`install_metrics` for
  additional information how to install and run Prometheus including the
  Grafana dashboard.

* By default, ENIs will be tagged with the cluster name, to allow Cilium
  Operator to garbage collect these ENIs if left dangling. The cluster name is
  either extracted from Cilium's own ``cluster-name`` flag or from the
  ``aws:eks:cluster-name`` tag on the operator's EC2 instance. If neither
  cluster names are available, a static default cluster name is assumed and
  ENI garbage collection will be performed across all such unnamed clusters.
  You may override this behavior by setting a cluster-specific ``--eni-gc-tags``
  tag set.

Custom ENI Configuration
========================

Custom ENI configuration can be defined with a custom CNI configuration
``ConfigMap``:

Create a CNI configuration
--------------------------

Create a ``cni-config.yaml`` file based on the template below. Fill in the
``subnet-tags`` field, assuming that the subnets in AWS have the tags applied
to them:

.. code-block:: yaml

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cni-configuration
     namespace: kube-system
   data:
     cni-config: |-
       {
         "cniVersion":"0.3.1",
         "name":"cilium",
{'fraction_non_alphanumeric': 0.08484408992023205, 'fraction_numerical': 0.0025380710659898475, 'mean_word_length': 4.003022974607013, 'pattern_counts': {'":': 4, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2547246', 'n_tokens_mistral': 2215, 'n_tokens_neox': 2164, 'n_words': 1129}