<commit_before>+++
title = "Setup Gitlab Runner with AWS ECR"
date = 2021-01-29T08:10:55+05:30
type = "post"
description = "How to setup Gitlab Runner with cross account AWS ECR access"
in_search_index = true
[taxonomies]
tags = ["Devops", "CI/CD"]
+++

There are some things you expect to just work. Sadly trying to make [Gitlab Runner](https://docs.gitlab.com/runner/) with [AWS ECR](https://aws.amazon.com/ecr/) turned out to be quite a daunting task and the little documentation in this area doesn't help. There's even a 4 years old [issue](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1583) and everyone there is echoing the sentiment that this is unnecessarily a lot harder than it should have been.

Anyway, since I spent a lot of time figuring out how to make a Private Registry work with a cross-account ECR, I'm documenting these steps hoping it'll help someone someday :).

### The Problem

There are mainly 2 seemingly same but different problems when it comes to using ECR. Let's discuss both of them separately:

- Pulling a private image from ECR using the Docker Executor. For eg, if your `gitlab-ci.yml` looks like:

```yml
test-pull:
  image: $PRIVATE_ECR_IMAGE
  script:
    - echo "Hello World!"
```

In this case, the Docker Executor needs to be "authenticated" to AWS ECR so that it can pull `$PRIVATE_ECR_IMAGE`.

- Pulling a private image _inside_ the job. For eg, if you're using Kaniko:

``` yaml
docker-build:
  image: $PRIVATE_ECR_KANIKO_IMAGE
  script:
    - |
      /kaniko ...
      # Inside this step, we use a PRIVATE_IMAGE defined in our `Dockerfile`.
```

In this case, [Kaniko](https://github.com/GoogleContainerTools/kaniko/) needs to be "authenticated" to AWS ECR so that it can pull `$PRIVATE_ECR_IMAGE`.

**NOTE** I prefer Kaniko over [DIND](https://docs.gitlab.com/ee/ci/docker/using_docker_build.html) as it is faster, doesn't require running the privileged container, caching is simplified, and is in general a lot simpler to setup.

### The Solution

So, for the first case, where you want to authenticate the Docker Executor to AWS ECR, you'll need 2 things:

1. Setup `DOCKER_AUTH_CONFIG` environment variable to  `{ "credsStore": "ecr-login" }` in the `config.toml` of the runner. For eg:

```
[[runners]]
  name = "Test"
  url = "https://gitlab.internal/"
  token = "<PASSWORD>"
  executor = "docker"
  environment = ["DOCKER_AUTH_CONFIG={ \"credsStore\": \"ecr-login\" }"]
```

2. Now, we've specified the Credential Store for Docker, but we don't have this binary `docker-credential-ecr-login` in our runner. AWS provides [amazon-ecr-credential-helper](https://github.com/awslabs/amazon-ecr-credential-helper) which is a neat way of automatically authenticating with AWS ECR based on your Access Keys/IAM role. What does automatic mean here? So, the normal `docker login` is a basic auth command, where if you've to log in to ECR, you need to do something like:

```sh
aws ecr get-login-password --region region | docker login --username AWS --password-stdin aws_account_id.dkr.ecr.region.amazonaws.com
```<commit_msg>chore: fix typos<commit_after>+++
title = "Setup Gitlab Runner with AWS ECR"
date = 2021-01-29T08:10:55+05:30
type = "post"
description = "How to setup Gitlab Runner with cross account AWS ECR access"
in_search_index = true
[taxonomies]
tags = ["Devops", "CI/CD"]
+++

There are some things you expect to just work. Sadly trying to make [Gitlab Runner](https://docs.gitlab.com/runner/) with [AWS ECR](https://aws.amazon.com/ecr/) turned out to be quite a daunting task and the little documentation in this area doesn't help. There's even a 4 years old [issue](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1583) and everyone there is echoing the sentiment that this is unnecessarily a lot harder than it should have been.

Anyway, since I spent a lot of time figuring out how to make a Private Registry work with a cross-account ECR, I'm documenting these steps hoping it'll help someone someday :).

### The Problem

There are mainly 2 seemingly same but different problems when it comes to using ECR. Let's discuss both of them separately:

- Pulling a private image from ECR using the Docker Executor. For eg, if your `gitlab-ci.yml` looks like:

```yml
test-pull:
  image: $PRIVATE_ECR_IMAGE
  script:
    - echo "Hello World!"
```

In this case, the Docker Executor needs to be "authenticated" to AWS ECR so that it can pull `$PRIVATE_ECR_IMAGE`.

- Pulling a private image _inside_ the job. For eg, if you're using Kaniko:

``` yaml
docker-build:
  image: gcr.io/kaniko-project/executor:debug
  script:
    - |
      /kaniko ...
      # Inside this step, we use a PRIVATE_ECR_IMAGE defined in our `Dockerfile`.
```

In this case, [Kaniko](https://github.com/GoogleContainerTools/kaniko/) needs to be "authenticated" to AWS ECR so that it can pull `$PRIVATE_ECR_IMAGE`.

**NOTE** I prefer Kaniko over [DIND](https://docs.gitlab.com/ee/ci/docker/using_docker_build.html) as it is faster, doesn't require running the privileged container, caching is simplified, and is in general a lot simpler to setup.

### The Solution

So, for the first case, where you want to authenticate the Docker Executor to AWS ECR, you'll need 2 things:

1. Setup `DOCKER_AUTH_CONFIG` environment variable to  `{ "credsStore": "ecr-login" }` in the `config.toml` of the runner. For eg:

```
[[runners]]
  name = "Test"
  url = "https://gitlab.internal/"
  token = "<PASSWORD>"
  executor = "docker"
  environment = ["DOCKER_AUTH_CONFIG={ \"credsStore\": \"ecr-login\" }"]
```

2. Now, we've specified the Credential Store for Docker, but we don't have this binary `docker-credential-ecr-login` in our runner. AWS provides [amazon-ecr-credential-helper](https://github.com/awslabs/amazon-ecr-credential-helper) which is a neat way of automatically authenticating with AWS ECR based on your Access Keys/IAM role. What does automatic mean here? So, the normal `docker login` is a basic auth command, where if you've to log in to ECR, you need to do something like:

```sh
aws ecr get-login-password --region region | docker login --username AWS --password-stdin aws_account_id.dkr.ecr.region.amazonaws.com
```
{'fraction_non_alphanumeric': 0.10551053484602918, 'fraction_numerical': 0.008752025931928687, 'mean_word_length': 4.190075693860387, 'pattern_counts': {'":': 4, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 14, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4768909', 'n_tokens_mistral': 2017, 'n_tokens_neox': 1868, 'n_words': 790}