<commit_before># Containerized Builds with Docker

The Buildkite Agent ships with built-in support for running your builds in Docker containers. Running your builds with Docker allows each project to define and document its testing environment, greatly simplifying your build servers, and provides build isolation when [parallelizing your build](parallelizing-builds).

<%= toc %>

## Creating a Docker Compose config

For Docker-based builds we recommend using [Docker Compose](https://docs.docker.com/compose/) as it allows each project to define it’s own `docker-compose.yml` with dependent containers and environment variables to be passed through.

Here's a example of a docker-compose.yml file for Ruby on Rails application that depends on Postgres, Redis and Memcache:

```yml
app:
  build: .
  volumes:
    - ./:/app
  links:
    - db:db
    - redis:redis
    - memcache:memcache
  environment:
    PGHOST: db
    PGUSER: postgres
    REDIS_URL: redis://redis
    MEMCACHE_SERVERS: memcache
db:
  image: postgres
redis:
  image: redis
memcache:
  image: tutum/memcached
```

**NOTE:** The `Dockerfile` must include a `WORKDIR` directive matching the mount point of the app. So you would use `WORKDIR /app` in the example above.

Mounting `.` (the directory of current build) as a volume in the container allows any changes from inside the container to be visible to the outside build agent, which is required if you want the agent to be able to access and upload artifacts created by the build.

Another example of a production docker-compose configuration is from the [buildkite-agent Golang source code](https://github.com/buildkite/agent):

```
app:
  build: .
  volumes:
    - ./:/go/src/github.com/buildkite/agent
    - /usr/bin/buildkite-agent:/usr/bin/buildkite-agent
  environment:
    - BUILDKITE_AGENT_ACCESS_TOKEN
    - BUILDKITE_JOB_ID
    - BUILDKITE_BUILD_ID
    - BUILDKITE_BUILD_NUMBER
    - GITHUB_RELEASE_ACCESS_TOKEN
    - DOCKER_HUB_TRIGGER_TOKEN
```

The above example has an additional volume: the `buildkite-agent` binary itself, so the build scripts can use the `buildkite-agent artifact` and `buildkite-agent meta-data` commands. It also passes through environment variables needed for the artifact command, as well as two secret keys set up by an [environment hook](/docs/agent/hooks).

## Testing the config

To test everything is set up correctly you can invoke the `docker-compose` command yourself on a development machine:

```bash
docker-compose run app script/tests
```

Once you've got everything working, you can configure your build pipeline and run it with Buildkite.

## Configuring the build step

Firstly we need to add the following environment variable to our build, either on an individual build step or as a project-level environment variable:

```bash
BUILDKITE_DOCKER_COMPOSE_CONTAINER=app
```

This will tell the agent to run each build step like so:

```bash
docker-compose -f docker-compose.buildkite.yml run app <command>
```

If you need, you can also specify an alternate Docker Compose config file:

```
BUILDKITE_DOCKER_COMPOSE_FILE=docker-compose.buildkite.yml
```

## Adding buildkite-agent to the docker group

To allow buildkite-agent to use the Docker client you’ll need to ensure its user has the necessary permissions. For most platforms this means adding the `buildkite-agent` user to your system’s `docker` group, and then restarting the Docker daemon. See your platform’s [Docker installation instructions](https://docs.docker.com/installation/) for more details.

## Adding a cleanup task

Even though the buildkite agent cleans up the images after each run, inevitably over time your Docker host can fill up with a lot of unused images.

We recommend running [docker-gc](https://github.com/spotify/docker-gc) or similar tool on a daily basis.

## Docker without Docker Compose

The Buildkite Agent also has support for running builds for projects that only need a single container and Dockerfile to run. To use plain Docker instead of Docker Compose, add the following environment variable in your build step configuration:

```
BUILDKITE_DOCKER=true
```

For each build job the agent will now build a Docker image and run the build step’s command inside a container.

If you need, you can also specify an alternate Dockerfile:

```
BUILDKITE_DOCKER_FILE=Dockerfile.ci
```

## Creating Your Own Docker Infrastructure

If your team has significant Docker experience you might find it worthwhile invoking your own runner scripts rather than using the simpler built-in Docker support.

To do this see the [agent hooks](/docs/agent/hooks) and [parallelizing builds](parallelizing-builds) documentation.
<commit_msg>Bring docker-compose syntax up to date<commit_after># Containerized Builds with Docker

The Buildkite Agent ships with built-in support for running your builds in Docker containers. Running your builds with Docker allows each project to define and document its testing environment, greatly simplifying your build servers, and provides build isolation when [parallelizing your build](parallelizing-builds).

<%= toc %>

## Creating a Docker Compose config

For Docker-based builds we recommend using [Docker Compose](https://docs.docker.com/compose/) as it allows each project to define it’s own `docker-compose.yml` with dependent containers and environment variables to be passed through.

Here's a example of a docker-compose.yml file for Ruby on Rails application that depends on Postgres, Redis and Memcache:

```yml
app:
  build: .
  volumes:
    - ./:/app
  links:
    - db
    - redis
    - memcache
  environment:
    PGHOST: db
    PGUSER: postgres
    REDIS_URL: redis://redis
    MEMCACHE_SERVERS: memcache
db:
  image: postgres
redis:
  image: redis
memcache:
  image: tutum/memcached
```

**NOTE:** The `Dockerfile` must include a `WORKDIR` directive matching the mount point of the app. So you would use `WORKDIR /app` in the example above.

Mounting `.` (the directory of current build) as a volume in the container allows any changes from inside the container to be visible to the outside build agent, which is required if you want the agent to be able to access and upload artifacts created by the build.

Another example of a production docker-compose configuration is from the [buildkite-agent Golang source code](https://github.com/buildkite/agent):

```
app:
  build: .
  volumes:
    - ./:/go/src/github.com/buildkite/agent
    - /usr/bin/buildkite-agent:/usr/bin/buildkite-agent
  environment:
    - BUILDKITE_AGENT_ACCESS_TOKEN: $BUILDKITE_AGENT_ACCESS_TOKEN
    - BUILDKITE_JOB_ID: $BUILDKITE_JOB_ID
    - BUILDKITE_BUILD_ID: $BUILDKITE_BUILD_ID
    - BUILDKITE_BUILD_NUMBER: $BUILDKITE_BUILD_NUMBER
    - GITHUB_RELEASE_ACCESS_TOKEN: $GITHUB_RELEASE_ACCESS_TOKEN
    - DOCKER_HUB_TRIGGER_TOKEN: $DOCKER_HUB_TRIGGER_TOKEN
```

The above example has an additional volume: the `buildkite-agent` binary itself, so the build scripts can use the `buildkite-agent artifact` and `buildkite-agent meta-data` commands. It also passes through environment variables needed for the artifact command, as well as two secret keys set up by an [environment hook](/docs/agent/hooks).

## Testing the config

To test everything is set up correctly you can invoke the `docker-compose` command yourself on a development machine:

```bash
docker-compose run app script/tests
```

Once you've got everything working, you can configure your build pipeline and run it with Buildkite.

## Configuring the build step

Firstly we need to add the following environment variable to our build, either on an individual build step or as a project-level environment variable:

```bash
BUILDKITE_DOCKER_COMPOSE_CONTAINER=app
```

This will tell the agent to run each build step like so:

```bash
docker-compose -f docker-compose.buildkite.yml run app <command>
```

If you need, you can also specify an alternate Docker Compose config file:

```
BUILDKITE_DOCKER_COMPOSE_FILE=docker-compose.buildkite.yml
```

## Adding buildkite-agent to the docker group

To allow buildkite-agent to use the Docker client you’ll need to ensure its user has the necessary permissions. For most platforms this means adding the `buildkite-agent` user to your system’s `docker` group, and then restarting the Docker daemon. See your platform’s [Docker installation instructions](https://docs.docker.com/installation/) for more details.

## Adding a cleanup task

Even though the buildkite agent cleans up the images after each run, inevitably over time your Docker host can fill up with a lot of unused images.

We recommend running [docker-gc](https://github.com/spotify/docker-gc) or similar tool on a daily basis.

## Docker without Docker Compose

The Buildkite Agent also has support for running builds for projects that only need a single container and Dockerfile to run. To use plain Docker instead of Docker Compose, add the following environment variable in your build step configuration:

```
BUILDKITE_DOCKER=true
```

For each build job the agent will now build a Docker image and run the build step’s command inside a container.

If you need, you can also specify an alternate Dockerfile:

```
BUILDKITE_DOCKER_FILE=Dockerfile.ci
```

## Creating Your Own Docker Infrastructure

If your team has significant Docker experience you might find it worthwhile invoking your own runner scripts rather than using the simpler built-in Docker support.

To do this see the [agent hooks](/docs/agent/hooks) and [parallelizing builds](parallelizing-builds) documentation.

{'fraction_non_alphanumeric': 0.06694648478488982, 'fraction_numerical': 0.0, 'mean_word_length': 4.487046632124352, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3741123', 'n_tokens_mistral': 2755, 'n_tokens_neox': 2594, 'n_words': 1189}