<commit_before># Parallelizing Builds

Running build jobs in parallel is a simple technique to decrease your build’s total running time. This guide will show you how to use Buildkite’s support for multiple agents and job parallelism to increase the speed of your builds.

<%= toc %>

## Running multiple agents

The Buildkite agent support the running of multiple instances on the one machine, as well as scaling your agents horizontally across any number of machines and networks.

Each platform’s [install instructions](/docs/agent/installation) has details for how to run multiple agents on a machine. In this case we'll use the [Ubuntu-based build agent](/docs/agent/ubuntu) to start 5 instances of the Buildkite agent:

```bash
# After running the standard install instructions...

sudo cp /etc/init/buildkite-agent.conf /etc/init/buildkite-agent-2.conf
sudo service buildkite-agent-2 start

sudo cp /etc/init/buildkite-agent.conf /etc/init/buildkite-agent-3.conf
sudo service buildkite-agent-3 start

sudo cp /etc/init/buildkite-agent.conf /etc/init/buildkite-agent-4.conf
sudo service buildkite-agent-4 start

sudo cp /etc/init/buildkite-agent.conf /etc/init/buildkite-agent-5.conf
sudo service buildkite-agent-5 start
```

You should now be able to see all 5 agents in the agent selector in your build pipeline editor:

<%= image 'agent-selector.png', size: '274x234', alt: 'Agent selector showing 5 agents' %>

## Job parallelism

To run the same step over all 5 of the agents we can set the parallelism field for a single build step:

<%= image 'parallelism.png', size: '555x138', alt: 'Step parallelism field' %>

Whilst we’re here we can also update the name of the step to include `%n`, which will automatically have the number substituted in and will help to differentiate between the different build jobs:

<%= image 'name.png', size: '555x80', alt: 'Step name field with %n' %>

Now the pipeline is configured we can create a new build:

<%= image 'build.png', size: '405x204', alt: 'The build' %>

If you inspect the first job’s environment variables you’ll find:

```
BUILDKITE_PARALLEL_JOB=0
BUILDKITE_PARALLEL_JOB_COUNT=5
```

The `BUILDKITE_PARALLEL_JOB` environment variable is the job number starting from 0, in this case each job’s value would be `0`, `1`, `2`, `3`, and `4`.

The `BUILDKITE_PARALLEL_JOB_COUNT` environment variable is the total number of jobs.

You can use these two environment variables to divide your application’s tests between the different jobs.

## Libraries

The following libraries have built-in support for the `BUILDKITE_PARALLEL_*` environment variables:

* [Knapsack](https://github.com/ArturT/knapsack)
<br>A ruby gem for that will automatically divide your tests between parallel jobs, as well as making sure each job runs in comparable time. Supports RSpec, Cucumber, Minitest, Spinach and Turnip.

* [Knapsack Pro](https://github.com/KnapsackPro/knapsack_pro-ruby)
<br>Is more advanced version of knapsack gem. Knapsack Pro has automated recording of tests time execution and can distribute tests across parallel jobs with predetermine split (regular mode) or in a dynamic way (queue mode) to get most optimal test suite split. You can learn more [how to configure project with Knapsack Pro and Buildkite](http://docs.knapsackpro.com/2017/auto-balancing-7-hours-tests-between-100-parallel-jobs-on-ci-buildkite-example) in the article.
<br>See an example repository of how to run Rails CI with [Knapsack Pro and test steps in parallel with Buildkite](https://github.com/KnapsackPro/buildkite-rails-parallel-example-with-knapsack_pro). Here is an example for [Docker, Knapsack Pro and Buildkite](https://github.com/KnapsackPro/buildkite-rails-docker-parallel-example-with-knapsack_pro).

## Isolated jobs

You can safely run multiple agents on a single machine, though you’ll still need to ensure your application’s code is able to be run in parallel. One convenient way of doing this is by using the agent’s [built in Docker and Docker Compose support](docker-containerized-builds) which will run each job in one or more completely isolated containers.

## Auto-scaling your build agents
<commit_msg>Simplify the Knapsack Pro docs<commit_after># Parallelizing Builds

Running build jobs in parallel is a simple technique to decrease your build’s total running time. This guide will show you how to use Buildkite’s support for multiple agents and job parallelism to increase the speed of your builds.

<%= toc %>

## Running multiple agents

The Buildkite agent support the running of multiple instances on the one machine, as well as scaling your agents horizontally across any number of machines and networks.

Each platform’s [install instructions](/docs/agent/installation) has details for how to run multiple agents on a machine. In this case we'll use the [Ubuntu-based build agent](/docs/agent/ubuntu) to start 5 instances of the Buildkite agent:

```bash
# After running the standard install instructions...

sudo cp /etc/init/buildkite-agent.conf /etc/init/buildkite-agent-2.conf
sudo service buildkite-agent-2 start

sudo cp /etc/init/buildkite-agent.conf /etc/init/buildkite-agent-3.conf
sudo service buildkite-agent-3 start

sudo cp /etc/init/buildkite-agent.conf /etc/init/buildkite-agent-4.conf
sudo service buildkite-agent-4 start

sudo cp /etc/init/buildkite-agent.conf /etc/init/buildkite-agent-5.conf
sudo service buildkite-agent-5 start
```

You should now be able to see all 5 agents in the agent selector in your build pipeline editor:

<%= image 'agent-selector.png', size: '274x234', alt: 'Agent selector showing 5 agents' %>

## Job parallelism

To run the same step over all 5 of the agents we can set the parallelism field for a single build step:

<%= image 'parallelism.png', size: '555x138', alt: 'Step parallelism field' %>

Whilst we’re here we can also update the name of the step to include `%n`, which will automatically have the number substituted in and will help to differentiate between the different build jobs:

<%= image 'name.png', size: '555x80', alt: 'Step name field with %n' %>

Now the pipeline is configured we can create a new build:

<%= image 'build.png', size: '405x204', alt: 'The build' %>

If you inspect the first job’s environment variables you’ll find:

```
BUILDKITE_PARALLEL_JOB=0
BUILDKITE_PARALLEL_JOB_COUNT=5
```

The `BUILDKITE_PARALLEL_JOB` environment variable is the job number starting from 0, in this case each job’s value would be `0`, `1`, `2`, `3`, and `4`.

The `BUILDKITE_PARALLEL_JOB_COUNT` environment variable is the total number of jobs.

You can use these two environment variables to divide your application’s tests between the different jobs.

## Libraries

The following libraries have built-in support for the `BUILDKITE_PARALLEL_*` environment variables:

* [Knapsack](https://github.com/ArturT/knapsack)
<br>A ruby gem for that will automatically divide your tests between parallel jobs, as well as making sure each job runs in comparable time. Supports RSpec, Cucumber, Minitest, Spinach and Turnip.

* [Knapsack Pro](https://github.com/KnapsackPro/knapsack_pro-ruby)
<br>A commercially supported version of Knapsack that provides a hosted service for test timing data and additional job distribution modes. See the [documentation](https://github.com/KnapsackPro/knapsack_pro-ruby#info-for-buildkitecom-users) and [step-by-step tutorial](http://docs.knapsackpro.com/2017/auto-balancing-7-hours-tests-between-100-parallel-jobs-on-ci-buildkite-example) for setup instructions and example pipelines.

## Isolated jobs

You can safely run multiple agents on a single machine, though you’ll still need to ensure your application’s code is able to be run in parallel. One convenient way of doing this is by using the agent’s [built in Docker and Docker Compose support](docker-containerized-builds) which will run each job in one or more completely isolated containers.

## Auto-scaling your build agents

{'fraction_non_alphanumeric': 0.08083606144547972, 'fraction_numerical': 0.012843112566104256, 'mean_word_length': 4.823313782991202, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6633432', 'n_tokens_mistral': 2407, 'n_tokens_neox': 2247, 'n_words': 998}