<commit_before># Defining Your Pipeline Steps

You define your build pipeline steps in your pipeline settings or, more powerfully, via a `pipeline.yml` file.

Defining your pipeline steps in a `pipeline.yml` file gives you access to more configuration options than is available in the web interface, and allows you to version, audit and review your build pipelines alongside your source code.

<%= toc %>

## Getting started

In your pipeline settings, click the ➕ to add a step and choose “Read steps from repository”:

<%= image "pipeline-upload-step.png", width: 612/2, height: 484/2, alt: "Screenshot of the 'Read steps from repository' menu item" %>

Your pipeline steps will now show a new :pipeline: step that runs a `buildkite-agent pipeline upload` command:

<%= image "pipeline-upload.png", width: 1174/2, height: 452/2, alt: "Screenshot of the pipeline upload step" %>

When this step runs on an agent, it reads in the pipeline file from `.buildkite/pipeline.yml` in your repository. For example, here is a pipeline file with a single step that prints “Hello!” to the build log:

```yml
steps:
  - label: Example Test
    command: echo "Hello!"
```

To test your `.buildkite/pipeline.yml` file, commit the pipeline file and push it to your repository to create a new build:

<%= image "show-example-test.png", width: 968/2, height: 698/2, alt: "Screenshot of the build passing with pipeline upload step first, and then the example step" %>

## Example pipeline

Here’s a more complete example based on [our own Buildkite Agent’s build pipeline](https://github.com/buildkite/agent/blob/master/.buildkite/pipeline.yml). It contains script commands, wait steps, block steps, and artifact uploading:

```yaml
steps:
  - label: '\:hammer\: Tests'
    command: 'scripts/tests.sh'
    env:
      BUILDKITE_DOCKER_COMPOSE_CONTAINER: app

  - wait

  - label: '\:package\: Package'
    command: 'scripts/build-binaries.sh'
    artifact_paths: 'pkg/*'
    env:
      BUILDKITE_DOCKER_COMPOSE_CONTAINER: app

  - wait

  - label: '\:debian\: Publish'
    command: 'scripts/build-debian-packages.sh'
    artifact_paths: 'deb/**/*'
    branches: 'master'
    agents:
      queue: 'deploy'

  - block: '\:shipit\: Release'
    branches: 'master'

  - label: '\:github\: Release'
    command: 'scripts/build-github-release.sh'
    artifact_paths: 'releases/**/*'
    branches: 'master'

  - wait

  - label: '\:whale\: Update images'
    command: 'scripts/release-docker.sh'
    branches: 'master'
    agents:
      queue: 'deploy'
```

## Step types

There are four different step types you can use in your Buildkite pipelines. See each step type for full documentation:

* [Command Step](/docs/pipelines/command-step)
* [Wait Step](/docs/pipelines/wait-step)
* [Block Step](/docs/pipelines/block-step)
* [Trigger Step](/docs/pipelines/trigger-step)

## Customizing the pipeline file path

By default the pipeline upload step reads your pipeline definition from `.buildkite/pipeline.yml` in your repository. You can specify a different file path by adding it as the first argument:

<%= image "custom-pipeline-file.png", width: 1174/2, height: 452/2, alt: "Screenshot of a custom pipeline file upload" %>

A common use for custom file paths is when separating test and deployment steps into two completely separate pipelines, both using the same repository URL. For example, your test pipeline’s upload command could be:

```
buildkite-agent pipeline upload .buildkite/pipeline.yml
```

And your deployment pipeline’s upload command could be:

```
buildkite-agent pipeline upload .buildkite/pipeline.deploy.yml
```

For a list of all command line options, see the [buildkite-agent pipeline upload](/docs/agent/cli-pipeline#uploading-pipelines) documentation.

## Dynamic pipelines

Because the pipeline upload step runs on your agent machine, you can generate pipelines dynamically using scripts from your source code. This provides you with the flexiblity to structure your pipelines as you wish.

The following example generates a list of parallel test steps based upon the `test/*` directories within your repository:

```
#!/bin/bash

# exit immediately on failure, or if an undefined variable is used
set -eu

# begin the pipeline.yml file
echo "steps:"

# add a new command step to run the tests in each test directory
for test_dir in test/*/; do
  echo "  - command: \"run_tests "${test_dir}"\"
done
```

To use this script, you'd save it to `.buildkite/pipeline.sh` inside your repository, ensure it is executable, and then update your pipeline upload step to use the new script: 

```bash
.buildkite/pipeline.sh | buildkite-agent pipeline upload
```

When a build is run it will execute the script and pipe its output to the `pipeline upload` command, dynamically adding build steps to the end of the build pipeline.

## Migrating existing steps to a pipeline.yml

If you have a pipeline that is defined in the web interface and would like to migrate it to a pipeline.yml without any interuption to builds, you can add the following command step at the start of your pipeline:

```bash
[[ -f ".buildkite/pipeline.yml" ]] && buildkite-agent pipeline upload --replace || true
```

This line of code will check if the pipeline file exists and, if it does, execute the `buildkite agent pipeline upload --replace` command to replace any steps that were defined via the web interface with those defined in the `pipeline.yml`.

## Further documentation

See the [buildkite-agent pipeline documentation](/docs/agent/cli-pipeline) for a full list of options.<commit_msg>Fix typo: interuption => interruption<commit_after># Defining Your Pipeline Steps

You define your build pipeline steps in your pipeline settings or, more powerfully, via a `pipeline.yml` file.

Defining your pipeline steps in a `pipeline.yml` file gives you access to more configuration options than is available in the web interface, and allows you to version, audit and review your build pipelines alongside your source code.

<%= toc %>

## Getting started

In your pipeline settings, click the ➕ to add a step and choose “Read steps from repository”:

<%= image "pipeline-upload-step.png", width: 612/2, height: 484/2, alt: "Screenshot of the 'Read steps from repository' menu item" %>

Your pipeline steps will now show a new :pipeline: step that runs a `buildkite-agent pipeline upload` command:

<%= image "pipeline-upload.png", width: 1174/2, height: 452/2, alt: "Screenshot of the pipeline upload step" %>

When this step runs on an agent, it reads in the pipeline file from `.buildkite/pipeline.yml` in your repository. For example, here is a pipeline file with a single step that prints “Hello!” to the build log:

```yml
steps:
  - label: Example Test
    command: echo "Hello!"
```

To test your `.buildkite/pipeline.yml` file, commit the pipeline file and push it to your repository to create a new build:

<%= image "show-example-test.png", width: 968/2, height: 698/2, alt: "Screenshot of the build passing with pipeline upload step first, and then the example step" %>

## Example pipeline

Here’s a more complete example based on [our own Buildkite Agent’s build pipeline](https://github.com/buildkite/agent/blob/master/.buildkite/pipeline.yml). It contains script commands, wait steps, block steps, and artifact uploading:

```yaml
steps:
  - label: '\:hammer\: Tests'
    command: 'scripts/tests.sh'
    env:
      BUILDKITE_DOCKER_COMPOSE_CONTAINER: app

  - wait

  - label: '\:package\: Package'
    command: 'scripts/build-binaries.sh'
    artifact_paths: 'pkg/*'
    env:
      BUILDKITE_DOCKER_COMPOSE_CONTAINER: app

  - wait

  - label: '\:debian\: Publish'
    command: 'scripts/build-debian-packages.sh'
    artifact_paths: 'deb/**/*'
    branches: 'master'
    agents:
      queue: 'deploy'

  - block: '\:shipit\: Release'
    branches: 'master'

  - label: '\:github\: Release'
    command: 'scripts/build-github-release.sh'
    artifact_paths: 'releases/**/*'
    branches: 'master'

  - wait

  - label: '\:whale\: Update images'
    command: 'scripts/release-docker.sh'
    branches: 'master'
    agents:
      queue: 'deploy'
```

## Step types

There are four different step types you can use in your Buildkite pipelines. See each step type for full documentation:

* [Command Step](/docs/pipelines/command-step)
* [Wait Step](/docs/pipelines/wait-step)
* [Block Step](/docs/pipelines/block-step)
* [Trigger Step](/docs/pipelines/trigger-step)

## Customizing the pipeline file path

By default the pipeline upload step reads your pipeline definition from `.buildkite/pipeline.yml` in your repository. You can specify a different file path by adding it as the first argument:

<%= image "custom-pipeline-file.png", width: 1174/2, height: 452/2, alt: "Screenshot of a custom pipeline file upload" %>

A common use for custom file paths is when separating test and deployment steps into two completely separate pipelines, both using the same repository URL. For example, your test pipeline’s upload command could be:

```
buildkite-agent pipeline upload .buildkite/pipeline.yml
```

And your deployment pipeline’s upload command could be:

```
buildkite-agent pipeline upload .buildkite/pipeline.deploy.yml
```

For a list of all command line options, see the [buildkite-agent pipeline upload](/docs/agent/cli-pipeline#uploading-pipelines) documentation.

## Dynamic pipelines

Because the pipeline upload step runs on your agent machine, you can generate pipelines dynamically using scripts from your source code. This provides you with the flexiblity to structure your pipelines as you wish.

The following example generates a list of parallel test steps based upon the `test/*` directories within your repository:

```
#!/bin/bash

# exit immediately on failure, or if an undefined variable is used
set -eu

# begin the pipeline.yml file
echo "steps:"

# add a new command step to run the tests in each test directory
for test_dir in test/*/; do
  echo "  - command: \"run_tests "${test_dir}"\"
done
```

To use this script, you'd save it to `.buildkite/pipeline.sh` inside your repository, ensure it is executable, and then update your pipeline upload step to use the new script: 

```bash
.buildkite/pipeline.sh | buildkite-agent pipeline upload
```

When a build is run it will execute the script and pipe its output to the `pipeline upload` command, dynamically adding build steps to the end of the build pipeline.

## Migrating existing steps to a pipeline.yml

If you have a pipeline that is defined in the web interface and would like to migrate it to a pipeline.yml without any interruption to builds, you can add the following command step at the start of your pipeline:

```bash
[[ -f ".buildkite/pipeline.yml" ]] && buildkite-agent pipeline upload --replace || true
```

This line of code will check if the pipeline file exists and, if it does, execute the `buildkite agent pipeline upload --replace` command to replace any steps that were defined via the web interface with those defined in the `pipeline.yml`.

## Further documentation

See the [buildkite-agent pipeline documentation](/docs/agent/cli-pipeline) for a full list of options.
{'fraction_non_alphanumeric': 0.09310870149654987, 'fraction_numerical': 0.00609373599784927, 'mean_word_length': 4.135757017947538, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 14, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1672465', 'n_tokens_mistral': 3298, 'n_tokens_neox': 3103, 'n_words': 1412}