<commit_before># buildkite-agent pipeline

The Buildkite Agent’s `pipeline` command allows you to add and replace build steps in the running build. The steps are defined using YML or JSON and can be read from a file or streamed from the output of a script.

See the [Defining Your Pipeline Steps](/docs/pipelines/uploading-pipelines) guide for a step-by-step example and list of step types.

<%= toc %>

## Uploading pipelines

```
$ buildkite-agent pipeline upload --help
Usage:

   buildkite-agent pipeline upload <file> [arguments...]

Description:

   Allows you to change the pipeline of a running build by uploading either a
   YAML (recommended) or JSON configuration file. If no configuration file is
   provided, the command looks for the file in the following locations:

   - buildkite.yml
   - buildkite.yaml
   - buildkite.json
   - .buildkite/pipeline.yml
   - .buildkite/pipeline.yaml
   - .buildkite/pipeline.json

   You can also pipe build pipelines to the command allowing you to create
   scripts that generate dynamic pipelines.

Example:

   $ buildkite-agent pipeline upload
   $ buildkite-agent pipeline upload my-custom-pipeline.yml
   $ ./script/dynamic_step_generator | buildkite-agent pipeline upload

Options:

   --replace                   Replace the rest of the existing pipeline with the steps uploaded. Jobs that are already running are not removed. [$BUILDKITE_PIPELINE_REPLACE]
   --job value                 The job that is making the changes to it's build [$BUILDKITE_JOB_ID]
   --agent-access-token value  The access token used to identify the agent [$BUILDKITE_AGENT_ACCESS_TOKEN]
   --endpoint value            The Agent API endpoint (default: "https://agent.buildkite.com/v3") [$BUILDKITE_AGENT_ENDPOINT]
   --no-color                  Don't show colors in logging [$BUILDKITE_AGENT_NO_COLOR]
   --debug                     Enable debug mode [$BUILDKITE_AGENT_DEBUG]
   --debug-http                Enable HTTP debug mode, which dumps all request and response bodies to the log [$BUILDKITE_AGENT_DEBUG_HTTP]
```

## Pipeline format

The pipeline can be written as YAML or JSON, but YAML is more common for its readability. There are two top level properties you can specify:

* `env` - A map of <a href="/docs/builds/environment-variables">environment variables</a> to apply to all steps
* `steps` - A list of [build pipeline steps](/docs/pipelines/defining-steps)

## Environment variable substitution

The `pipeline upload` command supports environment variable substitution using the syntax `$VAR` and `${VAR}`.

For example, the following pipeline substitutes a number of [Buildkite’s default environment variables](/docs/builds/environment-variables) into a [trigger step](/docs/pipelines/trigger-step):

```yml
- trigger: "app-deploy"
  label: "\:rocket\: Deploy"
  branches: "master"
  async: true
  build:
    message: "${BUILDKITE_MESSAGE}"
    commit: "${BUILDKITE_COMMIT}"
    branch: "${BUILDKITE_BRANCH}"
```

If you want an environment variable to be evaluated at run-time (for example, using the step’s environment variables) make sure to escape the `$` character using `$$` or `\$`. For example:

```yml
- command: "deploy.sh $$SERVER"
  env:
    SERVER: "server-a"
```

### Escaping the `$` character

If you need to prevent substitution, you can escape the `$` character by using `$$` or `\$`.

For example, using `$$USD` and `\$USD` will both result in the same value: `$USD`.

### Requiring environment variables

You can set required environment variables using the syntax `${VAR?}`. If `VAR` is not set, the `pipeline upload` command will print and error and exit with a status of 1.
<commit_msg>Document pipeline upload --no-interpolation<commit_after># buildkite-agent pipeline

The Buildkite Agent’s `pipeline` command allows you to add and replace build steps in the running build. The steps are defined using YML or JSON and can be read from a file or streamed from the output of a script.

See the [Defining Your Pipeline Steps](/docs/pipelines/uploading-pipelines) guide for a step-by-step example and list of step types.

<%= toc %>

## Uploading pipelines

```
$ buildkite-agent pipeline upload --help
Usage:

   buildkite-agent pipeline upload <file> [arguments...]

Description:

   Allows you to change the pipeline of a running build by uploading either a
   YAML (recommended) or JSON configuration file. If no configuration file is
   provided, the command looks for the file in the following locations:

   - buildkite.yml
   - buildkite.yaml
   - buildkite.json
   - .buildkite/pipeline.yml
   - .buildkite/pipeline.yaml
   - .buildkite/pipeline.json

   You can also pipe build pipelines to the command allowing you to create
   scripts that generate dynamic pipelines.

Example:

   $ buildkite-agent pipeline upload
   $ buildkite-agent pipeline upload my-custom-pipeline.yml
   $ ./script/dynamic_step_generator | buildkite-agent pipeline upload

Options:

   --replace                   Replace the rest of the existing pipeline with the steps uploaded. Jobs that are already running are not removed. [$BUILDKITE_PIPELINE_REPLACE]
   --job value                 The job that is making the changes to it's build [$BUILDKITE_JOB_ID]
   --no-interpolation          Skip variable interpolation the pipeline when uploaded [$BUILDKITE_PIPELINE_NO_INTERPOLATION]
   --agent-access-token value  The access token used to identify the agent [$BUILDKITE_AGENT_ACCESS_TOKEN]
   --endpoint value            The Agent API endpoint (default: "https://agent.buildkite.com/v3") [$BUILDKITE_AGENT_ENDPOINT]
   --no-color                  Don't show colors in logging [$BUILDKITE_AGENT_NO_COLOR]
   --debug                     Enable debug mode [$BUILDKITE_AGENT_DEBUG]
   --debug-http                Enable HTTP debug mode, which dumps all request and response bodies to the log [$BUILDKITE_AGENT_DEBUG_HTTP]
```

## Pipeline format

The pipeline can be written as YAML or JSON, but YAML is more common for its readability. There are two top level properties you can specify:

* `env` - A map of <a href="/docs/builds/environment-variables">environment variables</a> to apply to all steps
* `steps` - A list of [build pipeline steps](/docs/pipelines/defining-steps)

## Environment variable substitution

The `pipeline upload` command supports environment variable substitution using the syntax `$VAR` and `${VAR}`.

For example, the following pipeline substitutes a number of [Buildkite’s default environment variables](/docs/builds/environment-variables) into a [trigger step](/docs/pipelines/trigger-step):

```yml
- trigger: "app-deploy"
  label: "\:rocket\: Deploy"
  branches: "master"
  async: true
  build:
    message: "${BUILDKITE_MESSAGE}"
    commit: "${BUILDKITE_COMMIT}"
    branch: "${BUILDKITE_BRANCH}"
```

If you want an environment variable to be evaluated at run-time (for example, using the step’s environment variables) make sure to escape the `$` character using `$$` or `\$`. For example:

```yml
- command: "deploy.sh $$SERVER"
  env:
    SERVER: "server-a"
```

### Escaping the `$` character

If you need to prevent substitution, you can escape the `$` character by using `$$` or `\$`.

For example, using `$$USD` and `\$USD` will both result in the same value: `$USD`.

### Disabling interpolation

You can disable interpolation with the `--no-interpolation` flag, which was added in v3.1.1.

### Requiring environment variables

You can set required environment variables using the syntax `${VAR?}`. If `VAR` is not set, the `pipeline upload` command will print and error and exit with a status of 1.

{'fraction_non_alphanumeric': 0.09041853119241906, 'fraction_numerical': 0.0009212950776520137, 'mean_word_length': 3.670559311616472, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4021565', 'n_tokens_mistral': 2203, 'n_tokens_neox': 2103, 'n_words': 877}