<commit_before># Writing Build Scripts

One of the most common actions that Buildkite steps perform is running shell scripts. These scripts are checked in alongside your code and `pipeline.yml` file. 

The [Buildkite Agent](/docs/agent/v3) will run your scripts, capture and report the log output, and use the exit status to mark each job, as well as the overall build, as passed or failed.

{:toc}

## Configuring Bash

The shell that runs your scripts in Buildkite is a clean Bash prompt with no settings. If you rely on anything from your `~/.bash_profile` or `~/.bashrc` files when you run scripts locally, you'll need to explicitly add the relevant items to your build scripts.

When writing Bash shell scripts there are a number of options you can set to help prevent unexpected errors:

<table>
  <tbody>
  <tr>
    <th><code>e</code></th>
    <td>Exit script immediately if any command returns a non-zero exit status.</td>
  </tr>
  <tr>
    <th><code>u</code></th>
    <td>Exit script immediately if an undefined variable is used (e.g. <code>echo "$UNDEFINED_ENV_VAR"</code>).</td>
  </tr>
  <tr>
    <th><code>o pipefail</code></th>
    <td>Ensure Bash pipelines (e.g. <code>cmd | othercmd</code>) return a non-zero status if any of the commands fail, rather than returning the exit status of the last command in the pipeline.</td>
  </tr>
  <tr>
    <th><code>x</code></th>
    <td>Expand and print each command before executing. See <a href="/docs/builds/writing-build-scripts#debugging-your-environment">Debugging your environment</a> for more information. </td>
  </tr>
  </tbody>
</table>

Bash’s built-in `set` command can be used to enable and disable options. For example, `set -u` enables the `u` option, and `set +u` disables the `u` option. You can also set multiple options at once, for example `set -ue` enables both the `u` and `e` options.

The following example enables the most commonly used options for build scripts:

```bash
#!/bin/bash

set -euo pipefail

run_tests
```

For a full list of options, see the [Bash Reference Manual](https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html).

<section class="Docs__troubleshooting-note">
  <h3>Unbound variable errors</h3>
  <p>Note that while enabling the <code>u</code> option is generally a good default to use for all build scripts, it can cause some tools like <a href="https://rvm.io">rvm</a> to fail with “unbound variable” errors. If you encounter errors, you can either remove <code>u</code> from the list of options, or you can surround the command causing the error with <code>set +u</code> and <code>set -u</code> to temporarily disable the option.</p>
</section>

## Capturing exit status

Build scripts can sometimes contain commands that shouldn't affect the overall exit status. For example, take the following script:

```bash
#!/bin/bash

# Note that we don't enable the 'e' option, which would cause the script to
# immediately exit if 'run_tests' failed
set -uo pipefail

run_tests

clean_up
```

Running this script will exit with the status returned by the final command, `clean_up`. However, what we really care about is the exit status of the first command, `run_tests`.

By using a variable to store the exit status of `run_tests`, we can run additional commands while still returning the original exit status. For example:

```bash
#!/bin/bash

# Note that we don't enable the 'e' option, which would cause the script to
# immediately exit if 'run_tests' failed
set -uo pipefail

# Run the main command we're most interested in
run_tests

# Capture the exit status
TESTS_EXIT_STATUS=$?

# Run additional commands
clean_up

# Exit with the status of the original command
exit $TESTS_EXIT_STATUS
```

Using this technique gives you control over the exit code of your script, and the final success or failure of your build job.

## Debugging your environment

The first step in debugging your build script is to view the environment variables from the Buildkite web interface:

<%= image "viewing-job-environment-variables.png", width: 1044/2, height: 542/2, alt: 'Screenshot of viewing a job’s environment tab' %>

There may be additional environment variables available in your build job that
don’t appear in this list, such as ones set by your
[job lifecycle hooks](/docs/agent/v3/hooks#job-lifecycle-hooks).
To debug these, you can print them using `echo $SOME_VAR` before the command
you're wanting to run. For example:

```bash
#!/bin/bash

echo "$PATH"

some_command
```

If you require more environment information, you can execute `env` to print out all the environment variable names and their values. If you use `env` you should filter the output using a tool such as `grep` or `egrep`, to ensure you don't leak private keys or other information.

<section class="Docs__troubleshooting-note">
  <h3>Security recommendation</h3>
  <p>If you use environment variables to define sensitive data such as API keys or Secret Access Keys, you should always filter the output of <code>env</code> to ensure you’re not exposing any secrets in your build log.</p>
</section>

For example, the following prints all environment variable names and values containing the words "git" or "node", using a case-insensitive search:

```bash
#!/bin/bash

env | grep -i -E 'git|node'

some_command
```

Enabling Bash’s debug mode using `set -x` can also help to debug your build scripts. This debug output can be very noisy, so it’s best enable this before the command you want to debug, and then to disable it straight after. For example:

```bash
#!/bin/bash

set -x # Enable debugging
some_command
set +x # Disable debugging

some_other_command
```

For more information about the `x` option and debugging in general, see the [Bash Guide for Beginners' page on debugging Bash scripts](http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_03.html).

## Help with linting and debugging

To check your shell scripts for common errors and mistakes we highly recommend using a linting tool like [Shellcheck](http://www.shellcheck.net). Shellcheck is a shell script linter with a web-based front-end, a command line tool, and integrates directly with most code editors.

For an explanation of a shell code snippet, the [explainshell.com](http://explainshell.com) tool is extremely useful. Explainshell can tell you, in plain English, what a line of shell code does. It also integrates the man pages of common tools such as `ssh` and `git`.

## Managing log output

If your script is generating output that is too large, there are several strategies you can employ to reduce the output or redirect the log. Take a look at our guide to [managing log output guide](/docs/builds/managing-log-output) for a step by step introduction.
<commit_msg>Ignore the man in man page<commit_after># Writing Build Scripts

One of the most common actions that Buildkite steps perform is running shell scripts. These scripts are checked in alongside your code and `pipeline.yml` file. 

The [Buildkite Agent](/docs/agent/v3) will run your scripts, capture and report the log output, and use the exit status to mark each job, as well as the overall build, as passed or failed.

{:toc}

## Configuring Bash

The shell that runs your scripts in Buildkite is a clean Bash prompt with no settings. If you rely on anything from your `~/.bash_profile` or `~/.bashrc` files when you run scripts locally, you'll need to explicitly add the relevant items to your build scripts.

When writing Bash shell scripts there are a number of options you can set to help prevent unexpected errors:

<table>
  <tbody>
  <tr>
    <th><code>e</code></th>
    <td>Exit script immediately if any command returns a non-zero exit status.</td>
  </tr>
  <tr>
    <th><code>u</code></th>
    <td>Exit script immediately if an undefined variable is used (e.g. <code>echo "$UNDEFINED_ENV_VAR"</code>).</td>
  </tr>
  <tr>
    <th><code>o pipefail</code></th>
    <td>Ensure Bash pipelines (e.g. <code>cmd | othercmd</code>) return a non-zero status if any of the commands fail, rather than returning the exit status of the last command in the pipeline.</td>
  </tr>
  <tr>
    <th><code>x</code></th>
    <td>Expand and print each command before executing. See <a href="/docs/builds/writing-build-scripts#debugging-your-environment">Debugging your environment</a> for more information. </td>
  </tr>
  </tbody>
</table>

Bash’s built-in `set` command can be used to enable and disable options. For example, `set -u` enables the `u` option, and `set +u` disables the `u` option. You can also set multiple options at once, for example `set -ue` enables both the `u` and `e` options.

The following example enables the most commonly used options for build scripts:

```bash
#!/bin/bash

set -euo pipefail

run_tests
```

For a full list of options, see the [Bash Reference Manual](https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html).

<section class="Docs__troubleshooting-note">
  <h3>Unbound variable errors</h3>
  <p>Note that while enabling the <code>u</code> option is generally a good default to use for all build scripts, it can cause some tools like <a href="https://rvm.io">rvm</a> to fail with “unbound variable” errors. If you encounter errors, you can either remove <code>u</code> from the list of options, or you can surround the command causing the error with <code>set +u</code> and <code>set -u</code> to temporarily disable the option.</p>
</section>

## Capturing exit status

Build scripts can sometimes contain commands that shouldn't affect the overall exit status. For example, take the following script:

```bash
#!/bin/bash

# Note that we don't enable the 'e' option, which would cause the script to
# immediately exit if 'run_tests' failed
set -uo pipefail

run_tests

clean_up
```

Running this script will exit with the status returned by the final command, `clean_up`. However, what we really care about is the exit status of the first command, `run_tests`.

By using a variable to store the exit status of `run_tests`, we can run additional commands while still returning the original exit status. For example:

```bash
#!/bin/bash

# Note that we don't enable the 'e' option, which would cause the script to
# immediately exit if 'run_tests' failed
set -uo pipefail

# Run the main command we're most interested in
run_tests

# Capture the exit status
TESTS_EXIT_STATUS=$?

# Run additional commands
clean_up

# Exit with the status of the original command
exit $TESTS_EXIT_STATUS
```

Using this technique gives you control over the exit code of your script, and the final success or failure of your build job.

## Debugging your environment

The first step in debugging your build script is to view the environment variables from the Buildkite web interface:

<%= image "viewing-job-environment-variables.png", width: 1044/2, height: 542/2, alt: 'Screenshot of viewing a job’s environment tab' %>

There may be additional environment variables available in your build job that
don’t appear in this list, such as ones set by your
[job lifecycle hooks](/docs/agent/v3/hooks#job-lifecycle-hooks).
To debug these, you can print them using `echo $SOME_VAR` before the command
you're wanting to run. For example:

```bash
#!/bin/bash

echo "$PATH"

some_command
```

If you require more environment information, you can execute `env` to print out all the environment variable names and their values. If you use `env` you should filter the output using a tool such as `grep` or `egrep`, to ensure you don't leak private keys or other information.

<section class="Docs__troubleshooting-note">
  <h3>Security recommendation</h3>
  <p>If you use environment variables to define sensitive data such as API keys or Secret Access Keys, you should always filter the output of <code>env</code> to ensure you’re not exposing any secrets in your build log.</p>
</section>

For example, the following prints all environment variable names and values containing the words "git" or "node", using a case-insensitive search:

```bash
#!/bin/bash

env | grep -i -E 'git|node'

some_command
```

Enabling Bash’s debug mode using `set -x` can also help to debug your build scripts. This debug output can be very noisy, so it’s best enable this before the command you want to debug, and then to disable it straight after. For example:

```bash
#!/bin/bash

set -x # Enable debugging
some_command
set +x # Disable debugging

some_other_command
```

For more information about the `x` option and debugging in general, see the [Bash Guide for Beginners' page on debugging Bash scripts](http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_03.html).

## Help with linting and debugging

To check your shell scripts for common errors and mistakes we highly recommend using a linting tool like [Shellcheck](http://www.shellcheck.net). Shellcheck is a shell script linter with a web-based front-end, a command line tool, and integrates directly with most code editors.

<!--alex ignore gals-man -->

For an explanation of a shell code snippet, the [explainshell.com](http://explainshell.com) tool is extremely useful. Explainshell can tell you, in plain English, what a line of shell code does. It also integrates the man pages of common tools such as `ssh` and `git`.

## Managing log output

If your script is generating output that is too large, there are several strategies you can employ to reduce the output or redirect the log. Take a look at our guide to [managing log output guide](/docs/builds/managing-log-output) for a step by step introduction.

{'fraction_non_alphanumeric': 0.08646811026531669, 'fraction_numerical': 0.002808365974429089, 'mean_word_length': 4.357086302454474, 'pattern_counts': {'":': 0, '<': 138, '<?xml version=': 0, '>': 138, 'https://': 4, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1551511', 'n_tokens_mistral': 3925, 'n_tokens_neox': 3757, 'n_words': 1906}