<commit_before>---
layout: documentation
title: Testing
---

# Testing

There are several different approaches to testing Starlark code in Bazel. This
page gathers the current best practices and frameworks by use case.

* ToC
{:toc}

## For testing rules

[Skylib](https://github.com/bazelbuild/bazel-skylib) has a test framework called
[`unittest.bzl`](https://github.com/bazelbuild/bazel-skylib/blob/master/lib/unittest.bzl)
for checking the analysis-time behavior of rules, such as their actions and
providers. Such tests are called "analysis tests" and are currently the best
option for testing the inner workings of rules.

Some caveats:

* Test assertions occur within the build, not a separate test runner process.
  Targets that are created by the test must be named such that they do not
  collide with targets from other tests or from the build. An error that occurs
  during the test is seen by Bazel as a build breakage rather than a test
  failure.

* It requires a fair amount of boilerplate to set up the rules under test and
  the rules containing test assertions. This boilerplate may seem daunting at
  first. It helps to [keep in mind](concepts.md#evaluation-model) which code
  runs during the loading phase and which code runs during the analysis phase.

* Analysis tests are intended to be fairly small and lightweight. Certain
  features of the analysis testing framework are restricted to verifying
  targets with a maximum number of transitive dependencies (currently 500).
  This is due to performance implications of using these features with larger
  tests.

The basic principle is to define a testing rule that depends on the
rule-under-test. This gives the testing rule access to the rule-under-test's
providers.

The testing rule's implementation function carries out assertions. If there are
any failures, these are not raised immediately by calling `fail()` (which would
trigger an analysis-time build error), but rather by storing the errors in a
generated script that fails at test execution time.

See below for a minimal toy example, followed by an example that checks actions.

### Minimal example

`//mypkg/BUILD`:

```python
load(":myrules.bzl", "myrule")
load(":myrules_test.bzl", "myrules_test_suite")

# Production use of the rule.
myrule(
    name = "mytarget",
)

# Call a macro that defines targets that perform the tests at analysis time,
# and that can be executed with "bazel test" to return the result.
myrules_test_suite()
```

`//mypkg/myrules.bzl`:

```python
MyInfo = provider()

def _myrule_impl(ctx):
  """Rule that just generates a file and returns a provider."""
  ctx.actions.write(ctx.outputs.out, "abc")
  return [MyInfo(val="some value", out=ctx.outputs.out)]

myrule = rule(
    implementation = _myrule_impl,
    outputs = {"out": "%{name}.out"},
)
```

`//mypkg/myrules_test.bzl`:


```python
load("@bazel_skylib//lib:unittest.bzl", "asserts", "analysistest")
load(":myrules.bzl", "myrule", "MyInfo")

# ==== Check the provider contents ====

def _provider_contents_test_impl(ctx):
  # Analysis-time test logic; place assertions here. Always begins with begin()
  # and ends with returning end(). If you forget to return end(), you will get an
  # error about an analysis test needing to return an instance of AnalysisTestResultInfo.
  env = analysistest.begin(ctx)
  target_under_test = analysistest.target_under_test(env)
  asserts.equals(env, "some value", target_under_test[MyInfo].val)
  # You can also use keyword arguments for readability if you prefer.
  asserts.equals(env,
      expected="some value",
      actual=target_under_test[MyInfo].val)
  return analysistest.end(env)

# Create the testing rule to wrap the test logic. Note that this must be bound
# to a global variable due to restrictions on how rules can be defined. Also,
# its name must end with "_test".
provider_contents_test = analysistest.make(_provider_contents_test_impl)

# Macro to setup the test.
def test_provider_contents():
  # Rule under test.
  myrule(name = "provider_contents_subject")
  # Testing rule.
  provider_contents_test(name = "provider_contents",
                         target_under_test = ":provider_contents_subject")
  # Note the target_under_test attribute is how the test rule depends on
  # the real rule target.

# Entry point from the BUILD file; macro for running each test case's macro and
# declaring a test suite that wraps them together.
def myrules_test_suite():
  # Call all test functions and wrap their targets in a suite.
  test_provider_contents()
  # ...

  native.test_suite(
      name = "myrules_test",
      tests = [
          ":provider_contents",
          # ...
      ],
  )
```

The test can be run with `bazel test //mypkg:myrules_test`.

Aside from the initial `load()` statements, there are two main parts to the
file:

* The tests themselves, each of which consists of 1) an analysis-time
  implementation function for the testing rule, 2) a declaration of the testing
  rule via `analysistest.make()`, and 3) a loading-time function (macro) for
  declaring the rule-under-test (and its dependencies) and testing rule. If the
  assertions do not change between test cases, 1) and 2) may be shared by
  multiple test cases.

* The test suite function, which calls the loading-time functions for each test,
  and declares a `test_suite` target bundling all tests together.

We recommend the following naming convention. Let `foo` stand for the part of
the test name that describes what the test is checking (`provider_contents` in
the above example). For example, a JUnit test method would be named `testFoo`.
Then:

* the loading-time function should should be named `test_foo`
  (`test_provider_contents`)

* its testing rule type should be named `foo_test` (`provider_contents_test`)

* the label of the target of this rule type should be `foo`
  (`provider_contents`)

* the implementation function for the testing rule should be named
  `_foo_test_impl` (`_provider_contents_test_impl`)

* the labels of the targets of the rules under test and their dependencies
  should be prefixed with `foo_` (`provider_contents_`)

Note that the labels of all targets can conflict with other labels in the same
BUILD package, so it's helpful to use a unique name for the test.

### Failure Testing

It may be useful to verify that a rule fails given certain inputs or in
certain state. This can be done using the analysis test framework:

Firstly, the test rule created with `analysistest.make` should specify `expect_failure`:

```python
failure_testing_test = analysistest.make(
    _failure_testing_test_impl,
    expect_failure = True,
)
```

Secondly, the test rule implementation should make assertions on the nature
of the failure that took place (specifically, the failure message):

```python
def _failure_testing_test_impl(ctx):
    env = analysistest.begin(ctx)

    asserts.expect_failure(env, "This rule should never work")

    return analysistest.end(env)
```

### Verifying Registered Actions

You may want to write tests which make assertions about the actions that your
rule registers, for example, using `ctx.actions.run()`. This can be done in
your analysis test rule implementation function. An example:

```python
def _inspect_actions_test_impl(ctx):
    env = analysistest.begin(ctx)

    actions = analysistest.target_actions(env)
    asserts.equals(env, 1, len(actions))
    action_output = actions[0].outputs.to_list()[0]
    asserts.equals(env, "out.txt", action_output.basename)
    return analysistest.end(env)
```

Note that `analysistest.target_actions(env)` returns a list of
[`Action`](lib/Action.html) objects which represent actions registered by the
target under test.

### Verifying Rule Behavior Under Different Flags

You may want to verify your real rule behaves a certain way given certain<commit_msg>Documentation: suggest use of the 'manual' tag with failure testing<commit_after>---
layout: documentation
title: Testing
---

# Testing

There are several different approaches to testing Starlark code in Bazel. This
page gathers the current best practices and frameworks by use case.

* ToC
{:toc}

## For testing rules

[Skylib](https://github.com/bazelbuild/bazel-skylib) has a test framework called
[`unittest.bzl`](https://github.com/bazelbuild/bazel-skylib/blob/master/lib/unittest.bzl)
for checking the analysis-time behavior of rules, such as their actions and
providers. Such tests are called "analysis tests" and are currently the best
option for testing the inner workings of rules.

Some caveats:

* Test assertions occur within the build, not a separate test runner process.
  Targets that are created by the test must be named such that they do not
  collide with targets from other tests or from the build. An error that occurs
  during the test is seen by Bazel as a build breakage rather than a test
  failure.

* It requires a fair amount of boilerplate to set up the rules under test and
  the rules containing test assertions. This boilerplate may seem daunting at
  first. It helps to [keep in mind](concepts.md#evaluation-model) which code
  runs during the loading phase and which code runs during the analysis phase.

* Analysis tests are intended to be fairly small and lightweight. Certain
  features of the analysis testing framework are restricted to verifying
  targets with a maximum number of transitive dependencies (currently 500).
  This is due to performance implications of using these features with larger
  tests.

The basic principle is to define a testing rule that depends on the
rule-under-test. This gives the testing rule access to the rule-under-test's
providers.

The testing rule's implementation function carries out assertions. If there are
any failures, these are not raised immediately by calling `fail()` (which would
trigger an analysis-time build error), but rather by storing the errors in a
generated script that fails at test execution time.

See below for a minimal toy example, followed by an example that checks actions.

### Minimal example

`//mypkg/BUILD`:

```python
load(":myrules.bzl", "myrule")
load(":myrules_test.bzl", "myrules_test_suite")

# Production use of the rule.
myrule(
    name = "mytarget",
)

# Call a macro that defines targets that perform the tests at analysis time,
# and that can be executed with "bazel test" to return the result.
myrules_test_suite()
```

`//mypkg/myrules.bzl`:

```python
MyInfo = provider()

def _myrule_impl(ctx):
  """Rule that just generates a file and returns a provider."""
  ctx.actions.write(ctx.outputs.out, "abc")
  return [MyInfo(val="some value", out=ctx.outputs.out)]

myrule = rule(
    implementation = _myrule_impl,
    outputs = {"out": "%{name}.out"},
)
```

`//mypkg/myrules_test.bzl`:


```python
load("@bazel_skylib//lib:unittest.bzl", "asserts", "analysistest")
load(":myrules.bzl", "myrule", "MyInfo")

# ==== Check the provider contents ====

def _provider_contents_test_impl(ctx):
  # Analysis-time test logic; place assertions here. Always begins with begin()
  # and ends with returning end(). If you forget to return end(), you will get an
  # error about an analysis test needing to return an instance of AnalysisTestResultInfo.
  env = analysistest.begin(ctx)
  target_under_test = analysistest.target_under_test(env)
  asserts.equals(env, "some value", target_under_test[MyInfo].val)
  # You can also use keyword arguments for readability if you prefer.
  asserts.equals(env,
      expected="some value",
      actual=target_under_test[MyInfo].val)
  return analysistest.end(env)

# Create the testing rule to wrap the test logic. Note that this must be bound
# to a global variable due to restrictions on how rules can be defined. Also,
# its name must end with "_test".
provider_contents_test = analysistest.make(_provider_contents_test_impl)

# Macro to setup the test.
def test_provider_contents():
  # Rule under test. Be sure to tag 'manual', as this target should not be
  # built using `:all` except as a dependency of the test.
  myrule(name = "provider_contents_subject", tags = ["manual"])
  # Testing rule.
  provider_contents_test(name = "provider_contents",
                         target_under_test = ":provider_contents_subject")
  # Note the target_under_test attribute is how the test rule depends on
  # the real rule target.

# Entry point from the BUILD file; macro for running each test case's macro and
# declaring a test suite that wraps them together.
def myrules_test_suite():
  # Call all test functions and wrap their targets in a suite.
  test_provider_contents()
  # ...

  native.test_suite(
      name = "myrules_test",
      tests = [
          ":provider_contents",
          # ...
      ],
  )
```

The test can be run with `bazel test //mypkg:myrules_test`.

Aside from the initial `load()` statements, there are two main parts to the
file:

* The tests themselves, each of which consists of 1) an analysis-time
  implementation function for the testing rule, 2) a declaration of the testing
  rule via `analysistest.make()`, and 3) a loading-time function (macro) for
  declaring the rule-under-test (and its dependencies) and testing rule. If the
  assertions do not change between test cases, 1) and 2) may be shared by
  multiple test cases.

* The test suite function, which calls the loading-time functions for each test,
  and declares a `test_suite` target bundling all tests together.

We recommend the following naming convention. Let `foo` stand for the part of
the test name that describes what the test is checking (`provider_contents` in
the above example). For example, a JUnit test method would be named `testFoo`.
Then:

* the loading-time function should should be named `test_foo`
  (`test_provider_contents`)

* its testing rule type should be named `foo_test` (`provider_contents_test`)

* the label of the target of this rule type should be `foo`
  (`provider_contents`)

* the implementation function for the testing rule should be named
  `_foo_test_impl` (`_provider_contents_test_impl`)

* the labels of the targets of the rules under test and their dependencies
  should be prefixed with `foo_` (`provider_contents_`)

Note that the labels of all targets can conflict with other labels in the same
BUILD package, so it's helpful to use a unique name for the test.

### Failure Testing

It may be useful to verify that a rule fails given certain inputs or in
certain state. This can be done using the analysis test framework:

The test rule created with `analysistest.make` should specify `expect_failure`:

```python
failure_testing_test = analysistest.make(
    _failure_testing_test_impl,
    expect_failure = True,
)
```

The test rule implementation should make assertions on the nature
of the failure that took place (specifically, the failure message):

```python
def _failure_testing_test_impl(ctx):
    env = analysistest.begin(ctx)

    asserts.expect_failure(env, "This rule should never work")

    return analysistest.end(env)
```

Also make sure that your target under test is specifically tagged 'manual'.
Without this, building all targets in your package using `:all` will result
in a build of the intentionally-failing target and will exhibit a build failure.
With 'manual', your target under test will build only if explicitly specified,
or as a dependency of a non-manual target (such as your test rule):

```python
def failure_testing_suite():
    myrule(name = "this_should_fail", tags = ["manual"])

    failure_testing_test(name = "failure_testing_test",
                         target_under_test = ":this_should_fail")
```

### Verifying Registered Actions

You may want to write tests which make assertions about the actions that your
rule registers, for example, using `ctx.actions.run()`. This can be done in
your analysis test rule implementation function. An example:

```python
def _inspect_actions_test_impl(ctx):
    env = analysistest.begin(ctx)

    actions = analysistest.target_actions(env)
    asserts.equals(env, 1, len(actions))
    action_output = actions[0].outputs.to_list()[0]
    asserts.equals(env, "out.txt", action_output.basename)
    return analysistest.end(env)
```

Note that `analysistest.target_actions(env)` returns a list of
[`Action`](lib/Action.html) objects which represent actions registered by the
target under test.

### Verifying Rule Behavior Under Different Flags

You may want to verify your real rule behaves a certain way given certain
{'fraction_non_alphanumeric': 0.07847670032850712, 'fraction_numerical': 0.0013383623311838423, 'mean_word_length': 4.344278283485045, 'pattern_counts': {'":': 13, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7122445', 'n_tokens_mistral': 4683, 'n_tokens_neox': 4480, 'n_words': 2036}