<commit_before><commit_msg>qemu unit tests notes<commit_after>---
layout: post
title: Testing applications with qemu user mode, automake, and buildroot
---

Previously I discussed [using qemu user mode to run cross-compiled binaries](http://yurovsky.github.io/2016/12/14/qemu-user-mode/), now let's put a few things
together and run unit tests automatically with automake's `make check`. Then
we'll integrate everything into a build system (buildroot) and automatically
run cross-compiled tests as part of the build process. This has a number of
advantages including:

* no need to build for both x86 and ARM
* no need to deploy cross-compiled unit tests to a real target or even a `qemu-system-arm` machine, they'll just run on your machine
* easy integration and automation in your build system without requiring any additional steps or configurations

Many thanks to [<NAME>](https://github.com/ndreys) who explored this
further after discussing qemu user mode with me and figured all of this out
(I'm mostly just writing up notes on what he did).

# Running tests with automake

automake has a feature called `LOG_COMPILER` as part of its [Parallel Test Harness](https://www.gnu.org/software/automake/manual/html_node/Parallel-Test-Harness.html) infrastructure. If this variable is set, tests are executed
with the `LOG_COMPILER` as a wrapper. The Parallel Test Harness documentation
gives `perl` as an example wrapper but we can take advantage of this feature to
wrap our tests with a shell script calling out to qemu in order to run them
directly on our build machine.

We can capture and set this variable from the environment in `configure.ac`,
for instance:

    AC_SUBST([LOG_COMPILER], ${LOG_COMPILER})

It could be set like this when invoking `configure`:

    LOG_COMPILER=qemu-wrapper.sh ./configure

Of course we'd also include our cross-compilation options as well. We can then
run `make` and `make check` and we should see that the tests invoked by `make check` are now run using `qemu-wrapper.sh`.  That in turn needs to be some shell
script that runs its first command-line argument using `qemu-arm` with appropriate options (such as `-L`) set.

# Running tests with buildroot

Now let's put everything together into a buildroot recipe. I assume you're
familiar with how buildroot works (if not, please consult [the buildroot manual](https://buildroot.org/downloads/manual/manual.html) and familiarize yourself with how `external.mk` and recipes work).

## Add a test runner

We need something simple to point `LOG_COMPILER` to so we can add that as a
shell script somewhere in your `external/` tree. For example, `external/test-runner.sh` could look like:

    #!bin/sh
    
    exec ${HOST_DIR}/usr/bin/qemu-arm -L ${STAGING_DIR} $1

Keep in mind that `HOST_DIR` points to your build output's `host` directory,
containing the host's toolchain and, in this case, also the `host-qemu` executible. Meanwhile the `STAGING_DIR` gives us exactly what we need for qemu's `-L`
option.

We can then create a variable that points to this script. For instance, add:

    TEST_RUNNER := $(BR2_EXTERNAL)/test-runner.sh

to `external.mk`. Now package recipes have access to the `test-runner.sh`
script via `TEST_RUNNER`.

## Making sure qemu-arm is available

Packages needing this qemu-based test infrastructure simply need to select
`BR2_PACKAGE_HOST_QEMU` as a dependency in their `Config.in`. For example,

    config BR2_PACKAGE_EXAMPLE_APP
        bool "example-app"
        select BR2_PACKAGE_HOST_QEMU
        help
          This is a our example app.

Now when `example-app` is built, buildroot will build and deploy `qemu-arm`
(to `host`).

## Adding test hooks to a recipe

Given a typical buildroot recipe for our example app, `example-app.mk`, all
we need to do is set `LOG_COMPILER` for automake and tell buildroot to run
`make check` as a post-build step.  Setting `LOG_COMPILER` is easy by adding
to buildroot's `<package>_CONF_ENV` variable:

    EXAMPLE_APP_CONF_ENV += LOG_COMPILER=$(TEST_RUNNER)

Next, we add a post-build hook:

    define example-app-make-check
    $(MAKE) -C $(@D) check
    endef
    
    EXAMPLE_APP_POST_BUILD_HOOKS += example-app-make-check

Now we should see two things:

# buildroot will add our `LOG_COMPILER` variable to the environment when running `configure` in this package
# having built the package, buildroot will execute `make check` in the package build directory and that in turn will run any tests the package provides with `qemu-arm`

To try this out, build the package:

    make example-app

You should hopefully see the familiar `make check` output after the application
is built. Running `file` on binaries in the build directory should reveal that
they are in fact ARM binaries!

A failed unit test in the above example will break the build for that package.
You can adjust the implementation slightly if that's not what you want and you
could even introduce a configuration option in your `external/Config.in` menu
to make tests optional or control whether test failures break the build, as
usual there is a lot that can be customized with buildroot.
{'fraction_non_alphanumeric': 0.06675775912551239, 'fraction_numerical': 0.002927971891469842, 'mean_word_length': 4.266187050359712, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2215038', 'n_tokens_mistral': 1456, 'n_tokens_neox': 1403, 'n_words': 711}