<commit_before># The Firecracker Jailer

## Jailer Usage

The jailer is invoked in this manner:

``` bash
jailer --id <id> \
       --node <numa_node>\
       --exec-file <exec_file> \
       --uid <uid> \
       --gid <gid>
       [--chroot-base-dir <chroot_base>]
       [--netns <netns>]
       [--daemonize]
       [--seccomp-level <level>]
```

- `id` is the unique VM identification string, which may contain alphanumeric
  characters and hyphens. The maximum `id` length is currently 64 characters.
- `numa_node` represents the NUMA node the process gets assigned to. More
  details are available below.
- `exec_file` is the path to the Firecracker binary that will be exec-ed by the
  jailer. The user can provide a path to any binary, but the interaction with
  the jailer is mostly Firecracker specific.
- `uid` and `gid` are the uid and gid the jailer switches to as it execs the
  target binary.
- `chroot_base` represents the base folder where chroot jails are built. The
  default is `/srv/jailer`.
- `netns` represents the path to a network namespace handle. If present, the
  jailer will use this to join the associated network namespace.
- When present, the `--daemonize` flag causes the jailer to cal `setsid()` and
  redirect all three standard I/O file descriptors to `/dev/null`.
- `--seccomp-level` specifies whether seccomp filters should be installed and
  how restrictive they should be. Possible values are:
  - 0 (default): disabled.
  - 1 : basic filtering. This prohibits syscalls not whitelisted by
    Firecracker.
  - 2 : advanced filtering. This adds further checks on some of the parameters
    of the allowed syscalls.

## Jailer Operation

After starting, the Jailer goes through the following operations:

- Validate **all provided paths** and the VM `id`.
- Close all open file descriptors based on `sysconf(_SC_OPEN_MAX)` except input,
  output and error.
- Open `/dev/kvm` as *RW*, and bind a Unix domain socket listener to
  `<chroot_base>/<exec_file_name>/<id>/api.socket`. `exec_file_name` is the
  last path component of `exec_file` (for example, that would be `firecracker`
  for `/usr/bin/firecracker`). Both descriptors remain open across exec-ing
  into the target binary, which would be otherwise unable to open/create the
  associated files.
- Create the `<chroot_base>/<exec_file_name>/<id>/root` folder, which will be
  henceforth referred to as `chroot_dir`. Nothing is done if the path already
  exists (it should not, since `id` is supposed to be unique).
- Copy `exec_file` to
  `<chroot_base>/<exec_file_name>/<id>/root/<exec_file_name>`.
- Create the `cgroup` sub-folders. At the moment, the jailer uses three
  `cgroup v1` controllers: `cpu`, `cpuset`, and `pids`. On most systems, these
  (along with others) are mounted by default somewhere in `/sys/fs/cgroup`
  (they should be mounted by the user otherwise). The jailer will parse
  `/proc/mounts` to detect where each of the three controllers can be found
  (multiple controllers may share the same path). For each identified location
  (referred to as `<cgroup_base>`), the jailer creates the
  `<cgroup_base>/<exec_file_name>/<id>` subfolder, and writes the current pid
  to `<cgroup_base>/<exec_file_name>/<id>/tasks`. Also, the value of
  `numa_node` is written to the appropriate `cpuset.mems` file.
- Call `unshare()` into a new mount namespace, use `pivot_root()` to switch
  the old system root mount point with a new one base in `chroot_dir`, switch
  the current working directory to the new root, unmount the old root mount
  point, and call `chroot` into the current directory.
- Use `mknod` to create a `/dev/net/tun` equivalent inside the jail.
- If `--netns <netns>` is present, attempt to join the specified network
  namespace.
- If `--daemonize` is specified, call `setsid()` and redirect `STDIN`,
  `STDOUT`, and `STDERR` to `/dev/null`.
- Drop privileges via setting the provided `uid` and `gid`.
- Exec into `<exec_file_name> --context=<context_json>`. `context_json` is a
  structure to pass parameters from the jailer to Firecracker. The json string
  includes the following keys:
  - `id`: (`string`) - The `id` argument provided to the jailer.
  - `jailed`: (`boolean`) value (`true` in this case) to let Firecracker know
    that it's running inside a jail.
  - `seccomp_level`: (`number`) the `--seccomp-level` argument provided to the
    jailer.

## Example Run and Notes

Let’s assume Firecracker is available as `/usr/bin/firecracker`, and the jailer
can be found at `/usr/bin/jailer`. We pick the **unique id
551e7604-e35c-42b3-b825-416853441234**, and we choose to run on **NUMA node
0**, using **uid 123**, and **gid 100**. For this example, we are content with
the default `/srv/jailer` chroot base dir.

We start by running:

``` bash
/usr/bin/jailer --id 551e7604-e35c-42b3-b825-416853441234 --node 0 --exec-file
/usr/bin/firecracker --uid 123 --gid 100 --netns /var/run/netns/my_netns
--daemonize
```

After opening the file descriptors mentioned in the previous section, the
jailer will create the following resources (and all their prerequisites, such
as the path which contains them):

- `/srv/jailer/firecracker/551e7604-e35c-42b3-b825-416853441234/api.socket`
  (created via `bind`)
- `/srv/jailer/firecracker/551e7604-e35c-42b3-b825-416853441234/root/firecracker`
  (copied from `/usr/bin/firecracker`)

We are going to refer to
`/srv/jailer/firecracker/551e7604-e35c-42b3-b825-416853441234/root`
as `<chroot_dir>`.
<commit_msg>jailer doc example lacked '\' in some commands<commit_after># The Firecracker Jailer

## Jailer Usage

The jailer is invoked in this manner:

``` bash
jailer --id <id> \
       --node <numa_node>\
       --exec-file <exec_file> \
       --uid <uid> \
       --gid <gid>
       [--chroot-base-dir <chroot_base>]
       [--netns <netns>]
       [--daemonize]
       [--seccomp-level <level>]
```

- `id` is the unique VM identification string, which may contain alphanumeric
  characters and hyphens. The maximum `id` length is currently 64 characters.
- `numa_node` represents the NUMA node the process gets assigned to. More
  details are available below.
- `exec_file` is the path to the Firecracker binary that will be exec-ed by the
  jailer. The user can provide a path to any binary, but the interaction with
  the jailer is mostly Firecracker specific.
- `uid` and `gid` are the uid and gid the jailer switches to as it execs the
  target binary.
- `chroot_base` represents the base folder where chroot jails are built. The
  default is `/srv/jailer`.
- `netns` represents the path to a network namespace handle. If present, the
  jailer will use this to join the associated network namespace.
- When present, the `--daemonize` flag causes the jailer to cal `setsid()` and
  redirect all three standard I/O file descriptors to `/dev/null`.
- `--seccomp-level` specifies whether seccomp filters should be installed and
  how restrictive they should be. Possible values are:
  - 0 (default): disabled.
  - 1 : basic filtering. This prohibits syscalls not whitelisted by
    Firecracker.
  - 2 : advanced filtering. This adds further checks on some of the parameters
    of the allowed syscalls.

## Jailer Operation

After starting, the Jailer goes through the following operations:

- Validate **all provided paths** and the VM `id`.
- Close all open file descriptors based on `sysconf(_SC_OPEN_MAX)` except input,
  output and error.
- Open `/dev/kvm` as *RW*, and bind a Unix domain socket listener to
  `<chroot_base>/<exec_file_name>/<id>/api.socket`. `exec_file_name` is the
  last path component of `exec_file` (for example, that would be `firecracker`
  for `/usr/bin/firecracker`). Both descriptors remain open across exec-ing
  into the target binary, which would be otherwise unable to open/create the
  associated files.
- Create the `<chroot_base>/<exec_file_name>/<id>/root` folder, which will be
  henceforth referred to as `chroot_dir`. Nothing is done if the path already
  exists (it should not, since `id` is supposed to be unique).
- Copy `exec_file` to
  `<chroot_base>/<exec_file_name>/<id>/root/<exec_file_name>`.
- Create the `cgroup` sub-folders. At the moment, the jailer uses three
  `cgroup v1` controllers: `cpu`, `cpuset`, and `pids`. On most systems, these
  (along with others) are mounted by default somewhere in `/sys/fs/cgroup`
  (they should be mounted by the user otherwise). The jailer will parse
  `/proc/mounts` to detect where each of the three controllers can be found
  (multiple controllers may share the same path). For each identified location
  (referred to as `<cgroup_base>`), the jailer creates the
  `<cgroup_base>/<exec_file_name>/<id>` subfolder, and writes the current pid
  to `<cgroup_base>/<exec_file_name>/<id>/tasks`. Also, the value of
  `numa_node` is written to the appropriate `cpuset.mems` file.
- Call `unshare()` into a new mount namespace, use `pivot_root()` to switch
  the old system root mount point with a new one base in `chroot_dir`, switch
  the current working directory to the new root, unmount the old root mount
  point, and call `chroot` into the current directory.
- Use `mknod` to create a `/dev/net/tun` equivalent inside the jail.
- If `--netns <netns>` is present, attempt to join the specified network
  namespace.
- If `--daemonize` is specified, call `setsid()` and redirect `STDIN`,
  `STDOUT`, and `STDERR` to `/dev/null`.
- Drop privileges via setting the provided `uid` and `gid`.
- Exec into `<exec_file_name> --context=<context_json>`. `context_json` is a
  structure to pass parameters from the jailer to Firecracker. The json string
  includes the following keys:
  - `id`: (`string`) - The `id` argument provided to the jailer.
  - `jailed`: (`boolean`) value (`true` in this case) to let Firecracker know
    that it's running inside a jail.
  - `seccomp_level`: (`number`) the `--seccomp-level` argument provided to the
    jailer.

## Example Run and Notes

Let’s assume Firecracker is available as `/usr/bin/firecracker`, and the jailer
can be found at `/usr/bin/jailer`. We pick the **unique id
551e7604-e35c-42b3-b825-416853441234**, and we choose to run on **NUMA node
0**, using **uid 123**, and **gid 100**. For this example, we are content with
the default `/srv/jailer` chroot base dir.

We start by running:

``` bash
/usr/bin/jailer --id 551e7604-e35c-42b3-b825-416853441234 --node 0 --exec-file \
/usr/bin/firecracker --uid 123 --gid 100 --netns /var/run/netns/my_netns \
--daemonize
```

After opening the file descriptors mentioned in the previous section, the
jailer will create the following resources (and all their prerequisites, such
as the path which contains them):

- `/srv/jailer/firecracker/551e7604-e35c-42b3-b825-416853441234/api.socket`
  (created via `bind`)
- `/srv/jailer/firecracker/551e7604-e35c-42b3-b825-416853441234/root/firecracker`
  (copied from `/usr/bin/firecracker`)

We are going to refer to
`/srv/jailer/firecracker/551e7604-e35c-42b3-b825-416853441234/root`
as `<chroot_dir>`.

{'fraction_non_alphanumeric': 0.10436915251158144, 'fraction_numerical': 0.02815877918067036, 'mean_word_length': 3.7973856209150325, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 61, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '601395', 'n_tokens_mistral': 3770, 'n_tokens_neox': 3464, 'n_words': 1408}