This resolves #2189.

> build isolation would be a bit difficult to use in our case, as for instance when building scipy we need the patched numpy on the host and not the numpy version specified in pyproject.toml (which would be unpatched)

This is indeed the case, certain packages cannot be isolated. My strategy is to make a list of packages that shouldn't be isolated and add symlinks from the isolated build environment into the `.artifacts` directory to "unisolate" them. Then we remove the unisolated package requirements from the list of packages to install, in case pesky constraints aren't satisfied. In particular, packages that expect to be used with `pypa/build` often feel free to put very specific constraints on their build dependencies (often asking them to be == to a particular version). Specific version constraints is good for build reproducibility and with build isolation doesn't cost anything. So we just ignore the constraints. Hopefully nothing goes wrong.

In particular, any package that does stuff both at build time and at runtime and requires synchronization between the build time and run time environments needs the unisolation. This includes cffi with `_cffi_backend.so`, and of course numpy and scipy. pycparser needs to be unisolated because it is a dependency of cffi.

Currently I have also unisolated pythran and cython, though these are build time only tools and do not really need to be unisolated. Cython I unisolated specifically because numcodecs needs it but it isn't in the numcodecs build dependencies. Pythran I unisolated because of a problem with the scipy build which I don't fully understand (some problem with long double feature detection). I believe both of these are removable, particularly once #2132 is finished.
Use pypa/build. This resolves #2189.

> build isolation would be a bit difficult to use in our case, as for instance when building scipy we need the patched numpy on the host and not the numpy version specified in pyproject.toml (which would be unpatched)

This is indeed the case, certain packages cannot be isolated. My strategy is to make a list of packages that shouldn't be isolated and add symlinks from the isolated build environment into the `.artifacts` directory to "unisolate" them. Then we remove the unisolated package requirements from the list of packages to install, in case pesky constraints aren't satisfied. In particular, packages that expect to be used with `pypa/build` often feel free to put very specific constraints on their build dependencies (often asking them to be == to a particular version). Specific version constraints is good for build reproducibility and with build isolation doesn't cost anything. So we just ignore the constraints. Hopefully nothing goes wrong.

In particular, any package that does stuff both at build time and at runtime and requires synchronization between the build time and run time environments needs the unisolation. This includes cffi with `_cffi_backend.so`, and of course numpy and scipy. pycparser needs to be unisolated because it is a dependency of cffi.

Currently I have also unisolated pythran and cython, though these are build time only tools and do not really need to be unisolated. Cython I unisolated specifically because numcodecs needs it but it isn't in the numcodecs build dependencies. Pythran I unisolated because of a problem with the scipy build which I don't fully understand (some problem with long double feature detection). I believe both of these are removable, particularly once #2132 is finished.
"""
Generate a list of test modules in the CPython distribution.
"""

import os
from pathlib import Path
from sys import version_info

TEST_DIR = (
    Path(__file__).parents[2] / "cpython/installs"
    f"/python-{version_info.major}.{version_info.minor}.{version_info.micro}"
    f"/lib/python{version_info.major}.{version_info.minor}/test/"
)


explanation = """\
# This list is generated with test/make_test_list.py script, which needs
# to be re-run after each CPython update.
#
# Test modules with a failure reason after their name are either skipped
# or marked as a known failure in pytest.
#
# Following reason codes are skipped, as they lead to segfaults:
# - segfault-<syscall>: segfault in the corresponding system call.
#
# While the below reason codes are marked as a known failure. By default, they
# are also skipped. To run them, provide --run-xfail argument to pytest,
# - platform-specific: This is testing something about a particular platform
#   that isn't relevant here
# - async: relies on async
# - floating point: Failures caused by floating-point differences
# - threading: Failures due to lack of a threading implementation
# - subprocess: Failures caused by no subprocess module. Some of these are
#   because the underlying functionality depends on subprocess, and others are
#   just a side-effect of the way the test is written. The latter should
#   probably be marked as "skip" or rearchitected so we don't have to skip the
#   whole module.
# - networking: Fails because it tests low-level networking.
# - dbm: Failures due to no dbm module
# - strftime: Failures due to differences / shortcomings in WebAssembly's
#   implementation of date/time formatting in strftime and strptime
# - permissions: Issues with the test writing to the virtual filesystem
# - locale: Fails due to limitations in the included locale implementation.
# - multiprocessing: Fails due to no multiprocessing implementation.
# - fs: Fails due to virtual filesystem issues.
# - nonsense: This functionality doesn't make sense in this context. Includes
#   things like `pip`, `distutils`
# - crash: The Python interpreter just stopped without a traceback. Will require
#   further investigation. This usually seems to be caused by calling into a
#   system function that doesn't behave as one would expect.
# - crash-chrome: Same as crash but only affecting Chrome
# - crash-firefox: Same as crash but only affecting Firefox

"""


def collect_old_error_flags():
    old_error_flags = {}
    try:
        with open(Path(__file__).parent / "python_tests.txt") as fp:
            for line in fp:
                line = line.strip()
                if line.startswith("#") or not line:
                    continue
                error_flags = line.split()
                name = error_flags.pop(0)
                if error_flags:
                    old_error_flags[name] = error_flags
    except FileNotFoundError:
        pass
    return old_error_flags


def collect_tests(base_dir):
    """Collect CPython unit tests"""
    # Note: this functionality is somewhat equivalent to pytest test
    # collection.
    tests = []

    for root, _dirs, files in os.walk(base_dir):
        root = Path(root).relative_to(base_dir)

        if str(root) == ".":
            root = ""
        else:
            root = ".".join(str(root).split("/")) + "."

        for filename in files:
            filename = Path(filename)
            if str(filename).startswith("test_") and filename.suffix == ".py":
                tests.append(root + filename.stem)

    tests.sort()
    return tests


if __name__ == "__main__":
    old_error_flags = collect_old_error_flags()
    tests = collect_tests(TEST_DIR)
    with open("python_tests.txt", "w") as fp:
        fp.write(explanation)
        for test in tests:
            error_flags = " ".join(old_error_flags.get(test, []))
            line = test
            if error_flags:
                line += "    " + error_flags
            fp.write(line + "\n")

"""
Generate a list of test modules in the CPython distribution.
"""

import os
from pathlib import Path
from sys import version_info

TEST_DIR = (
    Path(__file__).parents[2] / "cpython/installs"
    f"/python-{version_info.major}.{version_info.minor}.{version_info.micro}"
    f"/lib/python{version_info.major}.{version_info.minor}/test/"
)


explanation = """\
# This list is generated with test/make_test_list.py script, which needs
# to be re-run after each CPython update.
#
# Test modules with a failure reason after their name are either skipped
# or marked as a known failure in pytest.
#
# Following reason codes are skipped, as they lead to segfaults:
# - segfault-<syscall>: segfault in the corresponding system call.
#
# While the below reason codes are marked as a known failure. By default, they
# are also skipped. To run them, provide --run-xfail argument to pytest,
# - platform-specific: This is testing something about a particular platform
#   that isn't relevant here
# - async: relies on async
# - floating point: Failures caused by floating-point differences
# - threading: Failures due to lack of a threading implementation
# - subprocess: Failures caused by no subprocess module. Some of these are
#   because the underlying functionality depends on subprocess, and others are
#   just a side-effect of the way the test is written. The latter should
#   probably be marked as "skip" or rearchitected so we don't have to skip the
#   whole module.
# - networking: Fails because it tests low-level networking.
# - dbm: Failures due to no dbm module
# - strftime: Failures due to differences / shortcomings in WebAssembly's
#   implementation of date/time formatting in strftime and strptime
# - permissions: Issues with the test writing to the virtual filesystem
# - locale: Fails due to limitations in the included locale implementation.
# - multiprocessing: Fails due to no multiprocessing implementation.
# - fs: Fails due to virtual filesystem issues.
# - nonsense: This functionality doesn't make sense in this context. Includes
#   things like `pip`, `distutils`
# - crash: The Python interpreter just stopped without a traceback. Will require
#   further investigation. This usually seems to be caused by calling into a
#   system function that doesn't behave as one would expect.
# - crash-chrome: Same as crash but only affecting Chrome
# - crash-firefox: Same as crash but only affecting Firefox

"""


def collect_old_error_flags():
    old_error_flags = {}
    try:
        with open(Path(__file__).parent / "python_tests.txt") as fp:
            for line in fp:
                line = line.strip()
                if line.startswith("#") or not line:
                    continue
                error_flags = line.split()
                name = error_flags.pop(0)
                if error_flags:
                    old_error_flags[name] = error_flags
    except FileNotFoundError:
        pass
    return old_error_flags


def collect_tests(base_dir):
    """Collect CPython unit tests"""
    # Note: this functionality is somewhat equivalent to pytest test
    # collection.
    tests = []

    for root, _dirs, files in os.walk(base_dir):
        root = Path(root).relative_to(base_dir)

        if str(root) == ".":
            root = ""
        else:
            root = ".".join(str(root).split("/")) + "."

        for filename in files:
            filename = Path(filename)
            if str(filename).startswith("test_") and filename.suffix == ".py":
                tests.append(root + filename.stem)

    tests.sort()
    return tests


if __name__ == "__main__":
    old_error_flags = collect_old_error_flags()
    tests = collect_tests(TEST_DIR)
    with open("python_tests.txt", "w") as fp:
        fp.write(explanation)
        for test in tests:
            error_flags = " ".join(old_error_flags.get(test, []))
            line = test
            if error_flags:
                line += "    " + error_flags
            fp.write(line + "\n")
