This resolves #2189.

> build isolation would be a bit difficult to use in our case, as for instance when building scipy we need the patched numpy on the host and not the numpy version specified in pyproject.toml (which would be unpatched)

This is indeed the case, certain packages cannot be isolated. My strategy is to make a list of packages that shouldn't be isolated and add symlinks from the isolated build environment into the `.artifacts` directory to "unisolate" them. Then we remove the unisolated package requirements from the list of packages to install, in case pesky constraints aren't satisfied. In particular, packages that expect to be used with `pypa/build` often feel free to put very specific constraints on their build dependencies (often asking them to be == to a particular version). Specific version constraints is good for build reproducibility and with build isolation doesn't cost anything. So we just ignore the constraints. Hopefully nothing goes wrong.

In particular, any package that does stuff both at build time and at runtime and requires synchronization between the build time and run time environments needs the unisolation. This includes cffi with `_cffi_backend.so`, and of course numpy and scipy. pycparser needs to be unisolated because it is a dependency of cffi.

Currently I have also unisolated pythran and cython, though these are build time only tools and do not really need to be unisolated. Cython I unisolated specifically because numcodecs needs it but it isn't in the numcodecs build dependencies. Pythran I unisolated because of a problem with the scipy build which I don't fully understand (some problem with long double feature detection). I believe both of these are removable, particularly once #2132 is finished.
Use pypa/build. This resolves #2189.

> build isolation would be a bit difficult to use in our case, as for instance when building scipy we need the patched numpy on the host and not the numpy version specified in pyproject.toml (which would be unpatched)

This is indeed the case, certain packages cannot be isolated. My strategy is to make a list of packages that shouldn't be isolated and add symlinks from the isolated build environment into the `.artifacts` directory to "unisolate" them. Then we remove the unisolated package requirements from the list of packages to install, in case pesky constraints aren't satisfied. In particular, packages that expect to be used with `pypa/build` often feel free to put very specific constraints on their build dependencies (often asking them to be == to a particular version). Specific version constraints is good for build reproducibility and with build isolation doesn't cost anything. So we just ignore the constraints. Hopefully nothing goes wrong.

In particular, any package that does stuff both at build time and at runtime and requires synchronization between the build time and run time environments needs the unisolation. This includes cffi with `_cffi_backend.so`, and of course numpy and scipy. pycparser needs to be unisolated because it is a dependency of cffi.

Currently I have also unisolated pythran and cython, though these are build time only tools and do not really need to be unisolated. Cython I unisolated specifically because numcodecs needs it but it isn't in the numcodecs build dependencies. Pythran I unisolated because of a problem with the scipy build which I don't fully understand (some problem with long double feature detection). I believe both of these are removable, particularly once #2132 is finished.
#ifndef PYPROXY_H
#define PYPROXY_H
#define PY_SSIZE_T_CLEAN
#include "Python.h"

/**
 * Makes Python objects usable from JavaScript.
 */

// This implements the JavaScript Proxy handler interface as defined here:
// https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Proxy

JsRef
pyproxy_new(PyObject* obj);

int
pyproxy_Check(JsRef x);

/**
 * Check if x is a PyProxy.
 *
 * Will fatally fail if x is not NULL or a valid JsRef.
 */
int
pyproxy_Check(JsRef x);

/**
 * Destroy a list of PyProxies.
 */
void
destroy_proxies(JsRef proxies_id, char* msg);

/**
 * Destroy a PyProxy.
 */
void
destroy_proxy(JsRef proxy, char* msg);

/**
 * Wrap a Python callable in a JavaScript function that can be called once.
 * After being called, the reference count of the python object is automatically
 * decremented. The Proxy also has a "destroy" API that can decrement the
 * reference count without calling the function.
 */
JsRef
create_once_callable(PyObject* obj);

/**
 * Wrap a pair of Python callables in a JavaScript function that can be called
 * once between the two of them. After being called, the reference counts of
 * both python objects are automatically decremented. The wrappers also have a
 * "destroy" API that can decrement the reference counts without calling the
 * function. Intended for use with `promise.then`.
 */
JsRef
create_promise_handles(PyObject* onfulfilled,
                       PyObject* onrejected,
                       JsRef done_callback_id);

int
pyproxy_init();

// These are defined as an enum in Python.h but we want to use them in
// pyproxy.ts.
#define PYGEN_NEXT 1
#define PYGEN_RETURN 0
#define PYGEN_ERROR -1

#endif /* PYPROXY_H */

#ifndef PYPROXY_H
#define PYPROXY_H
#define PY_SSIZE_T_CLEAN
#include "Python.h"

/**
 * Makes Python objects usable from JavaScript.
 */

// This implements the JavaScript Proxy handler interface as defined here:
// https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Proxy

JsRef
pyproxy_new(PyObject* obj);

int
pyproxy_Check(JsRef x);

/**
 * Check if x is a PyProxy.
 *
 * Will fatally fail if x is not NULL or a valid JsRef.
 */
int
pyproxy_Check(JsRef x);

/**
 * Destroy a list of PyProxies.
 */
void
destroy_proxies(JsRef proxies_id, char* msg);

/**
 * Destroy a PyProxy.
 */
void
destroy_proxy(JsRef proxy, char* msg);

/**
 * Wrap a Python callable in a JavaScript function that can be called once.
 * After being called, the reference count of the python object is automatically
 * decremented. The Proxy also has a "destroy" API that can decrement the
 * reference count without calling the function.
 */
JsRef
create_once_callable(PyObject* obj);

/**
 * Wrap a pair of Python callables in a JavaScript function that can be called
 * once between the two of them. After being called, the reference counts of
 * both python objects are automatically decremented. The wrappers also have a
 * "destroy" API that can decrement the reference counts without calling the
 * function. Intended for use with `promise.then`.
 */
JsRef
create_promise_handles(PyObject* onfulfilled,
                       PyObject* onrejected,
                       JsRef done_callback_id);

int
pyproxy_init();

// These are defined as an enum in Python.h but we want to use them in
// pyproxy.ts.
#define PYGEN_NEXT 1
#define PYGEN_RETURN 0
#define PYGEN_ERROR -1

#endif /* PYPROXY_H */
