This resolves #2189.

> build isolation would be a bit difficult to use in our case, as for instance when building scipy we need the patched numpy on the host and not the numpy version specified in pyproject.toml (which would be unpatched)

This is indeed the case, certain packages cannot be isolated. My strategy is to make a list of packages that shouldn't be isolated and add symlinks from the isolated build environment into the `.artifacts` directory to "unisolate" them. Then we remove the unisolated package requirements from the list of packages to install, in case pesky constraints aren't satisfied. In particular, packages that expect to be used with `pypa/build` often feel free to put very specific constraints on their build dependencies (often asking them to be == to a particular version). Specific version constraints is good for build reproducibility and with build isolation doesn't cost anything. So we just ignore the constraints. Hopefully nothing goes wrong.

In particular, any package that does stuff both at build time and at runtime and requires synchronization between the build time and run time environments needs the unisolation. This includes cffi with `_cffi_backend.so`, and of course numpy and scipy. pycparser needs to be unisolated because it is a dependency of cffi.

Currently I have also unisolated pythran and cython, though these are build time only tools and do not really need to be unisolated. Cython I unisolated specifically because numcodecs needs it but it isn't in the numcodecs build dependencies. Pythran I unisolated because of a problem with the scipy build which I don't fully understand (some problem with long double feature detection). I believe both of these are removable, particularly once #2132 is finished.
Use pypa/build. This resolves #2189.

> build isolation would be a bit difficult to use in our case, as for instance when building scipy we need the patched numpy on the host and not the numpy version specified in pyproject.toml (which would be unpatched)

This is indeed the case, certain packages cannot be isolated. My strategy is to make a list of packages that shouldn't be isolated and add symlinks from the isolated build environment into the `.artifacts` directory to "unisolate" them. Then we remove the unisolated package requirements from the list of packages to install, in case pesky constraints aren't satisfied. In particular, packages that expect to be used with `pypa/build` often feel free to put very specific constraints on their build dependencies (often asking them to be == to a particular version). Specific version constraints is good for build reproducibility and with build isolation doesn't cost anything. So we just ignore the constraints. Hopefully nothing goes wrong.

In particular, any package that does stuff both at build time and at runtime and requires synchronization between the build time and run time environments needs the unisolation. This includes cffi with `_cffi_backend.so`, and of course numpy and scipy. pycparser needs to be unisolated because it is a dependency of cffi.

Currently I have also unisolated pythran and cython, though these are build time only tools and do not really need to be unisolated. Cython I unisolated specifically because numcodecs needs it but it isn't in the numcodecs build dependencies. Pythran I unisolated because of a problem with the scipy build which I don't fully understand (some problem with long double feature detection). I believe both of these are removable, particularly once #2132 is finished.
export PYVERSION ?= 3.10.2
export PYODIDE_EMSCRIPTEN_VERSION ?= 2.0.16

# BASH_ENV tells bash to run pyodide_env.sh on startup, which sets various
# environment variables. The next line instructs make to use bash to run each
# command.
export BASH_ENV := $(PYODIDE_ROOT)/pyodide_env.sh
SHELL := /bin/bash

export TOOLSDIR=$(PYODIDE_ROOT)/tools

version_tuple := $(subst ., ,$(PYVERSION:v%=%))
export PYMAJOR=$(word 1,$(version_tuple))
export PYMINOR=$(word 2,$(version_tuple))
export PYMICRO=$(word 3,$(version_tuple))

export HOSTPYTHONROOT=$(shell python${PYMAJOR}.${PYMINOR} -c "import sys; print(sys.prefix)")
export HOSTPYTHON=$(HOSTPYTHONROOT)/bin/python$(PYMAJOR).$(PYMINOR)

export CPYTHONROOT=$(PYODIDE_ROOT)/cpython
export CPYTHONLIB=$(CPYTHONROOT)/installs/python-$(PYVERSION)/lib/python$(PYMAJOR).$(PYMINOR)
export CPYTHONBUILD=$(CPYTHONROOT)/build/Python-$(PYVERSION)/

export TARGETINSTALLDIR=$(PYODIDE_ROOT)/cpython/installs/python-$(PYVERSION)
export HOSTINSTALLDIR=$(PYODIDE_ROOT)/packages/.artifacts

export PYTHONINCLUDE=$(PYODIDE_ROOT)/cpython/installs/python-$(PYVERSION)/include/python$(PYMAJOR).$(PYMINOR)

# Use env variable if defined, otherwise fallback to './'
export PYODIDE_BASE_URL?=./

# For packages that depend on numpy.
# TODO: maybe move this somewhere else?
export NUMPY_LIB=$(HOSTINSTALLDIR)/numpy-wasm-libs

# This environment variable is used for packages to detect if they are built
# for pyodide during build time
export PYODIDE=1
# This is the legacy environment variable used for the aforementioned purpose
export PYODIDE_PACKAGE_ABI=1

# export DBGFLAGS=-g3 -gseparate-dwarf -sSEPARATE_DWARF_URL=http://localhost:8001/
export DBGFLAGS=-g0

export OPTFLAGS=-O2
export CFLAGS_BASE=\
	$(OPTFLAGS) \
	$(DBGFLAGS) \
	-fPIC \
	$(EXTRA_CFLAGS)

export LDFLAGS_BASE=\
	$(OPTFLAGS) \
	$(DBGFLAGS) \
	-s MODULARIZE=1 \
	-s LINKABLE=1 \
	-s EXPORT_ALL=1 \
	-s WASM=1 \
	-std=c++14 \
	-s LZ4=1 \
	-L $(CPYTHONROOT)/installs/python-$(PYVERSION)/lib/ \
	$(EXTRA_LDFLAGS)

export CXXFLAGS_BASE=

export SIDE_MODULE_LDFLAGS=	$(LDFLAGS_BASE) -s SIDE_MODULE=1
export MAIN_MODULE_LDFLAGS= $(LDFLAGS_BASE) \
	-s MAIN_MODULE=1 \
	-s EXPORT_NAME="'_createPyodideModule'" \
	-s EXPORTED_FUNCTIONS='["___cxa_guard_acquire", "__ZNSt3__28ios_base4initEPv", "_main"]' \
	-s EXCEPTION_CATCHING_ALLOWED=['we only want to allow exception handling in side modules'] \
	-s DEMANGLE_SUPPORT=1 \
	-s USE_FREETYPE=1 \
	-s USE_LIBPNG=1 \
	-s FORCE_FILESYSTEM=1 \
	-s TOTAL_MEMORY=20971520 \
	-s ALLOW_MEMORY_GROWTH=1 \
	\
	-lpython$(PYMAJOR).$(PYMINOR) \
	-lffi \
	-lsqlite3 \
	-lbz2 \
	-lstdc++ \
	-lidbfs.js \
	-lnodefs.js \
	-lproxyfs.js \
	-lworkerfs.js \
	\
	--use-preload-plugins \
	--preload-file $(CPYTHONLIB)@/lib/python$(PYMAJOR).$(PYMINOR) \
	--preload-file src/py/lib@/lib/python$(PYMAJOR).$(PYMINOR)/\
	--exclude-file "*__pycache__*" \
	--exclude-file "*/test/*" \
	--exclude-file "*/tests/*" \
	--exclude-file "*/distutils/*" \
	--pre-js src/core/pre.js


export SIDE_MODULE_CXXFLAGS = $(CXXFLAGS_BASE)

export SIDE_MODULE_CFLAGS= $(CFLAGS_BASE)
export MAIN_MODULE_CFLAGS= $(CFLAGS_BASE) \
	-Wall \
	-Wno-warn-absolute-paths \
	-Werror=unused-variable \
	-Werror=sometimes-uninitialized \
	-Werror=int-conversion \
	-Werror=incompatible-pointer-types \
	-Werror=unused-result \
	-I$(PYTHONINCLUDE) \
	-s EXCEPTION_CATCHING_ALLOWED=['we only want to allow exception handling in side modules']

export STDLIB_MODULE_CFLAGS= $(SIDE_MODULE_CFLAGS) -I Include/ -I .

.output_vars:
	set

export PYVERSION ?= 3.10.2
export PYODIDE_EMSCRIPTEN_VERSION ?= 2.0.16

# BASH_ENV tells bash to run pyodide_env.sh on startup, which sets various
# environment variables. The next line instructs make to use bash to run each
# command.
export BASH_ENV := $(PYODIDE_ROOT)/pyodide_env.sh
SHELL := /bin/bash

export TOOLSDIR=$(PYODIDE_ROOT)/tools

version_tuple := $(subst ., ,$(PYVERSION:v%=%))
export PYMAJOR=$(word 1,$(version_tuple))
export PYMINOR=$(word 2,$(version_tuple))
export PYMICRO=$(word 3,$(version_tuple))

export HOSTPYTHONROOT=$(shell python${PYMAJOR}.${PYMINOR} -c "import sys; print(sys.prefix)")
export HOSTPYTHON=$(HOSTPYTHONROOT)/bin/python$(PYMAJOR).$(PYMINOR)

export CPYTHONROOT=$(PYODIDE_ROOT)/cpython
export CPYTHONLIB=$(CPYTHONROOT)/installs/python-$(PYVERSION)/lib/python$(PYMAJOR).$(PYMINOR)
export CPYTHONBUILD=$(CPYTHONROOT)/build/Python-$(PYVERSION)/

export TARGETINSTALLDIR=$(PYODIDE_ROOT)/cpython/installs/python-$(PYVERSION)
export HOSTINSTALLDIR=$(PYODIDE_ROOT)/packages/.artifacts
export HOSTSITEPACKAGES=$(PYODIDE_ROOT)/packages/.artifacts/lib/python$(PYMAJOR).$(PYMINOR)/site-packages

export PYTHONINCLUDE=$(PYODIDE_ROOT)/cpython/installs/python-$(PYVERSION)/include/python$(PYMAJOR).$(PYMINOR)

# Use env variable if defined, otherwise fallback to './'
export PYODIDE_BASE_URL?=./

# For packages that depend on numpy.
# TODO: maybe move this somewhere else?
export NUMPY_LIB=$(HOSTINSTALLDIR)/numpy-wasm-libs

# This environment variable is used for packages to detect if they are built
# for pyodide during build time
export PYODIDE=1
# This is the legacy environment variable used for the aforementioned purpose
export PYODIDE_PACKAGE_ABI=1

# export DBGFLAGS=-g3 -gseparate-dwarf -sSEPARATE_DWARF_URL=http://localhost:8001/
export DBGFLAGS=-g0

export OPTFLAGS=-O2
export CFLAGS_BASE=\
	$(OPTFLAGS) \
	$(DBGFLAGS) \
	-fPIC \
	$(EXTRA_CFLAGS)

export LDFLAGS_BASE=\
	$(OPTFLAGS) \
	$(DBGFLAGS) \
	-s MODULARIZE=1 \
	-s LINKABLE=1 \
	-s EXPORT_ALL=1 \
	-s WASM=1 \
	-std=c++14 \
	-s LZ4=1 \
	-L $(CPYTHONROOT)/installs/python-$(PYVERSION)/lib/ \
	$(EXTRA_LDFLAGS)

export CXXFLAGS_BASE=

export SIDE_MODULE_LDFLAGS=	$(LDFLAGS_BASE) -s SIDE_MODULE=1
export MAIN_MODULE_LDFLAGS= $(LDFLAGS_BASE) \
	-s MAIN_MODULE=1 \
	-s EXPORT_NAME="'_createPyodideModule'" \
	-s EXPORTED_FUNCTIONS='["___cxa_guard_acquire", "__ZNSt3__28ios_base4initEPv", "_main"]' \
	-s EXCEPTION_CATCHING_ALLOWED=['we only want to allow exception handling in side modules'] \
	-s DEMANGLE_SUPPORT=1 \
	-s USE_FREETYPE=1 \
	-s USE_LIBPNG=1 \
	-s FORCE_FILESYSTEM=1 \
	-s TOTAL_MEMORY=20971520 \
	-s ALLOW_MEMORY_GROWTH=1 \
	\
	-lpython$(PYMAJOR).$(PYMINOR) \
	-lffi \
	-lsqlite3 \
	-lbz2 \
	-lstdc++ \
	-lidbfs.js \
	-lnodefs.js \
	-lproxyfs.js \
	-lworkerfs.js \
	\
	--use-preload-plugins \
	--preload-file $(CPYTHONLIB)@/lib/python$(PYMAJOR).$(PYMINOR) \
	--preload-file src/py/lib@/lib/python$(PYMAJOR).$(PYMINOR)/\
	--exclude-file "*__pycache__*" \
	--exclude-file "*/test/*" \
	--exclude-file "*/tests/*" \
	--exclude-file "*/distutils/*" \
	--pre-js src/core/pre.js


export SIDE_MODULE_CXXFLAGS = $(CXXFLAGS_BASE)

export SIDE_MODULE_CFLAGS= $(CFLAGS_BASE)
export MAIN_MODULE_CFLAGS= $(CFLAGS_BASE) \
	-Wall \
	-Wno-warn-absolute-paths \
	-Werror=unused-variable \
	-Werror=sometimes-uninitialized \
	-Werror=int-conversion \
	-Werror=incompatible-pointer-types \
	-Werror=unused-result \
	-I$(PYTHONINCLUDE) \
	-s EXCEPTION_CATCHING_ALLOWED=['we only want to allow exception handling in side modules']

export STDLIB_MODULE_CFLAGS= $(SIDE_MODULE_CFLAGS) -I Include/ -I .

.output_vars:
	set
