This resolves #2189.

> build isolation would be a bit difficult to use in our case, as for instance when building scipy we need the patched numpy on the host and not the numpy version specified in pyproject.toml (which would be unpatched)

This is indeed the case, certain packages cannot be isolated. My strategy is to make a list of packages that shouldn't be isolated and add symlinks from the isolated build environment into the `.artifacts` directory to "unisolate" them. Then we remove the unisolated package requirements from the list of packages to install, in case pesky constraints aren't satisfied. In particular, packages that expect to be used with `pypa/build` often feel free to put very specific constraints on their build dependencies (often asking them to be == to a particular version). Specific version constraints is good for build reproducibility and with build isolation doesn't cost anything. So we just ignore the constraints. Hopefully nothing goes wrong.

In particular, any package that does stuff both at build time and at runtime and requires synchronization between the build time and run time environments needs the unisolation. This includes cffi with `_cffi_backend.so`, and of course numpy and scipy. pycparser needs to be unisolated because it is a dependency of cffi.

Currently I have also unisolated pythran and cython, though these are build time only tools and do not really need to be unisolated. Cython I unisolated specifically because numcodecs needs it but it isn't in the numcodecs build dependencies. Pythran I unisolated because of a problem with the scipy build which I don't fully understand (some problem with long double feature detection). I believe both of these are removable, particularly once #2132 is finished.
Use pypa/build. This resolves #2189.

> build isolation would be a bit difficult to use in our case, as for instance when building scipy we need the patched numpy on the host and not the numpy version specified in pyproject.toml (which would be unpatched)

This is indeed the case, certain packages cannot be isolated. My strategy is to make a list of packages that shouldn't be isolated and add symlinks from the isolated build environment into the `.artifacts` directory to "unisolate" them. Then we remove the unisolated package requirements from the list of packages to install, in case pesky constraints aren't satisfied. In particular, packages that expect to be used with `pypa/build` often feel free to put very specific constraints on their build dependencies (often asking them to be == to a particular version). Specific version constraints is good for build reproducibility and with build isolation doesn't cost anything. So we just ignore the constraints. Hopefully nothing goes wrong.

In particular, any package that does stuff both at build time and at runtime and requires synchronization between the build time and run time environments needs the unisolation. This includes cffi with `_cffi_backend.so`, and of course numpy and scipy. pycparser needs to be unisolated because it is a dependency of cffi.

Currently I have also unisolated pythran and cython, though these are build time only tools and do not really need to be unisolated. Cython I unisolated specifically because numcodecs needs it but it isn't in the numcodecs build dependencies. Pythran I unisolated because of a problem with the scipy build which I don't fully understand (some problem with long double feature detection). I believe both of these are removable, particularly once #2132 is finished.
From 16abd462459b36c73db23b776f1eb5ab2c0c7813 Mon Sep 17 00:00:00 2001
From: Hood <hood@mit.edu>
Date: Thu, 24 Jun 2021 14:55:10 -0700
Subject: [PATCH 09/10] xfail core ctypes tests that don't work

PyCode_SetExtra doesn't work and ctypes doesn't seem to work with variadic functions including PyUnicode_FromFormat/
---
 Lib/test/test_code.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/Lib/test/test_code.py b/Lib/test/test_code.py
index b9c4f8bdcb..31cb8876c8 100644
--- a/Lib/test/test_code.py
+++ b/Lib/test/test_code.py
@@ -396,6 +396,7 @@ def test_bad_index(self):
                               ctypes.c_voidp(100)), 0)
 
         def test_free_called(self):
+            raise unittest.SkipTest("PyCode_SetExtra is broken")
             # Verify that the provided free function gets invoked
             # when the code object is cleaned up.
             f = self.get_func()
@@ -405,6 +406,7 @@ def test_free_called(self):
             self.assertEqual(LAST_FREED, 100)
 
         def test_get_set(self):
+            raise unittest.SkipTest("PyCode_SetExtra is broken")
             # Test basic get/set round tripping.
             f = self.get_func()
 
@@ -421,6 +423,7 @@ def test_get_set(self):
             del f
 
         def test_free_different_thread(self):
+            raise unittest.SkipTest("Requires threading")
             # Freeing a code object on a different thread then
             # where the co_extra was set should be safe.
             f = self.get_func()
@@ -445,8 +448,8 @@ def test_main(verbose=None):
     from test import test_code
     run_doctest(test_code, verbose)
     tests = [CodeTest, CodeConstsTest, CodeWeakRefTest]
-    if check_impl_detail(cpython=True) and ctypes is not None:
-        tests.append(CoExtra)
+    # if check_impl_detail(cpython=True) and ctypes is not None:
+    #     tests.append(CoExtra)
     run_unittest(*tests)
 
 if __name__ == "__main__":
-- 
2.25.1


From 16abd462459b36c73db23b776f1eb5ab2c0c7813 Mon Sep 17 00:00:00 2001
From: Hood <hood@mit.edu>
Date: Thu, 24 Jun 2021 14:55:10 -0700
Subject: [PATCH 09/10] xfail core ctypes tests that don't work

PyCode_SetExtra doesn't work and ctypes doesn't seem to work with variadic functions including PyUnicode_FromFormat/
---
 Lib/test/test_code.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/Lib/test/test_code.py b/Lib/test/test_code.py
index b9c4f8bdcb..31cb8876c8 100644
--- a/Lib/test/test_code.py
+++ b/Lib/test/test_code.py
@@ -396,6 +396,7 @@ def test_bad_index(self):
                               ctypes.c_voidp(100)), 0)
 
         def test_free_called(self):
+            raise unittest.SkipTest("PyCode_SetExtra is broken")
             # Verify that the provided free function gets invoked
             # when the code object is cleaned up.
             f = self.get_func()
@@ -405,6 +406,7 @@ def test_free_called(self):
             self.assertEqual(LAST_FREED, 100)
 
         def test_get_set(self):
+            raise unittest.SkipTest("PyCode_SetExtra is broken")
             # Test basic get/set round tripping.
             f = self.get_func()
 
@@ -421,6 +423,7 @@ def test_get_set(self):
             del f
 
         def test_free_different_thread(self):
+            raise unittest.SkipTest("Requires threading")
             # Freeing a code object on a different thread then
             # where the co_extra was set should be safe.
             f = self.get_func()
@@ -445,8 +448,8 @@ def test_main(verbose=None):
     from test import test_code
     run_doctest(test_code, verbose)
     tests = [CodeTest, CodeConstsTest, CodeWeakRefTest]
-    if check_impl_detail(cpython=True) and ctypes is not None:
-        tests.append(CoExtra)
+    # if check_impl_detail(cpython=True) and ctypes is not None:
+    #     tests.append(CoExtra)
     run_unittest(*tests)
 
 if __name__ == "__main__":
-- 
2.25.1

