This resolves #2189.

> build isolation would be a bit difficult to use in our case, as for instance when building scipy we need the patched numpy on the host and not the numpy version specified in pyproject.toml (which would be unpatched)

This is indeed the case, certain packages cannot be isolated. My strategy is to make a list of packages that shouldn't be isolated and add symlinks from the isolated build environment into the `.artifacts` directory to "unisolate" them. Then we remove the unisolated package requirements from the list of packages to install, in case pesky constraints aren't satisfied. In particular, packages that expect to be used with `pypa/build` often feel free to put very specific constraints on their build dependencies (often asking them to be == to a particular version). Specific version constraints is good for build reproducibility and with build isolation doesn't cost anything. So we just ignore the constraints. Hopefully nothing goes wrong.

In particular, any package that does stuff both at build time and at runtime and requires synchronization between the build time and run time environments needs the unisolation. This includes cffi with `_cffi_backend.so`, and of course numpy and scipy. pycparser needs to be unisolated because it is a dependency of cffi.

Currently I have also unisolated pythran and cython, though these are build time only tools and do not really need to be unisolated. Cython I unisolated specifically because numcodecs needs it but it isn't in the numcodecs build dependencies. Pythran I unisolated because of a problem with the scipy build which I don't fully understand (some problem with long double feature detection). I believe both of these are removable, particularly once #2132 is finished.
Use pypa/build. This resolves #2189.

> build isolation would be a bit difficult to use in our case, as for instance when building scipy we need the patched numpy on the host and not the numpy version specified in pyproject.toml (which would be unpatched)

This is indeed the case, certain packages cannot be isolated. My strategy is to make a list of packages that shouldn't be isolated and add symlinks from the isolated build environment into the `.artifacts` directory to "unisolate" them. Then we remove the unisolated package requirements from the list of packages to install, in case pesky constraints aren't satisfied. In particular, packages that expect to be used with `pypa/build` often feel free to put very specific constraints on their build dependencies (often asking them to be == to a particular version). Specific version constraints is good for build reproducibility and with build isolation doesn't cost anything. So we just ignore the constraints. Hopefully nothing goes wrong.

In particular, any package that does stuff both at build time and at runtime and requires synchronization between the build time and run time environments needs the unisolation. This includes cffi with `_cffi_backend.so`, and of course numpy and scipy. pycparser needs to be unisolated because it is a dependency of cffi.

Currently I have also unisolated pythran and cython, though these are build time only tools and do not really need to be unisolated. Cython I unisolated specifically because numcodecs needs it but it isn't in the numcodecs build dependencies. Pythran I unisolated because of a problem with the scipy build which I don't fully understand (some problem with long double feature detection). I believe both of these are removable, particularly once #2132 is finished.
name: main

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

concurrency:
  group: main-${{ github.head_ref }}
  cancel-in-progress: true

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v3
        with:
          python-version: 3.10.2
      - uses: pre-commit/action@v2.0.3

  build-core:
    runs-on: ubuntu-latest
    env:
      EMSDK_NUM_CORES: 3
      EMCC_CORES: 3
      PYODIDE_JOBS: 3
      CCACHE_DIR: /tmp/ccache

    steps:
      - uses: actions/checkout@v2

      - name: Cache ccache output
        uses: actions/cache@v2
        with:
          path: |
            /tmp/ccache
          key: ${{ hashFiles('Makefile.envs') }}-v20211025-

      - uses: conda-incubator/setup-miniconda@v2
        with:
          activate-environment: pyodide-env
          python-version: 3.10.2
          channels: conda-forge

      - name: Check Python versions
        shell: bash -l {0}
        run: |
          python --version
          which python

      - name: Install dependencies
        shell: bash -l {0}
        run: |
          sudo apt install -y build-essential git
          conda install -y nodejs ccache f2c pkg-config swig make patch pkg-config texinfo autoconf automake libtool
          pip install -r requirements.txt

      - name: Build emsdk
        shell: bash -l {0}
        run: |
          which ccache

          ccache -z
          make -C emsdk
          ccache -s

      - name: Build Cpython
        shell: bash -l {0}
        run: |
          # This is necessary to use the ccache from emsdk
          source pyodide_env.sh

          which ccache

          ccache -z
          make -C cpython
          ccache -s

      - name: build Pyodide core + numpy
        shell: bash -l {0}
        run: |
          # This is necessary to use the ccache from emsdk
          source pyodide_env.sh
          ccache -z
          PYODIDE_PACKAGES="core,numpy" make
          ccache -s

      - name: check-size
        run: ls -lh build/

      - name: Store artifacts build
        uses: actions/upload-artifact@v2
        with:
          name: core-build
          path: ./build/
          retention-days: 60

  test-core:
    runs-on: ubuntu-latest
    env:
      DISPLAY: :99

    needs: [build-core]
    strategy:
      matrix:
        selenium_runner: [firefox]

    steps:
      - uses: actions/checkout@v2

      - name: Download build artifact
        uses: actions/download-artifact@v2
        with:
          name: core-build
          path: ./build/

      - uses: conda-incubator/setup-miniconda@v2
        with:
          activate-environment: pyodide-env
          python-version: 3.10.2
          channels: conda-forge

      - name: install test requirements
        shell: bash -l {0}
        run: |
          pip install -r requirements.txt

      - name: run core tests
        env:
          SELENIUM_RUNNER: ${{ matrix.selenium_runner }}
        shell: bash -l {0}
        run: |
          ls -lh
          ls -lh build/
          tools/pytest_wrapper.py src packages/micropip/ -v -k "${SELENIUM_RUNNER}"

      - name: run package tests
        env:
          SELENIUM_RUNNER: ${{ matrix.selenium_runner }}
        shell: bash -l {0}
        run: |
          ls -lh
          ls -lh build/
          tools/pytest_wrapper.py packages/test* packages/*/test* -v -k "numpy and not joblib and ${SELENIUM_RUNNER}"

name: main

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

concurrency:
  group: main-${{ github.head_ref }}
  cancel-in-progress: true

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v3
        with:
          python-version: 3.10.2
      - uses: pre-commit/action@v2.0.3

  build-core:
    runs-on: ubuntu-latest
    env:
      EMSDK_NUM_CORES: 3
      EMCC_CORES: 3
      PYODIDE_JOBS: 3
      CCACHE_DIR: /tmp/ccache

    steps:
      - uses: actions/checkout@v2

      - name: Cache ccache output
        uses: actions/cache@v2
        with:
          path: |
            /tmp/ccache
          key: ${{ hashFiles('Makefile.envs') }}-v20211025-

      - uses: conda-incubator/setup-miniconda@v2
        with:
          activate-environment: pyodide-env
          python-version: 3.10.2
          channels: conda-forge

      - name: Check Python versions
        shell: bash -l {0}
        run: |
          python --version
          which python

      - name: Install dependencies
        shell: bash -l {0}
        run: |
          sudo apt install -y build-essential git
          conda install -y nodejs ccache f2c pkg-config swig make patch pkg-config texinfo autoconf automake libtool
          pip install -r requirements.txt

      - name: Build emsdk
        shell: bash -l {0}
        run: |
          which ccache

          ccache -z
          make -C emsdk
          ccache -s

      - name: Build Cpython
        shell: bash -l {0}
        run: |
          # This is necessary to use the ccache from emsdk
          source pyodide_env.sh

          which ccache

          ccache -z
          make -C cpython
          ccache -s

      - name: build Pyodide core + numpy
        shell: bash -l {0}
        run: |
          # This is necessary to use the ccache from emsdk
          source pyodide_env.sh
          ccache -z
          PYODIDE_PACKAGES="core,numpy" make
          ccache -s

      - name: check-size
        run: ls -lh build/

      - name: Store artifacts build
        uses: actions/upload-artifact@v2
        with:
          name: core-build
          path: ./build/
          retention-days: 60

  test-core:
    runs-on: ubuntu-latest
    env:
      DISPLAY: :99

    needs: [build-core]
    strategy:
      matrix:
        selenium_runner: [firefox]

    steps:
      - uses: actions/checkout@v2

      - name: Download build artifact
        uses: actions/download-artifact@v2
        with:
          name: core-build
          path: ./build/

      - uses: conda-incubator/setup-miniconda@v2
        with:
          activate-environment: pyodide-env
          python-version: 3.10.2
          channels: conda-forge

      - name: install test requirements
        shell: bash -l {0}
        run: |
          pip install -r requirements.txt

      - name: run core tests
        env:
          SELENIUM_RUNNER: ${{ matrix.selenium_runner }}
        shell: bash -l {0}
        run: |
          ls -lh
          ls -lh build/
          tools/pytest_wrapper.py src packages/micropip/ -v -k "${SELENIUM_RUNNER}"

      - name: run package tests
        env:
          SELENIUM_RUNNER: ${{ matrix.selenium_runner }}
        shell: bash -l {0}
        run: |
          ls -lh
          ls -lh build/
          tools/pytest_wrapper.py packages/test* packages/*/test* -v -k "numpy and not joblib and ${SELENIUM_RUNNER}"
