Instead of trying to convince setup.py to do what we want, the wheel
package empowers us to unpack and directly manipulate the wheel.
In particular this is important because `--skip-build` really does
not work.

First we build a native wheel. If skip_host is True, then this
"native" wheel has defective empty files as binaries, if it's False,
we build a real native wheel. If skip_host is False, then we use
pip to install our native wheel into host directory.

Then we unpack the wheel, replace the platform tag with
wasm32_emscripten, remove the non-metadata files, and copy in the
new files with the cross compiled binaries.

Of course, this is all only necessary for binary extensions.


Hopefully this will help get us towards using alternative build
backends. The one concern is that this approach depends on
implementation details of where the build backend puts the files
into the directory tree before copying them into the wheel. If
the build backend moved the linked binaries after building them,
we would be in trouble. Or worse if the build backend deleted the
files after packing them into the wheel, we'd be in trouble.

Basically, it seems clear that we can only support alternative
build backends on a case-by-case basis.

@henryiii I'm curious what your thoughts are cross compilation + arbitrary build backends.

Better package cross compilation. Instead of trying to convince setup.py to do what we want, the wheel
package empowers us to unpack and directly manipulate the wheel.
In particular this is important because `--skip-build` really does
not work.

First we build a native wheel. If skip_host is True, then this
"native" wheel has defective empty files as binaries, if it's False,
we build a real native wheel. If skip_host is False, then we use
pip to install our native wheel into host directory.

Then we unpack the wheel, replace the platform tag with
wasm32_emscripten, remove the non-metadata files, and copy in the
new files with the cross compiled binaries.

Of course, this is all only necessary for binary extensions.


Hopefully this will help get us towards using alternative build
backends. The one concern is that this approach depends on
implementation details of where the build backend puts the files
into the directory tree before copying them into the wheel. If
the build backend moved the linked binaries after building them,
we would be in trouble. Or worse if the build backend deleted the
files after packing them into the wheel, we'd be in trouble.

Basically, it seems clear that we can only support alternative
build backends on a case-by-case basis.

@henryiii I'm curious what your thoughts are cross compilation + arbitrary build backends.

From 320cc07c0a30436759cde8f4c926e76ed8dc918a Mon Sep 17 00:00:00 2001
From: Hood <hood@mit.edu>
Date: Fri, 3 Sep 2021 18:08:26 -0700
Subject: [PATCH] Patch in keyboard interrupt handling

---
 Include/cpython/ceval.h   |  1 +
 Include/cpython/pystate.h |  3 +++
 Modules/signalmodule.c    |  3 +++
 Python/ceval.c            | 13 ++++++++++++-
 Python/pystate.c          |  2 ++
 5 files changed, 21 insertions(+), 1 deletion(-)

This patch adds a callback called pyodide_callback with signature 
`int callback(void)` to the main loop in `ceval.c`. This function gets called once 
per opcode except after opcodes `SETUP_FINALLY`, `SETUP_WITH`, `BEFORE_ASYNC_WITH`,
and `YIELD_FROM`. The main loop normally prevents signal handling, etc from happening
after these instructions, so I figured this should apply to my callback too.
(There is an extra layer of protection here though because in the callback we use 
`PyErr_SetInterrupt` which simulates a SIGINT signal and triggers the KeyboardInterrupt
via the standard mechanism.)

Note that we call the callback outside of the normal 
`if (_Py_atomic_load_relaxed(eval_breaker))` 
block where most of the "periodic things" happen. This is because normally when a 
"periodic thing" is queued, the `eval_breaker` flag is set signalling the need for
handling. The whole point of this patch though is that we need to be able to set an
interrupt from a remote thread and the `eval_breaker` flag lives on the WASM heap.
Unless the whole WASM heap is made into a `SharedArrayBuffer` and shared between 
webworkers, we have no way to set the `eval_breaker` flag. We still want to skip
the callback after the sensitive opcodes `SETUP_FINALLY`, `SETUP_WITH`, 
`BEFORE_ASYNC_WITH`, and `YIELD_FROM`, so we hoisted the check for that condition
out of the `eval_breaker` conditional.

We also patched the `threadstate` struct to include a `pyodide_callback` field, patch
`new_threadstate` to initialize the `pyodide_callback` field to `NULL`, and add a new
API `PyPyodide_SetPyodideCallback` to set `pyodide_callback`.

Lastly, we patched PyErr_CheckSignals to check our custom source of keyboard interrupts
so that C extensions that use PyErr_CheckSignals to allow keyboard interrupts will be 
interruptable in pyodide too.

diff --git a/Include/cpython/ceval.h b/Include/cpython/ceval.h
index e1922a6..a4acd9b 100644
--- a/Include/cpython/ceval.h
+++ b/Include/cpython/ceval.h
@@ -10,6 +10,7 @@ PyAPI_FUNC(void) PyEval_SetProfile(Py_tracefunc, PyObject *);
 PyAPI_DATA(int) _PyEval_SetProfile(PyThreadState *tstate, Py_tracefunc func, PyObject *arg);
 PyAPI_FUNC(void) PyEval_SetTrace(Py_tracefunc, PyObject *);
 PyAPI_FUNC(int) _PyEval_SetTrace(PyThreadState *tstate, Py_tracefunc func, PyObject *arg);
+PyAPI_FUNC(void) PyPyodide_SetPyodideCallback(PyPyodide_callback);
 PyAPI_FUNC(int) _PyEval_GetCoroutineOriginTrackingDepth(void);
 PyAPI_FUNC(int) _PyEval_SetAsyncGenFirstiter(PyObject *);
 PyAPI_FUNC(PyObject *) _PyEval_GetAsyncGenFirstiter(void);
diff --git a/Include/cpython/pystate.h b/Include/cpython/pystate.h
index f292da1..3577ac8 100644
--- a/Include/cpython/pystate.h
+++ b/Include/cpython/pystate.h
@@ -17,6 +17,7 @@ PyAPI_FUNC(PyObject *) _PyInterpreterState_GetMainModule(PyInterpreterState *);
 
 /* Py_tracefunc return -1 when raising an exception, or 0 for success. */
 typedef int (*Py_tracefunc)(PyObject *, PyFrameObject *, int, PyObject *);
+typedef int (*PyPyodide_callback)(void);
 
 /* The following values are used for 'what' for tracefunc functions
  *
@@ -75,6 +76,8 @@ struct _ts {
     PyObject *c_profileobj;
     PyObject *c_traceobj;
 
+    PyPyodide_callback pyodide_callback;
+
     /* The exception currently being raised */
     PyObject *curexc_type;
     PyObject *curexc_value;
diff --git a/Modules/signalmodule.c b/Modules/signalmodule.c
index de564c2..bfe2fd1 100644
--- a/Modules/signalmodule.c
+++ b/Modules/signalmodule.c
@@ -1683,6 +1683,9 @@ PyErr_CheckSignals(void)
 int
 _PyErr_CheckSignalsTstate(PyThreadState *tstate)
 {
+    if (tstate->pyodide_callback != NULL && tstate->pyodide_callback() == -1)
+        return -1;
+
     if (!_Py_atomic_load(&is_tripped)) {
         return 0;
     }
diff --git a/Python/ceval.c b/Python/ceval.c
index 91e879e..f8d424d 100644
--- a/Python/ceval.c
+++ b/Python/ceval.c
@@ -1374,7 +1374,6 @@ main_loop:
            async I/O handler); see Py_AddPendingCall() and
            Py_MakePendingCalls() above. */
 
-        if (_Py_atomic_load_relaxed(eval_breaker)) {
             opcode = _Py_OPCODE(*next_instr);
             if (opcode == SETUP_FINALLY ||
                 opcode == SETUP_WITH ||
@@ -1399,11 +1398,18 @@ main_loop:
                 goto fast_next_opcode;
             }
 
+        if (_Py_atomic_load_relaxed(eval_breaker)) {
             if (eval_frame_handle_pending(tstate) != 0) {
                 goto error;
             }
         }
 
+        if(tstate->pyodide_callback != NULL){
+            if(tstate->pyodide_callback() != 0){
+                goto error;
+            }
+        }
+
     fast_next_opcode:
         f->f_lasti = INSTR_OFFSET();
 
@@ -4816,6 +4822,11 @@ PyEval_SetTrace(Py_tracefunc func, PyObject *arg)
     }
 }
 
+void
+PyPyodide_SetPyodideCallback(PyPyodide_callback pyodide_callback){
+    PyThreadState *tstate = _PyThreadState_GET();
+    tstate->pyodide_callback = pyodide_callback;
+}
 
 void
 _PyEval_SetCoroutineOriginTrackingDepth(PyThreadState *tstate, int new_depth)
diff --git a/Python/pystate.c b/Python/pystate.c
index 9beefa8..408b7ce 100644
--- a/Python/pystate.c
+++ b/Python/pystate.c
@@ -602,6 +602,8 @@ new_threadstate(PyInterpreterState *interp, int init)
     tstate->c_profileobj = NULL;
     tstate->c_traceobj = NULL;
 
+    tstate->pyodide_callback = NULL;
+
     tstate->trash_delete_nesting = 0;
     tstate->trash_delete_later = NULL;
     tstate->on_delete = NULL;
-- 
2.17.1


From 320cc07c0a30436759cde8f4c926e76ed8dc918a Mon Sep 17 00:00:00 2001
From: Hood <hood@mit.edu>
Date: Fri, 3 Sep 2021 18:08:26 -0700
Subject: [PATCH] Patch in keyboard interrupt handling

---
 Include/cpython/ceval.h   |  1 +
 Include/cpython/pystate.h |  3 +++
 Modules/signalmodule.c    |  3 +++
 Python/ceval.c            | 13 ++++++++++++-
 Python/pystate.c          |  2 ++
 5 files changed, 21 insertions(+), 1 deletion(-)

This patch adds a callback called pyodide_callback with signature 
`int callback(void)` to the main loop in `ceval.c`. This function gets called once 
per opcode except after opcodes `SETUP_FINALLY`, `SETUP_WITH`, `BEFORE_ASYNC_WITH`,
and `YIELD_FROM`. The main loop normally prevents signal handling, etc from happening
after these instructions, so I figured this should apply to my callback too.
(There is an extra layer of protection here though because in the callback we use 
`PyErr_SetInterrupt` which simulates a SIGINT signal and triggers the KeyboardInterrupt
via the standard mechanism.)

Note that we call the callback outside of the normal 
`if (_Py_atomic_load_relaxed(eval_breaker))` 
block where most of the "periodic things" happen. This is because normally when a 
"periodic thing" is queued, the `eval_breaker` flag is set signalling the need for
handling. The whole point of this patch though is that we need to be able to set an
interrupt from a remote thread and the `eval_breaker` flag lives on the WASM heap.
Unless the whole WASM heap is made into a `SharedArrayBuffer` and shared between 
webworkers, we have no way to set the `eval_breaker` flag. We still want to skip
the callback after the sensitive opcodes `SETUP_FINALLY`, `SETUP_WITH`, 
`BEFORE_ASYNC_WITH`, and `YIELD_FROM`, so we hoisted the check for that condition
out of the `eval_breaker` conditional.

We also patched the `threadstate` struct to include a `pyodide_callback` field, patch
`new_threadstate` to initialize the `pyodide_callback` field to `NULL`, and add a new
API `PyPyodide_SetPyodideCallback` to set `pyodide_callback`.

Lastly, we patched PyErr_CheckSignals to check our custom source of keyboard interrupts
so that C extensions that use PyErr_CheckSignals to allow keyboard interrupts will be 
interruptable in pyodide too.

diff --git a/Include/cpython/ceval.h b/Include/cpython/ceval.h
index e1922a6..a4acd9b 100644
--- a/Include/cpython/ceval.h
+++ b/Include/cpython/ceval.h
@@ -10,6 +10,7 @@ PyAPI_FUNC(void) PyEval_SetProfile(Py_tracefunc, PyObject *);
 PyAPI_DATA(int) _PyEval_SetProfile(PyThreadState *tstate, Py_tracefunc func, PyObject *arg);
 PyAPI_FUNC(void) PyEval_SetTrace(Py_tracefunc, PyObject *);
 PyAPI_FUNC(int) _PyEval_SetTrace(PyThreadState *tstate, Py_tracefunc func, PyObject *arg);
+PyAPI_FUNC(void) PyPyodide_SetPyodideCallback(PyPyodide_callback);
 PyAPI_FUNC(int) _PyEval_GetCoroutineOriginTrackingDepth(void);
 PyAPI_FUNC(int) _PyEval_SetAsyncGenFirstiter(PyObject *);
 PyAPI_FUNC(PyObject *) _PyEval_GetAsyncGenFirstiter(void);
diff --git a/Include/cpython/pystate.h b/Include/cpython/pystate.h
index f292da1..3577ac8 100644
--- a/Include/cpython/pystate.h
+++ b/Include/cpython/pystate.h
@@ -17,6 +17,7 @@ PyAPI_FUNC(PyObject *) _PyInterpreterState_GetMainModule(PyInterpreterState *);
 
 /* Py_tracefunc return -1 when raising an exception, or 0 for success. */
 typedef int (*Py_tracefunc)(PyObject *, PyFrameObject *, int, PyObject *);
+typedef int (*PyPyodide_callback)(void);
 
 /* The following values are used for 'what' for tracefunc functions
  *
@@ -75,6 +76,8 @@ struct _ts {
     PyObject *c_profileobj;
     PyObject *c_traceobj;
 
+    PyPyodide_callback pyodide_callback;
+
     /* The exception currently being raised */
     PyObject *curexc_type;
     PyObject *curexc_value;
diff --git a/Modules/signalmodule.c b/Modules/signalmodule.c
index de564c2..bfe2fd1 100644
--- a/Modules/signalmodule.c
+++ b/Modules/signalmodule.c
@@ -1683,6 +1683,9 @@ PyErr_CheckSignals(void)
 int
 _PyErr_CheckSignalsTstate(PyThreadState *tstate)
 {
+    if (tstate->pyodide_callback != NULL && tstate->pyodide_callback() == -1)
+        return -1;
+
     if (!_Py_atomic_load(&is_tripped)) {
         return 0;
     }
diff --git a/Python/ceval.c b/Python/ceval.c
index 91e879e..f8d424d 100644
--- a/Python/ceval.c
+++ b/Python/ceval.c
@@ -1374,7 +1374,6 @@ main_loop:
            async I/O handler); see Py_AddPendingCall() and
            Py_MakePendingCalls() above. */
 
-        if (_Py_atomic_load_relaxed(eval_breaker)) {
             opcode = _Py_OPCODE(*next_instr);
             if (opcode == SETUP_FINALLY ||
                 opcode == SETUP_WITH ||
@@ -1399,11 +1398,18 @@ main_loop:
                 goto fast_next_opcode;
             }
 
+        if (_Py_atomic_load_relaxed(eval_breaker)) {
             if (eval_frame_handle_pending(tstate) != 0) {
                 goto error;
             }
         }
 
+        if(tstate->pyodide_callback != NULL){
+            if(tstate->pyodide_callback() != 0){
+                goto error;
+            }
+        }
+
     fast_next_opcode:
         f->f_lasti = INSTR_OFFSET();
 
@@ -4816,6 +4822,11 @@ PyEval_SetTrace(Py_tracefunc func, PyObject *arg)
     }
 }
 
+void
+PyPyodide_SetPyodideCallback(PyPyodide_callback pyodide_callback){
+    PyThreadState *tstate = _PyThreadState_GET();
+    tstate->pyodide_callback = pyodide_callback;
+}
 
 void
 _PyEval_SetCoroutineOriginTrackingDepth(PyThreadState *tstate, int new_depth)
diff --git a/Python/pystate.c b/Python/pystate.c
index 9beefa8..408b7ce 100644
--- a/Python/pystate.c
+++ b/Python/pystate.c
@@ -602,6 +602,8 @@ new_threadstate(PyInterpreterState *interp, int init)
     tstate->c_profileobj = NULL;
     tstate->c_traceobj = NULL;
 
+    tstate->pyodide_callback = NULL;
+
     tstate->trash_delete_nesting = 0;
     tstate->trash_delete_later = NULL;
     tstate->on_delete = NULL;
-- 
2.17.1

