## Elevator Pitch

Enable file system persistence with IndexedDB (`IDBFS`), Node.js (`NODEFS`), blobs (`WORKERFS`), proxy (`PROXYFS`).

![Screenshot from 2021-05-19 23-34-01](https://user-images.githubusercontent.com/45380/118983166-238b9200-b94a-11eb-8bb7-11ea96bc5674.png)

## Motivation

Continuing the discussion from #328, this would allow a pyodide instance to work more naturally with persistent files. 

Some external use cases:
- persisting files between user sessions
- sharing files between concurrent browser tabs

Some internal use cases:
- caching downloaded wheels and API responses?

## Changes

- [x] add ` -l*fs.js` to `Makefile` to restore all backends
  - ~~considered [other backends](https://emscripten.org/docs/api_reference/Filesystem-API.html#persistent-data), but as this has some impact on build time and size without a clear use case, I have not included them for now, but an eventual API should probably plan for supporting them explicitly~~
    - [x] #1689 enabled node, and i've already though about uses cases for WORKERFS
- [x] #1692 ~~hoist `FS.mount`, `FS.mkdir` and `FS.sync` and `IDBFS` up to the `pyodide` API?~~
  - rather than doing something fancy, this has the highest likelihood of working with other language runtimes
  - as discussed below, the preferred approach is to hoist `FS` to the `pyodide` namespace, pending a better name...
- [x] ~~add a convenience API in e.g. `loadPyodide({mount: "/home/user1"})`?~~
  - a lot of use cases would be covered by "simple" persistence
  - could also add it to `sys.prefix`
  - moved to https://github.com/pyodide/pyodide/issues/1715
- [x] tests?
  - haven't looked into what this will take...
- [x] docs?
  - depends on what features we end up landing...
  - focus  on documenting notional API https://github.com/pyodide/pyodide/issues/1715

## Findings

- you can make as many mount points as you want
- they all get synced into the `MEMFS` (which must exist) with `FS.sync(true, (err)=>{...})` and likewise back out with `FS.sync(false, (err) => {...})`
  - the example guidance suggests sync-in at the start of the process and sync-out before exiting, but this is pretty limiting!

## Dead-ends

The image above was gotten by hacking `main.c`, emulating a naive/broken approach for polling. A better path is being investigated!
```c

  // Create and mount userfs immediately.
	EM_ASM({
		FS.mkdir('/home/jo');
		FS.mount(IDBFS, {}, '/home/jo');
    FS.syncfs(true, function (err) {
      err && console.error('error syncing FROM IndexedDB', err);
    });
    setInterval(function() {
        FS.syncfs(false, function (err) {
          err && console.error('error syncing TO IndexedDB', err);
      });
    }, 5000);
	});

```
Enable all emscripten file systems: IDBFS, NODEFS, PROXYFS and WORKERFS. ## Elevator Pitch

Enable file system persistence with IndexedDB (`IDBFS`), Node.js (`NODEFS`), blobs (`WORKERFS`), proxy (`PROXYFS`).

![Screenshot from 2021-05-19 23-34-01](https://user-images.githubusercontent.com/45380/118983166-238b9200-b94a-11eb-8bb7-11ea96bc5674.png)

## Motivation

Continuing the discussion from #328, this would allow a pyodide instance to work more naturally with persistent files. 

Some external use cases:
- persisting files between user sessions
- sharing files between concurrent browser tabs

Some internal use cases:
- caching downloaded wheels and API responses?

## Changes

- [x] add ` -l*fs.js` to `Makefile` to restore all backends
  - ~~considered [other backends](https://emscripten.org/docs/api_reference/Filesystem-API.html#persistent-data), but as this has some impact on build time and size without a clear use case, I have not included them for now, but an eventual API should probably plan for supporting them explicitly~~
    - [x] #1689 enabled node, and i've already though about uses cases for WORKERFS
- [x] #1692 ~~hoist `FS.mount`, `FS.mkdir` and `FS.sync` and `IDBFS` up to the `pyodide` API?~~
  - rather than doing something fancy, this has the highest likelihood of working with other language runtimes
  - as discussed below, the preferred approach is to hoist `FS` to the `pyodide` namespace, pending a better name...
- [x] ~~add a convenience API in e.g. `loadPyodide({mount: "/home/user1"})`?~~
  - a lot of use cases would be covered by "simple" persistence
  - could also add it to `sys.prefix`
  - moved to https://github.com/pyodide/pyodide/issues/1715
- [x] tests?
  - haven't looked into what this will take...
- [x] docs?
  - depends on what features we end up landing...
  - focus  on documenting notional API https://github.com/pyodide/pyodide/issues/1715

## Findings

- you can make as many mount points as you want
- they all get synced into the `MEMFS` (which must exist) with `FS.sync(true, (err)=>{...})` and likewise back out with `FS.sync(false, (err) => {...})`
  - the example guidance suggests sync-in at the start of the process and sync-out before exiting, but this is pretty limiting!

## Dead-ends

The image above was gotten by hacking `main.c`, emulating a naive/broken approach for polling. A better path is being investigated!
```c

  // Create and mount userfs immediately.
	EM_ASM({
		FS.mkdir('/home/jo');
		FS.mount(IDBFS, {}, '/home/jo');
    FS.syncfs(true, function (err) {
      err && console.error('error syncing FROM IndexedDB', err);
    });
    setInterval(function() {
        FS.syncfs(false, function (err) {
          err && console.error('error syncing TO IndexedDB', err);
      });
    }, 5000);
	});

```
# core/pyodide

The C and Javascript code in this package is responsible for embedding the
Python interpreter in our emscripten js/wasm application. The primary purpose of
this code is to define the foreign function interface between Python and
Javascript. Once this foreign function interface is defined, more complex
behaviors are better defined in Python when possible for easier development and
debugging.

In particular, when possible logic should be moved from core/pyodide to
py/\_pyodide or to py/pyodide.

The core/pyodide code is responsible for the following main steps:

1. Initialize the CPython interpreter
2. Import py/\_pyodide
3. Initialize `_pyodide_core` which is a Python C extension that we use to make
   functionality available to py/pyodide.
4. Set up functionality to automatically convert functions from Javascript to
   CPython calling conventions (`error_handling.h`).
5. Set up the "hiwire" side table to hold references to Javascript objects --
   necessary because wasm variables can only hold numbers (`hiwire.c`).
6. Set up type conversions of basic types between Python and Javascript
   (`js2python.c` and `python2js.c`).
7. Set up Proxying of remaining types between Python and Javascript (`jsproxy.c`
   and `pyproxy.c`). This is the most complicated part of the Pyodide runtime
   and involves careful conversion between [abstract Javascript object
   protocols](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Proxy)
   (see also [Javascript Iteration
   Protocols](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Iteration_protocols)
   )
   and [Python object protocols](https://docs.python.org/3/c-api/abstract.html).
8. Add `_pyodide_core` to `sys.modules` and import py/pyodide.

# core/pyodide

The C and Javascript code in this package is responsible for embedding the
Python interpreter in our emscripten js/wasm application. The primary purpose of
this code is to define the foreign function interface between Python and
Javascript. Once this foreign function interface is defined, more complex
behaviors are better defined in Python when possible for easier development and
debugging.

In particular, when possible logic should be moved from core/pyodide to
py/\_pyodide or to py/pyodide.

The core/pyodide code is responsible for the following main steps:

1. Initialize the CPython interpreter
2. Import py/\_pyodide
3. Initialize `_pyodide_core` which is a Python C extension that we use to make
   functionality available to py/pyodide.
4. Set up functionality to automatically convert functions from Javascript to
   CPython calling conventions (`error_handling.h`).
5. Set up the "hiwire" side table to hold references to Javascript objects --
   necessary because wasm variables can only hold numbers (`hiwire.c`).
6. Set up type conversions of basic types between Python and Javascript
   (`js2python.c` and `python2js.c`).
7. Set up Proxying of remaining types between Python and Javascript (`jsproxy.c`
   and `pyproxy.c`). This is the most complicated part of the Pyodide runtime
   and involves careful conversion between [abstract Javascript object
   protocols](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Proxy)
   (see also [Javascript Iteration
   Protocols](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Iteration_protocols)
   )
   and [Python object protocols](https://docs.python.org/3/c-api/abstract.html).
8. Add `_pyodide_core` to `sys.modules` and import py/pyodide.
