## Elevator Pitch

Enable file system persistence with IndexedDB (`IDBFS`), Node.js (`NODEFS`), blobs (`WORKERFS`), proxy (`PROXYFS`).

![Screenshot from 2021-05-19 23-34-01](https://user-images.githubusercontent.com/45380/118983166-238b9200-b94a-11eb-8bb7-11ea96bc5674.png)

## Motivation

Continuing the discussion from #328, this would allow a pyodide instance to work more naturally with persistent files. 

Some external use cases:
- persisting files between user sessions
- sharing files between concurrent browser tabs

Some internal use cases:
- caching downloaded wheels and API responses?

## Changes

- [x] add ` -l*fs.js` to `Makefile` to restore all backends
  - ~~considered [other backends](https://emscripten.org/docs/api_reference/Filesystem-API.html#persistent-data), but as this has some impact on build time and size without a clear use case, I have not included them for now, but an eventual API should probably plan for supporting them explicitly~~
    - [x] #1689 enabled node, and i've already though about uses cases for WORKERFS
- [x] #1692 ~~hoist `FS.mount`, `FS.mkdir` and `FS.sync` and `IDBFS` up to the `pyodide` API?~~
  - rather than doing something fancy, this has the highest likelihood of working with other language runtimes
  - as discussed below, the preferred approach is to hoist `FS` to the `pyodide` namespace, pending a better name...
- [x] ~~add a convenience API in e.g. `loadPyodide({mount: "/home/user1"})`?~~
  - a lot of use cases would be covered by "simple" persistence
  - could also add it to `sys.prefix`
  - moved to https://github.com/pyodide/pyodide/issues/1715
- [x] tests?
  - haven't looked into what this will take...
- [x] docs?
  - depends on what features we end up landing...
  - focus  on documenting notional API https://github.com/pyodide/pyodide/issues/1715

## Findings

- you can make as many mount points as you want
- they all get synced into the `MEMFS` (which must exist) with `FS.sync(true, (err)=>{...})` and likewise back out with `FS.sync(false, (err) => {...})`
  - the example guidance suggests sync-in at the start of the process and sync-out before exiting, but this is pretty limiting!

## Dead-ends

The image above was gotten by hacking `main.c`, emulating a naive/broken approach for polling. A better path is being investigated!
```c

  // Create and mount userfs immediately.
	EM_ASM({
		FS.mkdir('/home/jo');
		FS.mount(IDBFS, {}, '/home/jo');
    FS.syncfs(true, function (err) {
      err && console.error('error syncing FROM IndexedDB', err);
    });
    setInterval(function() {
        FS.syncfs(false, function (err) {
          err && console.error('error syncing TO IndexedDB', err);
      });
    }, 5000);
	});

```
Enable all emscripten file systems: IDBFS, NODEFS, PROXYFS and WORKERFS. ## Elevator Pitch

Enable file system persistence with IndexedDB (`IDBFS`), Node.js (`NODEFS`), blobs (`WORKERFS`), proxy (`PROXYFS`).

![Screenshot from 2021-05-19 23-34-01](https://user-images.githubusercontent.com/45380/118983166-238b9200-b94a-11eb-8bb7-11ea96bc5674.png)

## Motivation

Continuing the discussion from #328, this would allow a pyodide instance to work more naturally with persistent files. 

Some external use cases:
- persisting files between user sessions
- sharing files between concurrent browser tabs

Some internal use cases:
- caching downloaded wheels and API responses?

## Changes

- [x] add ` -l*fs.js` to `Makefile` to restore all backends
  - ~~considered [other backends](https://emscripten.org/docs/api_reference/Filesystem-API.html#persistent-data), but as this has some impact on build time and size without a clear use case, I have not included them for now, but an eventual API should probably plan for supporting them explicitly~~
    - [x] #1689 enabled node, and i've already though about uses cases for WORKERFS
- [x] #1692 ~~hoist `FS.mount`, `FS.mkdir` and `FS.sync` and `IDBFS` up to the `pyodide` API?~~
  - rather than doing something fancy, this has the highest likelihood of working with other language runtimes
  - as discussed below, the preferred approach is to hoist `FS` to the `pyodide` namespace, pending a better name...
- [x] ~~add a convenience API in e.g. `loadPyodide({mount: "/home/user1"})`?~~
  - a lot of use cases would be covered by "simple" persistence
  - could also add it to `sys.prefix`
  - moved to https://github.com/pyodide/pyodide/issues/1715
- [x] tests?
  - haven't looked into what this will take...
- [x] docs?
  - depends on what features we end up landing...
  - focus  on documenting notional API https://github.com/pyodide/pyodide/issues/1715

## Findings

- you can make as many mount points as you want
- they all get synced into the `MEMFS` (which must exist) with `FS.sync(true, (err)=>{...})` and likewise back out with `FS.sync(false, (err) => {...})`
  - the example guidance suggests sync-in at the start of the process and sync-out before exiting, but this is pretty limiting!

## Dead-ends

The image above was gotten by hacking `main.c`, emulating a naive/broken approach for polling. A better path is being investigated!
```c

  // Create and mount userfs immediately.
	EM_ASM({
		FS.mkdir('/home/jo');
		FS.mount(IDBFS, {}, '/home/jo');
    FS.syncfs(true, function (err) {
      err && console.error('error syncing FROM IndexedDB', err);
    });
    setInterval(function() {
        FS.syncfs(false, function (err) {
          err && console.error('error syncing TO IndexedDB', err);
      });
    }, 5000);
	});

```
(debugging)=
# Interactive Debugging
See [Emscripten's page about
debugging](https://emscripten.org/docs/porting/Debugging.html) which has
extensive info about the various debugging options available.

[Chromium has support for DWARF
info](https://developer.chrome.com/blog/webassembly/) which can be very helpful
for debugging in certain circumstances. To build Pyodide with DWARF, you should
set `EXTRA_CFLAGS="-g4"` and `EXTRA_LD_FLAGS="-g4 --source-map-base
"http://localhost:<port>/"` (substitute in your favorite port). Make sure to
rebuild CPython with these flags set (it isn't necessary to rebuild emsdk).

Once you have done this, when you load Pyodide, you will see 404 errors for
requests for the various source maps. In order to load the source maps
correctly, you will need to run a custom server that "fixes" the source map
urls. The following debugging server seems to work for both CPython and numpy.
Run it in the Pyodide root directory. If you need to debug other C extensions,
you will need to update the server. It should be clear what to do based by
looking at the 404s generated by the server and locating those files in the file
tree, perhaps by using `find`.
```py
import socket
import socketserver
from http.server import SimpleHTTPRequestHandler
import pathlib

alternative_bases=["cpython/build/Python-3.9.5/","src/", "build/"]
def fixup_url(path):
    if pathlib.Path("." + path).exists():
        return path
    for base in alternative_bases:
        q = pathlib.Path(base + path)
        if q.exists():
            return str(q)
    # Numpy source maps can be in a bunch of different places inside of the
    # directory tree, so we need to glob for them.
    dir = list(
        pathlib.Path("packages/numpy/build/numpy-1.17.5/").glob("**" + path)
    )
    if dir:
        return str(dir[0])
    return path


class MyTCPServer(socketserver.TCPServer):
    def server_bind(self):
        """Use socket.SO_REUSEADDR to allow faster restart"""
        self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        self.socket.bind(self.server_address)

class Handler(SimpleHTTPRequestHandler):
    def end_headers(self):
        # Enable Cross-Origin Resource Sharing (CORS)
        self.send_header('Access-Control-Allow-Origin', '*')
        super().end_headers()

    def do_GET(self):
        self.path = fixup_url(self.path)
        super().do_GET()


if __name__ == '__main__':
    port = 8000
    with MyTCPServer(("", port), Handler) as httpd:
        print("Serving at: http://127.0.0.1:{}".format(port))
        httpd.serve_forever()
```

(debugging)=
# Interactive Debugging
See [Emscripten's page about
debugging](https://emscripten.org/docs/porting/Debugging.html) which has
extensive info about the various debugging options available.

[Chromium has support for DWARF
info](https://developer.chrome.com/blog/webassembly/) which can be very helpful
for debugging in certain circumstances. To build Pyodide with DWARF, you should
set `EXTRA_CFLAGS="-g4"` and `EXTRA_LD_FLAGS="-g4 --source-map-base
"http://localhost:<port>/"` (substitute in your favorite port). Make sure to
rebuild CPython with these flags set (it isn't necessary to rebuild emsdk).

Once you have done this, when you load Pyodide, you will see 404 errors for
requests for the various source maps. In order to load the source maps
correctly, you will need to run a custom server that "fixes" the source map
urls. The following debugging server seems to work for both CPython and numpy.
Run it in the Pyodide root directory. If you need to debug other C extensions,
you will need to update the server. It should be clear what to do based by
looking at the 404s generated by the server and locating those files in the file
tree, perhaps by using `find`.
```py
import socket
import socketserver
from http.server import SimpleHTTPRequestHandler
import pathlib

alternative_bases=["cpython/build/Python-3.9.5/","src/", "build/"]
def fixup_url(path):
    if pathlib.Path("." + path).exists():
        return path
    for base in alternative_bases:
        q = pathlib.Path(base + path)
        if q.exists():
            return str(q)
    # Numpy source maps can be in a bunch of different places inside of the
    # directory tree, so we need to glob for them.
    dir = list(
        pathlib.Path("packages/numpy/build/numpy-1.17.5/").glob("**" + path)
    )
    if dir:
        return str(dir[0])
    return path


class MyTCPServer(socketserver.TCPServer):
    def server_bind(self):
        """Use socket.SO_REUSEADDR to allow faster restart"""
        self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        self.socket.bind(self.server_address)

class Handler(SimpleHTTPRequestHandler):
    def end_headers(self):
        # Enable Cross-Origin Resource Sharing (CORS)
        self.send_header('Access-Control-Allow-Origin', '*')
        super().end_headers()

    def do_GET(self):
        self.path = fixup_url(self.path)
        super().do_GET()


if __name__ == '__main__':
    port = 8000
    with MyTCPServer(("", port), Handler) as httpd:
        print("Serving at: http://127.0.0.1:{}".format(port))
        httpd.serve_forever()
```
