## Elevator Pitch

Enable file system persistence with IndexedDB (`IDBFS`), Node.js (`NODEFS`), blobs (`WORKERFS`), proxy (`PROXYFS`).

![Screenshot from 2021-05-19 23-34-01](https://user-images.githubusercontent.com/45380/118983166-238b9200-b94a-11eb-8bb7-11ea96bc5674.png)

## Motivation

Continuing the discussion from #328, this would allow a pyodide instance to work more naturally with persistent files. 

Some external use cases:
- persisting files between user sessions
- sharing files between concurrent browser tabs

Some internal use cases:
- caching downloaded wheels and API responses?

## Changes

- [x] add ` -l*fs.js` to `Makefile` to restore all backends
  - ~~considered [other backends](https://emscripten.org/docs/api_reference/Filesystem-API.html#persistent-data), but as this has some impact on build time and size without a clear use case, I have not included them for now, but an eventual API should probably plan for supporting them explicitly~~
    - [x] #1689 enabled node, and i've already though about uses cases for WORKERFS
- [x] #1692 ~~hoist `FS.mount`, `FS.mkdir` and `FS.sync` and `IDBFS` up to the `pyodide` API?~~
  - rather than doing something fancy, this has the highest likelihood of working with other language runtimes
  - as discussed below, the preferred approach is to hoist `FS` to the `pyodide` namespace, pending a better name...
- [x] ~~add a convenience API in e.g. `loadPyodide({mount: "/home/user1"})`?~~
  - a lot of use cases would be covered by "simple" persistence
  - could also add it to `sys.prefix`
  - moved to https://github.com/pyodide/pyodide/issues/1715
- [x] tests?
  - haven't looked into what this will take...
- [x] docs?
  - depends on what features we end up landing...
  - focus  on documenting notional API https://github.com/pyodide/pyodide/issues/1715

## Findings

- you can make as many mount points as you want
- they all get synced into the `MEMFS` (which must exist) with `FS.sync(true, (err)=>{...})` and likewise back out with `FS.sync(false, (err) => {...})`
  - the example guidance suggests sync-in at the start of the process and sync-out before exiting, but this is pretty limiting!

## Dead-ends

The image above was gotten by hacking `main.c`, emulating a naive/broken approach for polling. A better path is being investigated!
```c

  // Create and mount userfs immediately.
	EM_ASM({
		FS.mkdir('/home/jo');
		FS.mount(IDBFS, {}, '/home/jo');
    FS.syncfs(true, function (err) {
      err && console.error('error syncing FROM IndexedDB', err);
    });
    setInterval(function() {
        FS.syncfs(false, function (err) {
          err && console.error('error syncing TO IndexedDB', err);
      });
    }, 5000);
	});

```
Enable all emscripten file systems: IDBFS, NODEFS, PROXYFS and WORKERFS. ## Elevator Pitch

Enable file system persistence with IndexedDB (`IDBFS`), Node.js (`NODEFS`), blobs (`WORKERFS`), proxy (`PROXYFS`).

![Screenshot from 2021-05-19 23-34-01](https://user-images.githubusercontent.com/45380/118983166-238b9200-b94a-11eb-8bb7-11ea96bc5674.png)

## Motivation

Continuing the discussion from #328, this would allow a pyodide instance to work more naturally with persistent files. 

Some external use cases:
- persisting files between user sessions
- sharing files between concurrent browser tabs

Some internal use cases:
- caching downloaded wheels and API responses?

## Changes

- [x] add ` -l*fs.js` to `Makefile` to restore all backends
  - ~~considered [other backends](https://emscripten.org/docs/api_reference/Filesystem-API.html#persistent-data), but as this has some impact on build time and size without a clear use case, I have not included them for now, but an eventual API should probably plan for supporting them explicitly~~
    - [x] #1689 enabled node, and i've already though about uses cases for WORKERFS
- [x] #1692 ~~hoist `FS.mount`, `FS.mkdir` and `FS.sync` and `IDBFS` up to the `pyodide` API?~~
  - rather than doing something fancy, this has the highest likelihood of working with other language runtimes
  - as discussed below, the preferred approach is to hoist `FS` to the `pyodide` namespace, pending a better name...
- [x] ~~add a convenience API in e.g. `loadPyodide({mount: "/home/user1"})`?~~
  - a lot of use cases would be covered by "simple" persistence
  - could also add it to `sys.prefix`
  - moved to https://github.com/pyodide/pyodide/issues/1715
- [x] tests?
  - haven't looked into what this will take...
- [x] docs?
  - depends on what features we end up landing...
  - focus  on documenting notional API https://github.com/pyodide/pyodide/issues/1715

## Findings

- you can make as many mount points as you want
- they all get synced into the `MEMFS` (which must exist) with `FS.sync(true, (err)=>{...})` and likewise back out with `FS.sync(false, (err) => {...})`
  - the example guidance suggests sync-in at the start of the process and sync-out before exiting, but this is pretty limiting!

## Dead-ends

The image above was gotten by hacking `main.c`, emulating a naive/broken approach for polling. A better path is being investigated!
```c

  // Create and mount userfs immediately.
	EM_ASM({
		FS.mkdir('/home/jo');
		FS.mount(IDBFS, {}, '/home/jo');
    FS.syncfs(true, function (err) {
      err && console.error('error syncing FROM IndexedDB', err);
    });
    setInterval(function() {
        FS.syncfs(false, function (err) {
          err && console.error('error syncing TO IndexedDB', err);
      });
    }, 5000);
	});

```
PYODIDE_ROOT=$(abspath .)

include Makefile.envs

.PHONY=check


CPYTHONROOT=cpython
CPYTHONLIB=$(CPYTHONROOT)/installs/python-$(PYVERSION)/lib/python$(PYMAJOR).$(PYMINOR)

CC=emcc
CXX=em++


all: check \
	build/pyodide.asm.js \
	build/pyodide.js \
	build/console.html \
	build/test.data \
	build/distutils.data \
	build/packages.json \
	build/test.html \
	build/webworker.js \
	build/webworker_dev.js
	echo -e "\nSUCCESS!"

$(CPYTHONLIB)/tzdata :
	pip install tzdata --target=$(CPYTHONLIB)

build/pyodide.asm.js: \
	src/core/docstring.o \
	src/core/error_handling.o \
	src/core/numpy_patch.o \
	src/core/hiwire.o \
	src/core/js2python.o \
	src/core/jsproxy.o \
	src/core/keyboard_interrupt.o \
	src/core/main.o  \
	src/core/pyproxy.o \
	src/core/python2js_buffer.o \
	src/core/python2js.o \
	src/pystone.py \
	src/_testcapi.py \
	src/_testinternalcapi.py \
	src/webbrowser.py \
	$(CPYTHONLIB)/tzdata \
	$(wildcard src/py/pyodide/*.py) \
	$(wildcard src/py/_pyodide/*.py) \
	$(CPYTHONLIB)
	date +"[%F %T] Building pyodide.asm.js..."
	[ -d build ] || mkdir build
	$(CXX) -s EXPORT_NAME="'_createPyodideModule'" -o build/pyodide.asm.js $(filter %.o,$^) \
		$(MAIN_MODULE_LDFLAGS) -s FORCE_FILESYSTEM=1 \
		--preload-file $(CPYTHONLIB)@/lib/python$(PYMAJOR).$(PYMINOR) \
		--preload-file src/webbrowser.py@/lib/python$(PYMAJOR).$(PYMINOR)/webbrowser.py \
		--preload-file src/_testcapi.py@/lib/python$(PYMAJOR).$(PYMINOR)/_testcapi.py \
		--preload-file src/_testinternalcapi.py@/lib/python$(PYMAJOR).$(PYMINOR)/_testinternalcapi.py \
		--preload-file src/pystone.py@/lib/python$(PYMAJOR).$(PYMINOR)/pystone.py \
		--preload-file src/py/pyodide@/lib/python$(PYMAJOR).$(PYMINOR)/site-packages/pyodide \
		--preload-file src/py/_pyodide@/lib/python$(PYMAJOR).$(PYMINOR)/site-packages/_pyodide \
		--exclude-file "*__pycache__*" \
		--exclude-file "*/test/*" \
		--exclude-file "*/tests/*" \
		--exclude-file "*/distutils/*"
   # Strip out C++ symbols which all start __Z.
   # There are 4821 of these and they have VERY VERY long names.
   # To show some stats on the symbols you can use the following:
   # cat build/pyodide.asm.js | grep -ohE 'var _{0,5}.' | sort | uniq -c | sort -nr | head -n 20
	sed -i -E 's/var __Z[^;]*;//g' build/pyodide.asm.js
	sed -i '1i\
		"use strict";\
		let setImmediate = globalThis.setImmediate;\
		let clearImmediate = globalThis.clearImmediate;\
		let baseName, fpcGOT, dyncallGOT, fpVal, dcVal;\
	' build/pyodide.asm.js
	echo "globalThis._createPyodideModule = _createPyodideModule;" >> build/pyodide.asm.js
	date +"[%F %T] done building pyodide.asm.js."


env:
	env


node_modules/.installed : src/js/package.json
	cd src/js && npm install --save-dev
	ln -sfn src/js/node_modules/ node_modules
	touch node_modules/.installed

build/pyodide.js: src/js/*.js src/js/pyproxy.gen.js node_modules/.installed
	npx typescript --project src/js
	npx rollup -c src/js/rollup.config.js

src/js/pyproxy.gen.js : src/core/pyproxy.* src/core/*.h
	# We can't input pyproxy.js directly because CC will be unhappy about the file
	# extension. Instead cat it and have CC read from stdin.
	# -E : Only apply prepreocessor
	# -C : Leave comments alone (this allows them to be preserved in typescript
	#      definition files, rollup will strip them out)
	# -P : Don't put in macro debug info
	# -imacros pyproxy.c : include all of the macros definitions from pyproxy.c
	rm -f $@
	echo "// This file is generated by applying the C preprocessor to core/pyproxy.js" >> $@
	echo "// It uses the macros defined in core/pyproxy.c" >> $@
	echo "// Do not edit it directly!" >> $@
	cat $< | $(CC) -E -C -P -imacros src/core/pyproxy.c $(MAIN_MODULE_CFLAGS) - >> $@

build/test.html: src/templates/test.html
	cp $< $@


.PHONY: build/console.html
build/console.html: src/templates/console.html
	cp $< $@
	sed -i -e 's#{{ PYODIDE_BASE_URL }}#$(PYODIDE_BASE_URL)#g' $@


.PHONY: docs/_build/html/console.html
docs/_build/html/console.html: src/templates/console.html
	mkdir -p docs/_build/html
	cp $< $@
	sed -i -e 's#{{ PYODIDE_BASE_URL }}#$(PYODIDE_BASE_URL)#g' $@


.PHONY: build/webworker.js
build/webworker.js: src/webworker.js
	cp $< $@
	sed -i -e 's#{{ PYODIDE_BASE_URL }}#$(PYODIDE_BASE_URL)#g' $@


.PHONY: build/webworker_dev.js
build/webworker_dev.js: src/webworker.js
	cp $< $@
	sed -i -e 's#{{ PYODIDE_BASE_URL }}#./#g' $@


update_base_url: \
	build/console.html \
	build/webworker.js


test: all
	pytest src emsdk/tests packages/*/test* pyodide-build -v

lint: node_modules/.installed
	# check for unused imports, the rest is done by black
	flake8 --select=F401 src tools pyodide-build benchmark conftest.py docs packages/matplotlib/src/
	find src -type f -regex '.*\.\(c\|h\)' \
		| xargs clang-format-6.0 -output-replacements-xml \
		| (! grep '<replacement ')
	npx prettier --check `find src -type f -name '*.js' -not -name '*.gen.js'`
	npx prettier --check `find src -type f -name '*.html'`
	black --check .
	mypy --ignore-missing-imports    \
		pyodide-build/pyodide_build/ \
		src/ 					     \
		packages/micropip/micropip/  \
		packages/*/test* 			 \
		conftest.py 				 \
		docs



benchmark: all
	$(HOSTPYTHON) benchmark/benchmark.py $(HOSTPYTHON) build/benchmarks.json
	$(HOSTPYTHON) benchmark/plot_benchmark.py build/benchmarks.json build/benchmarks.png


clean:
	rm -fr build/*
	rm -fr src/*/*.o
	rm -fr node_modules
	make -C packages clean
	echo "The Emsdk, CPython are not cleaned. cd into those directories to do so."


clean-all: clean
	make -C emsdk clean
	make -C cpython clean
	rm -fr cpython/build

%.o: %.c $(CPYTHONLIB) $(wildcard src/core/*.h src/core/python2js_buffer.js)
	$(CC) -o $@ -c $< $(MAIN_MODULE_CFLAGS) -Isrc/core/


# Stdlib modules that we repackage as standalone packages

# TODO: also include test directories included in other stdlib modules
build/test.data: $(CPYTHONLIB) node_modules/.installed
	./tools/file_packager.sh build/test.data --js-output=build/test.js \
		--preload $(CPYTHONLIB)/test@/lib/python$(PYMAJOR).$(PYMINOR)/test
	npx terser build/test.js -o build/test.js


build/distutils.data: $(CPYTHONLIB) node_modules/.installed
	./tools/file_packager.sh build/distutils.data --js-output=build/distutils.js \
		--preload $(CPYTHONLIB)/distutils@/lib/python$(PYMAJOR).$(PYMINOR)/distutils \
		--exclude tests
	npx terser build/distutils.js -o build/distutils.js


$(CPYTHONLIB): emsdk/emsdk/.complete $(PYODIDE_EMCC) $(PYODIDE_CXX)
	date +"[%F %T] Building cpython..."
	make -C $(CPYTHONROOT)
	date +"[%F %T] done building cpython..."


build/packages.json: FORCE
	date +"[%F %T] Building packages..."
	make -C packages
	date +"[%F %T] done building packages..."


emsdk/emsdk/.complete:
	date +"[%F %T] Building emsdk..."
	make -C emsdk
	date +"[%F %T] done building emsdk."


FORCE:


check:
	./tools/dependency-check.sh


minimal :
	PYODIDE_PACKAGES+=",micropip" make


debug :
	EXTRA_CFLAGS+=" -D DEBUG_F" \
	PYODIDE_PACKAGES+=", micropip, pyparsing, pytz, packaging, " \
	make

PYODIDE_ROOT=$(abspath .)

include Makefile.envs

.PHONY=check


CPYTHONROOT=cpython
CPYTHONLIB=$(CPYTHONROOT)/installs/python-$(PYVERSION)/lib/python$(PYMAJOR).$(PYMINOR)

CC=emcc
CXX=em++


all: check \
	build/pyodide.asm.js \
	build/pyodide.js \
	build/console.html \
	build/test.data \
	build/distutils.data \
	build/packages.json \
	build/test.html \
	build/webworker.js \
	build/webworker_dev.js
	echo -e "\nSUCCESS!"

$(CPYTHONLIB)/tzdata :
	pip install tzdata --target=$(CPYTHONLIB)

build/pyodide.asm.js: \
	src/core/docstring.o \
	src/core/error_handling.o \
	src/core/numpy_patch.o \
	src/core/hiwire.o \
	src/core/js2python.o \
	src/core/jsproxy.o \
	src/core/keyboard_interrupt.o \
	src/core/main.o  \
	src/core/pyproxy.o \
	src/core/python2js_buffer.o \
	src/core/python2js.o \
	src/pystone.py \
	src/_testcapi.py \
	src/_testinternalcapi.py \
	src/webbrowser.py \
	$(CPYTHONLIB)/tzdata \
	$(wildcard src/py/pyodide/*.py) \
	$(wildcard src/py/_pyodide/*.py) \
	$(CPYTHONLIB)
	date +"[%F %T] Building pyodide.asm.js..."
	[ -d build ] || mkdir build
	$(CXX) -s EXPORT_NAME="'_createPyodideModule'" -o build/pyodide.asm.js $(filter %.o,$^) \
		$(MAIN_MODULE_LDFLAGS) -s FORCE_FILESYSTEM=1 \
		-lidbfs.js \
		-lnodefs.js \
		-lproxyfs.js \
		-lworkerfs.js \
		--preload-file $(CPYTHONLIB)@/lib/python$(PYMAJOR).$(PYMINOR) \
		--preload-file src/webbrowser.py@/lib/python$(PYMAJOR).$(PYMINOR)/webbrowser.py \
		--preload-file src/_testcapi.py@/lib/python$(PYMAJOR).$(PYMINOR)/_testcapi.py \
		--preload-file src/_testinternalcapi.py@/lib/python$(PYMAJOR).$(PYMINOR)/_testinternalcapi.py \
		--preload-file src/pystone.py@/lib/python$(PYMAJOR).$(PYMINOR)/pystone.py \
		--preload-file src/py/pyodide@/lib/python$(PYMAJOR).$(PYMINOR)/site-packages/pyodide \
		--preload-file src/py/_pyodide@/lib/python$(PYMAJOR).$(PYMINOR)/site-packages/_pyodide \
		--exclude-file "*__pycache__*" \
		--exclude-file "*/test/*" \
		--exclude-file "*/tests/*" \
		--exclude-file "*/distutils/*"
   # Strip out C++ symbols which all start __Z.
   # There are 4821 of these and they have VERY VERY long names.
   # To show some stats on the symbols you can use the following:
   # cat build/pyodide.asm.js | grep -ohE 'var _{0,5}.' | sort | uniq -c | sort -nr | head -n 20
	sed -i -E 's/var __Z[^;]*;//g' build/pyodide.asm.js
	sed -i '1i\
		"use strict";\
		let setImmediate = globalThis.setImmediate;\
		let clearImmediate = globalThis.clearImmediate;\
		let baseName, fpcGOT, dyncallGOT, fpVal, dcVal;\
	' build/pyodide.asm.js
	echo "globalThis._createPyodideModule = _createPyodideModule;" >> build/pyodide.asm.js
	date +"[%F %T] done building pyodide.asm.js."


env:
	env


node_modules/.installed : src/js/package.json
	cd src/js && npm install --save-dev
	ln -sfn src/js/node_modules/ node_modules
	touch node_modules/.installed

build/pyodide.js: src/js/*.js src/js/pyproxy.gen.js node_modules/.installed
	npx typescript --project src/js
	npx rollup -c src/js/rollup.config.js

src/js/pyproxy.gen.js : src/core/pyproxy.* src/core/*.h
	# We can't input pyproxy.js directly because CC will be unhappy about the file
	# extension. Instead cat it and have CC read from stdin.
	# -E : Only apply prepreocessor
	# -C : Leave comments alone (this allows them to be preserved in typescript
	#      definition files, rollup will strip them out)
	# -P : Don't put in macro debug info
	# -imacros pyproxy.c : include all of the macros definitions from pyproxy.c
	rm -f $@
	echo "// This file is generated by applying the C preprocessor to core/pyproxy.js" >> $@
	echo "// It uses the macros defined in core/pyproxy.c" >> $@
	echo "// Do not edit it directly!" >> $@
	cat $< | $(CC) -E -C -P -imacros src/core/pyproxy.c $(MAIN_MODULE_CFLAGS) - >> $@

build/test.html: src/templates/test.html
	cp $< $@


.PHONY: build/console.html
build/console.html: src/templates/console.html
	cp $< $@
	sed -i -e 's#{{ PYODIDE_BASE_URL }}#$(PYODIDE_BASE_URL)#g' $@


.PHONY: docs/_build/html/console.html
docs/_build/html/console.html: src/templates/console.html
	mkdir -p docs/_build/html
	cp $< $@
	sed -i -e 's#{{ PYODIDE_BASE_URL }}#$(PYODIDE_BASE_URL)#g' $@


.PHONY: build/webworker.js
build/webworker.js: src/webworker.js
	cp $< $@
	sed -i -e 's#{{ PYODIDE_BASE_URL }}#$(PYODIDE_BASE_URL)#g' $@


.PHONY: build/webworker_dev.js
build/webworker_dev.js: src/webworker.js
	cp $< $@
	sed -i -e 's#{{ PYODIDE_BASE_URL }}#./#g' $@


update_base_url: \
	build/console.html \
	build/webworker.js


test: all
	pytest src emsdk/tests packages/*/test* pyodide-build -v

lint: node_modules/.installed
	# check for unused imports, the rest is done by black
	flake8 --select=F401 src tools pyodide-build benchmark conftest.py docs packages/matplotlib/src/
	find src -type f -regex '.*\.\(c\|h\)' \
		| xargs clang-format-6.0 -output-replacements-xml \
		| (! grep '<replacement ')
	npx prettier --check `find src -type f -name '*.js' -not -name '*.gen.js'`
	npx prettier --check `find src -type f -name '*.html'`
	black --check .
	mypy --ignore-missing-imports    \
		pyodide-build/pyodide_build/ \
		src/ 					     \
		packages/micropip/micropip/  \
		packages/*/test* 			 \
		conftest.py 				 \
		docs



benchmark: all
	$(HOSTPYTHON) benchmark/benchmark.py $(HOSTPYTHON) build/benchmarks.json
	$(HOSTPYTHON) benchmark/plot_benchmark.py build/benchmarks.json build/benchmarks.png


clean:
	rm -fr build/*
	rm -fr src/*/*.o
	rm -fr node_modules
	make -C packages clean
	echo "The Emsdk, CPython are not cleaned. cd into those directories to do so."


clean-all: clean
	make -C emsdk clean
	make -C cpython clean
	rm -fr cpython/build

%.o: %.c $(CPYTHONLIB) $(wildcard src/core/*.h src/core/python2js_buffer.js)
	$(CC) -o $@ -c $< $(MAIN_MODULE_CFLAGS) -Isrc/core/


# Stdlib modules that we repackage as standalone packages

# TODO: also include test directories included in other stdlib modules
build/test.data: $(CPYTHONLIB) node_modules/.installed
	./tools/file_packager.sh build/test.data --js-output=build/test.js \
		--preload $(CPYTHONLIB)/test@/lib/python$(PYMAJOR).$(PYMINOR)/test
	npx terser build/test.js -o build/test.js


build/distutils.data: $(CPYTHONLIB) node_modules/.installed
	./tools/file_packager.sh build/distutils.data --js-output=build/distutils.js \
		--preload $(CPYTHONLIB)/distutils@/lib/python$(PYMAJOR).$(PYMINOR)/distutils \
		--exclude tests
	npx terser build/distutils.js -o build/distutils.js


$(CPYTHONLIB): emsdk/emsdk/.complete $(PYODIDE_EMCC) $(PYODIDE_CXX)
	date +"[%F %T] Building cpython..."
	make -C $(CPYTHONROOT)
	date +"[%F %T] done building cpython..."


build/packages.json: FORCE
	date +"[%F %T] Building packages..."
	make -C packages
	date +"[%F %T] done building packages..."


emsdk/emsdk/.complete:
	date +"[%F %T] Building emsdk..."
	make -C emsdk
	date +"[%F %T] done building emsdk."


FORCE:


check:
	./tools/dependency-check.sh


minimal :
	PYODIDE_PACKAGES+=",micropip" make


debug :
	EXTRA_CFLAGS+=" -D DEBUG_F" \
	PYODIDE_PACKAGES+=", micropip, pyparsing, pytz, packaging, " \
	make
