This PR finishes adding types (partial types were added in #1592, #1593 did some prerequisite rearrangement).

To generate the types correctly in `pyproxy.js`, I changed the protocols like `PyProxyLengthMethods` into ES6 classes (they used to just be objects). This was needed because otherwise Typescript makes the protocol into a namespace but I want it to be a type to indicate that an object has certain methods. To compensate for this change, I had to modify `getPyProxyClass` accordingly. This change required a number of syntactical changes.

I also added a bunch of predicate methods to the base `PyProxy` class to test whether the proxy supports certain extra methods. These are needed to use as Typescript type guards, but they probably are also useful by themselves when code needs to do reflection on the proxies.

I don't know whether the exposed types or the type guard predicates are well named, I would be happy for recommendations of improvements to the names -- both the type names and the predicates are public and deserve to be named with care.

JsDoc does not recognize typescript type guards or typescript type definitions, so I added a JsDoc plugin which strips out type definitions and transforms typeguards `x is y` ==> `boolean`. There's one JsDoc typescript parse error that still happens but it doesn't cause any harm to the output.

I also changed the way that we collect javascript API docs to be a bit more robust. In particular, it used to be organized by file. I changed it as follows: 
1. any API part of `globalThis` or marked with `@memberof globalThis` is treated as a global, 
2. any API of a class whose name starts with `PyProxy` is treated as a part of the `PyProxy` API,
3. anything else is assumed to be part of the pyodide API

As before, things marked `@private` are left out.

I also added tests for the type declarations using `tpd`. They are a new CI step.


ENH More complete typescript types. This PR finishes adding types (partial types were added in #1592, #1593 did some prerequisite rearrangement).

To generate the types correctly in `pyproxy.js`, I changed the protocols like `PyProxyLengthMethods` into ES6 classes (they used to just be objects). This was needed because otherwise Typescript makes the protocol into a namespace but I want it to be a type to indicate that an object has certain methods. To compensate for this change, I had to modify `getPyProxyClass` accordingly. This change required a number of syntactical changes.

I also added a bunch of predicate methods to the base `PyProxy` class to test whether the proxy supports certain extra methods. These are needed to use as Typescript type guards, but they probably are also useful by themselves when code needs to do reflection on the proxies.

I don't know whether the exposed types or the type guard predicates are well named, I would be happy for recommendations of improvements to the names -- both the type names and the predicates are public and deserve to be named with care.

JsDoc does not recognize typescript type guards or typescript type definitions, so I added a JsDoc plugin which strips out type definitions and transforms typeguards `x is y` ==> `boolean`. There's one JsDoc typescript parse error that still happens but it doesn't cause any harm to the output.

I also changed the way that we collect javascript API docs to be a bit more robust. In particular, it used to be organized by file. I changed it as follows: 
1. any API part of `globalThis` or marked with `@memberof globalThis` is treated as a global, 
2. any API of a class whose name starts with `PyProxy` is treated as a part of the `PyProxy` API,
3. anything else is assumed to be part of the pyodide API

As before, things marked `@private` are left out.

I also added tests for the type declarations using `tpd`. They are a new CI step.


PYODIDE_ROOT=$(abspath .)

include Makefile.envs

.PHONY=check


CPYTHONROOT=cpython
CPYTHONLIB=$(CPYTHONROOT)/installs/python-$(PYVERSION)/lib/python$(PYMINOR)

CC=emcc
CXX=em++

all: check \
	build/pyodide.asm.js \
	build/pyodide.js \
	build/console.html \
	build/test.data \
	build/distutils.data \
	build/packages.json \
	build/test.html \
	build/webworker.js \
	build/webworker_dev.js
	echo -e "\nSUCCESS!"


build/pyodide.asm.js: \
	src/core/docstring.o \
	src/core/error_handling.o \
	src/core/numpy_patch.o \
	src/core/hiwire.o \
	src/core/js2python.o \
	src/core/jsproxy.o \
	src/core/keyboard_interrupt.o \
	src/core/main.o  \
	src/core/pyproxy.o \
	src/core/python2js_buffer.o \
	src/core/python2js.o \
	src/pystone.py \
	src/_testcapi.py \
	src/webbrowser.py \
	$(wildcard src/py/pyodide/*.py) \
	$(CPYTHONLIB)
	date +"[%F %T] Building pyodide.asm.js..."
	[ -d build ] || mkdir build
	$(CXX) -s EXPORT_NAME="'_createPyodideModule'" -o build/pyodide.asm.js $(filter %.o,$^) \
		$(MAIN_MODULE_LDFLAGS) -s FORCE_FILESYSTEM=1 \
		--preload-file $(CPYTHONLIB)@/lib/python$(PYMINOR) \
		--preload-file src/webbrowser.py@/lib/python$(PYMINOR)/webbrowser.py \
		--preload-file src/_testcapi.py@/lib/python$(PYMINOR)/_testcapi.py \
		--preload-file src/pystone.py@/lib/python$(PYMINOR)/pystone.py \
		--preload-file src/py/pyodide@/lib/python$(PYMINOR)/site-packages/pyodide \
		--preload-file src/py/_pyodide@/lib/python$(PYMINOR)/site-packages/_pyodide \
		--exclude-file "*__pycache__*" \
		--exclude-file "*/test/*"		\
		--exclude-file "*/tests/*" \
		--exclude-file "*/distutils/*"
   # Strip out C++ symbols which all start __Z.
   # There are 4821 of these and they have VERY VERY long names.
   # To show some stats on the symbols you can use the following:
   # cat build/pyodide.asm.js | grep -ohE 'var _{0,5}.' | sort | uniq -c | sort -nr | head -n 20
	sed -i -E 's/var __Z[^;]*;//g' build/pyodide.asm.js
	sed -i '1i\
		"use strict";\
		let setImmediate = globalThis.setImmediate;\
		let clearImmediate = globalThis.clearImmediate;\
		let baseName, fpcGOT, dyncallGOT, fpVal, dcVal;\
	' build/pyodide.asm.js
	echo "globalThis._createPyodideModule = _createPyodideModule;" >> build/pyodide.asm.js
	date +"[%F %T] done building pyodide.asm.js."


env:
	env


node_modules/.installed : src/js/package.json
	cd src/js && npm install --save-dev
	ln -sfn src/js/node_modules/ node_modules
	touch node_modules/.installed

build/pyodide.js: src/js/*.js src/js/pyproxy.gen.js node_modules/.installed
	npx typescript src/js/pyodide.js \
		--lib ES2018 --allowJs \
		--declaration --emitDeclarationOnly \
		--outDir build
	npx rollup -c src/js/rollup.config.js

src/js/pyproxy.gen.js : src/core/pyproxy.* src/core/*.h
	# We can't input pyproxy.js directly because CC will be unhappy about the file
	# extension. Instead cat it and have CC read from stdin.
	# -E : Only apply prepreocessor
	# -C : Leave comments alone (this allows them to be preserved in typescript
	#      definition files, rollup will strip them out)
	# -P : Don't put in macro debug info
	# -imacros pyproxy.c : include all of the macros definitions from pyproxy.c
	rm -f $@
	echo "// This file is generated by applying the C preprocessor to core/pyproxy.js" >> $@
	echo "// It uses the macros defined in core/pyproxy.c" >> $@
	echo "// Do not edit it directly!" >> $@
	cat $< | $(CC) -E -C -P -imacros src/core/pyproxy.c $(MAIN_MODULE_CFLAGS) - >> $@

build/test.html: src/templates/test.html
	cp $< $@


.PHONY: build/console.html
build/console.html: src/templates/console.html
	cp $< $@
	sed -i -e 's#{{ PYODIDE_BASE_URL }}#$(PYODIDE_BASE_URL)#g' $@


.PHONY: docs/_build/html/console.html
docs/_build/html/console.html: src/templates/console.html
	mkdir -p docs/_build/html
	cp $< $@
	sed -i -e 's#{{ PYODIDE_BASE_URL }}#$(PYODIDE_BASE_URL)#g' $@


.PHONY: build/webworker.js
build/webworker.js: src/webworker.js
	cp $< $@
	sed -i -e 's#{{ PYODIDE_BASE_URL }}#$(PYODIDE_BASE_URL)#g' $@


.PHONY: build/webworker_dev.js
build/webworker_dev.js: src/webworker.js
	cp $< $@
	sed -i -e 's#{{ PYODIDE_BASE_URL }}#./#g' $@

update_base_url: \
	build/console.html \
	build/webworker.js

test: all
	pytest src emsdk/tests packages/*/test* pyodide-build -v

lint: node_modules/.installed
	# check for unused imports, the rest is done by black
	flake8 --select=F401 src tools pyodide-build benchmark conftest.py docs
	find src -type f -regex '.*\.\(c\|h\)' \
		| xargs clang-format-6.0 -output-replacements-xml \
		| (! grep '<replacement ')
	npx prettier --check `find src -type f -name '*.js' -not -name '*.gen.js'`
	black --check .
	mypy --ignore-missing-imports    \
		pyodide-build/pyodide_build/ \
		src/ 					     \
		packages/micropip/micropip/  \
		packages/*/test* 			 \
		conftest.py 				 \
		docs

apply-lint:
	./tools/apply-lint.sh

benchmark: all
	python benchmark/benchmark.py $(HOSTPYTHON) build/benchmarks.json
	python benchmark/plot_benchmark.py build/benchmarks.json build/benchmarks.png


clean:
	rm -fr build/*
	rm -fr src/*/*.o
	rm -fr node_modules
	make -C packages clean
	echo "The Emsdk, CPython are not cleaned. cd into those directories to do so."

clean-all: clean
	make -C emsdk clean
	make -C cpython clean
	rm -fr cpython/build

%.o: %.c $(CPYTHONLIB) $(wildcard src/core/*.h src/core/python2js_buffer.js)
	$(CC) -o $@ -c $< $(MAIN_MODULE_CFLAGS) -Isrc/core/

# Stdlib modules that we repackage as standalone packages

# TODO: also include test directories included in other stdlib modules
build/test.data: $(CPYTHONLIB) node_modules/.installed
	./tools/file_packager.sh build/test.data --js-output=build/test.js \
		--preload $(CPYTHONLIB)/test@/lib/python$(PYMINOR)/test
	npx terser build/test.js -o build/test.js


build/distutils.data: $(CPYTHONLIB) node_modules/.installed
	./tools/file_packager.sh build/distutils.data --js-output=build/distutils.js \
		--preload $(CPYTHONLIB)/distutils@/lib/python$(PYMINOR)/distutils \
		--exclude tests
	npx terser build/distutils.js -o build/distutils.js


$(CPYTHONLIB): emsdk/emsdk/.complete $(PYODIDE_EMCC) $(PYODIDE_CXX)
	date +"[%F %T] Building cpython..."
	make -C $(CPYTHONROOT)
	date +"[%F %T] done building cpython..."

build/packages.json: FORCE
	date +"[%F %T] Building packages..."
	make -C packages
	date +"[%F %T] done building packages..."

emsdk/emsdk/.complete:
	date +"[%F %T] Building emsdk..."
	make -C emsdk
	date +"[%F %T] done building emsdk."

FORCE:

check:
	./tools/dependency-check.sh

minimal :
	PYODIDE_PACKAGES+=",micropip" make

debug :
	EXTRA_CFLAGS+=" -D DEBUG_F" \
	PYODIDE_PACKAGES+=", micropip, pyparsing, pytz, packaging, kiwisolver, " \
	make

PYODIDE_ROOT=$(abspath .)

include Makefile.envs

.PHONY=check


CPYTHONROOT=cpython
CPYTHONLIB=$(CPYTHONROOT)/installs/python-$(PYVERSION)/lib/python$(PYMINOR)

CC=emcc
CXX=em++

all: check \
	build/pyodide.asm.js \
	build/pyodide.js \
	build/console.html \
	build/test.data \
	build/distutils.data \
	build/packages.json \
	build/test.html \
	build/webworker.js \
	build/webworker_dev.js
	echo -e "\nSUCCESS!"


build/pyodide.asm.js: \
	src/core/docstring.o \
	src/core/error_handling.o \
	src/core/numpy_patch.o \
	src/core/hiwire.o \
	src/core/js2python.o \
	src/core/jsproxy.o \
	src/core/keyboard_interrupt.o \
	src/core/main.o  \
	src/core/pyproxy.o \
	src/core/python2js_buffer.o \
	src/core/python2js.o \
	src/pystone.py \
	src/_testcapi.py \
	src/webbrowser.py \
	$(wildcard src/py/pyodide/*.py) \
	$(CPYTHONLIB)
	date +"[%F %T] Building pyodide.asm.js..."
	[ -d build ] || mkdir build
	$(CXX) -s EXPORT_NAME="'_createPyodideModule'" -o build/pyodide.asm.js $(filter %.o,$^) \
		$(MAIN_MODULE_LDFLAGS) -s FORCE_FILESYSTEM=1 \
		--preload-file $(CPYTHONLIB)@/lib/python$(PYMINOR) \
		--preload-file src/webbrowser.py@/lib/python$(PYMINOR)/webbrowser.py \
		--preload-file src/_testcapi.py@/lib/python$(PYMINOR)/_testcapi.py \
		--preload-file src/pystone.py@/lib/python$(PYMINOR)/pystone.py \
		--preload-file src/py/pyodide@/lib/python$(PYMINOR)/site-packages/pyodide \
		--preload-file src/py/_pyodide@/lib/python$(PYMINOR)/site-packages/_pyodide \
		--exclude-file "*__pycache__*" \
		--exclude-file "*/test/*"		\
		--exclude-file "*/tests/*" \
		--exclude-file "*/distutils/*"
   # Strip out C++ symbols which all start __Z.
   # There are 4821 of these and they have VERY VERY long names.
   # To show some stats on the symbols you can use the following:
   # cat build/pyodide.asm.js | grep -ohE 'var _{0,5}.' | sort | uniq -c | sort -nr | head -n 20
	sed -i -E 's/var __Z[^;]*;//g' build/pyodide.asm.js
	sed -i '1i\
		"use strict";\
		let setImmediate = globalThis.setImmediate;\
		let clearImmediate = globalThis.clearImmediate;\
		let baseName, fpcGOT, dyncallGOT, fpVal, dcVal;\
	' build/pyodide.asm.js
	echo "globalThis._createPyodideModule = _createPyodideModule;" >> build/pyodide.asm.js
	date +"[%F %T] done building pyodide.asm.js."


env:
	env


node_modules/.installed : src/js/package.json
	cd src/js && npm install --save-dev
	ln -sfn src/js/node_modules/ node_modules
	touch node_modules/.installed

build/pyodide.js: src/js/*.js src/js/pyproxy.gen.js node_modules/.installed
	npx typescript src/js/pyodide.js \
		--lib ES2018 --allowJs \
		--declaration --emitDeclarationOnly \
		--outDir build
	npx rollup -c src/js/rollup.config.js

src/js/pyproxy.gen.js : src/core/pyproxy.* src/core/*.h
	# We can't input pyproxy.js directly because CC will be unhappy about the file
	# extension. Instead cat it and have CC read from stdin.
	# -E : Only apply prepreocessor
	# -C : Leave comments alone (this allows them to be preserved in typescript
	#      definition files, rollup will strip them out)
	# -P : Don't put in macro debug info
	# -imacros pyproxy.c : include all of the macros definitions from pyproxy.c
	rm -f $@
	echo "// This file is generated by applying the C preprocessor to core/pyproxy.js" >> $@
	echo "// It uses the macros defined in core/pyproxy.c" >> $@
	echo "// Do not edit it directly!" >> $@
	cat $< | $(CC) -E -C -P -imacros src/core/pyproxy.c $(MAIN_MODULE_CFLAGS) - >> $@

build/test.html: src/templates/test.html
	cp $< $@


.PHONY: build/console.html
build/console.html: src/templates/console.html
	cp $< $@
	sed -i -e 's#{{ PYODIDE_BASE_URL }}#$(PYODIDE_BASE_URL)#g' $@


.PHONY: docs/_build/html/console.html
docs/_build/html/console.html: src/templates/console.html
	mkdir -p docs/_build/html
	cp $< $@
	sed -i -e 's#{{ PYODIDE_BASE_URL }}#$(PYODIDE_BASE_URL)#g' $@


.PHONY: build/webworker.js
build/webworker.js: src/webworker.js
	cp $< $@
	sed -i -e 's#{{ PYODIDE_BASE_URL }}#$(PYODIDE_BASE_URL)#g' $@


.PHONY: build/webworker_dev.js
build/webworker_dev.js: src/webworker.js
	cp $< $@
	sed -i -e 's#{{ PYODIDE_BASE_URL }}#./#g' $@

update_base_url: \
	build/console.html \
	build/webworker.js

test: all
	pytest src emsdk/tests packages/*/test* pyodide-build -v

lint: node_modules/.installed
	# check for unused imports, the rest is done by black
	flake8 --select=F401 src tools pyodide-build benchmark conftest.py docs
	find src -type f -regex '.*\.\(c\|h\)' \
		| xargs clang-format-6.0 -output-replacements-xml \
		| (! grep '<replacement ')
	npx prettier --check `find src -type f -name '*.js' -not -name '*.gen.js'`
	black --check .
	mypy --ignore-missing-imports    \
		pyodide-build/pyodide_build/ \
		src/ 					     \
		packages/micropip/micropip/  \
		packages/*/test* 			 \
		conftest.py 				 \
		docs

apply-lint:
	./tools/apply-lint.sh

benchmark: all
	python benchmark/benchmark.py $(HOSTPYTHON) build/benchmarks.json
	python benchmark/plot_benchmark.py build/benchmarks.json build/benchmarks.png


clean:
	rm -fr build/*
	rm -fr src/*/*.o
	rm -fr node_modules
	make -C packages clean
	echo "The Emsdk, CPython are not cleaned. cd into those directories to do so."

clean-all: clean
	make -C emsdk clean
	make -C cpython clean
	rm -fr cpython/build

%.o: %.c $(CPYTHONLIB) $(wildcard src/core/*.h src/core/python2js_buffer.js)
	$(CC) -o $@ -c $< $(MAIN_MODULE_CFLAGS) -Isrc/core/

# Stdlib modules that we repackage as standalone packages

# TODO: also include test directories included in other stdlib modules
build/test.data: $(CPYTHONLIB) node_modules/.installed
	./tools/file_packager.sh build/test.data --js-output=build/test.js \
		--preload $(CPYTHONLIB)/test@/lib/python$(PYMINOR)/test
	npx terser build/test.js -o build/test.js


build/distutils.data: $(CPYTHONLIB) node_modules/.installed
	./tools/file_packager.sh build/distutils.data --js-output=build/distutils.js \
		--preload $(CPYTHONLIB)/distutils@/lib/python$(PYMINOR)/distutils \
		--exclude tests
	npx terser build/distutils.js -o build/distutils.js


$(CPYTHONLIB): emsdk/emsdk/.complete $(PYODIDE_EMCC) $(PYODIDE_CXX)
	date +"[%F %T] Building cpython..."
	make -C $(CPYTHONROOT)
	date +"[%F %T] done building cpython..."

build/packages.json: FORCE
	date +"[%F %T] Building packages..."
	make -C packages
	date +"[%F %T] done building packages..."

emsdk/emsdk/.complete:
	date +"[%F %T] Building emsdk..."
	make -C emsdk
	date +"[%F %T] done building emsdk."

FORCE:

check:
	./tools/dependency-check.sh

minimal :
	PYODIDE_PACKAGES+=",micropip" make

debug :
	EXTRA_CFLAGS+=" -D DEBUG_F" \
	PYODIDE_PACKAGES+=", micropip, pyparsing, pytz, packaging, kiwisolver, " \
	make
