Download standard libraries in a zipfile. ### Description

This is a re-work of #3166. After this PR, the standard libraries are not vendored with `pyodide.asm.data`. Instead, it is now downloaded separately in a zip file similar to how we download pyodide_py.tar.

~~The hard part of this PR is to bootstrap Python without Python source codes. This requires deep-freezing all modules that are required during the bootstrap.~~

~~As mentioned in #3539, `encodings` module is needed during the bootstrap stage, but it is a large module (> 1.0MB) containing a bunch of encodings in the world, so I tried to freeze a subset of the encodings module that is used in the bootstrap (thanks for the idea from Hood), so the size increase of pyodide.asm.wasm is only 70KB.~~

~~However, the problem of freezing only part of the module is that the current Python import system does not support it. Therefore, I added some hacks to make it work, which is explained in a comment in pyodide.ts.~~

### File size change after this PR:

Before

```
- pyodide.asm.data:
    Original size: 4555 KB
    Gzip compressed size (level 1): 3062 KB
    Gzip compressed size (level 6): 2911 KB
    Gzip compressed size (level 9): 2911 KB
    Brotli compressed size: 2522 KB
- pyodide.asm.js:
    Original size: 1186 KB
    Gzip compressed size (level 1): 246 KB
    Gzip compressed size (level 6): 208 KB
    Gzip compressed size (level 9): 208 KB
    Brotli compressed size: 169 KB
- pyodide.asm.wasm:
    Original size: 8715 KB
    Gzip compressed size (level 1): 3142 KB
    Gzip compressed size (level 6): 2791 KB
    Gzip compressed size (level 9): 2791 KB
    Brotli compressed size: 2096 KB
```

After

```
- pyodide.asm.data:
    Original size: 4 KB
    Gzip compressed size (level 1): 0 KB
    Gzip compressed size (level 6): 0 KB
    Gzip compressed size (level 9): 0 KB
    Brotli compressed size: 0 KB
- pyodide.asm.js:
    Original size: 1086 KB
    Gzip compressed size (level 1): 213 KB
    Gzip compressed size (level 6): 177 KB
    Gzip compressed size (level 9): 177 KB
    Brotli compressed size: 145 KB
- pyodide.asm.wasm:
    Original size: 8775 KB
    Gzip compressed size (level 1): 3162 KB
    Gzip compressed size (level 6): 2806 KB
    Gzip compressed size (level 9): 2806 KB
    Brotli compressed size: 2105 KB
- python_stdlib.zip:
    Original size: 2230 KB
    Gzip compressed size (level 1): 2196 KB
    Gzip compressed size (level 6): 2194 KB
    Gzip compressed size (level 9): 2194 KB
    Brotli compressed size: 2181 KB
```


### Remaining works:

- remove pyodide.asm.data: the only remaining parts inside pyodide.asm.data is our [webbrowser.py](https://github.com/pyodide/pyodide/blob/main/src/py/lib/webbrowser.py) stub, I think we can either remove it or move it into pyodide_py.
- maybe merge pyodide_py.tar into the standard libraries zipfile.
- add option to use pycompiled standard libraries.

### Checklists

- [x] Add a [CHANGELOG](https://github.com/pyodide/pyodide/blob/main/docs/project/changelog.md) entry
- [x] Add / update tests

from pathlib import Path
from textwrap import dedent
from typing import TypedDict


class PackageInfo(TypedDict):
    file_name: str
    sha256: str


def create_pypa_index(
    packages: dict[str, PackageInfo], target_dir: Path, dist_url: str
) -> None:
    """Create a pip-compatible Python package (pypa) index to be used with a Pyodide virtual
    environment.

    To use, pass as an `--index-url` or `--extra-index-url` parameter to pip.
    The argument should be a `file:` url pointing to the `pypa_index` folder (or
    if you serve `pypa_index` it can be a normal url). It is also used
    automatically by Pyodide virtual environments created from a release version
    of Pyodide.

    Parameters
    ----------
    packages:
        A dictionary of packages that we want to index. This should be the
        "packages" field from repodata.json.

    target_dir:
        Where to put the  index. It will be placed in a subfolder of
        target_dir called `pypa_index`. `target_dir` should exist but
        `target_dir/pypa_index` should not exist.

    dist_url:
        The CDN url to download packages from. This will be hard coded into the
        generated index. If you wish to install from local files, then prefix
        with `file:` e.g., `f"file:{pyodide_root}/dist"`.
    """
    # We only want to index the wheels
    packages = {
        pkgname: pkginfo
        for (pkgname, pkginfo) in packages.items()
        if pkginfo["file_name"].endswith(".whl")
    }
    if not target_dir.exists():
        raise RuntimeError(f"target_dir={target_dir} does not exist")

    index_dir = target_dir / "pypa_index"
    if index_dir.exists():
        raise RuntimeError(f"{index_dir} already exists")
    index_dir.mkdir()

    # Create top level index
    packages_str = "\n".join(f'<a href="{x}/">{x}</a>' for x in packages.keys())
    index_html = dedent(
        f"""
        <!DOCTYPE html>
        <html>
        <head>
        <meta name="pypi:repository-version" content="1.0">
        <title>Simple index</title>
        </head>
        <body>
        {packages_str}
        </body>
        </html>
        """
    ).strip()

    (index_dir / "index.html").write_text(index_html)

    files_template = dedent(
        """
        <!DOCTYPE html>
        <html>
        <head>
        <meta name="pypi:repository-version" content="1.0">
        <title>Links for {pkgname}</title>
        </head>
        <body>
        <h1>Links for {pkgname}</h1>
        {links}
        </body>
        </html>
        """
    ).strip()

    for pkgname, pkginfo in packages.items():
        pkgdir = index_dir / pkgname
        filename = pkginfo["file_name"]
        shasum = pkginfo["sha256"]
        href = f"{dist_url}{filename}#sha256={shasum}"
        links_str = f'<a href="{href}">{pkgname}</a>\n'
        files_html = files_template.format(pkgname=pkgname, links=links_str)

        pkgdir.mkdir()
        (pkgdir / "index.html").write_text(files_html)

from pathlib import Path
from textwrap import dedent
from typing import TypedDict


class PackageInfo(TypedDict):
    file_name: str
    sha256: str


def create_pypa_index(
    packages: dict[str, PackageInfo], target_dir: Path, dist_url: str
) -> None:
    """Create a pip-compatible Python package (pypa) index to be used with a Pyodide virtual
    environment.

    To use, pass as an `--index-url` or `--extra-index-url` parameter to pip.
    The argument should be a `file:` url pointing to the `pypa_index` folder (or
    if you serve `pypa_index` it can be a normal url). It is also used
    automatically by Pyodide virtual environments created from a release version
    of Pyodide.

    Parameters
    ----------
    packages:
        A dictionary of packages that we want to index. This should be the
        "packages" field from repodata.json.

    target_dir:
        Where to put the  index. It will be placed in a subfolder of
        target_dir called `pypa_index`. `target_dir` should exist but
        `target_dir/pypa_index` should not exist.

    dist_url:
        The CDN url to download packages from. This will be hard coded into the
        generated index. If you wish to install from local files, then prefix
        with `file:` e.g., `f"file:{pyodide_root}/dist"`.
    """
    # We only want to index the wheels
    packages = {
        pkgname: pkginfo
        for (pkgname, pkginfo) in packages.items()
        if pkginfo["file_name"].endswith(".whl")
    }
    if not target_dir.exists():
        raise RuntimeError(f"target_dir={target_dir} does not exist")

    index_dir = target_dir / "pypa_index"
    if index_dir.exists():
        raise RuntimeError(f"{index_dir} already exists")
    index_dir.mkdir()

    # Create top level index
    packages_str = "\n".join(f'<a href="{x}/">{x}</a>' for x in packages.keys())
    index_html = dedent(
        f"""
        <!DOCTYPE html>
        <html>
        <head>
        <meta name="pypi:repository-version" content="1.0">
        <title>Simple index</title>
        </head>
        <body>
        {packages_str}
        </body>
        </html>
        """
    ).strip()

    (index_dir / "index.html").write_text(index_html)

    files_template = dedent(
        """
        <!DOCTYPE html>
        <html>
        <head>
        <meta name="pypi:repository-version" content="1.0">
        <title>Links for {pkgname}</title>
        </head>
        <body>
        <h1>Links for {pkgname}</h1>
        {links}
        </body>
        </html>
        """
    ).strip()

    for pkgname, pkginfo in packages.items():
        pkgdir = index_dir / pkgname
        filename = pkginfo["file_name"]
        shasum = pkginfo["sha256"]
        href = f"{dist_url}{filename}#sha256={shasum}"
        links_str = f'<a href="{href}">{pkgname}</a>\n'
        files_html = files_template.format(pkgname=pkgname, links=links_str)

        pkgdir.mkdir()
        (pkgdir / "index.html").write_text(files_html)
