Wauplin HF staff commited on
Commit
0fc235e
0 Parent(s):

first commit

Browse files
.gitignore ADDED
@@ -0,0 +1,144 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Byte-compiled / optimized / DLL files
2
+ __pycache__/
3
+ *.py[cod]
4
+ *$py.class
5
+
6
+ # C extensions
7
+ *.so
8
+
9
+ # Distribution / packaging
10
+ .Python
11
+ build/
12
+ develop-eggs/
13
+ dist/
14
+ downloads/
15
+ eggs/
16
+ .eggs/
17
+ lib64/
18
+ parts/
19
+ sdist/
20
+ var/
21
+ wheels/
22
+ pip-wheel-metadata/
23
+ share/python-wheels/
24
+ *.egg-info/
25
+ .installed.cfg
26
+ *.egg
27
+ MANIFEST
28
+
29
+ # PyInstaller
30
+ # Usually these files are written by a python script from a template
31
+ # before PyInstaller builds the exe, so as to inject date/other infos into it.
32
+ *.manifest
33
+ *.spec
34
+
35
+ # Installer logs
36
+ pip-log.txt
37
+ pip-delete-this-directory.txt
38
+
39
+ # Unit test / coverage reports
40
+ htmlcov/
41
+ .tox/
42
+ .nox/
43
+ .coverage
44
+ .coverage.*
45
+ .cache
46
+ nosetests.xml
47
+ coverage.xml
48
+ *.cover
49
+ *.py,cover
50
+ .hypothesis/
51
+ .pytest_cache/
52
+
53
+ # Translations
54
+ *.mo
55
+ *.pot
56
+
57
+ # Django stuff:
58
+ *.log
59
+ local_settings.py
60
+ db.sqlite3
61
+ db.sqlite3-journal
62
+
63
+ # Flask stuff:
64
+ instance/
65
+ .webassets-cache
66
+
67
+ # Scrapy stuff:
68
+ .scrapy
69
+
70
+ # Sphinx documentation
71
+ docs/_build/
72
+
73
+ # PyBuilder
74
+ target/
75
+
76
+ # Jupyter Notebook
77
+ .ipynb_checkpoints
78
+
79
+ # IPython
80
+ profile_default/
81
+ ipython_config.py
82
+
83
+ # pyenv
84
+ .python-version
85
+
86
+ # pipenv
87
+ # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
88
+ # However, in case of collaboration, if having platform-specific dependencies or dependencies
89
+ # having no cross-platform support, pipenv may install dependencies that don't work, or not
90
+ # install all needed dependencies.
91
+ #Pipfile.lock
92
+
93
+ # PEP 582; used by e.g. github.com/David-OConnor/pyflow
94
+ __pypackages__/
95
+
96
+ # Celery stuff
97
+ celerybeat-schedule
98
+ celerybeat.pid
99
+
100
+ # SageMath parsed files
101
+ *.sage.py
102
+
103
+ # Environments
104
+ .env
105
+ .venv
106
+ .venv*
107
+ env/
108
+ venv/
109
+ ENV/
110
+ env.bak/
111
+ venv.bak/
112
+ .venv*
113
+
114
+ # Spyder project settings
115
+ .spyderproject
116
+ .spyproject
117
+
118
+ # Rope project settings
119
+ .ropeproject
120
+
121
+ # mkdocs documentation
122
+ /site
123
+
124
+ # mypy
125
+ .mypy_cache/
126
+ .dmypy.json
127
+ dmypy.json
128
+
129
+ # Pyre type checker
130
+ .pyre/
131
+ .vscode/
132
+ .idea/
133
+
134
+ .DS_Store
135
+
136
+ # Ruff
137
+ .ruff_cache
138
+
139
+ # Spell checker config
140
+ cspell.json
141
+
142
+ mock
143
+ _user_history
144
+ _user_history_exports
Makefile ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ .PHONY: quality style
2
+
3
+ quality:
4
+ ruff format . --check
5
+ ruff check .
6
+ mypy --install-types .
7
+
8
+ style:
9
+ ruff format .
10
+ ruff check --fix .
README.md ADDED
@@ -0,0 +1,92 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ title: Gradio Space CI
3
+ emoji: 🤖
4
+ colorFrom: gray
5
+ colorTo: gray
6
+ sdk: gradio
7
+ sdk_version: 4.7.1
8
+ app_file: app.py
9
+ pinned: false
10
+ ---
11
+
12
+ # Build a CI for your Spaces 🚀
13
+
14
+ **Gradio Space CI** is a plugin (and package) to create ephemeral Spaces for each PR opened on your Space repo.
15
+ The goal is to foster community contributions by making the review process as lean as possible.
16
+
17
+ ## Key features
18
+
19
+ 1. **Listen to Pull Requests**.
20
+ 2. **Launch ephemeral Spaces** on new PRs.
21
+ 1. When a commit is pushed to a PR, the ephemeral Space gets synchronized.
22
+ 2. When the PR is closed, the ephemeral Space is deleted.
23
+ 3. Configure ephemeral Spaces automatically
24
+ 1. All **variables** are copied from the main Space.
25
+ 2. **Secrets** are copied from the main Space, based on CI configuration.
26
+ 3. **Hardware** and **storage** are set, based on CI configuration.
27
+ 4. Only **trusted authors** are able to access secrets
28
+ 1. By default, repo owners are trusted authors
29
+ 2. More authors can be added in CI configuration
30
+ 3. untrusted authors can start ephemeral Space but without secrets or custom hardware
31
+
32
+ Want more? Please open an issue in the [Community Tab](https://huggingface.co/spaces/Wauplin/gradio-space-ci/discussions)! This is meant to be a community-driven implementation, enhanced by user feedback and contributions!
33
+
34
+ ## Integration
35
+
36
+ Integrate *Gradio Space CI* in just a few steps:
37
+
38
+ ### 1. Update your `requirements.txt`
39
+
40
+ If you don't have a `requirements.txt` file yet, create one in your Space repo.
41
+ Add the following line to it:
42
+
43
+ ```bash
44
+ # requirements.txt
45
+ git+https://huggingface.co/spaces/Wauplin/gradio-space-ci
46
+ ```
47
+
48
+
49
+ ### 2. Add a user token as `HF_TOKEN` secret
50
+
51
+ 1. Go to your [user settings page](https://huggingface.co/settings/tokens).
52
+ 2. Create a new token with **write** permissions.
53
+ 3. Go to your Space settings page.
54
+ 4. Add `HF_TOKEN` as a Space secret with your newly created token.
55
+
56
+ (optional) You can also define a `SPACE_CI_SECRET` secret value that will be used to authenticate webhook calls. If you
57
+ don't define one, a random secret value will be generated for you.
58
+
59
+
60
+ ### 3. Configure CI in `app.py`
61
+
62
+ Import the `gradio_space_ci` package and update the last line of your `app.py` script, just before launching the Gradio app.
63
+
64
+ ```py
65
+ # app.py
66
+ import gradio as gr
67
+ from gradio_space_ci import configure_space_ci
68
+
69
+ # ANY gradio app
70
+ with gr.Blocks() as demo:
71
+ ...
72
+
73
+ # Replace `demo.launch()` by `configure_space_ci(demo).launch()`.
74
+ configure_space_ci(
75
+ demo,
76
+ trusted_authors=[], # space owners + manually trusted authors
77
+ private="auto", # ephemeral spaces will have same visibility as the main space. Otherwise, set to `True` or `False` explicitly.
78
+ variables="auto", # same variables as the main space. Otherwise, set to a `Dict[str, str]`.
79
+ secrets=["HF_TOKEN"], # which secret do I want to copy from the main space? Can be a `List[str]`.
80
+ hardware=None, # "cpu-basic" by default. Otherwise set to "auto" to have same hardware as the main space or any valid string value.
81
+ storage=None, # no storage by default. Otherwise set to "auto" to have same storage as the main space or any valid string value.
82
+ ).launch()
83
+ ```
84
+
85
+ And you're done! Ephemeral Spaces will be launched for each and every PR on your repo.
86
+
87
+
88
+ ## Useful links
89
+
90
+ - **Demo:** https://huggingface.co/spaces/Wauplin/gradio-space-ci
91
+ - **README:** https://huggingface.co/spaces/Wauplin/gradio-space-ci/blob/main/README.md
92
+ - **Questions and feedback:** https://huggingface.co/spaces/Wauplin/gradio-space-ci/discussions
app.py ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python
2
+ from pathlib import Path
3
+
4
+ import gradio as gr
5
+ from gradio_space_ci import configure_space_ci
6
+
7
+
8
+ def greet(name: str) -> str:
9
+ return "Hello " + name + "!"
10
+
11
+
12
+ with gr.Blocks() as demo:
13
+ gr.Markdown("## Dummy gradio app to showcase gradio-space-ci")
14
+ name = gr.Textbox(label="Name")
15
+ output = gr.Textbox(label="Output Box")
16
+ greet_btn = gr.Button("Greet")
17
+ greet_btn.click(fn=greet, inputs=name, outputs=output, api_name="greet")
18
+
19
+
20
+ with gr.Blocks() as demo_with_readme:
21
+ with gr.Tab("README"):
22
+ gr.Markdown(Path("README.md").read_text().split("---")[-1])
23
+ with gr.Tab("Demo"):
24
+ demo.render()
25
+
26
+ if __name__ == "__main__":
27
+ configure_space_ci(
28
+ blocks=demo_with_readme.queue(), # ANY gradio app
29
+ trusted_authors=["clefourrier"], # space owners + manually trusted authors
30
+ private="auto", # ephemeral spaces will have same visibility as the main space. Otherwise, set to `True` or `False` explicitly.
31
+ variables="auto", # same variables as the main space. Otherwise, set to a `Dict[str, str]`.
32
+ secrets=["HF_TOKEN"], # which secret do I want to copy from the main space? Can be a `List[str]`.
33
+ hardware=None, # "cpu-basic" by default. Otherwise set to "auto" to have same hardware as the main space or any valid string value.
34
+ storage=None, # no storage by default. Otherwise set to "auto" to have same storage as the main space or any valid string value.
35
+ ).launch()
pyproject.toml ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [tool.ruff]
2
+ # Enable pycodestyle (`E`) and Pyflakes (`F`) codes by default.
3
+ select = ["E", "F"]
4
+ ignore = ["E501"] # line too long (black is taking care of this)
5
+ line-length = 119
6
+ fixable = ["A", "B", "C", "D", "E", "F", "G", "I", "N", "Q", "S", "T", "W", "ANN", "ARG", "BLE", "COM", "DJ", "DTZ", "EM", "ERA", "EXE", "FBT", "ICN", "INP", "ISC", "NPY", "PD", "PGH", "PIE", "PL", "PT", "PTH", "PYI", "RET", "RSE", "RUF", "SIM", "SLF", "TCH", "TID", "TRY", "UP", "YTT"]
7
+
8
+ [tool.isort]
9
+ profile = "black"
10
+ line_length = 119
11
+
12
+ [tool.black]
13
+ line-length = 119
14
+
15
+ [tool.mypy]
16
+ ignore_missing_imports = true
17
+ no_implicit_optional = true
18
+ scripts_are_modules = true
19
+
requirements-dev.txt ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ mypy
2
+ ruff
3
+ types-requests
4
+ types-ujson
requirements.txt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ gradio
2
+ # TODO: update once merged
3
+ git+https://github.com/huggingface/huggingface_hub@main
setup.py ADDED
@@ -0,0 +1,57 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from setuptools import find_packages, setup
2
+
3
+
4
+ def get_version() -> str:
5
+ rel_path = "src/gradio_space_ci/__init__.py"
6
+ with open(rel_path, "r") as fp:
7
+ for line in fp.read().splitlines():
8
+ if line.startswith("__version__"):
9
+ delim = '"' if '"' in line else "'"
10
+ return line.split(delim)[1]
11
+ raise RuntimeError("Unable to find version string.")
12
+
13
+
14
+ install_requires = [
15
+ "gradio[oauth]>=3.44",
16
+ ]
17
+
18
+ extras = {}
19
+
20
+ extras["dev"] = [
21
+ "ruff",
22
+ "black",
23
+ "mypy",
24
+ ]
25
+
26
+
27
+ setup(
28
+ name="gradio_space_ci",
29
+ version=get_version(),
30
+ author="Lucain Pouget",
31
+ author_email="[email protected]",
32
+ description="A package to enable Space CI (ephemeral Spaces on when PR is created).",
33
+ long_description=open("README.md", "r", encoding="utf-8").read(),
34
+ long_description_content_type="text/markdown",
35
+ keywords="gradio spaces ci machine-learning",
36
+ license="Apache",
37
+ url="https://huggingface.co/spaces/Wauplin/gradio-space-ci",
38
+ package_dir={"": "src"},
39
+ packages=find_packages("src"),
40
+ extras_require=extras,
41
+ python_requires=">=3.8.0",
42
+ install_requires=install_requires,
43
+ classifiers=[
44
+ "Intended Audience :: Developers",
45
+ "Intended Audience :: Education",
46
+ "Intended Audience :: Science/Research",
47
+ "License :: OSI Approved :: Apache Software License",
48
+ "Operating System :: OS Independent",
49
+ "Programming Language :: Python :: 3",
50
+ "Programming Language :: Python :: 3 :: Only",
51
+ "Programming Language :: Python :: 3.8",
52
+ "Programming Language :: Python :: 3.9",
53
+ "Programming Language :: Python :: 3.10",
54
+ "Programming Language :: Python :: 3.11",
55
+ "Topic :: Scientific/Engineering :: Artificial Intelligence",
56
+ ],
57
+ )
src/gradio_space_ci/__init__.py ADDED
@@ -0,0 +1,40 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Gradio Space CI is a plugin (and package) to create ephemeral Spaces for each PR opened on your Space repo.
3
+ The goal is to foster community contributions by making the review process as lean as possible.
4
+
5
+ # Key features:
6
+ 1. Listen to Pull Requests.
7
+ 2. Launch ephemeral Spaces on new PRs.
8
+ 1. When a commit is pushed to a PR, the ephemeral Space gets synchronized.
9
+ 2. When the PR is closed, the ephemeral Space is deleted.
10
+ 3. Configure ephemeral Spaces automatically
11
+ 1. All variables are copied from the main Space.
12
+ 2. Secrets are copied from the main Space, based on CI configuration.
13
+ 3. Hardware and storage are set, based on CI configuration.
14
+ 4. Only trusted authors are able to access secrets
15
+ 1. By default, repo owners are trusted authors
16
+ 2. More authors can be added in CI configuration
17
+ 3. untrusted authors can start ephemeral Space but without secrets or custom hardware
18
+
19
+ # Useful links:
20
+ - Demo: https://huggingface.co/spaces/Wauplin/gradio-space-ci
21
+ - README: https://huggingface.co/spaces/Wauplin/gradio-space-ci/blob/main/README.md
22
+ - Questions and feedback: https://huggingface.co/spaces/Wauplin/gradio-space-ci/discussions
23
+ """
24
+ import warnings
25
+
26
+ from huggingface_hub import HfFolder
27
+
28
+ # Check if `HF_TOKEN` is set. If not, Space CI will be disabled but no error is raised.
29
+ if HfFolder.get_token() is None:
30
+ warnings.warn(
31
+ "Cannot find `HF_TOKEN` in environment variables. Please set a token in your Space secrets to enable ephemeral Spaces."
32
+ )
33
+
34
+ def configure_space_ci(blocks, *args, kwargs):
35
+ return blocks
36
+ else:
37
+ from .webhook import configure_space_ci # noqa: F401
38
+
39
+
40
+ __version__ = "0.1.0"
src/gradio_space_ci/webhook.py ADDED
@@ -0,0 +1,526 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import os
2
+ import uuid
3
+ import warnings
4
+ from pathlib import Path
5
+ from typing import Any, Dict, List, Literal, Optional, TypedDict, Union
6
+
7
+ import gradio as gr
8
+ from fastapi import BackgroundTasks, HTTPException, Response, status
9
+ from huggingface_hub import (
10
+ SpaceHardware,
11
+ SpaceStorage,
12
+ WebhookPayload,
13
+ WebhooksServer,
14
+ add_space_secret,
15
+ add_space_variable,
16
+ comment_discussion,
17
+ create_repo,
18
+ delete_repo,
19
+ get_discussion_details,
20
+ get_repo_discussions,
21
+ get_space_runtime,
22
+ get_space_variables,
23
+ request_space_hardware,
24
+ request_space_storage,
25
+ snapshot_download,
26
+ space_info,
27
+ upload_folder,
28
+ )
29
+ from huggingface_hub.repocard import RepoCard
30
+ from huggingface_hub.utils import (
31
+ RepositoryNotFoundError,
32
+ build_hf_headers,
33
+ get_session,
34
+ hf_raise_for_status,
35
+ )
36
+ from requests import HTTPError
37
+
38
+ SPACE_ID = os.environ.get("SPACE_ID")
39
+ IS_EPHEMERAL_SPACE = SPACE_ID is not None and "-ci-pr-" in SPACE_ID
40
+ WEBHOOK_SECRET = os.environ.get("SPACE_CI_SECRET")
41
+
42
+ if SPACE_ID is not None: # If running in a Space (i.e. not locally)
43
+ if WEBHOOK_SECRET is None: # No secret set yet => generate one => restart space
44
+ WEBHOOK_SECRET = str(uuid.uuid4())
45
+ add_space_secret(
46
+ repo_id=SPACE_ID,
47
+ key="SPACE_CI_SECRET",
48
+ value=WEBHOOK_SECRET,
49
+ description="This value is used by the SpaceCI. It is automatically generated and should not be changed.",
50
+ )
51
+
52
+ EPHEMERAL_SPACES_CONFIG: Dict[str, Any] = {}
53
+
54
+
55
+ def configure_space_ci(
56
+ blocks: Optional["gr.Blocks"] = None,
57
+ trusted_authors: Optional[List[str]] = None,
58
+ private: Union[bool, Literal["auto"]] = "auto",
59
+ variables: Union[Dict[str, str], Literal["auto"]] = "auto",
60
+ secrets: Optional[List[str]] = None,
61
+ hardware: Union[SpaceHardware, Literal["auto"], None] = None,
62
+ storage: Union[SpaceStorage, Literal["auto"], None] = None,
63
+ ) -> WebhooksServer:
64
+ if SPACE_ID is None or IS_EPHEMERAL_SPACE:
65
+ # Runs locally => don't configure webhook
66
+ # Runs in an ephemeral Space => don't configure webhook
67
+ return WebhooksServer(ui=blocks)
68
+
69
+ # Authors
70
+ trusted_authors = trusted_authors or []
71
+ namespace = SPACE_ID.split("/")[0]
72
+ try: # Check if namespace is an organization => in this case all members are allowed to trigger CI by default
73
+ response = get_session().get(
74
+ f"https://huggingface.co/api/organizations/{namespace}/members", headers=build_hf_headers()
75
+ )
76
+ response.raise_for_status()
77
+ trusted_authors += [user["user"] for user in response.json()]
78
+ except Exception: # Otherwise, it's a single user => only this user is allowed to trigger CI by default
79
+ trusted_authors += [namespace]
80
+ trusted_authors = sorted(set(trusted_authors))
81
+ EPHEMERAL_SPACES_CONFIG["trusted_authors"] = trusted_authors
82
+
83
+ # Private
84
+ if private == "auto":
85
+ private = space_info(SPACE_ID).private
86
+ EPHEMERAL_SPACES_CONFIG["private"] = private
87
+
88
+ # Variables
89
+ if variables == "auto":
90
+ variables = {value.key: value.value for value in get_space_variables(SPACE_ID).values()}
91
+ EPHEMERAL_SPACES_CONFIG["variables"] = variables
92
+
93
+ # Secrets
94
+ secrets_with_values: Dict[str, str] = {}
95
+ if secrets is not None:
96
+ for secret in secrets:
97
+ secret_value = os.environ.get(secret)
98
+ if secret_value is None:
99
+ warnings.warn(f"Secret {secret} not found in environment variables. Will skip it in ephemeral Space.")
100
+ continue
101
+ secrets_with_values[secret] = secret_value
102
+ EPHEMERAL_SPACES_CONFIG["secrets"] = secrets_with_values
103
+
104
+ # Hardware and storage
105
+ if hardware == "auto" or storage == "auto":
106
+ runtime = get_space_runtime(SPACE_ID)
107
+ if hardware == "auto":
108
+ hardware = runtime.hardware
109
+ if storage == "auto":
110
+ storage = runtime.storage
111
+ EPHEMERAL_SPACES_CONFIG["hardware"] = hardware
112
+ EPHEMERAL_SPACES_CONFIG["storage"] = storage
113
+
114
+ # Summary
115
+ print(
116
+ "Ephemeral Spaces config:"
117
+ f"\n - trusted authors: {trusted_authors}"
118
+ f"\n - private: {private}"
119
+ f"\n - secrets: {', '.join(sorted(secrets_with_values.keys()))}"
120
+ f"\n - variables: {variables}"
121
+ f"\n - storage: {storage}"
122
+ f"\n - hardware: {hardware}"
123
+ )
124
+
125
+ # Configure webhook
126
+ server = WebhooksServer(ui=blocks, webhook_secret=WEBHOOK_SECRET)
127
+ server.add_webhook()(trigger_ci_on_pr)
128
+ configure_webhook_on_hub()
129
+ return server
130
+
131
+
132
+ ###
133
+ # Define webhook on the Hub logic
134
+ ###
135
+
136
+
137
+ def configure_webhook_on_hub():
138
+ url = "https://" + os.environ.get("SPACE_HOST").strip("/") + "/webhooks/trigger_ci_on_pr"
139
+
140
+ # Check if webhook already exists
141
+ webhooks = list_webhooks()
142
+ for webhook in webhooks:
143
+ if webhook["url"] == url:
144
+ print("Webhook already configured")
145
+ return
146
+
147
+ # If not => create it
148
+ create_webhook(
149
+ watched=[{"type": "space", "name": SPACE_ID}], url=url, domains=["repo", "discussion"], secret=WEBHOOK_SECRET
150
+ )
151
+ print("New webhook already configured!")
152
+
153
+
154
+ ###
155
+ # Webhook logic
156
+ ###
157
+
158
+
159
+ async def trigger_ci_on_pr(payload: WebhookPayload, task_queue: BackgroundTasks):
160
+ if payload.repo.type != "space":
161
+ raise HTTPException(400, f"Must be a Space, not {payload.repo.type}")
162
+
163
+ space_id = payload.repo.name
164
+
165
+ has_task = False
166
+ if (
167
+ # Means "a new PR has been opened"
168
+ payload.event.scope.startswith("discussion")
169
+ and payload.event.action == "create"
170
+ and payload.discussion is not None
171
+ and payload.discussion.isPullRequest
172
+ and payload.discussion.status == "open"
173
+ ):
174
+ if not is_pr_synced(space_id=space_id, pr_num=payload.discussion.num):
175
+ # New PR! Sync task scheduled
176
+ task_queue.add_task(
177
+ sync_ci_space,
178
+ space_id=space_id,
179
+ pr_num=payload.discussion.num,
180
+ private=payload.repo.private,
181
+ )
182
+ has_task = True
183
+ elif (
184
+ # Means "a PR has been merged or closed"
185
+ payload.event.scope.startswith("discussion")
186
+ and payload.event.action == "update"
187
+ and payload.discussion is not None
188
+ and payload.discussion.isPullRequest
189
+ and (payload.discussion.status == "merged" or payload.discussion.status == "closed")
190
+ ):
191
+ task_queue.add_task(
192
+ delete_ci_space,
193
+ space_id=space_id,
194
+ pr_num=payload.discussion.num,
195
+ )
196
+ has_task = True
197
+ elif (
198
+ # Means "some content has been pushed to the Space" (any branch)
199
+ payload.event.scope.startswith("repo.content") and payload.event.action == "update"
200
+ ):
201
+ # New repo change. Is it a commit on a PR?
202
+ # => loop through all PRs and check if new changes happened
203
+ for discussion in get_repo_discussions(repo_id=space_id, repo_type="space"):
204
+ if discussion.is_pull_request and discussion.status == "open":
205
+ if not is_pr_synced(space_id=space_id, pr_num=discussion.num):
206
+ # Found a PR that is not yet synced
207
+ task_queue.add_task(
208
+ sync_ci_space,
209
+ space_id=space_id,
210
+ pr_num=discussion.num,
211
+ private=payload.repo.private,
212
+ )
213
+ has_task = True
214
+
215
+ if has_task:
216
+ return Response("Task scheduled to sync/delete Space", status_code=status.HTTP_202_ACCEPTED)
217
+ else:
218
+ return Response("No task scheduled", status_code=status.HTTP_200_OK)
219
+
220
+
221
+ ###
222
+ # Internal logic
223
+ ###
224
+
225
+
226
+ def is_pr_synced(space_id: str, pr_num: int) -> bool:
227
+ # What is the last synced commit for this PR?
228
+ ci_space_id = _get_ci_space_id(space_id=space_id, pr_num=pr_num)
229
+ try:
230
+ card = RepoCard.load(repo_id_or_path=ci_space_id, repo_type="space")
231
+ last_synced_sha = getattr(card.data, "synced_sha", None)
232
+ except HTTPError:
233
+ return False
234
+
235
+ # What is the last commit id for this PR?
236
+ info = space_info(repo_id=space_id, revision=f"refs/pr/{pr_num}")
237
+ last_pr_sha = info.sha
238
+
239
+ # Is it up to date ?
240
+ return last_synced_sha == last_pr_sha
241
+
242
+
243
+ def sync_ci_space(space_id: str, pr_num: int, private: bool) -> None:
244
+ print(f"New task: sync ephemeral env for {space_id} (PR {pr_num})")
245
+ ci_space_id = _get_ci_space_id(space_id=space_id, pr_num=pr_num)
246
+
247
+ # Create a temporary space for CI if didn't exist
248
+ is_new = create_ephemeral_space(space_id=space_id, pr_num=pr_num)
249
+
250
+ # Configure ephemeral Space if trusted author
251
+ is_configured = False
252
+ if is_new:
253
+ is_configured = configure_ephemeral_space(space_id=space_id, pr_num=pr_num)
254
+
255
+ # Download space codebase from PR revision
256
+ snapshot_path = Path(snapshot_download(repo_id=space_id, revision=f"refs/pr/{pr_num}", repo_type="space"))
257
+
258
+ # Overwrite README file in cache (/!\)
259
+ readme_path = snapshot_path / "README.md"
260
+ card = RepoCard.load(readme_path)
261
+ setattr(card.data, "synced_sha", snapshot_path.name) # latest sha
262
+ card.data.title = f"{card.data.title} (ephemeral #{pr_num})"
263
+ card.save(readme_path)
264
+
265
+ # Sync space codebase with PR revision
266
+ upload_folder(
267
+ repo_id=ci_space_id,
268
+ repo_type="space",
269
+ commit_message=f"Sync CI Space with PR {pr_num}.",
270
+ folder_path=snapshot_path,
271
+ delete_patterns="*",
272
+ )
273
+
274
+ # Delete readme file from cache (just in case)
275
+ readme_path.unlink(missing_ok=True)
276
+
277
+ # Post a comment on the PR
278
+ if is_new and is_configured:
279
+ notify_pr(space_id=space_id, pr_num=pr_num, action="created_and_configured")
280
+ elif is_new:
281
+ notify_pr(space_id=space_id, pr_num=pr_num, action="created_not_configured")
282
+ else:
283
+ notify_pr(space_id=space_id, pr_num=pr_num, action="updated")
284
+
285
+
286
+ def create_ephemeral_space(space_id: str, pr_num: int) -> bool:
287
+ # Config values
288
+ ci_space_id = _get_ci_space_id(space_id=space_id, pr_num=pr_num)
289
+ private: bool = EPHEMERAL_SPACES_CONFIG["private"]
290
+
291
+ # Create space
292
+ try:
293
+ create_repo(
294
+ ci_space_id,
295
+ repo_type="space",
296
+ space_sdk="docker", # Will be overwritten by sync
297
+ private=private,
298
+ exist_ok=False,
299
+ )
300
+ return True
301
+ except HTTPError as err:
302
+ if err.response is not None and err.response.status_code == 409: # already exists
303
+ return False
304
+ else:
305
+ raise
306
+
307
+
308
+ def configure_ephemeral_space(space_id: str, pr_num: int) -> bool:
309
+ # Config values
310
+ ci_space_id = _get_ci_space_id(space_id=space_id, pr_num=pr_num)
311
+ trusted_authors: List[str] = EPHEMERAL_SPACES_CONFIG["trusted_authors"]
312
+ variables: Dict[str, str] = EPHEMERAL_SPACES_CONFIG["variables"]
313
+ secrets: Dict[str, str] = EPHEMERAL_SPACES_CONFIG["secrets"]
314
+ hardware: Optional[SpaceHardware] = EPHEMERAL_SPACES_CONFIG["hardware"]
315
+ storage: Optional[SpaceHardware] = EPHEMERAL_SPACES_CONFIG["storage"]
316
+
317
+ # Check if trusted author
318
+ details = get_discussion_details(repo_id=space_id, repo_type="space", discussion_num=pr_num)
319
+ if details.author not in trusted_authors:
320
+ return False # not a trusted author => do NOT set secrets, hardware, storage, etc.
321
+
322
+ # Configure space
323
+ for key, value in variables.items():
324
+ add_space_variable(ci_space_id, key, value)
325
+ for key, value in secrets.items():
326
+ add_space_secret(ci_space_id, key, value)
327
+
328
+ # Request hardware/storage for space
329
+ if hardware is not None and hardware != SpaceHardware.CPU_BASIC:
330
+ request_space_hardware(ci_space_id, hardware, sleep_time=5 * 60) # sleep after 5min on PR Spaces with GPU
331
+ if storage is not None:
332
+ request_space_storage(ci_space_id, storage)
333
+
334
+ return True
335
+
336
+
337
+ def delete_ci_space(space_id: str, pr_num: int) -> None:
338
+ print(f"New task: delete ephemeral env for {space_id} (PR {pr_num})")
339
+
340
+ # Delete
341
+ ci_space_id = _get_ci_space_id(space_id=space_id, pr_num=pr_num)
342
+ try:
343
+ delete_repo(repo_id=ci_space_id, repo_type="space")
344
+ except RepositoryNotFoundError:
345
+ # Repo did not exist: no need to notify
346
+ return
347
+
348
+ # Notify about deletion
349
+ notify_pr(space_id=space_id, pr_num=pr_num, action="deleted")
350
+
351
+
352
+ def notify_pr(
353
+ space_id: str,
354
+ pr_num: int,
355
+ action: Literal["created_not_configured", "created_and_configured", "updated", "deleted"],
356
+ ) -> None:
357
+ ci_space_id = _get_ci_space_id(space_id=space_id, pr_num=pr_num)
358
+ if action == "created_not_configured":
359
+ comment = NOTIFICATION_TEMPLATE_CREATED_NOT_CONFIGURED.format(ci_space_id=ci_space_id)
360
+ elif action == "created_and_configured":
361
+ comment = NOTIFICATION_TEMPLATE_CREATED_AND_CONFIGURED.format(ci_space_id=ci_space_id)
362
+ elif action == "updated":
363
+ comment = NOTIFICATION_TEMPLATE_UPDATED.format(ci_space_id=ci_space_id)
364
+ elif action == "deleted":
365
+ comment = NOTIFICATION_TEMPLATE_DELETED
366
+ else:
367
+ raise ValueError(f"Status {action} not handled.")
368
+
369
+ comment_discussion(repo_id=space_id, repo_type="space", discussion_num=pr_num, comment=comment)
370
+
371
+
372
+ def _get_ci_space_id(space_id: str, pr_num: int) -> str:
373
+ return f"{space_id}-ci-pr-{pr_num}"
374
+
375
+
376
+ NOTIFICATION_TEMPLATE_CREATED_AND_CONFIGURED = """\
377
+ Following the creation of this PR, an ephemeral Space [{ci_space_id}](https://huggingface.co/spaces/{ci_space_id}) has been started. Any changes pushed to this PR will be synced with the test Space.
378
+ Since this PR has been created by a trusted author, the ephemeral Space has been configured with the correct hardware, storage, and secrets.
379
+ _(This is an automated message.)_
380
+ """
381
+
382
+ NOTIFICATION_TEMPLATE_CREATED_NOT_CONFIGURED = """\
383
+ Following the creation of this PR, an ephemeral Space [{ci_space_id}](https://huggingface.co/spaces/{ci_space_id}) has been started. Any changes pushed to this PR will be synced with the test Space.
384
+ Since this PR has not been created by a trusted author, the ephemeral Space has not been configured with the correct hardware, storage, and secrets. An admin must configure it manually.
385
+ _(This is an automated message.)_
386
+ """
387
+
388
+ NOTIFICATION_TEMPLATE_UPDATED = """\
389
+ Following new commits that happened in this PR, the ephemeral Space [{ci_space_id}](https://huggingface.co/spaces/{ci_space_id}) has been updated.
390
+ _(This is an automated message.)_
391
+ """
392
+
393
+ NOTIFICATION_TEMPLATE_DELETED = """\
394
+ PR is now merged/closed. The ephemeral Space has been deleted.
395
+ _(This is an automated message.)_
396
+ """
397
+
398
+ ### TO MOVE TO ITS OWN MODULE
399
+ # Taken from https://github.com/huggingface/huggingface_hub/issues/1808#issuecomment-1802341663
400
+
401
+
402
+ headers = build_hf_headers()
403
+
404
+
405
+ class WatchedItem(TypedDict):
406
+ # Examples:
407
+ # {"type": "user", "name": "julien-c"}
408
+ # {"type": "org", "name": "HuggingFaceH4"}
409
+ # {"type": "model", "name": "HuggingFaceH4/zephyr-7b-beta"}
410
+ # {"type": "dataset", "name": "HuggingFaceH4/ultrachat_200k"}
411
+ # {"type": "space", "name": "HuggingFaceH4/zephyr-chat"}
412
+ type: Literal["model", "dataset", "space", "org", "user"]
413
+ name: str
414
+
415
+
416
+ # Do you want to subscribe to repo updates (code changes), discussion updates (issues, PRs, comments), or both?
417
+ DOMAIN_T = Literal["repo", "discussion"]
418
+
419
+
420
+ def get_webhook(webhook_id: str) -> Dict:
421
+ """Get a webhook by its id."""
422
+ response = get_session().get(f"https://huggingface.co/api/settings/webhooks/{webhook_id}", headers=headers)
423
+ hf_raise_for_status(response)
424
+ return response.json()
425
+
426
+
427
+ def list_webhooks() -> List[Dict]:
428
+ """List all configured webhooks."""
429
+ response = get_session().get("https://huggingface.co/api/settings/webhooks", headers=headers)
430
+ hf_raise_for_status(response)
431
+ return response.json()
432
+
433
+
434
+ def create_webhook(watched: List[WatchedItem], url: str, domains: List[DOMAIN_T], secret: Optional[str]) -> Dict:
435
+ """Create a new webhook.
436
+
437
+ Args:
438
+ watched (List[WatchedItem]):
439
+ List of items to watch. It an be users, orgs, models, datasets or spaces.
440
+ See `WatchedItem` for more details.
441
+ url (str):
442
+ URL to send the payload to.
443
+ domains (List[Literal["repo", "discussion"]]):
444
+ List of domains to watch. It can be "repo", "discussion" or both.
445
+ secret (str, optional):
446
+ Secret to use to sign the payload.
447
+
448
+ Returns:
449
+ dict: The created webhook.
450
+
451
+ Example:
452
+ ```python
453
+ >>> payload = create_webhook(
454
+ ... watched=[{"type": "user", "name": "julien-c"}, {"type": "org", "name": "HuggingFaceH4"}],
455
+ ... url="https://webhook.site/a2176e82-5720-43ee-9e06-f91cb4c91548",
456
+ ... domains=["repo", "discussion"],
457
+ ... secret="my-secret",
458
+ ... )
459
+ {
460
+ "webhook": {
461
+ "id": "654bbbc16f2ec14d77f109cc",
462
+ "watched": [{"type": "user", "name": "julien-c"}, {"type": "org", "name": "HuggingFaceH4"}],
463
+ "url": "https://webhook.site/a2176e82-5720-43ee-9e06-f91cb4c91548",
464
+ "secret": "my-secret",
465
+ "domains": ["repo", "discussion"],
466
+ "disabled": False,
467
+ },
468
+ }
469
+ ```
470
+ """
471
+ print("Creating webhook")
472
+ print({"watched": watched, "url": url, "domains": domains, "secret": str(type(secret))})
473
+
474
+ response = get_session().post(
475
+ "https://huggingface.co/api/settings/webhooks",
476
+ json={"watched": watched, "url": url, "domains": domains, "secret": secret},
477
+ headers=headers,
478
+ )
479
+ hf_raise_for_status(response)
480
+ return response.json()
481
+
482
+
483
+ def update_webhook(
484
+ webhook_id: str, watched: List[WatchedItem], url: str, domains: List[DOMAIN_T], secret: Optional[str]
485
+ ) -> Dict:
486
+ """Update an existing webhook.
487
+
488
+ Exact same usage as `create_webhook` but you must know the `webhook_id`.
489
+ All fields are updated.
490
+ """
491
+ response = get_session().post(
492
+ f"https://huggingface.co/api/settings/webhooks/{webhook_id}",
493
+ json={"watched": watched, "url": url, "domains": domains, "secret": secret},
494
+ headers=headers,
495
+ )
496
+ hf_raise_for_status(response)
497
+ return response.json()
498
+
499
+
500
+ def enable_webhook(webhook_id: str) -> Dict:
501
+ """Enable a webhook (makes it "active")."""
502
+ response = get_session().post(
503
+ f"https://huggingface.co/api/settings/webhooks/{webhook_id}/enable",
504
+ headers=headers,
505
+ )
506
+ hf_raise_for_status(response)
507
+ return response.json()
508
+
509
+
510
+ def disable_webhook(webhook_id: str) -> Dict:
511
+ """Disable a webhook (makes it "disabled")."""
512
+ response = get_session().post(
513
+ f"https://huggingface.co/api/settings/webhooks/{webhook_id}/disable",
514
+ headers=headers,
515
+ )
516
+ hf_raise_for_status(response)
517
+ return response.json()
518
+
519
+
520
+ def delete_webhook(webhook_id: str):
521
+ """Delete a webhook."""
522
+ response = get_session().delete(
523
+ f"https://huggingface.co/api/settings/webhooks/{webhook_id}",
524
+ headers=headers,
525
+ )
526
+ hf_raise_for_status(response)