Spaces:
Runtime error
Runtime error
Improve README
Browse files- README.md +50 -2
- images/autotrain_job.png +0 -0
- images/autotrain_projects.png +0 -0
README.md
CHANGED
@@ -44,15 +44,44 @@ Next, copy the example file of environment variables:
|
|
44 |
cp .env.template .env
|
45 |
```
|
46 |
|
47 |
-
and set the `HF_TOKEN` variable with a valid API token from the `autoevaluator` user. Finally, spin up the application by running:
|
48 |
|
49 |
```
|
50 |
streamlit run app.py
|
51 |
```
|
52 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
53 |
## AutoTrain configuration details
|
54 |
|
55 |
-
Models are evaluated by AutoTrain, with the payload sent to the `AUTOTRAIN_BACKEND_API` environment variable.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
56 |
|
57 |
```
|
58 |
AUTOTRAIN_BACKEND_API=https://api.autotrain.huggingface.co
|
@@ -63,3 +92,22 @@ To evaluate models with a _local_ instance of AutoTrain, change the environment
|
|
63 |
```
|
64 |
AUTOTRAIN_BACKEND_API=http://localhost:8000
|
65 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
44 |
cp .env.template .env
|
45 |
```
|
46 |
|
47 |
+
and set the `HF_TOKEN` variable with a valid API token from the [`autoevaluator`](https://huggingface.co/autoevaluator) bot user. Finally, spin up the application by running:
|
48 |
|
49 |
```
|
50 |
streamlit run app.py
|
51 |
```
|
52 |
|
53 |
+
## Usage
|
54 |
+
|
55 |
+
Evaluation on the Hub involves two main steps:
|
56 |
+
|
57 |
+
1. Submitting an evaluation job via the UI. This creates an AutoTrain project with `N` models for evaluation. At this stage, the dataset is also processed and prepared for evaluation.
|
58 |
+
2. Triggering the evaluation itself once the dataset is processed.
|
59 |
+
|
60 |
+
From the user perspective, only step (1) is needed since step (2) is handled by a cron job on GitHub Actions that executes the `run_evaluation_jobs.py` script every 15 minutes.
|
61 |
+
|
62 |
+
See below for details on manually triggering evaluation jobs.
|
63 |
+
|
64 |
+
### Triggering an evaluation
|
65 |
+
|
66 |
+
To evaluate the models in an AutoTrain project, run:
|
67 |
+
|
68 |
+
```
|
69 |
+
python run_evaluation_jobs.py
|
70 |
+
```
|
71 |
+
|
72 |
+
This will download the [`autoevaluate/evaluation-job-logs`](https://huggingface.co/datasets/autoevaluate/evaluation-job-logs) dataset from the Hub and check which evaluation projects are ready for evaluation (i.e. those whose dataset has been processed).
|
73 |
+
|
74 |
## AutoTrain configuration details
|
75 |
|
76 |
+
Models are evaluated by the [`autoevaluator`](https://huggingface.co/autoevaluator) bot user in AutoTrain, with the payload sent to the `AUTOTRAIN_BACKEND_API` environment variable. Evaluation projects are created and run on either the `prod` or `staging` environments. You can view the status of projects in the AutoTrain UI by navigating to one of the links below (ask internally for access to the staging UI):
|
77 |
+
|
78 |
+
| AutoTrain environment | AutoTrain UI URL | `AUTOTRAIN_BACKEND_API` |
|
79 |
+
|:---------------------:|:--------------------------------------------------------------------------------------------------------------:|:--------------------------------------------:|
|
80 |
+
| `prod` | [`https://ui.autotrain.huggingface.co/projects`](https://ui.autotrain.huggingface.co/projects) | https://api.autotrain.huggingface.co |
|
81 |
+
| `staging` | [`https://ui-staging.autotrain.huggingface.co/projects`](https://ui-staging.autotrain.huggingface.co/projects) | https://api-staging.autotrain.huggingface.co |
|
82 |
+
|
83 |
+
|
84 |
+
The current configuration for evaluation jobs running on [Spaces](https://huggingface.co/spaces/autoevaluate/model-evaluator) is:
|
85 |
|
86 |
```
|
87 |
AUTOTRAIN_BACKEND_API=https://api.autotrain.huggingface.co
|
|
|
92 |
```
|
93 |
AUTOTRAIN_BACKEND_API=http://localhost:8000
|
94 |
```
|
95 |
+
|
96 |
+
### Migrating from staging to production (and vice versa)
|
97 |
+
|
98 |
+
In general, evaluation jobs should run in AutoTrain's `prod` environment, which is defined by the following environment variable:
|
99 |
+
|
100 |
+
```
|
101 |
+
AUTOTRAIN_BACKEND_API=https://api.autotrain.huggingface.co
|
102 |
+
```
|
103 |
+
|
104 |
+
However, there are times when it is necessary to run evaluation jobs in AutoTrain's `staging` environment (e.g. because a new evaluation pipeline is being deployed). In these cases the corresponding environement variable is:
|
105 |
+
|
106 |
+
```
|
107 |
+
AUTOTRAIN_BACKEND_API=https://api-staging.autotrain.huggingface.co
|
108 |
+
```
|
109 |
+
|
110 |
+
To migrate between these two environments, update the `AUTOTRAIN_BACKEND_API` in two places:
|
111 |
+
|
112 |
+
* In the [repo secrets](https://huggingface.co/spaces/autoevaluate/model-evaluator/settings) associated with the `model-evaluator` Space. This will ensure evaluation projects are created in the desired environment.
|
113 |
+
* In the [GitHub Actions secrets](https://github.com/huggingface/model-evaluator/settings/secrets/actions) associated with this repo. This will ensure that the correct evaluation jobs are approved and launched via the `run_evaluation_jobs.py` script.
|
images/autotrain_job.png
ADDED
images/autotrain_projects.png
ADDED