text and tuto

app.py

@@ -209,90 +209,6 @@ Be the first to [submit your model]()!
 """
 
 
-HEADING = """
-# 🥇 Open RL Leaderboard 🥇
-
-Welcome to the Open RL Leaderboard! This is a community-driven benchmark for reinforcement learning models.
-"""
-
-ABOUT_TEXT = r"""
-The Open RL Leaderboard is a community-driven benchmark for reinforcement learning models.
-
-## 🔌 How to have your agent evaluated?
-
-The Open RL Leaderboard constantly scans the 🤗 Hub to detect new models to be evaluated. For your model to be evaluated, it must meet the following criteria.
-
-1. The model must be public on the 🤗 Hub
-2. The model must contain an `agent.pt` file.
-3. The model must be [tagged](https://huggingface.co/docs/hub/model-cards#model-cards) `reinforcement-learning`
-4. The model must be [tagged](https://huggingface.co/docs/hub/model-cards#model-cards) with the name of the environment you want to evaluate (for example `MountainCar-v0`)
-
-Once your model meets these criteria, it will be automatically evaluated on the Open RL Leaderboard. It usually takes a few minutes for the evaluation to be completed.
-That's it!
-
-## 🏗️ How do I build the `agent.pt`?
-
-The `agent.pt` file is a [TorchScript module](https://pytorch.org/docs/stable/jit.html#). It must be loadable using `torch.jit.load`.
-The module must take batch of observations as input and return batch of actions. To check if your model is compatible with the Open RL Leaderboard, you can run the following code:
-
-```python
-import gymnasium as gym
-import numpy as np
-import torch
-
-agent_path = "path/to/agent.pt"
-env_id = ... # e.g. "MountainCar-v0"
-
-agent = torch.jit.load(agent_path)
-env = gym.make(env_id)
-observations = np.array([env.observation_space.sample()])
-observations = torch.from_numpy(observations)
-actions = agent(observations)
-actions = actions.numpy()[0]
-assert env.action_space.contains(actions)
-```
-
-## 🕵 How are the models evaluated?
-
-The evaluation is done by running the agent on the environment for 100 episodes. 
-
-For further information, please refer to the [Open RL Leaderboard evaulation script](https://huggingface.co/spaces/open-rl-leaderboard/leaderboard/blob/main/src/evaluation.py).
-
-### The particular case of Atari environments
-
-Atari environments are evaluated on the `NoFrameskip-v4` version of the environment. For example, to evaluate an agent on the `Pong` environment, you must tag your model with `PongNoFrameskip-v4`. The environment is then wrapped to match the standard Atari preprocessing pipeline.
-
-- No-op reset with a maximum of 30 no-ops
-- Max and skip with a skip of 4
-- Episodic life (although the reported score is for the full episode, not the life)
-- Fire reset
-- Clip reward (although the reported score is not clipped)
-- Resize observation to 84x84
-- Grayscale observation
-- Frame stack of 4
-
-## 🚑 Troubleshooting
-
-If you encounter any issue, please [open an issue](https://huggingface.co/spaces/open-rl-leaderboard/leaderboard/discussions/new) on the Open RL Leaderboard repository.
-
-## 🏃 Next steps
-
-We are working on adding more environments and metrics to the Open RL Leaderboard.
-If you have any suggestions, please [open an discussion](https://huggingface.co/spaces/open-rl-leaderboard/leaderboard/discussions/new) on the Open RL Leaderboard repository.
-
-## 📜 Citation
-
-```bibtex
-@misc{open-rl-leaderboard,
-  author = {Quentin Gallouédec and TODO},
-  title = {Open RL Leaderboard},
-  year = {2024},
-  publisher = {Hugging Face},
-  howpublished = "\url{https://huggingface.co/spaces/open-rl-leaderboard/leaderboard}",
-}
-```
-"""
-
 css = """
 .generating {
     border: none;
@@ -325,7 +241,8 @@ def refresh():
 
 
 with gr.Blocks(css=css) as demo:
-    gr.Markdown(HEADING)
+    with open("texts/heading.md") as fp:
+        gr.Markdown(fp.read())
     with gr.Tabs(elem_classes="tab-buttons") as tabs:
         with gr.TabItem("🏅 Leaderboard"):
             all_gr_dfs = {}
@@ -365,8 +282,14 @@ with gr.Blocks(css=css) as demo:
                 # Load the first video of the first environment
                 demo.load(refresh_one_video(df, env_ids[0]), outputs=[all_gr_videos[env_ids[0]]])
 
+
+        with gr.TabItem("🚀 Getting my agent evaluated"):
+            with open("texts/getting_my_agent_evaluated.md") as fp:
+                gr.Markdown(fp.read())
+
         with gr.TabItem("📝 About"):
-            gr.Markdown(ABOUT_TEXT)
+            with open("texts/about.md") as fp:
+                gr.Markdown(fp.read())
 
     demo.load(refresh, outputs=list(all_gr_dfs.values()) + list(all_gr_winners.values()))
 

texts/about.md

@@ -0,0 +1,53 @@
+The Open RL Leaderboard is a community-driven benchmark for reinforcement learning models.
+
+## 🔌 How to have your agent evaluated?
+
+The Open RL Leaderboard constantly scans the 🤗 Hub to detect new models to be evaluated. For your model to be evaluated, it must meet the following criteria.
+
+1. The model must be public on the 🤗 Hub
+2. The model must contain an `agent.pt` file.
+3. The model must be [tagged](https://huggingface.co/docs/hub/model-cards#model-cards) `reinforcement-learning`
+4. The model must be [tagged](https://huggingface.co/docs/hub/model-cards#model-cards) with the name of the environment you want to evaluate (for example `MountainCar-v0`)
+
+Once your model meets these criteria, it will be automatically evaluated on the Open RL Leaderboard. It usually takes a few minutes for the evaluation to be completed.
+That's it!
+
+## 🕵 How are the models evaluated?
+
+The evaluation is done by running the agent on the environment for 100 episodes. 
+
+For further information, please refer to the [Open RL Leaderboard evaulation script](https://huggingface.co/spaces/open-rl-leaderboard/leaderboard/blob/main/src/evaluation.py).
+
+### The particular case of Atari environments
+
+Atari environments are evaluated on the `NoFrameskip-v4` version of the environment. For example, to evaluate an agent on the `Pong` environment, you must tag your model with `PongNoFrameskip-v4`. The environment is then wrapped to match the standard Atari preprocessing pipeline.
+
+- No-op reset with a maximum of 30 no-ops
+- Max and skip with a skip of 4
+- Episodic life (although the reported score is for the full episode, not the life)
+- Fire reset
+- Clip reward (although the reported score is not clipped)
+- Resize observation to 84x84
+- Grayscale observation
+- Frame stack of 4
+
+## 🚑 Troubleshooting
+
+If you encounter any issue, please [open an issue](https://huggingface.co/spaces/open-rl-leaderboard/leaderboard/discussions/new) on the Open RL Leaderboard repository.
+
+## 🏃 Next steps
+
+We are working on adding more environments and metrics to the Open RL Leaderboard.
+If you have any suggestions, please [open an discussion](https://huggingface.co/spaces/open-rl-leaderboard/leaderboard/discussions/new) on the Open RL Leaderboard repository.
+
+## 📜 Citation
+
+```bibtex
+@misc{open-rl-leaderboard,
+  author = {Quentin Gallouédec and TODO},
+  title = {Open RL Leaderboard},
+  year = {2024},
+  publisher = {Hugging Face},
+  howpublished = "\url{https://huggingface.co/spaces/open-rl-leaderboard/leaderboard}",
+}
+```

texts/getting_my_agent_evaluated.md

@@ -0,0 +1,133 @@
+In this guide, we explain how to get your agent evaluated by the [Open RL Leaderboard](https://huggingface.co/spaces/open-rl-leaderboard/leaderboard). For the sake of demonstration, we'll train a simple agent, but if you already have a trained agent, you can of course skip this step.
+
+## 🛠️ Prerequisites
+
+Ensure you have the necessary packages installed:
+
+```bash
+pip install torch huggingface-hub
+```
+
+## 🏋️‍♂️ Training the agent (optinal, just for demonstration)
+
+Here is a simple example of training a reinforcement learning agent using the `CartPole-v1` environment from Gymnasium. You can skip this step if you already have a trained model. For this example, you'll also need the `gymnasium` package:
+
+```bash
+pip install gymnasium
+```
+
+Now, let's train the agent with a simple policy gradient algorithm:
+
+```python
+import gymnasium as gym
+import torch
+from torch import nn, optim
+from torch.distributions import Categorical
+
+# Environment setup
+env_id = "CartPole-v1"
+env = gym.make(env_id, render_mode="human")
+env = gym.wrappers.RecordEpisodeStatistics(env)
+
+# Agent setup
+policy = nn.Sequential(
+    nn.Linear(4, 128),
+    nn.Dropout(p=0.6),
+    nn.ReLU(),
+    nn.Linear(128, 2),
+    nn.Softmax(-1),
+)
+optimizer = optim.Adam(policy.parameters(), lr=1e-2)
+
+# Training loop
+global_step = 0
+for episode_idx in range(10):
+    log_probs = torch.zeros((env.spec.max_episode_steps + 1))
+    returns = torch.zeros((env.spec.max_episode_steps + 1))
+    observation, info = env.reset()
+    terminated = truncated = False
+    step = 0
+    while not terminated and not truncated:
+        probs = policy(torch.tensor(observation))
+        distribution = Categorical(probs)  # Create distribution
+        action = distribution.sample()  # Sample action
+        log_probs[step] = distribution.log_prob(action)  # Store log probability
+        action = action.cpu().numpy()  # Convert to numpy array
+        observation, reward, terminated, truncated, info = env.step(action)
+        step += 1
+        global_step += 1
+        returns[:step] += 0.99 ** torch.flip(torch.arange(step), (0,)) * reward  # return = sum(gamma^i * reward_i)
+
+    episodic_return = info["episode"]["r"][0]
+    print(f"Episode: {episode_idx}  Global step: {global_step}  Episodic return: {episodic_return:.2f}")
+
+    batch_returns = returns[:step]
+    batch_log_probs = log_probs[:step]
+    batch_returns = (batch_returns - batch_returns.mean()) / (batch_returns.std() + 10**-5)
+    policy_loss = torch.sum(-batch_log_probs * batch_returns)
+    optimizer.zero_grad()
+    policy_loss.backward()
+    optimizer.step()
+```
+
+That's it! You've trained a simple policy gradient agent. Now let's see how to upload the agent to the 🤗 Hub so that the [Open RL Leaderboard](https://huggingface.co/spaces/open-rl-leaderboard/leaderboard) can evaluate it.
+
+## 🤖 From policy to agent
+
+To make the agent compatible with the Open RL Leaderboard, you need your model to take a batch of observations as input and return a batch of actions. Here's how you can wrap your policy model into an agent class:
+
+```python
+class Agent(nn.Module):
+    def __init__(self, policy):
+        super().__init__()
+        self.policy = policy
+
+    def forward(self, observations):
+        probs = self.policy(observations)
+        distribution = Categorical(probs)
+        return distribution.sample()
+
+
+agent = Agent(policy) # instantiate the agent
+
+# A few tests to check if the agent is working
+observations = torch.tensor(env.observation_space.sample()).unsqueeze(0)  # dummy batch of observations
+actions = agent(observations)
+actions = actions.numpy()[0]
+assert env.action_space.contains(actions)
+```
+
+## 💾 Saving the agent
+
+For the Open RL Leaderboard to evaluate your agent, you need to save it as a [TorchScript module](https://pytorch.org/docs/stable/jit.html#) under the name `agent.pt`.
+It must be loadable using `torch.jit.load`. Then you can push it to the 🤗 Hub.
+
+```python
+from huggingface_hub import metadata_save, HfApi
+
+# Save model along with its card
+metadata_save("model_card.md", {"tags": ["reinforcement-learning", env_id]})
+dummy_input = torch.tensor(env.observation_space.sample()).unsqueeze(0)  # dummy batch of observations
+agent = torch.jit.trace(agent.eval(), dummy_input)
+agent = torch.jit.freeze(agent)  # required for for the model not to depend on the training library
+agent = torch.jit.optimize_for_inference(agent)
+torch.jit.save(agent, "agent.pt")
+
+# Upload model and card to the 🤗 Hub
+api = HfApi()
+repo_id = "username/REINFORCE-CartPole-v1"  # can be any name
+model_path = api.create_repo(repo_id, repo_type="model")
+api.upload_file(path_or_fileobj="agent.pt", path_in_repo="agent.pt", repo_id=repo_id)
+api.upload_file(path_or_fileobj="model_card.md", path_in_repo="README.md", repo_id=repo_id)
+```
+
+Now, you can find your agent on the 🤗 Hub at `https://huggingface.co/username/REINFORCE-CartPole-v1`.
+
+## 📊 Open RL Leaderboard evaluation
+
+At this point, all you have to do is to wait for the Open RL Leaderboard to evaluate your agent. It usually takes less than 10 minutes.
+Speaking of which, my agent has just appeared on the leaderboard:
+
+![Leaderboard](img.png)
+
+Last place 😢. Next time, our agent will do better 💪!

texts/heading.md

@@ -0,0 +1,3 @@
+# 🥇 Open RL Leaderboard 🥇
+
+Welcome to the Open RL Leaderboard! This is a community-driven benchmark for reinforcement learning models.