sync README

README.md

@@ -1,6 +1,6 @@
 # SUPERB Submission Template
 
-Welcome to the [SUPERB Challenge](https://superbbenchmark.org/challenge)! SUPERB is a collection of benchmarking resources to evaluate the capability of a universal shared representation for speech processing. It comes with a benchmark on the publicly available datasets and a challenge on a secret/not released hidden dataset. In SUPERB Challenge, a challenging hidden dataset is newly recorded to evaluate the ultimate generaliziblity across various tasks and data.
+Welcome to the [SUPERB Challenge](https://superbbenchmark.org/challenge-slt2022/challenge_overview)! SUPERB is a collection of benchmarking resources to evaluate the capability of a universal shared representation for speech processing. It comes with a benchmark on the publicly available datasets and a challenge on a secret/not released hidden dataset. In SUPERB Challenge, a challenging hidden dataset is newly recorded to evaluate the ultimate generaliziblity across various tasks and data.
 
 You can participate the challenge by simply submitting your self-supervised (SSL) pretrained models (model definition & pretrained weights), and we benchmark it with the hidden dataset. This repository constains useful tools to let you easliy [submit](https://superbbenchmark.org/submit) your models ***privately*** for evaluation to [the challenge hidden-set leaderboard](https://superbbenchmark.org/leaderboard?track=constrained&subset=Hidden+Dev+Set).
 
@@ -148,7 +148,7 @@ That's it! You're now all set to start pretraining your speech models - see the
 
 To make a submission to the [leaderboard](https://superbbenchmark.org/leaderboard?subset=Hidden+Dev+Set), there are 4 main steps:
 
-1. Modify `expert.py` and change `model.pt` so we can initialize an upstream model following the [challenge policy](https://superbbenchmark.org/challenge#Upstream-Specification) by:
+1. Modify `expert.py` and change `model.pt` so we can initialize an upstream model following the [challenge policy](https://superbbenchmark.org/challenge-slt2022/upstream) by:
 
     ```python
     upstream = UpstreamExpert(ckpt="./model.pt")
@@ -156,7 +156,7 @@ To make a submission to the [leaderboard](https://superbbenchmark.org/leaderboar
 
     ***Package Dependency:*** Note that we only install `torch` package so far by following the above steps. If your model needs more packages, you can modify the `requirement.txt` to meet your need and install them inside the current conda environment. We will install the packages you list in the `requirement.txt` before initializing the upstream model.
 
-2. Validate the upstream model's interface meets the requirements in the [challenge policy](https://superbbenchmark.org/challenge#Upstream-Specification). If everything is correct, you should see the following message: "All submission files validated! Now you can make a submission."
+2. Validate the upstream model's interface meets the requirements in the [challenge policy](https://superbbenchmark.org/challenge-slt2022/upstream). If everything is correct, you should see the following message: "All submission files validated! Now you can make a submission."
 
     ```
     python cli.py validate

{{cookiecutter.repo_name}}/README.md

@@ -11,21 +11,79 @@ You can participate the challenge by simply submitting your self-supervised (SSL
 
 #### Note 1.
 
-We accept pre-trained models in PyTorch by default. If you wish to submit upstreams in non-PyTorch frameworks, please [contant us](#contact)!
+We accept pre-trained models in PyTorch by default. If you wish to submit upstreams in non-PyTorch frameworks, please mail to [[email protected]](mailto:[email protected])!
 
 #### Note 2.
 
-If you are not feasible to submit the pre-trained model, please [contant us](#contact) for us to see how to help!
+If you are not feasible to submit the pre-trained model, please mail to [[email protected]](mailto:[email protected]) for us to see how to help!
 
 ## Quickstart
 
-### 1. Create an account and organization on the Hugging Face Hub
+### 1. Add model interfaces
+
+#### forward
+
+Extract features from waveforms.
+
+- **Input:** A list of waveforms in 16000 Hz
+
+    ```python
+    SAMPLE_RATE = 16000
+    BATCH_SIZE = 8
+    EXAMPLE_SEC = 10
+    wavs = [torch.randn(SAMPLE_RATE * EXAMPLE_SEC).cuda() for _ in range(BATCH_SIZE)]
+    results = upstream(wavs)
+    ```
+
+- **Output:** A dictionary with a key for each task. If any task-specific key is not presented, a "hidden_states" key should be provided as the default key. The value for each key is **a list** of padded sequences in the same shape of **(batch_size, max_sequence_length_of_batch, hidden_size)** for weighted-sum to work. It is welcome to perform some preprocessing on the upstream's raw hidden-sets, including upsampling and downsampling. However, all the values must come from **a single upstream model**:
+
+    ```python
+    assert isinstance(results, dict)
+    tasks = ["PR", "SID", "ER", "ASR", "ASV", "SD", "QbE", "ST", "SS", "SE"]
+    for task in tasks:
+        hidden_states = results.get(task, results["hidden_states"])
+        assert isinstance(hidden_states, list)
+
+        for state in hidden_states:
+            assert isinstance(state, torch.Tensor)
+            assert state.dim() == 3, "(batch_size, max_sequence_length_of_batch, hidden_size)"
+            assert state.shape == hidden_states[0].shape
+    ```
+
+#### get_downsample_rates
+
+Provide the downsample rate **from 16000 Hz waveforms** for each task's representation in the dict. For the standard 10ms stride representation, the downsample rate is 160.
+
+```python
+SAMPLE_RATE = 16000
+MSEC_PER_SEC = 1000
+downsample_rate = SAMPLE_RATE * 10 / MSEC_PER_SEC  # 160
+```
+
+The downsample rate will be used to:
+
+1. Calculate the valid representation length of each utterance in the output padded representation.
+2. Prepare the training materials according to the representation's downsample rate for frame-level tasks, e.g. SD, SE, and SS.
+
+- **Input:** the task key (str)
+- **Output:** the downsample rate (int) of the representation for that task
+
+```python
+for task in tasks:
+    assert isinstance(task, str)
+    downsample_rate = upstream.get_downsample_rate(task)
+    assert isinstance(downsample_rate, int)
+    print("The upstream's representation for {task}"
+        f" has the downsample rate of {downsample_rate}.")
+```
+
+### 2. Create an account and organization on the Hugging Face Hub
 
 First create an account on the Hugging Face Hub and you can sign up [here](https://huggingface.co/join) if you haven't already! Next, create a new organization and invite the SUPERB Hidden Set Committee to join. You will upload your model to a repository under this organization so that members inside it can access the model which is not publicly available.
 
 * [superb-hidden-set](https://huggingface.co/superb-hidden-set)
 
-### 2. Create a template repository on your machine
+### 3. Create a template repository on your machine
 
 The next step is to create a template repository on your local machine that contains various files and a CLI to help you validate and submit your pretrained models. The Hugging Face Hub uses [Git Large File Storage (LFS)](https://git-lfs.github.com) to manage large files, so first install it if you don't have it already. For example, on macOS you can run:
 
@@ -72,7 +130,7 @@ my-superb-submission
 └── model.pt                <- Your model weights
 ```
 
-### 3. Install the dependencies
+### 4. Install the dependencies
 
 The final step is to install the project's dependencies:
 
@@ -121,7 +179,3 @@ To make a submission to the [leaderboard](https://superbbenchmark.org/leaderboar
     - Commit Hash (full 40 characters)
 
 After you finish the above 4 steps. You will see a new entry in your [SUPERB profile page](https://superbbenchmark.org/profile) (need login) which does not have any benchmark numbers yet. Please wait for us to finetuned it on the hidden dataset and get the benchmark results. The results will be revealed within one week. Please stay tuned!
-
-## Contact
-
-[[email protected]]([email protected])