zh translation

naacl_demo/main.py

@@ -40,11 +40,11 @@ widget_size = 400
 # Patternification widget
 ########################################################################################################################
 
-passage = TextAreaInput(title="Passage", rows=3, value=initial_passage, max_width=text_width)
+passage = TextAreaInput(title="篇章", rows=3, value=initial_passage, max_width=text_width)
 passage.align = "center"
-question = TextInput(title="Question", value=initial_question, max_width=text_width)
+question = TextInput(title="问题", value=initial_question, max_width=text_width)
 question.align = "center"
-radio_button_group = RadioButtonGroup(labels=["Pattern 1", "Pattern 2", "Pattern 3"], active=0, max_width=text_width)
+radio_button_group = RadioButtonGroup(labels=["模板 1", "模板 2", "模板 3"], active=0, max_width=text_width)
 radio_button_group.align = "center"
 
 box_style = {
@@ -291,4 +291,4 @@ main_body = column(textbox1, patternification, textbox2, advantage_column, textb
 main_body.align = "center"
 
 curdoc().add_root(main_body)
-curdoc().title = "How many data points is a prompt worth ?"
+curdoc().title = "提示抵得上多少样本数据？"

naacl_demo/text.md

@@ -1,82 +1,90 @@
-Pre-trained language models, fine-tuned with task-specific heads, are the backbone of applied NLP, and bigger and bigger language models are coming. With this in mind, alternative methods are emerging to compete with the classifier heads used in BERT, UniLM and GPT. In particular, GPT-3 has popularized prompts, natural language inputs designed to steer the pre-trained language model itself into solving the task, rather than a classifier built on top of it. 
+当前 NLP 应用的主流方法是针对各式各样的特定任务，分别对预训练语言模型的分类头进行微调。随着语言模型变得越来越大，各种替代方法相继涌现，开始叫板在 BERT、UniLM 以及 GPT 中广泛使用的分类头法。特别地，GPT-3 向大家普及了提示法，该方法通过自然语言输入来引导预训练语言模型依靠自身解决任务，而不再需要额外的分类头。
 
-Prompts are interesting because they allow a practitioner to give information to the model, although in a very different fashion from standard ML supervision. In our NAACL 2021 paper, we investigate prompt-based fine-tuning, a promising alternative fine-tuning approach, and find that prompts often yield an edge over the standard approach. As we interpret a prompt as additional human-crafted information for the model, we measure that edge in terms of data points and quantify: **how many data points is a prompt worth?** 
+提示很有意思，用户可以通过它向模型提供信息，这与传统的 ML 监督学习有很大不同。在我们的 NAACL 2021 论文中，我们研究了基于提示的微调，该方法可用于替代当前标准的有监督微调法，我们的实验表明提示通常比标准方法更有优势，因此该方法很有前途。在分析这种优势时，我们认为提示为模型带来了额外的信息，这促使我们想用数据点这个指标来量化这种优势，也就是说：**提示可以抵多少个数据点？**
 
-## Prompting
+## 提示法
 
-In order to adapt pre-trained language models to a task, the main method is to replace the final token prediction layer of the original model with a randomly initialized linear classifier head. Supervised task data is then used to train the modified model via backpropagation, learning weights for this new head but also modifying weights deeper in the model. In this work, we call this a _head_ model. 
+为了使预训练语言模型能够完成特定任务，当前的主流方法是用随机初始化的线性分类头替换原模型的最后一层：词预测层。然后使用有监督的任务数据通过反向传播来训练修改后的模型，主要学习这个新分类头的权重，同时也可以更新模型其他层的权重。我们将这种方法称为*分类头*法。
 
-A competing approach is _prompting_: a broad class of methods that attempt to use the initial language model to answer the task by predicting words correlated with the classes instead of a class label. This allows them to perform classification while preserving the language model functionality. For this, _prompts_ are used: input sequences designed to produce the desired answer as textual output. 
+一种与之相竞争的方法是*提示法*：这类方法主要尝试使用原语言模型来预测目标类相应的单词来“回答”分类问题，而不是像传统方法那样“预测”类标签。这使得我们可以直接使用语言模型本身来执行分类任务。在这里，*提示*就是精心设计的、用于生成所需的答案文本的输入文本。
 
-Although this may sound abstract, this is a very natural way to reason about text for humans in practice: school exercises, for example, tend to be presented as a text input (for example, an article about Mars) and a question ("Is there life on Mars?") with an expected answer in natural text ("No"<sup>1</sup>) that maps to one of the classes of the task (presumably here, "No" to `False` and "Yes" to `True`). In this paradigm, task-specific data is presented to the model much like a grammar exercise where a student would need to fill in blanks in a fixed way over a list of sequences. Prompting attempts to use the pre-training information contained in the language model explicitly, rather than implicitly through hidden representations that get fed into the linear classifier head.  
+这听起来可能很抽象，但这其实恰恰就是人类在实际生活中进行文本推理时所使用的非常自然的方法：例如，学校练习往往以一个文本输入（例如，一篇关于火星的文章）加上一个问题（“火星上有生命吗？”）的形式呈现，并期望你提供一个自然语言的答案（“否”<sup>1</sup>），该答案其实就可以映射到分类任务的某个类别（这里，“否”对应假，“是”对应真，本例就是个二分类问题）。在这种范式中，就像做语法练习一样，我们把特定于任务的数据输入给模型，而模型就像学生一样，需要以固定的方式进行填空。提示法希望能��式利用语言模型中包含的预训练信息，而不是仅以将其隐含表征馈送给线性分类头的方式隐式利用这些信息。
 
-
-Here's an example for SuperGLUE task BoolQ, which provides a text <span style="color: #0c593d">passage</span> and a <span style="color: #031154">question</span> and expects a boolean yes-or-no answer. This data is combined with a <span style="color: #910713">**pattern**</span> into a sequence with a single <span style="color: #ba9004">**masked token**</span> that the model must predict. This prediction is turned into a classification prediction with a pre-set *verbalizer*, a mapping between tokens and classes: the model probabilities on this token for *yes* and *no* are compared, with the final prediction being `True` if *yes* dominates and `False` if *no* does.
+以下是 SuperGLUE 任务 BoolQ 的示例，其题型为判断题，每条数据包括文本 <span style="color: #0c593d">passage</span> 和问题 <span style="color: #031154">question</span> ，其答案为布尔值，要么为真，要么为假。每条数据可以和 <span style="color: #910713">**模板（pattern）**</span> 一起组装成一个文本序列，该序列只有一个需预测的 <span style="color: #ba9004">**掩码词**</span>。对该掩码作出预测后，预测词会被一个预设的 *言语器（verbalizer）* 转换为类，也就是说*言语器*负责输出词与类别之间的映射：比较该词被映射为*真*和*假*的概率，如果*真*的概率高，则最终预测为真，反之则为假。
 
 ![image](mockups/boolqpatterns.png)
 
-## Fine-tuning
+## 微调
 
-With this, we have turned our general language model into a task-specific classifier. These language model classifiers based on prompts have been used in very diverse ways:  
- 
-- The preserved language modeling functionality from the pre-trained model allows them to perform without additional data, as opposed to linear classifier _heads_ that are initialized from scratch and always start at random performance. A variety of papers have used this for zero-shot classification.  
-- In order to incorporate supervised task data, they can use backpropagation with the usual language modeling cross-entropy loss objective: the verbalizer token associated with the correct class then serves as the correct token prediction. This is a component of PET, and is the objective used by T5 - although T5 uses prefixes to indicate the task rather than describing it with a natural-language prompt.  
-- They can also use _priming_, where the sequence that needs to be filled in is prefixed with a list of correctly-filled examples. No backpropagation is used, and the weights of the language model are never modified: instead, it can attend to correct examples at inference time. This is the method used by GPT3.  
-- Finally, PET uses prompt models to pseudo-label unlabeled data that is then fed to a linear head model.  
+这样，我们就把通用语言模型转变成了针对特定任务的分类器。这种基于提示的语言模型分类器的用法很多：
  
-In this paper, our goal is to present the fairest comparison possible with head models, so we fine-tune with backpropagation.
+- 预训练模型中保留的语言建模功能允许它们在没有额外数据的情况下执行，这与**以随机初始化开始因此初始性能也随机**的*线性分类头*模型相反。因此，许多论文将其用于零样本分类。 
+
+- 为了将有监督的任务数据引入模型，我们可以使用反向传播及语言建模中的交叉熵损失目标来微调：将与正确类别相关联的言语器词作为正确预测。 PET 使用了这个方法，T5 也使用了这个目标函数 - 尽管 T5 使用任务前缀来指示任务，而未使用自然语言提示来描述它。
+
+- 还有一种方法是使用*潜觉（priming）*，此时，我们需要为当前问题找到若干正确的示例，将其作为原输入文本的前缀一起输入给模型。它没有反向传播，所以永远不会修改语言模型的权重永远；相反，它主要靠在推理时使用注意力机制去利用正确的示例。GPT3 使用了该方法。
 
-## How many data points is a prompt worth?
+- 最后，PET 的方法是使用提示模型预测未标注数据的软标签（或称为伪标签），然后将其作为标签去训练线性分类头。
 
-As we have seen, both heads and prompting can be used in a task specific supervised setting. The core difference is that the prompted model is given a specific sentence that roughly describes the task in addition to supervised examples. In some sense, this sentence is supervision as it tells the model about the task, but it is qualitatively a very different form of supervision than is standard in ML. How should we think about this supervision? How do we quantify how “zero-shot” this setup really is?  
+在本文中，我们想在提示法和分类头法之间进行一个公平的比较，因此我们统一采用基于反向传播的微调方法。
 
-We do this by comparing the _head_ and _prompt_ setups on the SuperGLUE tasks and MNLI. For each task, we extract subsets of the dataset of growing size, and repeat fine-tuning on `RoBERTa-large` with both methods on every subset, keeping everything else the same. For fairness, we tune the hyperparameters on the head baseline until they've attained the level of performance of the BERT++ baseline from the SuperGLUE leaderboard, and keep them the same for the _prompt_ model. 
+## 一个提示可以抵多少个数据点？
 
-The curves of final performance (on each task's metric) vs dataset size are plotted below for each task <sup>2</sup>. They allow us to contrast the amount of data required to attain a certain level of performance with both setups on a given task. We call this difference the _data advantage_ of a training setup over the other at that level of performance. We call the range of performance that has been attained by both models the _comparison window_. By integrating over it we get the _average data advantage_ of a method over the other on the task. Graphically, that is simply the area between the curves, divided by the height of the comparison window. <sup>3</sup>  
+正如我们所看到的，分类头法和提示法都可以用于针对下游任务进行有监督微调。二者的核心区别在于，除了带标注的原始样本外，提示法还给了模型一个用于对特定任务进行粗略描述的句子。从某种意义上说，这句话也是一种监督数据，因为它告诉模型任务的信息，但它在本质上与机器学习中标准的监督数据又截然不同。我们应该如何看待这种新的监督数据？又该如何量化这种方法的“零样本”程度？
+
+我们通过在 SuperGLUE 任务和 MNLI 上比较*分类头法*和*提示法*来尝试回答上面的问题。我们使用的具体方法是：对每个任务，我们通过从数据集中选取样本数不断增加的子集，然后在每个子集上使用这两种方法对 `RoBERTa-large` 进行微调，同时其他所有配置保持不变，最后对评估各自的微调模型的性能。为了公平起见，我们先调整基线分类头模型的超参，并使它们的性能达到 SuperGLUE 排行榜中 BERT++ 的性能水平，然后在对应的*提示法*模型中采用相同的超参。
+
+下图绘制了每个任务 <sup>2</sup> 的最终性能（指标随任务而不同）随数据集大小的变化曲线。有了这个图，我们就能够对两种方法在给定任务上达到一定性能水平所需的数据量进行对比。我们将这种差异称为在该性能水平上其中一个方法相对于其他方法的“数据优势”。我们将两种方法都能达到的性能的范围称为*比较窗口*。通过在该范围内进行积分，我们可以获得在某任务上一种方法相对于另一种方法的“平均数据优势”。从图上看，这即是两条曲线所夹区域的的面积除以比较窗口的高度。<sup>3</sup>
 
 ![image](mockups/advantage.png)
 
-Here's a recapitulative table of the average data advantage of the prompt model over the head model per task, with error bounds obtained by a bootstrapping approach where we hold out one of the 4 head runs and 4 prompt runs (16 combinations total for every data size), and compute the standard deviation of those outcomes. Results are very different from task to task; they even vary for the same task on different dataset, for example for MNLI and RTE, both entailment tasks. However, on every task but WiC <sup>4</sup>, the prompt method has a significant edge. **The additional information provided by the prompt is consistently equivalent to hundreds of data points**.  
+下表总结了在每个任务上提示法相对于分类头法的平均数据优势，其误差范围由自助采样法（bootstrapping）获得，具体做法是对每个数据规模，我们运行 4 次分类头法和 4 次提示法（即每个数据规模共 16 种组合），然后计算这些结果的标准差。不同任务的结果有很大不同；甚至对于同一任务的不同数据集，结果也会有所不同，例如 MNLI 和 RTE，这俩数据集虽然同属蕴涵任务，但结果就很不同。但，除 WiC <sup>4</sup> 之外，提示方法在其他任务中都具有显著的优势。 **提示提供的附加信息一直相当于数百个数据点**。
 
 |                | MNLI     | BoolQ  | CB   | COPA    | MultiRC<sup>5</sup> | RTE    | WiC     | WSC     |
 |----------------|----------|--------|------|---------|----------|--------|---------|---------|
-| Prompt vs Head | 3506±536 | 752±46 | 90±2 | 288±242 | 384±378  | 282±34 | -424±74 | 281±137 |
+| 提示法 vs 分类头法 | 3506±536 | 752±46 | 90±2 | 288±242 | 384±378  | 282±34 | -424±74 | 281±137 |
 
 
-## Patterns and verbalizers
+## 模板与言语器
 
-#### Control verbalizers
+#### 对言语器进行控制
 
-Prompting has for now mostly been used as a tool for zero-shot classification, which is a natural use case. However, zero-shot is usually tricky and requires perfectly aligning the prompt and verbalizer. We have already shown that prompting could be applied more generally, including in the full-data regime. In order to contrast the zero-shot and adaptive natures of prompts, we consider a _null verbalizer_, a control with a verbalizer that is completely decorrelated from the task. For tasks that only require filling in one token (thus excluding the more free-form COPA and WSC), we replace the verbalizers, for example, "yes", "no", "maybe", "right" or "wrong",  with random first names. This makes the model unusable without training data, much like a head model. We plot the corresponding curves and perform the same advantage analysis below:
+当前，提示主要被用作零样本分类的工具，这是一个很自然的用法。然而，真正操作起来，零样本一般会很棘手，因为需要对提示和言语器进行完美对齐。在上文中，我们已经表明，提示可以应用到更广泛的场景中，包括在全数据场景中。为了对比提示法的零样本性和自适应性，我们考虑一个*空言语器（null verbalizer）*，该言语器与任务完全无关。对于只需要填写一个词的任务（因此 COPA 和 WSC 数据集不在此列），我们把器中的映射（例如“是”、“否”、“也许”、“对”或“错”）替换成随机的。这样的话，模型在没有训练数据的情况下就无法使用了，就像分类头模型一样。我们对空言语器配置进行与上文相同的优势分析，并绘制出相应的曲线，如下：
 
 ![image](mockups/nullverbalizer.png)
 
-|                | MNLI     | BoolQ  | CB   | MultiRC<sup>4</sup> | RTE    | WiC     |
+|                | MNLI     | BoolQ  | CB   | MultiRC<sup>5</sup> | RTE    | WiC     |
 |----------------|----------|--------|------|----------|--------|---------|
-| Prompt vs Head | 3506±536 | 752±46 | 90±2 | 384±378  | 282±34 | -424±74 |
-| Prompt vs Null | 150±252  | 299±81 | 78±2 | 74±56    | 404±68 | -354±166 |
-| Null vs Head   | 3355±612 | 453±90 | 12±1 | 309±320  | -122±62 | -70±160 |
+| 提示法 vs 分类头法 | 3506±536 | 752±46 | 90±2 | 384±378  | 282±34 | -424±74 |
+| 提示法 vs 空言语器 | 150±252  | 299±81 | 78±2 | 74±56    | 404±68 | -354±166 |
+| 空言语器 vs 分类头法 | 3355±612 | 453±90 | 12±1 | 309±320  | -122±62 | -70±160 |
 
-Results are noisier than for the straight prompt vs head comparison; however, we find that even with a null verbalizer, the language model is able to adapt to the task, generally catching up with the proper prompted model even with a few data points, and generally doing either on par with or better than the head model, showing the inductive bias of the prompt patterns is beneficial even without an informative verbalizer.  
+从结果来看，其数据优势噪声比直接提示法与分类头法的数据优势的噪声更大。然而，我们也发现，即使仅使用空言语器，语言模型也能够适应任务，即使只有几个数据点，其也能凭借恰当的提示取得与分类头模型相当或更好的性能。因此，我们可以认为，即使没有信息丰富的言语器，提示法带来的归纳偏差也是有益的。
 
-#### Influence of the pattern choice
+#### 模板选择带来的影响
 
-Another choice that can make or break zero-shot classification is that of the pattern, and we investigate whether that still holds in our setting. In all of our experiments, we have re-used the pattern choices from PET - two or three quite different formulations per task - and repeated all of our prompt experiments with every pattern available on the task. We plot results below; they show that the choice of prompt does not have a significant influence, being always within random seed variance.  
+另一个可能影响提示法在零样本分类场景下的成败的因素是：提示模板的选择。这里，我们看一下该因素对我们的影响。我们复用了 PET 中的模板（每个任务有两到三个完全不同的模板），并对每个任务的每种模板进行了实验，结果如下。我们可以看到提示的选择对结果没有显著影响，其方差小于随机种子带来的方差。
 
 ![image](mockups/prompts.png)
 
-## Mot de la fin
+## 最后的话
 
-In this work, we investigate alternate methods of fine-tuning based on natural language prompts, that aim to use the language modeling ability of pre-trained models explicitly through word predictions, instead of implicitly through linear classifiers based on the model's internal representations. We isolate the problem of fine-tuning prompt-based classifier language models with backpropagation, and find that they generally outperform standard fine-tuned linear classifiers. We estimate this advantage in terms of data point to measure the additional information provided by the human via the prompt, and find that **writing a prompt is consistently worth hundreds of data points**. Furthermore, this advantage holds even with non-informative target tokens and is fairly robust to the choice of prompt. 
+通过这项工作，我���研究了一种新的、基于自然语言提示的微调方法，其目的是通过单词预测显式地利用预训练模型的语言建模能力，而不是通过基于模型中间表征的线性分类器隐式地利用它。为了公平比较，我们把问题建模为用反向传播来微调基于提示的分类器语言模型，我们发现提示法通常优于标准的微调线性分类头的方法。我们用数据点来估计这种优势，以衡量人类通过提示提供的附加信息，并发现**编写提示始终抵得上数百个数据点**。此外，即使没有言语器带来的信息量（即使用空言语器），这种优势仍然存在，并且这种方法对于提示的选择相当鲁棒。
 
-For practitioners, we believe that prompt-based fine-tuning should become a standard tool: especially for small- and middle-size task-specific datasets, designing a prompt yourself is a small effort for a sizable data advantage. For researchers, we believe that a lot of questions remain unexplored in this space: Why is the same prompt worth 3500 MNLI data points but only 282 RTE data points? How are prompts related to standard ML supervision? Do they react differently to adversarial or out-of domain examples, since they have some zero-shot behaviour?
+对于从业人员而言，我们相信基于提示的微调应当而且会成为一种标准工​​具：特别是对于中小型的特定任务数据集，自己设计提示只需付出很小的努力即可获得相当大的数据优势。而对于研究人员而言，我们认为这个领域还有很多问题尚待探索：为什么相同的提示在 MNLI 数据集上抵得上 3500 个样本，而在 RTE 数据集上却只抵得上 282 个样本？提示与标准 ML 监督有何关系？由于它们具有一些零样本特性，因此它们对对抗样本或领域外样本的反应是否有所不同？
 
-<sup>1</sup>: Or at least not that we know of.
+<sup>1</sup>：或者严格点说，至少据我们所知为否。
 
-<sup>2</sup>: A sharp-eyed reader will have noticed that all those curves are monotonous. We've performed 4 runs for every experiment (i.e. every data size of every task for head and prompt models). For clarity, and because fine-tuning can sometimes fail for both methods, resulting in negative outliers, we report for every data size the maximum performance that has been attained at this data size or smaller, which we call the _accumulated maximum_ aggregate. This does not have a big impact on the reported data advantage besides reducing variance, and the graphical interpretation would still hold even with non-monotonous curves. 
+<sup>2</sup>：眼尖的读者会注意到所有这些曲线都是单调的。我们为每个实验执行了 4 次运行（即对每个任务的每个数据规模，分别各运行分类头法和提示法 4 次，并用得到的模型测试）。为了清楚起见，并且由于两种方法的微调有时都会失败，从而导致负异常值，因此针对每个数据规模我们报告在此数据规模或更小的数据规模下获得的最大性能，我们将其称为*累积最大*聚合。除了减少方差之外，这不会对报告的数据优势产生太大影响，且即使对于非单调曲线，对图形的解读仍然成立。
 
 <sup>3</sup>: We treat each metric linearly to calculate advantage; alternatively, we could re-parameterize the y axis for each task. This choice does not have a consistent effect for or against prompting. For example, emphasizing gains close to convergence increases prompting advantage on CB and MNLI but decreases it on COPA or BoolQ. 
 
-<sup>4</sup>: where, interestingly, PET had already found prompting to be ineffective
+<sup>3</sup>：在计算每个指标的数据优势时，我们为每个数据赋予相同的权重；我们还可以针对每个任务重新参数化 y 轴。这种做法到底是会对提示法相对于分类头法的数据优势起促进作用还是阻碍作用不好说，与数据集相关。举个例子，强调接近收敛时的增益会增加 CB 和 MNLI 上的提示法的数据优势，但会降低 COPA 或 BoolQ 上的优势。
+
+<sup>4</sup>：有趣的是，PET 已经发现提示对该数据集无效
+
+<sup>5</sup>：MultiRC 的比较窗口太小，因为分类头基线模型无法学习到多数类之外知识；我们使用整个区域来获得实际结果的下界。
 
-<sup>5</sup>: The comparison window of MultiRC is too small as the head baseline fails to learn beyond majority class; we use the full region for a lower-bound result.
+> 英文原文: <url> https://huggingface.co/spaces/teven-projects/how_many_data_points </url>
+> 原文作者：Teven Le Scao
+> 译者: Matrix Yao (姚伟峰)，英特尔深度学习工程师，工作方向为 transformer-family 模型在各模态数据上的应用及大规模模型的训练推理。