"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_random_elements(datasets[\"train\"])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "n9qywopnIrJH"
},
"source": [
"## 预处理数据"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"id": "eXNLu_-nIrJI"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"tokenizer_config.json: 100%|██████████| 28.0/28.0 [00:00<00:00, 117kB/s]\n",
"config.json: 100%|██████████| 483/483 [00:00<00:00, 2.07MB/s]\n",
"vocab.txt: 100%|██████████| 232k/232k [00:00<00:00, 1.66MB/s]\n",
"tokenizer.json: 100%|██████████| 466k/466k [00:00<00:00, 6.44MB/s]\n"
]
}
],
"source": [
"from transformers import AutoTokenizer\n",
" \n",
"tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Vl6IidfdIrJK"
},
"source": [
"以下断言确保我们的 Tokenizers 使用的是 FastTokenizer（Rust 实现，速度和功能性上有一定优势）。"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"import transformers\n",
"assert isinstance(tokenizer, transformers.PreTrainedTokenizerFast)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"您可以在大模型表上查看哪种类型的模型具有可用的快速标记器，哪种类型没有。\n",
"\n",
"您可以直接在两个句子上调用此标记器（一个用于答案，一个用于上下文）："
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"id": "a5hBlsrHIrJL",
"outputId": "acdaa98a-a8cd-4a20-89b8-cc26437bbe90"
},
"outputs": [
{
"data": {
"text/plain": [
"{'input_ids': [[101, 2054, 2003, 2115, 2171, 1029, 102], [101, 2026, 2171, 2003, 25353, 22144, 2378, 1012, 102]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1]]}"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"test_str = \"What is your name?\", \"My name is Sylvain.\"\n",
"tokenizer(test_str)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Tokenizer 进阶操作\n",
"\n",
"在问答预处理中的一个特定问题是如何处理非常长的文档。\n",
"\n",
"在其他任务中，当文档的长度超过模型最大句子长度时，我们通常会截断它们，但在这里，删除上下文的一部分可能会导致我们丢失正在寻找的答案。\n",
"\n",
"为了解决这个问题，我们允许数据集中的一个（长）示例生成多个输入特征，每个特征的长度都小于模型的最大长度（或我们设置的超参数）。"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"# The maximum length of a feature (question and context)\n",
"max_length = 384 \n",
"# The authorized overlap between two part of the context when splitting it is needed.\n",
"doc_stride = 128 "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### 超出最大长度的文本数据处理\n",
"\n",
"下面，我们从训练集中找出一个超过最大长度（384）的文本："
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"for i, example in enumerate(datasets[\"train\"]):\n",
" if len(tokenizer(example[\"question\"], example[\"context\"])[\"input_ids\"]) > 384:\n",
" break\n",
"example = datasets[\"train\"][i]"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"396"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(tokenizer(example[\"question\"], example[\"context\"])[\"input_ids\"])"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"384"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(\n",
" tokenizer(example[\"question\"],\n",
" example[\"context\"],\n",
" max_length=max_length,\n",
" truncation=\"only_second\")[\"input_ids\"]\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### 关于截断的策略\n",
"\n",
"- 直接截断超出部分: truncation=`only_second`\n",
"- 仅截断上下文（context），保留问题（question）：`return_overflowing_tokens=True` & 设置`stride`\n"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"tokenized_example = tokenizer(\n",
" example[\"question\"],\n",
" example[\"context\"],\n",
" max_length=max_length,\n",
" truncation=\"only_second\",\n",
" return_overflowing_tokens=True,\n",
" stride=doc_stride\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"使用此策略截断后，Tokenizer 将返回多个 `input_ids` 列表。"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[384, 157]"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"[len(x) for x in tokenized_example[\"input_ids\"]]\n",
"# tokenized_example"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"解码两个输入特征，可以看到重叠的部分："
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[CLS] how many wins does the notre dame men's basketball team have? [SEP] the men's basketball team has over 1, 600 wins, one of only 12 schools who have reached that mark, and have appeared in 28 ncaa tournaments. former player austin carr holds the record for most points scored in a single game of the tournament with 61. although the team has never won the ncaa tournament, they were named by the helms athletic foundation as national champions twice. the team has orchestrated a number of upsets of number one ranked teams, the most notable of which was ending ucla's record 88 - game winning streak in 1974. the team has beaten an additional eight number - one teams, and those nine wins rank second, to ucla's 10, all - time in wins against the top team. the team plays in newly renovated purcell pavilion ( within the edmund p. joyce center ), which reopened for the beginning of the 2009 – 2010 season. the team is coached by mike brey, who, as of the 2014 – 15 season, his fifteenth at notre dame, has achieved a 332 - 165 record. in 2009 they were invited to the nit, where they advanced to the semifinals but were beaten by penn state who went on and beat baylor in the championship. the 2010 – 11 team concluded its regular season ranked number seven in the country, with a record of 25 – 5, brey's fifth straight 20 - win season, and a second - place finish in the big east. during the 2014 - 15 season, the team went 32 - 6 and won the acc conference tournament, later advancing to the elite 8, where the fighting irish lost on a missed buzzer - beater against then undefeated kentucky. led by nba draft picks jerian grant and pat connaughton, the fighting irish beat the eventual national champion duke blue devils twice during the season. the 32 wins were [SEP]\n",
"[CLS] how many wins does the notre dame men's basketball team have? [SEP] championship. the 2010 – 11 team concluded its regular season ranked number seven in the country, with a record of 25 – 5, brey's fifth straight 20 - win season, and a second - place finish in the big east. during the 2014 - 15 season, the team went 32 - 6 and won the acc conference tournament, later advancing to the elite 8, where the fighting irish lost on a missed buzzer - beater against then undefeated kentucky. led by nba draft picks jerian grant and pat connaughton, the fighting irish beat the eventual national champion duke blue devils twice during the season. the 32 wins were the most by the fighting irish team since 1908 - 09. [SEP]\n"
]
}
],
"source": [
"for x in tokenized_example[\"input_ids\"][:2]:\n",
" print(tokenizer.decode(x))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### 使用 offsets_mapping 获取原始的 input_ids\n",
"\n",
"设置 `return_offsets_mapping=True`，将使得截断分割生成的多个 input_ids 列表中的 token，通过映射保留原始文本的 input_ids。\n",
"\n",
"如下所示：第一个标记（[CLS]）的起始和结束字符都是（0, 0），因为它不对应问题/答案的任何部分，然后第二个标记与问题(question)的字符0到3相同."
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[(1455, 1457), (1457, 1458), (1458, 1462), (1462, 1464), (1465, 1472), (1473, 1477), (1478, 1488), (1489, 1497), (1497, 1498), (1499, 1502), (1503, 1505), (1506, 1509), (1510, 1515), (1516, 1521), (1522, 1524), (1524, 1528), (1529, 1534), (1535, 1538), (1539, 1542), (1543, 1546), (1546, 1548), (1548, 1552), (1552, 1554), (1554, 1555), (1556, 1559), (1560, 1568), (1569, 1574), (1575, 1579), (1580, 1583), (1584, 1592), (1593, 1601), (1602, 1610), (1611, 1615), (1616, 1620), (1621, 1627), (1628, 1633), (1634, 1640), (1641, 1644), (1645, 1651), (1651, 1652), (1653, 1656), (1657, 1659), (1660, 1664), (1665, 1669), (1670, 1673), (1674, 1678), (1679, 1681), (1682, 1685), (1686, 1694), (1695, 1700), (1701, 1705), (1706, 1711), (1712, 1716), (1716, 1717), (1717, 1719), (1719, 1720), (0, 0)]\n"
]
}
],
"source": [
"tokenized_example = tokenizer(\n",
" example[\"question\"],\n",
" example[\"context\"],\n",
" max_length=max_length,\n",
" truncation=\"only_second\",\n",
" return_overflowing_tokens=True,\n",
" return_offsets_mapping=True,\n",
" stride=doc_stride\n",
")\n",
"print(tokenized_example[\"offset_mapping\"][1][100:])\n",
"# import json\n",
"# json.dumps(tokenized_example.__dict__, indent=\" \")\n",
"# tokenized_example"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"因此，我们可以使用这个映射来找到答案在给定特征中的起始和结束标记的位置。\n",
"\n",
"我们只需区分偏移的哪些部分对应于问题，哪些部分对应于上下文。"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"how How\n"
]
}
],
"source": [
"first_token_id = tokenized_example[\"input_ids\"][0][1]\n",
"offsets = tokenized_example[\"offset_mapping\"][0][1]\n",
"print(tokenizer.convert_ids_to_tokens([first_token_id])[0], example[\"question\"][offsets[0]:offsets[1]])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"借助`tokenized_example`的`sequence_ids`方法，我们可以方便的区分token的来源编号：\n",
"\n",
"- 对于特殊标记：返回None，\n",
"- 对于正文Token：返回句子编号（从0开始编号）。\n",
"\n",
"综上，现在我们可以很方便的在一个输入特征中找到答案的起始和结束 Token。"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[None, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, None, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, None]\n"
]
}
],
"source": [
"sequence_ids = tokenized_example.sequence_ids(1)\n",
"print(sequence_ids)\n",
"# tokenized_example"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'id': '5733caf74776f4190066124c',\n",
" 'title': 'University_of_Notre_Dame',\n",
" 'context': \"The men's basketball team has over 1,600 wins, one of only 12 schools who have reached that mark, and have appeared in 28 NCAA tournaments. Former player Austin Carr holds the record for most points scored in a single game of the tournament with 61. Although the team has never won the NCAA Tournament, they were named by the Helms Athletic Foundation as national champions twice. The team has orchestrated a number of upsets of number one ranked teams, the most notable of which was ending UCLA's record 88-game winning streak in 1974. The team has beaten an additional eight number-one teams, and those nine wins rank second, to UCLA's 10, all-time in wins against the top team. The team plays in newly renovated Purcell Pavilion (within the Edmund P. Joyce Center), which reopened for the beginning of the 2009–2010 season. The team is coached by Mike Brey, who, as of the 2014–15 season, his fifteenth at Notre Dame, has achieved a 332-165 record. In 2009 they were invited to the NIT, where they advanced to the semifinals but were beaten by Penn State who went on and beat Baylor in the championship. The 2010–11 team concluded its regular season ranked number seven in the country, with a record of 25–5, Brey's fifth straight 20-win season, and a second-place finish in the Big East. During the 2014-15 season, the team went 32-6 and won the ACC conference tournament, later advancing to the Elite 8, where the Fighting Irish lost on a missed buzzer-beater against then undefeated Kentucky. Led by NBA draft picks Jerian Grant and Pat Connaughton, the Fighting Irish beat the eventual national champion Duke Blue Devils twice during the season. The 32 wins were the most by the Fighting Irish team since 1908-09.\",\n",
" 'question': \"How many wins does the Notre Dame men's basketball team have?\",\n",
" 'answers': {'text': ['over 1,600'], 'answer_start': [30]}}"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"23 26\n"
]
}
],
"source": [
"answers = example[\"answers\"]\n",
"start_char = answers[\"answer_start\"][0]\n",
"end_char = start_char + len(answers[\"text\"][0])\n",
"sequence_ids = tokenized_example.sequence_ids()\n",
"\n",
"# 当前span在文本中的起始标记索引。\n",
"token_start_index = 0\n",
"while sequence_ids[token_start_index] != 1:\n",
" token_start_index += 1\n",
"\n",
"# 当前span在文本中的结束标记索引。\n",
"token_end_index = len(tokenized_example[\"input_ids\"][0]) - 1\n",
"while sequence_ids[token_end_index] != 1:\n",
" token_end_index -= 1\n",
"\n",
"# 检测答案是否超出span范围（如果超出范围，该特征将以CLS标记索引标记）。\n",
"offsets = tokenized_example[\"offset_mapping\"][0]\n",
"if (offsets[token_start_index][0] <= start_char and offsets[token_end_index][1] >= end_char):\n",
" # 将token_start_index和token_end_index移动到答案的两端。\n",
" # 注意：如果答案是最后一个单词，我们可以移到最后一个标记之后（边界情况）。\n",
" while token_start_index < len(offsets) and offsets[token_start_index][0] <= start_char:\n",
" token_start_index += 1\n",
" start_position = token_start_index - 1\n",
" while offsets[token_end_index][1] >= end_char:\n",
" token_end_index -= 1\n",
" end_position = token_end_index + 1\n",
" print(start_position, end_position)\n",
"else:\n",
" print(\"答案不在此特征中。\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"打印检查是否准确找到了起始位置："
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"over 1, 600\n",
"over 1,600\n"
]
},
{
"data": {
"text/plain": [
"'over 1,600'"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"print(tokenizer.decode(tokenized_example[\"input_ids\"][0][start_position: end_position+1]))\n",
"print(answers[\"text\"][0])\n",
"example['context'][start_char:end_char]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### 关于填充的策略\n",
"\n",
"- 对于没有超过最大长度的文本，填充补齐长度。\n",
"- 对于需要左侧填充的模型，交换 question 和 context 顺序"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'right'"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pad_on_right = tokenizer.padding_side == \"right\"\n",
"tokenizer.padding_side"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 整合以上所有预处理步骤\n",
"\n",
"让我们将所有内容整合到一个函数中，并将其应用到训练集。\n",
"\n",
"针对不可回答的情况（上下文过长，答案在另一个特征中），我们为开始和结束位置都设置了cls索引。\n",
"\n",
"如果allow_impossible_answers标志为False，我们还可以简单地从训练集中丢弃这些示例。"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [],
"source": [
"def prepare_train_features(examples):\n",
" # 一些问题的左侧可能有很多空白字符，这对我们没有用，而且会导致上下文的截断失败\n",
" # （标记化的问题将占用大量空间）。因此，我们删除左侧的空白字符。\n",
" examples[\"question\"] = [q.lstrip() for q in examples[\"question\"]]\n",
"\n",
" # 使用截断和填充对我们的示例进行标记化，但保留溢出部分，使用步幅（stride）。\n",
" # 当上下文很长时，这会导致一个示例可能提供多个特征，其中每个特征的上下文都与前一个特征的上下文有一些重叠。\n",
" tokenized_examples = tokenizer(\n",
" examples[\"question\" if pad_on_right else \"context\"],\n",
" examples[\"context\" if pad_on_right else \"question\"],\n",
" truncation=\"only_second\" if pad_on_right else \"only_first\",\n",
" max_length=max_length,\n",
" stride=doc_stride,\n",
" return_overflowing_tokens=True,\n",
" return_offsets_mapping=True,\n",
" padding=\"max_length\",\n",
" )\n",
"\n",
" # 由于一个示例可能给我们提供多个特征（如果它具有很长的上下文），我们需要一个从特征到其对应示例的映射。这个键就提供了这个映射关系。\n",
" sample_mapping = tokenized_examples.pop(\"overflow_to_sample_mapping\")\n",
" # 偏移映射将为我们提供从令牌到原始上下文中的字符位置的映射。这将帮助我们计算开始位置和结束位置。\n",
" offset_mapping = tokenized_examples.pop(\"offset_mapping\")\n",
"\n",
" # 让我们为这些示例进行标记！\n",
" tokenized_examples[\"start_positions\"] = []\n",
" tokenized_examples[\"end_positions\"] = []\n",
"\n",
" for i, offsets in enumerate(offset_mapping):\n",
" # 我们将使用CLS令牌的索引来标记不可能的答案。\n",
" input_ids = tokenized_examples[\"input_ids\"][i]\n",
" cls_index = input_ids.index(tokenizer.cls_token_id)\n",
"\n",
" # 获取与该示例对应的序列（以了解上下文和问题是什么）。\n",
" sequence_ids = tokenized_examples.sequence_ids(i)\n",
"\n",
" # 一个示例可以提供多个跨度，这是包含此文本跨度的示例的索引。\n",
" sample_index = sample_mapping[i]\n",
" answers = examples[\"answers\"][sample_index]\n",
" # 如果没有给出答案，则将cls_index设置为答案。\n",
" if len(answers[\"text\"]) == 0:\n",
" tokenized_examples[\"start_positions\"].append(cls_index)\n",
" tokenized_examples[\"end_positions\"].append(cls_index)\n",
" else:\n",
" # 答案在文本中的开始和结束字符索引。\n",
" start_char = answers[\"answer_start\"][0]\n",
" end_char = start_char + len(answers[\"text\"][0])\n",
"\n",
" # 当前跨度在文本中的开始令牌索引。\n",
" token_start_index = 0\n",
" while sequence_ids[token_start_index] != (1 if pad_on_right else 0):\n",
" token_start_index += 1\n",
"\n",
" # 当前跨度在文本中的结束令牌索引。\n",
" token_end_index = len(input_ids) - 1\n",
" while sequence_ids[token_end_index] != (1 if pad_on_right else 0):\n",
" token_end_index -= 1\n",
"\n",
" # 检测答案是否超出跨度（在这种情况下，该特征的标签将使用CLS索引）。\n",
" if not (offsets[token_start_index][0] <= start_char and offsets[token_end_index][1] >= end_char):\n",
" tokenized_examples[\"start_positions\"].append(cls_index)\n",
" tokenized_examples[\"end_positions\"].append(cls_index)\n",
" else:\n",
" # 否则，将token_start_index和token_end_index移到答案的两端。\n",
" # 注意：如果答案是最后一个单词（边缘情况），我们可以在最后一个偏移之后继续。\n",
" while token_start_index < len(offsets) and offsets[token_start_index][0] <= start_char:\n",
" token_start_index += 1\n",
" tokenized_examples[\"start_positions\"].append(token_start_index - 1)\n",
" while offsets[token_end_index][1] >= end_char:\n",
" token_end_index -= 1\n",
" tokenized_examples[\"end_positions\"].append(token_end_index + 1)\n",
" return tokenized_examples"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zS-6iXTkIrJT"
},
"source": [
"#### datasets.map 的进阶使用\n",
"\n",
"使用 `datasets.map` 方法将 `prepare_train_features` 应用于所有训练、验证和测试数据：\n",
"\n",
"- batched: 批量处理数据。\n",
"- remove_columns: 因为预处理更改了样本的数量，所以在应用它时需要删除旧列。\n",
"- load_from_cache_file：是否使用datasets库的自动缓存\n",
"\n",
"datasets 库针对大规模数据，实现了高效缓存机制，能够自动检测传递给 map 的函数是否已更改（因此需要不使用缓存数据）。如果在调用 map 时设置 `load_from_cache_file=False`，可以强制重新应用预处理。"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [],
"source": [
"# prepare_train_features(datasets['train'][i:i+1])"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"id": "DDtsaJeVIrJT",
"outputId": "aa4734bf-4ef5-4437-9948-2c16363da719"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Map: 100%|██████████| 87599/87599 [00:34<00:00, 2550.59 examples/s]\n",
"Map: 100%|██████████| 10570/10570 [00:04<00:00, 2602.77 examples/s]\n"
]
}
],
"source": [
"tokenized_datasets = datasets.map(\n",
" prepare_train_features,\n",
" batched=True,\n",
" remove_columns=datasets[\"train\"].column_names,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "545PP3o8IrJV"
},
"source": [
"## 微调模型"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FBiW8UpKIrJW"
},
"source": [
"现在我们的数据已经准备好用于训练，我们可以下载预训练模型并进行微调。\n",
"\n",
"由于我们的任务是问答，我们使用 `AutoModelForQuestionAnswering` 类。(对比 Yelp 评论打分使用的是 `AutoModelForSequenceClassification` 类）\n",
"\n",
"警告通知我们正在丢弃一些权重（`vocab_transform` 和 `vocab_layer_norm` 层），并随机初始化其他一些权重（`pre_classifier` 和 `classifier` 层）。在微调模型情况下是绝对正常的，因为我们正在删除用于预训练模型的掩码语言建模任务的头部，并用一个新的头部替换它，对于这个新头部，我们没有预训练的权重，所以库会警告我们在用它进行推理之前应该对这个模型进行微调，而这正是我们要做的事情。"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"editable": true,
"id": "TlqNaB8jIrJW",
"outputId": "84916cf3-6e6c-47f3-d081-032ec30a4132",
"slideshow": {
"slide_type": ""
},
"tags": []
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"model.safetensors: 100%|██████████| 268M/268M [00:01<00:00, 190MB/s] \n",
"Some weights of DistilBertForQuestionAnswering were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['qa_outputs.weight', 'qa_outputs.bias']\n",
"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
]
}
],
"source": [
"from transformers import AutoModelForQuestionAnswering, TrainingArguments, Trainer\n",
"\n",
"model = AutoModelForQuestionAnswering.from_pretrained(model_checkpoint)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_N8urzhyIrJY"
},
"source": [
"#### 训练超参数（TrainingArguments）"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {
"id": "Bliy8zgjIrJY"
},
"outputs": [],
"source": [
"batch_size=64\n",
"model_dir = \"models\"\n",
"model_name = model_checkpoint.split(\"/\")[-1]\n",
"\n",
"args = TrainingArguments(\n",
" f\"{model_dir}/{model_name}-finetuned-squad\",\n",
" evaluation_strategy = \"epoch\",\n",
" learning_rate=2e-5,\n",
" per_device_train_batch_size=batch_size,\n",
" per_device_eval_batch_size=batch_size,\n",
" num_train_epochs=3,\n",
" weight_decay=0.01,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Data Collator（数据整理器）\n",
"\n",
"数据整理器将训练数据整理为批次数据，用于模型训练时的批次处理。本教程使用默认的 `default_data_collator`。\n"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [],
"source": [
"from transformers import default_data_collator\n",
"\n",
"data_collator = default_data_collator"
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"id": "rXuFTAzDIrJe",
"slideshow": {
"slide_type": ""
},
"tags": []
},
"source": [
"### 实例化训练器（Trainer）\n",
"\n",
"为了减少训练时间（需要大量算力支持），我们不在本教程的训练模型过程中计算模型评估指标。\n",
"\n",
"而是训练完成后，再独立进行模型评估。"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {
"editable": true,
"id": "imY1oC3SIrJf",
"slideshow": {
"slide_type": ""
},
"tags": []
},
"outputs": [],
"source": [
"small_train_dataset = tokenized_datasets[\"train\"].shuffle(seed=42).select(range(1000))\n",
"small_validation_dataset = tokenized_datasets[\"validation\"].shuffle(seed=42).select(range(1000))\n",
"\n",
"trainer = Trainer(\n",
" model,\n",
" args,\n",
" train_dataset=tokenized_datasets[\"train\"],\n",
" eval_dataset=tokenized_datasets[\"validation\"],\n",
" data_collator=data_collator,\n",
" tokenizer=tokenizer,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### GPU 使用情况\n",
"\n",
"训练数据与模型配置：\n",
"\n",
"- SQUAD v1.1\n",
"- model_checkpoint = \"distilbert-base-uncased\"\n",
"- batch_size = 64\n",
"\n",
"NVIDIA GPU 使用情况：\n",
"\n",
"```shell\n",
"Every 1.0s: nvidia-smi Wed Dec 20 15:39:57 2023\n",
"\n",
"Wed Dec 20 15:39:57 2023\n",
"+---------------------------------------------------------------------------------------+\n",
"| NVIDIA-SMI 535.129.03 Driver Version: 535.129.03 CUDA Version: 12.2 |\n",
"|-----------------------------------------+----------------------+----------------------+\n",
"| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |\n",
"| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |\n",
"| | | MIG M. |\n",
"|=========================================+======================+======================|\n",
"| 0 Tesla T4 Off | 00000000:00:0D.0 Off | 0 |\n",
"| N/A 67C P0 67W / 70W | 14617MiB / 15360MiB | 100% Default |\n",
"| | | N/A |\n",
"+-----------------------------------------+----------------------+----------------------+\n",
"\n",
"+---------------------------------------------------------------------------------------+\n",
"| Processes: |\n",
"| GPU GI CI PID Type Process name GPU Memory |\n",
"| ID ID Usage |\n",
"|=======================================================================================|\n",
"| 0 N/A N/A 16384 C /root/miniconda3/bin/python 14612MiB |\n",
"+---------------------------------------------------------------------------------------+\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {
"editable": true,
"id": "uNx5pyRlIrJh",
"outputId": "077e661e-d36c-469b-89b8-7ff7f73541ec",
"slideshow": {
"slide_type": ""
},
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
"
\n",
" [ 40/2076 00:39 < 34:52, 0.97 it/s, Epoch 0.06/3]\n",
"
\n",
" \n",
" \n",
" | Epoch | \n",
" Training Loss | \n",
" Validation Loss | \n",
"
\n",
" \n",
" \n",
" \n",
"
"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"ename": "KeyboardInterrupt",
"evalue": "",
"output_type": "error",
"traceback": [
"\u001B[0;31m---------------------------------------------------------------------------\u001B[0m",
"\u001B[0;31mKeyboardInterrupt\u001B[0m Traceback (most recent call last)",
"Cell \u001B[0;32mIn[38], line 1\u001B[0m\n\u001B[0;32m----> 1\u001B[0m \u001B[43mtrainer\u001B[49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mtrain\u001B[49m\u001B[43m(\u001B[49m\u001B[43m)\u001B[49m\n",
"File \u001B[0;32m/data/miniconda3/envs/jxzhang/lib/python3.11/site-packages/transformers/trainer.py:1537\u001B[0m, in \u001B[0;36mTrainer.train\u001B[0;34m(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)\u001B[0m\n\u001B[1;32m 1535\u001B[0m hf_hub_utils\u001B[38;5;241m.\u001B[39menable_progress_bars()\n\u001B[1;32m 1536\u001B[0m \u001B[38;5;28;01melse\u001B[39;00m:\n\u001B[0;32m-> 1537\u001B[0m \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[43minner_training_loop\u001B[49m\u001B[43m(\u001B[49m\n\u001B[1;32m 1538\u001B[0m \u001B[43m \u001B[49m\u001B[43margs\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43margs\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m 1539\u001B[0m \u001B[43m \u001B[49m\u001B[43mresume_from_checkpoint\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43mresume_from_checkpoint\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m 1540\u001B[0m \u001B[43m \u001B[49m\u001B[43mtrial\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43mtrial\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m 1541\u001B[0m \u001B[43m \u001B[49m\u001B[43mignore_keys_for_eval\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43mignore_keys_for_eval\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m 1542\u001B[0m \u001B[43m \u001B[49m\u001B[43m)\u001B[49m\n",
"File \u001B[0;32m/data/miniconda3/envs/jxzhang/lib/python3.11/site-packages/transformers/trainer.py:1859\u001B[0m, in \u001B[0;36mTrainer._inner_training_loop\u001B[0;34m(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)\u001B[0m\n\u001B[1;32m 1853\u001B[0m \u001B[38;5;28;01mwith\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39maccelerator\u001B[38;5;241m.\u001B[39maccumulate(model):\n\u001B[1;32m 1854\u001B[0m tr_loss_step \u001B[38;5;241m=\u001B[39m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39mtraining_step(model, inputs)\n\u001B[1;32m 1856\u001B[0m \u001B[38;5;28;01mif\u001B[39;00m (\n\u001B[1;32m 1857\u001B[0m args\u001B[38;5;241m.\u001B[39mlogging_nan_inf_filter\n\u001B[1;32m 1858\u001B[0m \u001B[38;5;129;01mand\u001B[39;00m \u001B[38;5;129;01mnot\u001B[39;00m is_torch_tpu_available()\n\u001B[0;32m-> 1859\u001B[0m \u001B[38;5;129;01mand\u001B[39;00m (torch\u001B[38;5;241m.\u001B[39misnan(tr_loss_step) \u001B[38;5;129;01mor\u001B[39;00m \u001B[43mtorch\u001B[49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43misinf\u001B[49m\u001B[43m(\u001B[49m\u001B[43mtr_loss_step\u001B[49m\u001B[43m)\u001B[49m)\n\u001B[1;32m 1860\u001B[0m ):\n\u001B[1;32m 1861\u001B[0m \u001B[38;5;66;03m# if loss is nan or inf simply add the average of previous logged losses\u001B[39;00m\n\u001B[1;32m 1862\u001B[0m tr_loss \u001B[38;5;241m+\u001B[39m\u001B[38;5;241m=\u001B[39m tr_loss \u001B[38;5;241m/\u001B[39m (\u001B[38;5;241m1\u001B[39m \u001B[38;5;241m+\u001B[39m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39mstate\u001B[38;5;241m.\u001B[39mglobal_step \u001B[38;5;241m-\u001B[39m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_globalstep_last_logged)\n\u001B[1;32m 1863\u001B[0m \u001B[38;5;28;01melse\u001B[39;00m:\n",
"\u001B[0;31mKeyboardInterrupt\u001B[0m: "
]
}
],
"source": [
"trainer.train()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 训练完成后，第一时间保存模型权重文件。"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [],
"source": [
"trained_model_path = f\"{model_dir}/{model_name}-finetuned-squad-trained\"\n",
"model_to_save = trainer.save_model(trained_model_path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 模型评估"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**评估模型输出需要一些额外的处理：将模型的预测映射回上下文的部分。**\n",
"\n",
"模型直接输出的是预测答案的`起始位置`和`结束位置`的**logits**"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"odict_keys(['loss', 'start_logits', 'end_logits'])"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import torch\n",
"\n",
"for batch in trainer.get_eval_dataloader():\n",
" break\n",
"batch = {k: v.to(trainer.args.device) for k, v in batch.items()}\n",
"with torch.no_grad():\n",
" output = trainer.model(**batch)\n",
"output.keys()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"模型的输出是一个类似字典的对象，其中包含损失（因为我们提供了标签），以及起始和结束logits。我们不需要损失来进行预测，让我们看一下logits："
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(torch.Size([32, 384]), torch.Size([32, 384]))"
]
},
"execution_count": 40,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"output.start_logits.shape, output.end_logits.shape"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(tensor([ 46, 57, 78, 43, 118, 107, 72, 35, 107, 34, 73, 41, 80, 91,\n",
" 156, 35, 83, 91, 80, 58, 77, 31, 42, 53, 41, 35, 42, 77,\n",
" 11, 44, 27, 133], device='cuda:0'),\n",
" tensor([ 47, 58, 81, 44, 118, 109, 75, 37, 109, 36, 76, 42, 83, 94,\n",
" 158, 35, 83, 94, 83, 60, 80, 74, 43, 54, 42, 35, 43, 80,\n",
" 13, 45, 28, 133], device='cuda:0'))"
]
},
"execution_count": 41,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"output.start_logits.argmax(dim=-1), output.end_logits.argmax(dim=-1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### 如何从模型输出的位置 logit 组合成答案\n",
"\n",
"我们有每个特征和每个标记的logit。在每个特征中为每个标记预测答案最明显的方法是，将起始logits的最大索引作为起始位置，将结束logits的最大索引作为结束位置。\n",
"\n",
"在许多情况下这种方式效果很好，但是如果此预测给出了不可能的结果该怎么办？比如：起始位置可能大于结束位置，或者指向问题中的文本片段而不是答案。在这种情况下，我们可能希望查看第二好的预测，看它是否给出了一个可能的答案，并选择它。\n",
"\n",
"选择第二好的答案并不像选择最佳答案那么容易：\n",
"- 它是起始logits中第二佳索引与结束logits中最佳索引吗？\n",
"- 还是起始logits中最佳索引与结束logits中第二佳索引？\n",
"- 如果第二好的答案也不可能，那么对于第三好的答案，情况会更加棘手。\n",
"\n",
"为了对答案进行分类，\n",
"1. 将使用通过添加起始和结束logits获得的分数\n",
"1. 设计一个名为`n_best_size`的超参数，限制不对所有可能的答案进行排序。\n",
"1. 我们将选择起始和结束logits中的最佳索引，并收集这些预测的所有答案。\n",
"1. 在检查每一个是否有效后，我们将按照其分数对它们进行排序，并保留最佳的答案。\n",
"\n",
"以下是我们如何在批次中的第一个特征上执行此操作的示例："
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [],
"source": [
"n_best_size = 20"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"start_logits = output.start_logits[0].cpu().numpy()\n",
"end_logits = output.end_logits[0].cpu().numpy()\n",
"\n",
"# 获取最佳的起始和结束位置的索引：\n",
"start_indexes = np.argsort(start_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
"end_indexes = np.argsort(end_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
"\n",
"valid_answers = []\n",
"\n",
"# 遍历起始位置和结束位置的索引组合\n",
"for start_index in start_indexes:\n",
" for end_index in end_indexes:\n",
" if start_index <= end_index: # 需要进一步测试以检查答案是否在上下文中\n",
" valid_answers.append(\n",
" {\n",
" \"score\": start_logits[start_index] + end_logits[end_index],\n",
" \"text\": \"\" # 我们需要找到一种方法来获取与上下文中答案对应的原始子字符串\n",
" }\n",
" )\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"然后，我们可以根据它们的得分对`valid_answers`进行排序，并仅保留最佳答案。唯一剩下的问题是如何检查给定的跨度是否在上下文中（而不是问题中），以及如何获取其中的文本。为此，我们需要向我们的验证特征添加两个内容：\n",
"\n",
"- 生成该特征的示例的ID（因为每个示例可以生成多个特征，如前所示）；\n",
"- 偏移映射，它将为我们提供从标记索引到上下文中字符位置的映射。\n",
"\n",
"这就是为什么我们将使用以下函数稍微不同于`prepare_train_features`来重新处理验证集："
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {},
"outputs": [],
"source": [
"def prepare_validation_features(examples):\n",
" # 一些问题的左侧有很多空白，这些空白并不有用且会导致上下文截断失败（分词后的问题会占用很多空间）。\n",
" # 因此我们移除这些左侧空白\n",
" examples[\"question\"] = [q.lstrip() for q in examples[\"question\"]]\n",
"\n",
" # 使用截断和可能的填充对我们的示例进行分词，但使用步长保留溢出的令牌。这导致一个长上下文的示例可能产生\n",
" # 几个特征，每个特征的上下文都会稍微与前一个特征的上下文重叠。\n",
" tokenized_examples = tokenizer(\n",
" examples[\"question\" if pad_on_right else \"context\"],\n",
" examples[\"context\" if pad_on_right else \"question\"],\n",
" truncation=\"only_second\" if pad_on_right else \"only_first\",\n",
" max_length=max_length,\n",
" stride=doc_stride,\n",
" return_overflowing_tokens=True,\n",
" return_offsets_mapping=True,\n",
" padding=\"max_length\",\n",
" )\n",
"\n",
" # 由于一个示例在上下文很长时可能会产生几个特征，我们需要一个从特征映射到其对应示例的映射。这个键就是为了这个目的。\n",
" sample_mapping = tokenized_examples.pop(\"overflow_to_sample_mapping\")\n",
"\n",
" # 我们保留产生这个特征的示例ID，并且会存储偏移映射。\n",
" tokenized_examples[\"example_id\"] = []\n",
"\n",
" for i in range(len(tokenized_examples[\"input_ids\"])):\n",
" # 获取与该示例对应的序列（以了解哪些是上下文，哪些是问题）。\n",
" sequence_ids = tokenized_examples.sequence_ids(i)\n",
" context_index = 1 if pad_on_right else 0\n",
"\n",
" # 一个示例可以产生几个文本段，这里是包含该文本段的示例的索引。\n",
" sample_index = sample_mapping[i]\n",
" tokenized_examples[\"example_id\"].append(examples[\"id\"][sample_index])\n",
"\n",
" # 将不属于上下文的偏移映射设置为None，以便容易确定一个令牌位置是否属于上下文。\n",
" tokenized_examples[\"offset_mapping\"][i] = [\n",
" (o if sequence_ids[k] == context_index else None)\n",
" for k, o in enumerate(tokenized_examples[\"offset_mapping\"][i])\n",
" ]\n",
"\n",
" return tokenized_examples\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"将`prepare_validation_features`应用到整个验证集："
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Map: 100%|██████████| 10570/10570 [00:03<00:00, 2680.59 examples/s]\n"
]
}
],
"source": [
"validation_features = datasets[\"validation\"].map(\n",
" prepare_validation_features,\n",
" batched=True,\n",
" remove_columns=datasets[\"validation\"].column_names\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true,
"is_executing": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we can grab the predictions for all features by using the `Trainer.predict` method:"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"raw_predictions = trainer.predict(validation_features)"
]
},
{
"cell_type": "code",
"execution_count": 60,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'input_ids': Sequence(feature=Value(dtype='int32', id=None), length=-1, id=None),\n",
" 'attention_mask': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None),\n",
" 'offset_mapping': Sequence(feature=Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None), length=-1, id=None),\n",
" 'example_id': Value(dtype='string', id=None)}"
]
},
"execution_count": 60,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"validation_features.features"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`Trainer`会隐藏模型不使用的列（在这里是`example_id`和`offset_mapping`，我们需要它们进行后处理），所以我们需要将它们重新设置回来："
]
},
{
"cell_type": "code",
"execution_count": 61,
"metadata": {},
"outputs": [],
"source": [
"validation_features.set_format(\n",
" type=validation_features.format[\"type\"], \n",
" columns=list(validation_features.features.keys())\n",
")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"现在，我们可以改进之前的测试：\n",
"\n",
"由于在偏移映射中，当它对应于问题的一部分时，我们将其设置为None，因此可以轻松检查答案是否完全在上下文中。我们还可以从考虑中排除非常长的答案（可以调整的超参数）。\n",
"\n",
"展开说下具体实现：\n",
"- 首先从模型输出中获取起始和结束的逻辑值（logits），这些值表明答案在文本中可能开始和结束的位置。\n",
"- 然后，它使用偏移映射（offset_mapping）来找到这些逻辑值在原始文本中的具体位置。\n",
"- 接下来，代码遍历可能的开始和结束索引组合，排除那些不在上下文范围内或长度不合适的答案。\n",
"- 对于有效的答案，它计算出一个分数（基于开始和结束逻辑值的和），并将答案及其分数存储起来。\n",
"- 最后，它根据分数对答案进行排序，并返回得分最高的几个答案。"
]
},
{
"cell_type": "code",
"execution_count": 62,
"metadata": {},
"outputs": [],
"source": [
"max_answer_length = 30"
]
},
{
"cell_type": "code",
"execution_count": 63,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'score': 14.460105, 'text': 'Denver Broncos'},\n",
" {'score': 12.578646,\n",
" 'text': 'Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers'},\n",
" {'score': 11.292594, 'text': 'Carolina Panthers'},\n",
" {'score': 10.702299,\n",
" 'text': 'American Football Conference (AFC) champion Denver Broncos'},\n",
" {'score': 9.912531, 'text': 'Broncos'},\n",
" {'score': 9.225191,\n",
" 'text': 'The American Football Conference (AFC) champion Denver Broncos'},\n",
" {'score': 8.820841,\n",
" 'text': 'American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers'},\n",
" {'score': 8.585277, 'text': 'Denver'},\n",
" {'score': 8.031073,\n",
" 'text': 'Broncos defeated the National Football Conference (NFC) champion Carolina Panthers'},\n",
" {'score': 8.028643,\n",
" 'text': 'Denver Broncos defeated the National Football Conference'},\n",
" {'score': 7.7296076,\n",
" 'text': 'Denver Broncos defeated the National Football Conference (NFC)'},\n",
" {'score': 7.343733,\n",
" 'text': 'The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers'},\n",
" {'score': 7.267834,\n",
" 'text': 'Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10'},\n",
" {'score': 7.0992393, 'text': 'champion Denver Broncos'},\n",
" {'score': 6.209029,\n",
" 'text': 'National Football Conference (NFC) champion Carolina Panthers'},\n",
" {'score': 6.190158,\n",
" 'text': 'Denver Broncos defeated the National Football Conference (NFC'},\n",
" {'score': 6.1445813,\n",
" 'text': 'Denver Broncos defeated the National Football Conference (NFC) champion Carolina'},\n",
" {'score': 6.1090403,\n",
" 'text': 'Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super Bowl title.'},\n",
" {'score': 6.0088606, 'text': 'Panthers'},\n",
" {'score': 5.981782, 'text': 'Carolina Panthers 24–10'}]"
]
},
"execution_count": 63,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"start_logits = output.start_logits[0].cpu().numpy()\n",
"end_logits = output.end_logits[0].cpu().numpy()\n",
"offset_mapping = validation_features[0][\"offset_mapping\"]\n",
"\n",
"# 第一个特征来自第一个示例。对于更一般的情况，我们需要将example_id匹配到一个示例索引\n",
"context = datasets[\"validation\"][0][\"context\"]\n",
"\n",
"# 收集最佳开始/结束逻辑的索引：\n",
"start_indexes = np.argsort(start_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
"end_indexes = np.argsort(end_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
"valid_answers = []\n",
"for start_index in start_indexes:\n",
" for end_index in end_indexes:\n",
" # 不考虑超出范围的答案，原因是索引超出范围或对应于输入ID的部分不在上下文中。\n",
" if (\n",
" start_index >= len(offset_mapping)\n",
" or end_index >= len(offset_mapping)\n",
" or offset_mapping[start_index] is None\n",
" or offset_mapping[end_index] is None\n",
" ):\n",
" continue\n",
" # 不考虑长度小于0或大于max_answer_length的答案。\n",
" if end_index < start_index or end_index - start_index + 1 > max_answer_length:\n",
" continue\n",
" if start_index <= end_index: # 我们需要细化这个测试，以检查答案是否在上下文中\n",
" start_char = offset_mapping[start_index][0]\n",
" end_char = offset_mapping[end_index][1]\n",
" valid_answers.append(\n",
" {\n",
" \"score\": start_logits[start_index] + end_logits[end_index],\n",
" \"text\": context[start_char: end_char]\n",
" }\n",
" )\n",
"\n",
"valid_answers = sorted(valid_answers, key=lambda x: x[\"score\"], reverse=True)[:n_best_size]\n",
"valid_answers\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"打印比较模型输出和标准答案（Ground-truth）是否一致:"
]
},
{
"cell_type": "code",
"execution_count": 64,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'text': ['Denver Broncos', 'Denver Broncos', 'Denver Broncos'],\n",
" 'answer_start': [177, 177, 177]}"
]
},
"execution_count": 64,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"datasets[\"validation\"][0][\"answers\"]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**模型最高概率的输出与标准答案一致**\n",
"\n",
"正如上面的代码所示，这在第一个特征上很容易，因为我们知道它来自第一个示例。\n",
"\n",
"对于其他特征，我们需要建立一个示例与其对应特征的映射关系。\n",
"\n",
"此外，由于一个示例可以生成多个特征，我们需要将由给定示例生成的所有特征中的所有答案汇集在一起，然后选择最佳答案。\n",
"\n",
"下面的代码构建了一个示例索引到其对应特征索引的映射关系："
]
},
{
"cell_type": "code",
"execution_count": 73,
"metadata": {},
"outputs": [],
"source": [
"import collections\n",
"\n",
"examples = datasets[\"validation\"]\n",
"features = validation_features\n",
"\n",
"example_id_to_index = {k: i for i, k in enumerate(examples[\"id\"])}\n",
"features_per_example = collections.defaultdict(list)\n",
"for i, feature in enumerate(features):\n",
" features_per_example[example_id_to_index[feature[\"example_id\"]]].append(i)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"当`squad_v2 = True`时，有一定概率出现不可能的答案（impossible answer)。\n",
"\n",
"上面的代码仅保留在上下文中的答案，我们还需要获取不可能答案的分数（其起始和结束索引对应于CLS标记的索引）。\n",
"\n",
"当一个示例生成多个特征时，我们必须在所有特征中的不可能答案都预测出现不可能答案时（因为一个特征可能之所以能够预测出不可能答案，是因为答案不在它可以访问的上下文部分），这就是为什么一个示例中不可能答案的分数是该示例生成的每个特征中的不可能答案的分数的最小值。"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"outputs": [],
"source": [
"from tqdm.auto import tqdm\n",
"\n",
"def postprocess_qa_predictions(examples, features, raw_predictions, n_best_size = 20, max_answer_length = 30):\n",
" all_start_logits, all_end_logits = raw_predictions\n",
" # 构建一个从示例到其对应特征的映射。\n",
" example_id_to_index = {k: i for i, k in enumerate(examples[\"id\"])}\n",
" features_per_example = collections.defaultdict(list)\n",
" for i, feature in enumerate(features):\n",
" features_per_example[example_id_to_index[feature[\"example_id\"]]].append(i)\n",
"\n",
" # 我们需要填充的字典。\n",
" predictions = collections.OrderedDict()\n",
"\n",
" # 日志记录。\n",
" print(f\"正在后处理 {len(examples)} 个示例的预测，这些预测分散在 {len(features)} 个特征中。\")\n",
"\n",
" # 遍历所有示例！\n",
" for example_index, example in enumerate(tqdm(examples)):\n",
" # 这些是与当前示例关联的特征的索引。\n",
" feature_indices = features_per_example[example_index]\n",
"\n",
" min_null_score = None # 仅在squad_v2为True时使用。\n",
" valid_answers = []\n",
" \n",
" context = example[\"context\"]\n",
" # 遍历与当前示例关联的所有特征。\n",
" for feature_index in feature_indices:\n",
" # 我们获取模型对这个特征的预测。\n",
" start_logits = all_start_logits[feature_index]\n",
" end_logits = all_end_logits[feature_index]\n",
" # 这将允许我们将logits中的某些位置映射到原始上下文中的文本跨度。\n",
" offset_mapping = features[feature_index][\"offset_mapping\"]\n",
"\n",
" # 更新最小空预测。\n",
" cls_index = features[feature_index][\"input_ids\"].index(tokenizer.cls_token_id)\n",
" feature_null_score = start_logits[cls_index] + end_logits[cls_index]\n",
" if min_null_score is None or min_null_score < feature_null_score:\n",
" min_null_score = feature_null_score\n",
"\n",
" # 浏览所有的最佳开始和结束logits，为 `n_best_size` 个最佳选择。\n",
" start_indexes = np.argsort(start_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
" end_indexes = np.argsort(end_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
" for start_index in start_indexes:\n",
" for end_index in end_indexes:\n",
" # 不考虑超出范围的答案，原因是索引超出范围或对应于输入ID的部分不在上下文中。\n",
" if (\n",
" start_index >= len(offset_mapping)\n",
" or end_index >= len(offset_mapping)\n",
" or offset_mapping[start_index] is None\n",
" or offset_mapping[end_index] is None\n",
" ):\n",
" continue\n",
" # 不考虑长度小于0或大于max_answer_length的答案。\n",
" if end_index < start_index or end_index - start_index + 1 > max_answer_length:\n",
" continue\n",
"\n",
" start_char = offset_mapping[start_index][0]\n",
" end_char = offset_mapping[end_index][1]\n",
" valid_answers.append(\n",
" {\n",
" \"score\": start_logits[start_index] + end_logits[end_index],\n",
" \"text\": context[start_char: end_char]\n",
" }\n",
" )\n",
" \n",
" if len(valid_answers) > 0:\n",
" best_answer = sorted(valid_answers, key=lambda x: x[\"score\"], reverse=True)[0]\n",
" else:\n",
" # 在极少数情况下我们没有一个非空预测，我们创建一个假预测以避免失败。\n",
" best_answer = {\"text\": \"\", \"score\": 0.0}\n",
" \n",
" # 选择我们的最终答案：最佳答案或空答案（仅适用于squad_v2）\n",
" if not squad_v2:\n",
" predictions[example[\"id\"]] = best_answer[\"text\"]\n",
" else:\n",
" answer = best_answer[\"text\"] if best_answer[\"score\"] > min_null_score else \"\"\n",
" predictions[example[\"id\"]] = answer\n",
"\n",
" return predictions\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"在原始结果上应用后处理问答结果："
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"正在后处理 10570 个示例的预测，这些预测分散在 10784 个特征中。\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████| 10570/10570 [00:39<00:00, 266.00it/s]\n"
]
}
],
"source": [
"final_predictions = postprocess_qa_predictions(\n",
" datasets[\"validation\"], validation_features, raw_predictions.predictions\n",
")\n"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"10784"
]
},
"execution_count": 34,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# final_predictions\n",
"len(raw_predictions.predictions[0])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"使用 `datasets.load_metric` 中加载 `SQuAD v2` 的评估指标"
]
},
{
"cell_type": "code",
"execution_count": 94,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/tmp/ipykernel_384469/2330875496.py:3: FutureWarning: load_metric is deprecated and will be removed in the next major version of datasets. Use 'evaluate.load' instead, from the new library 🤗 Evaluate: https://huggingface.co/docs/evaluate\n",
" metric = load_metric(\"squad_v2\" if squad_v2 else \"squad\")\n",
"/usr/local/lib/python3.9/dist-packages/datasets/load.py:752: FutureWarning: The repository for squad contains custom code which must be executed to correctly load the metric. You can inspect the repository content at https://raw.githubusercontent.com/huggingface/datasets/2.16.1/metrics/squad/squad.py\n",
"You can avoid this message in future by passing the argument `trust_remote_code=True`.\n",
"Passing `trust_remote_code=True` will be mandatory to load this metric from the next major release of `datasets`.\n",
" warnings.warn(\n",
"Downloading builder script: 4.50kB [00:00, 11.7MB/s] \n",
"Downloading extra modules: 3.30kB [00:00, 9.04MB/s] \n"
]
}
],
"source": [
"from datasets import load_metric\n",
"\n",
"metric = load_metric(\"squad_v2\" if squad_v2 else \"squad\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"接下来，我们可以调用上面定义的函数进行评估。\n",
"\n",
"只需稍微调整一下预测和标签的格式，因为它期望的是一系列字典而不是一个大字典。\n",
"\n",
"在使用`squad_v2`数据集时，我们还需要设置`no_answer_probability`参数（我们在这里将其设置为0.0，因为如果我们选择了答案，我们已经将答案设置为空）。"
]
},
{
"cell_type": "code",
"execution_count": 95,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'exact_match': 73.91674550614948, 'f1': 82.85029550924814}"
]
},
"execution_count": 95,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"if squad_v2:\n",
" formatted_predictions = [{\"id\": k, \"prediction_text\": v, \"no_answer_probability\": 0.0} \n",
" for k, v in final_predictions.items()]\n",
"else:\n",
" formatted_predictions = [{\"id\": k, \"prediction_text\": v} for k, v in final_predictions.items()]\n",
"references = [{\"id\": ex[\"id\"], \"answers\": ex[\"answers\"]} for ex in datasets[\"validation\"]]\n",
"metric.compute(predictions=formatted_predictions, references=references)\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Homework：加载本地保存的模型，进行评估和再训练更高的 F1 Score"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"1"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"trained_model_path = f\"models/distilbert-base-uncased-finetuned-squad-trained\"\n",
"\n",
"# 根据你使用的模型和GPU资源情况，调整以下关键参数\n",
"squad_v2 = False\n",
"model_checkpoint = \"distilbert-base-uncased\"\n",
"batch_size = 8\n",
"\n",
"import os\n",
"os.environ[\"CUDA_DEVICE_ORDER\"]=\"PCI_BUS_ID\" # see issue #152\n",
"os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"7\"\n",
"\n",
"import torch\n",
"torch.cuda.device_count()"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"# import boto3\n",
"# s3_resource = boto3.resource(\"s3\")\n",
"# s3_bucket = s3_resource.Bucket(\"wish-test-temp\")\n",
"# s3_bucket.upload_file(\"models/squad_trained.zip\", \"jxzhang/squad_trained.zip\")\n",
"# s3_bucket.upload_file(\"models/yelp.zip\", \"jxzhang/yelp.zip\")\n",
"\n",
"# s3_bucket.download_file(\"jxzhang/squad_trained.zip\", \"models/squad_trained.zip\")\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/data/miniconda3/envs/jxzhang/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
" from .autonotebook import tqdm as notebook_tqdm\n"
]
}
],
"source": [
"from transformers import AutoTokenizer\n",
" \n",
"tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)\n",
"pad_on_right = tokenizer.padding_side == \"right\"\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"# The maximum length of a feature (question and context)\n",
"max_length = 384 \n",
"# The authorized overlap between two part of the context when splitting it is needed.\n",
"doc_stride = 128 \n",
"\n",
"def prepare_train_features(examples):\n",
" # 一些问题的左侧可能有很多空白字符，这对我们没有用，而且会导致上下文的截断失败\n",
" # （标记化的问题将占用大量空间）。因此，我们删除左侧的空白字符。\n",
" examples[\"question\"] = [q.lstrip() for q in examples[\"question\"]]\n",
"\n",
" # 使用截断和填充对我们的示例进行标记化，但保留溢出部分，使用步幅（stride）。\n",
" # 当上下文很长时，这会导致一个示例可能提供多个特征，其中每个特征的上下文都与前一个特征的上下文有一些重叠。\n",
" tokenized_examples = tokenizer(\n",
" examples[\"question\" if pad_on_right else \"context\"],\n",
" examples[\"context\" if pad_on_right else \"question\"],\n",
" truncation=\"only_second\" if pad_on_right else \"only_first\",\n",
" max_length=max_length,\n",
" stride=doc_stride,\n",
" return_overflowing_tokens=True,\n",
" return_offsets_mapping=True,\n",
" padding=\"max_length\",\n",
" )\n",
"\n",
" # 由于一个示例可能给我们提供多个特征（如果它具有很长的上下文），我们需要一个从特征到其对应示例的映射。这个键就提供了这个映射关系。\n",
" sample_mapping = tokenized_examples.pop(\"overflow_to_sample_mapping\")\n",
" # 偏移映射将为我们提供从令牌到原始上下文中的字符位置的映射。这将帮助我们计算开始位置和结束位置。\n",
" offset_mapping = tokenized_examples.pop(\"offset_mapping\")\n",
"\n",
" # 让我们为这些示例进行标记！\n",
" tokenized_examples[\"start_positions\"] = []\n",
" tokenized_examples[\"end_positions\"] = []\n",
"\n",
" for i, offsets in enumerate(offset_mapping):\n",
" # 我们将使用CLS令牌的索引来标记不可能的答案。\n",
" input_ids = tokenized_examples[\"input_ids\"][i]\n",
" cls_index = input_ids.index(tokenizer.cls_token_id)\n",
"\n",
" # 获取与该示例对应的序列（以了解上下文和问题是什么）。\n",
" sequence_ids = tokenized_examples.sequence_ids(i)\n",
"\n",
" # 一个示例可以提供多个跨度，这是包含此文本跨度的示例的索引。\n",
" sample_index = sample_mapping[i]\n",
" answers = examples[\"answers\"][sample_index]\n",
" # 如果没有给出答案，则将cls_index设置为答案。\n",
" if len(answers[\"text\"]) == 0:\n",
" tokenized_examples[\"start_positions\"].append(cls_index)\n",
" tokenized_examples[\"end_positions\"].append(cls_index)\n",
" else:\n",
" # 答案在文本中的开始和结束字符索引。\n",
" start_char = answers[\"answer_start\"][0]\n",
" end_char = start_char + len(answers[\"text\"][0])\n",
"\n",
" # 当前跨度在文本中的开始令牌索引。\n",
" token_start_index = 0\n",
" while sequence_ids[token_start_index] != (1 if pad_on_right else 0):\n",
" token_start_index += 1\n",
"\n",
" # 当前跨度在文本中的结束令牌索引。\n",
" token_end_index = len(input_ids) - 1\n",
" while sequence_ids[token_end_index] != (1 if pad_on_right else 0):\n",
" token_end_index -= 1\n",
"\n",
" # 检测答案是否超出跨度（在这种情况下，该特征的标签将使用CLS索引）。\n",
" if not (offsets[token_start_index][0] <= start_char and offsets[token_end_index][1] >= end_char):\n",
" tokenized_examples[\"start_positions\"].append(cls_index)\n",
" tokenized_examples[\"end_positions\"].append(cls_index)\n",
" else:\n",
" # 否则，将token_start_index和token_end_index移到答案的两端。\n",
" # 注意：如果答案是最后一个单词（边缘情况），我们可以在最后一个偏移之后继续。\n",
" while token_start_index < len(offsets) and offsets[token_start_index][0] <= start_char:\n",
" token_start_index += 1\n",
" tokenized_examples[\"start_positions\"].append(token_start_index - 1)\n",
" while offsets[token_end_index][1] >= end_char:\n",
" token_end_index -= 1\n",
" tokenized_examples[\"end_positions\"].append(token_end_index + 1)\n",
" return tokenized_examples"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"from datasets import load_dataset\n",
"datasets = load_dataset(\"squad_v2\" if squad_v2 else \"squad\")"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Map: 100%|██████████| 10570/10570 [00:04<00:00, 2436.88 examples/s]\n"
]
}
],
"source": [
"tokenized_datasets = datasets.map(\n",
" prepare_train_features,\n",
" batched=True,\n",
" remove_columns=datasets[\"train\"].column_names,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"from transformers import AutoModelForQuestionAnswering, TrainingArguments, Trainer\n",
"from transformers import default_data_collator\n",
"\n",
"data_collator = default_data_collator\n",
"trained_model = AutoModelForQuestionAnswering.from_pretrained(trained_model_path)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"batch_size=64\n",
"model_dir = \"models\"\n",
"model_name = model_checkpoint.split(\"/\")[-1]\n",
"\n",
"args = TrainingArguments(\n",
" f\"{model_dir}/{model_name}-finetuned-squad\",\n",
" evaluation_strategy = \"epoch\",\n",
" learning_rate=2e-5,\n",
" per_device_train_batch_size=batch_size,\n",
" per_device_eval_batch_size=batch_size,\n",
" num_train_epochs=3,\n",
" weight_decay=0.01,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"trained_trainer = Trainer(\n",
" trained_model,\n",
" args,\n",
" train_dataset=tokenized_datasets[\"train\"],\n",
" eval_dataset=tokenized_datasets[\"validation\"],\n",
" data_collator=data_collator,\n",
" tokenizer=tokenizer,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" \n",
"
\n",
" [4152/4152 37:00, Epoch 3/3]\n",
"
\n",
" \n",
" \n",
" | Epoch | \n",
" Training Loss | \n",
" Validation Loss | \n",
"
\n",
" \n",
" \n",
" \n",
" | 1 | \n",
" 1.012300 | \n",
" 1.163603 | \n",
"
\n",
" \n",
" | 2 | \n",
" 0.839100 | \n",
" 1.161118 | \n",
"
\n",
" \n",
" | 3 | \n",
" 0.739800 | \n",
" 1.187968 | \n",
"
\n",
" \n",
"
"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"TrainOutput(global_step=4152, training_loss=0.8671502350382723, metrics={'train_runtime': 2223.6661, 'train_samples_per_second': 119.43, 'train_steps_per_second': 1.867, 'total_flos': 2.602335381127373e+16, 'train_loss': 0.8671502350382723, 'epoch': 3.0})"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"trained_trainer.train(True)"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"trained_model_path = f\"{model_dir}/{model_name}-finetuned-squad-trained-2\"\n",
"model_to_save = trained_trainer.save_model(trained_model_path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 模型评估"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
"n_best_size = 20\n",
"\n",
"def prepare_validation_features(examples):\n",
" # 一些问题的左侧有很多空白，这些空白并不有用且会导致上下文截断失败（分词后的问题会占用很多空间）。\n",
" # 因此我们移除这些左侧空白\n",
" examples[\"question\"] = [q.lstrip() for q in examples[\"question\"]]\n",
"\n",
" # 使用截断和可能的填充对我们的示例进行分词，但使用步长保留溢出的令牌。这导致一个长上下文的示例可能产生\n",
" # 几个特征，每个特征的上下文都会稍微与前一个特征的上下文重叠。\n",
" tokenized_examples = tokenizer(\n",
" examples[\"question\" if pad_on_right else \"context\"],\n",
" examples[\"context\" if pad_on_right else \"question\"],\n",
" truncation=\"only_second\" if pad_on_right else \"only_first\",\n",
" max_length=max_length,\n",
" stride=doc_stride,\n",
" return_overflowing_tokens=True,\n",
" return_offsets_mapping=True,\n",
" padding=\"max_length\",\n",
" )\n",
"\n",
" # 由于一个示例在上下文很长时可能会产生几个特征，我们需要一个从特征映射到其对应示例的映射。这个键就是为了这个目的。\n",
" sample_mapping = tokenized_examples.pop(\"overflow_to_sample_mapping\")\n",
"\n",
" # 我们保留产生这个特征的示例ID，并且会存储偏移映射。\n",
" tokenized_examples[\"example_id\"] = []\n",
"\n",
" for i in range(len(tokenized_examples[\"input_ids\"])):\n",
" # 获取与该示例对应的序列（以了解哪些是上下文，哪些是问题）。\n",
" sequence_ids = tokenized_examples.sequence_ids(i)\n",
" context_index = 1 if pad_on_right else 0\n",
"\n",
" # 一个示例可以产生几个文本段，这里是包含该文本段的示例的索引。\n",
" sample_index = sample_mapping[i]\n",
" tokenized_examples[\"example_id\"].append(examples[\"id\"][sample_index])\n",
"\n",
" # 将不属于上下文的偏移映射设置为None，以便容易确定一个令牌位置是否属于上下文。\n",
" tokenized_examples[\"offset_mapping\"][i] = [\n",
" (o if sequence_ids[k] == context_index else None)\n",
" for k, o in enumerate(tokenized_examples[\"offset_mapping\"][i])\n",
" ]\n",
"\n",
" return tokenized_examples"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Map: 100%|██████████| 10570/10570 [00:07<00:00, 1392.07 examples/s]\n"
]
}
],
"source": [
"validation_features = datasets[\"validation\"].map(\n",
" prepare_validation_features,\n",
" batched=True,\n",
" remove_columns=datasets[\"validation\"].column_names\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"raw_predictions = trained_trainer.predict(validation_features)"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [],
"source": [
"validation_features.set_format(\n",
" type=validation_features.format[\"type\"], \n",
" columns=list(validation_features.features.keys())\n",
")\n"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [],
"source": [
"max_answer_length = 30"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [],
"source": [
"from tqdm.auto import tqdm\n",
"import collections\n",
"import numpy as np\n",
"def postprocess_qa_predictions(examples, features, raw_predictions, n_best_size = 20, max_answer_length = 30):\n",
" all_start_logits, all_end_logits = raw_predictions\n",
" # 构建一个从示例到其对应特征的映射。\n",
" example_id_to_index = {k: i for i, k in enumerate(examples[\"id\"])}\n",
" features_per_example = collections.defaultdict(list)\n",
" for i, feature in enumerate(features):\n",
" features_per_example[example_id_to_index[feature[\"example_id\"]]].append(i)\n",
"\n",
" # 我们需要填充的字典。\n",
" predictions = collections.OrderedDict()\n",
"\n",
" # 日志记录。\n",
" print(f\"正在后处理 {len(examples)} 个示例的预测，这些预测分散在 {len(features)} 个特征中。\")\n",
"\n",
" # 遍历所有示例！\n",
" for example_index, example in enumerate(tqdm(examples)):\n",
" # 这些是与当前示例关联的特征的索引。\n",
" feature_indices = features_per_example[example_index]\n",
"\n",
" min_null_score = None # 仅在squad_v2为True时使用。\n",
" valid_answers = []\n",
" \n",
" context = example[\"context\"]\n",
" # 遍历与当前示例关联的所有特征。\n",
" for feature_index in feature_indices:\n",
" # 我们获取模型对这个特征的预测。\n",
" start_logits = all_start_logits[feature_index]\n",
" end_logits = all_end_logits[feature_index]\n",
" # 这将允许我们将logits中的某些位置映射到原始上下文中的文本跨度。\n",
" offset_mapping = features[feature_index][\"offset_mapping\"]\n",
"\n",
" # 更新最小空预测。\n",
" cls_index = features[feature_index][\"input_ids\"].index(tokenizer.cls_token_id)\n",
" feature_null_score = start_logits[cls_index] + end_logits[cls_index]\n",
" if min_null_score is None or min_null_score < feature_null_score:\n",
" min_null_score = feature_null_score\n",
"\n",
" # 浏览所有的最佳开始和结束logits，为 `n_best_size` 个最佳选择。\n",
" start_indexes = np.argsort(start_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
" end_indexes = np.argsort(end_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
" for start_index in start_indexes:\n",
" for end_index in end_indexes:\n",
" # 不考虑超出范围的答案，原因是索引超出范围或对应于输入ID的部分不在上下文中。\n",
" if (\n",
" start_index >= len(offset_mapping)\n",
" or end_index >= len(offset_mapping)\n",
" or offset_mapping[start_index] is None\n",
" or offset_mapping[end_index] is None\n",
" ):\n",
" continue\n",
" # 不考虑长度小于0或大于max_answer_length的答案。\n",
" if end_index < start_index or end_index - start_index + 1 > max_answer_length:\n",
" continue\n",
"\n",
" start_char = offset_mapping[start_index][0]\n",
" end_char = offset_mapping[end_index][1]\n",
" valid_answers.append(\n",
" {\n",
" \"score\": start_logits[start_index] + end_logits[end_index],\n",
" \"text\": context[start_char: end_char]\n",
" }\n",
" )\n",
" \n",
" if len(valid_answers) > 0:\n",
" best_answer = sorted(valid_answers, key=lambda x: x[\"score\"], reverse=True)[0]\n",
" else:\n",
" # 在极少数情况下我们没有一个非空预测，我们创建一个假预测以避免失败。\n",
" best_answer = {\"text\": \"\", \"score\": 0.0}\n",
" \n",
" # 选择我们的最终答案：最佳答案或空答案（仅适用于squad_v2）\n",
" if not squad_v2:\n",
" predictions[example[\"id\"]] = best_answer[\"text\"]\n",
" else:\n",
" answer = best_answer[\"text\"] if best_answer[\"score\"] > min_null_score else \"\"\n",
" predictions[example[\"id\"]] = answer\n",
"\n",
" return predictions\n"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"正在后处理 10570 个示例的预测，这些预测分散在 10784 个特征中。\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████| 10570/10570 [00:39<00:00, 266.86it/s]\n"
]
}
],
"source": [
"final_predictions = postprocess_qa_predictions(\n",
" datasets[\"validation\"], validation_features, raw_predictions.predictions\n",
")\n"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/tmp/ipykernel_479474/2330875496.py:3: FutureWarning: load_metric is deprecated and will be removed in the next major version of datasets. Use 'evaluate.load' instead, from the new library 🤗 Evaluate: https://huggingface.co/docs/evaluate\n",
" metric = load_metric(\"squad_v2\" if squad_v2 else \"squad\")\n",
"/data/miniconda3/envs/jxzhang/lib/python3.11/site-packages/datasets/load.py:752: FutureWarning: The repository for squad contains custom code which must be executed to correctly load the metric. You can inspect the repository content at https://raw.githubusercontent.com/huggingface/datasets/2.16.1/metrics/squad/squad.py\n",
"You can avoid this message in future by passing the argument `trust_remote_code=True`.\n",
"Passing `trust_remote_code=True` will be mandatory to load this metric from the next major release of `datasets`.\n",
" warnings.warn(\n",
"Downloading builder script: 4.50kB [00:00, 8.30MB/s] \n",
"Downloading extra modules: 3.30kB [00:00, 6.72MB/s] \n"
]
}
],
"source": [
"from datasets import load_metric\n",
"\n",
"metric = load_metric(\"squad_v2\" if squad_v2 else \"squad\")"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'exact_match': 76.13055818353831, 'f1': 84.69880055947921}"
]
},
"execution_count": 37,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"if squad_v2:\n",
" formatted_predictions = [{\"id\": k, \"prediction_text\": v, \"no_answer_probability\": 0.0} \n",
" for k, v in final_predictions.items()]\n",
"else:\n",
" formatted_predictions = [{\"id\": k, \"prediction_text\": v} for k, v in final_predictions.items()]\n",
"references = [{\"id\": ex[\"id\"], \"answers\": ex[\"answers\"]} for ex in datasets[\"validation\"]]\n",
"metric.compute(predictions=formatted_predictions, references=references)\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"trained_model_path = f\"models/distilbert-base-uncased-finetuned-squad-trained-2\"\n",
"from transformers import AutoModelForQuestionAnswering, TrainingArguments, Trainer\n",
"trained_model = AutoModelForQuestionAnswering.from_pretrained(trained_model_path)"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"CommitInfo(commit_url='https://huggingface.co/yqzhangjx/distilbert-base-uncased-finetuned-for-squad/commit/70ad585c1fe33692188d8a98c43528bf88a15728', commit_message='Upload DistilBertForQuestionAnswering', commit_description='', oid='70ad585c1fe33692188d8a98c43528bf88a15728', pr_url=None, pr_revision=None, pr_num=None)"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"trained_model.push_to_hub(\"yqzhangjx/distilbert-base-uncased-finetuned-for-squad\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"colab": {
"name": "Question Answering on SQUAD",
"provenance": []
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.7"
}
},
"nbformat": 4,
"nbformat_minor": 4
}