<jupyter_start><jupyter_text>作业一：语言模型 Part1 * 本次作业通过PyTorch搭建LSTM语言模型* 数据集采用ROC故事语料库，为减少单词量已将大部分名字转换为Bob和Sue* 数据集包含以下内容：文件名|说明:-:|:-:bobsue.lm.train.txt | 语言模型训练数据bobsue.lm.dev.txt | 语言模型验证数据bobsue.lm.test.txt | 语言模型测试数据bobsue.voc.txt | 词汇表数据 **首先导入这次作业需要的包，并设置随机种子**<jupyter_code>import random
from collections import defaultdict, Counter
from pathlib import Path

import numpy as np
import torch
import torch.nn as nn

def set_random_seed(seed):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

set_random_seed(2020)<jupyter_output><empty_output><jupyter_text>**设定计算设备与数据集路径**<jupyter_code>device = torch.device('cuda' if torch.cuda.is_available else 'cpu')
data_path = Path('/media/bnu/data/nlp-practice/language-model')

print('PyTorch Version:', torch.__version__)
print('-' * 60)
if torch.cuda.is_available():
    print('CUDA Device Count:', torch.cuda.device_count())
    print('CUDA Device Name:')
    for i in range(torch.cuda.device_count()):
        print('\t', torch.cuda.get_device_name(i))
    print('CUDA Current Device Index:', torch.cuda.current_device())
    print('-' * 60)
print('Data Path:', data_path)<jupyter_output>PyTorch Version: 1.4.0
------------------------------------------------------------
CUDA Device Count: 2
CUDA Device Name:
	 GeForce RTX 2080 Ti
	 GeForce RTX 2080 Ti
CUDA Current Device Index: 0
------------------------------------------------------------
Data Path: /media/bnu/data/nlp-practice/language-model<jupyter_text>数据处理  定义单词表类 * 定义`Vocab`类用于存储单词表* `Vocab`类当中包含了单词(token)与索引(index)之间的映射<jupyter_code>class Vocab:
    def __init__(self, vocab_path):
        self.stoi = {}  # token -> index (dict)
        self.itos = []  # index -> token (list)
        
        with open(vocab_path) as f:
            # bobsue.voc.txt中，每一行是一个单词
            for w in f.readlines():
                w = w.strip()
                if w not in self.stoi:
                    self.stoi[w] = len(self.itos)
                    self.itos.append(w)

    def __len__(self):
        return len(self.itos)<jupyter_output><empty_output><jupyter_text>**简单测试**<jupyter_code>vocab = Vocab(data_path / 'bobsue.voc.txt')
print('单词表大小：', len(vocab))
print('-' * 60)
print('样例（单词 -> 索引）：')
print(list(vocab.stoi.items())[:5])
print('-' * 60)
print('样例（索引 -> 单词）：')
print(list(enumerate(vocab.itos))[:5])<jupyter_output>单词表大小： 1498
------------------------------------------------------------
样例（单词 -> 索引）：
[('<s>', 0), ('</s>', 1), ('.', 2), ('to', 3), ('Bob', 4)]
------------------------------------------------------------
样例（索引 -> 单词）：
[(0, '<s>'), (1, '</s>'), (2, '.'), (3, 'to'), (4, 'Bob')]<jupyter_text>定义语料库 * 定义`Corpus`类读取训练集、验证集、测试集语料* 语料文件中每一行都是一个句子，也就是我们训练时的一份样本* 将语料中的句子读入后，根据`Vocab`转换成索引列表<jupyter_code>class Corpus:
    def __init__(self, data_path, sort_by_len=False):
        self.vocab = Vocab(data_path / 'bobsue.voc.txt')
        self.sort_by_len = sort_by_len
        self.train_data = self.tokenize(data_path / 'bobsue.lm.train.txt')
        self.valid_data = self.tokenize(data_path / 'bobsue.lm.dev.txt')
        self.test_data = self.tokenize(data_path / 'bobsue.lm.test.txt')
        
    def tokenize(self, text_path):
        with open(text_path) as f:
            index_data = []  # 索引数据，存储每个样本的单词索引列表
            for s in f.readlines():
                index_data.append(
                    self.sentence_to_index(s)
                )
        if self.sort_by_len:  # 为了提升训练速度，可以考虑将样本按照长度排序，这样可以减少padding
            index_data = sorted(index_data, key=lambda x: len(x), reverse=True)
        return index_data
    
    def sentence_to_index(self, s):
        return [self.vocab.stoi[w] for w in s.split()]
    
    def index_to_sentence(self, x):
        return ' '.join([self.vocab.itos[i] for i in x])<jupyter_output><empty_output><jupyter_text>**简单测试**<jupyter_code>corpus = Corpus(data_path, sort_by_len=False)
print('训练集句子数目：', len(corpus.train_data))
print('验证集句子数目：', len(corpus.valid_data))
print('测试集句子数目：', len(corpus.test_data))
print('-' * 60)
print('训练集总共单词数目：', sum([len(x) for x in corpus.train_data]))
print('验证集总共单词数目：', sum([len(x) for x in corpus.valid_data]))
print('测试集总共单词数目：', sum([len(x) for x in corpus.test_data]))
print('-' * 60)
print('训练集预测单词数目：', sum([len(x) - 1 for x in corpus.train_data]))
print('验证集预测单词数目：', sum([len(x) - 1 for x in corpus.valid_data]))
print('测试集预测单词数目：', sum([len(x) - 1 for x in corpus.test_data])) 
print('-' * 60)
print('数据样本：')
for i in range(5):
    print(corpus.train_data[i])
    print(corpus.index_to_sentence(corpus.train_data[i]))<jupyter_output>训练集句子数目： 6036
验证集句子数目： 750
测试集句子数目： 750
------------------------------------------------------------
训练集总共单词数目： 71367
验证集总共单词数目： 8707
测试集总共单词数目： 8809
------------------------------------------------------------
训练集预测单词数目： 65331
验证集预测单词数目： 7957
测试集预测单词数目： 8059
------------------------------------------------------------
数据样本：
[0, 16, 235, 372, 10, 60, 3, 75, 618, 39, 2, 1]
<s> She ate quickly and asked to be taken home . </s>
[0, 38, 192, 222, 32, 31, 4, 2, 1]
<s> The girl broke up with Bob . </s>
[0, 7, 842, 2, 1]
<s> Sue apologized . </s>
[0, 12, 150, 18, 8, 261, 3, 546, 102, 5, 1097, 2, 1]
<s> He tried for a year to break into the market . </s>
[0, 200, 706, 14, 5, 26, 15, 427, 228, 2, 1]
<s> So far , the day had gone well . </s><jupyter_text>定义语言模型的DataSet * 这里使用PyTorch中的`DataSet`来构建我们自己的语言模型数据集* 我们自定义的类继承`DataSet`后，要实现`__len__`与`__getitem__`方法* 根据语言模型定义，每个样本的输入是前n-1个单词，预测目标为后n-1个单词<jupyter_code>class BobSueLMDataSet(torch.utils.data.Dataset):
    
    def __init__(self, index_data):
        self.index_data = index_data
        
    def __getitem__(self, i):
        # 根据语言模型定义，这里我们要用前n-1个单词预测后n-1个单词
        return self.index_data[i][:-1], self.index_data[i][1:]
        
    def __len__(self):
        return len(self.index_data)<jupyter_output><empty_output><jupyter_text>**简单测试**<jupyter_code>train_set = BobSueLMDataSet(corpus.train_data)
print('训练集大小：', len(train_set))
print('训练集样本：')
print('\t输入：', train_set[10][0])
print('\t     ', corpus.index_to_sentence(train_set[10][0]))
print('\t目标：', train_set[10][1])
print('\t     ', corpus.index_to_sentence(train_set[10][1]))<jupyter_output>训练集大小： 6036
训练集样本：
	输入： [0, 7, 208, 601, 28, 10, 1276, 25, 5, 709, 507, 2]
	      <s> Sue sat behind him and stared at the cute guy .
	目标： [7, 208, 601, 28, 10, 1276, 25, 5, 709, 507, 2, 1]
	      Sue sat behind him and stared at the cute guy . </s><jupyter_text>定义语言模型的DataLoader * 因为数据集中每个样本的长度不同，我们需要自定义`collate_fn`来处理这个问题* 为了解决这个问题我搞了一下午，尝试了很多办法，现在这个应该是相对“优雅”的解决方案* 这部分我参考了[PyTorch论坛中的方法](https://discuss.pytorch.org/t/dataloader-for-various-length-of-data/6418/11)<jupyter_code>def lm_collate_fn(batch):
    # 这里输入的batch格式为[(input_1, target_1), ... ,(input_n, target_n)]
    # 我们要将其格式转换为[(input_1, ... , input_n), (target_1, ... , target_n)]
    batch = list(zip(*batch))
    
    # 生成长度列表
    lengths = torch.LongTensor([len(x) for x in batch[0]]).to(device)
    
    # 对输入和目标进行padding
    inputs = [torch.LongTensor(x).to(device) for x in batch[0]]
    inputs = nn.utils.rnn.pad_sequence(inputs, batch_first=True)
    targets = [torch.LongTensor(x).to(device) for x in batch[1]]
    targets = nn.utils.rnn.pad_sequence(targets, batch_first=True)
    
    # 因为目标中不存在编号为0的单词，所以目标中为0的位置为padding，由此生成mask矩阵
    mask = (targets != 0).float().to(device)
    
    # 在之后的训练中因为还要进行pack_padded_sequence操作，所以在这里按照长度降序排列
    lengths, perm_index = lengths.sort(descending=True)
    inputs = inputs[perm_index]
    targets = targets[perm_index]
    mask = mask[perm_index]
    
    return inputs, targets, lengths, mask<jupyter_output><empty_output><jupyter_text>**简单测试**<jupyter_code>test_loader = torch.utils.data.DataLoader(
    dataset=train_set,
    batch_size=8,
    shuffle=True,
    collate_fn=lm_collate_fn
)
inputs, targets, lengths, mask = next(iter(test_loader))
print('输入：')
print(inputs)
print('-' * 60)
print('目标：')
print(targets)
print('-' * 60)
print('Mask：')
print(mask)
print('-' * 60)
print('每个样本的实际长度：')
print(lengths)
print('-' * 60)<jupyter_output>输入：
tensor([[   0,   12,   77,    5,   62,    3,    5, 1250,    3,  108,  111,    6,
          500,   31,   20,    2],
        [   0,  159,  713,  353,   17,   33,    3,  108,  111,    6,  553,   28,
           55,  164,    2,    0],
        [   0,    4,   35,  113,  260,  652,    3,   36,    3,    5,  363,  141,
           67,    0,    0,    0],
        [   0,   12,  231,   25,    5,  538,   31,   11,   95,   25,  660,    2,
            0,    0,    0,    0],
        [   0,   12,  599,   10,   13,   77,    5,  267,    3,   58,    0,    0,
            0,    0,    0,    0],
        [   0,    7,  255,   32,   31,    8,  731, 1383,    2,    0,    0,    0,
            0,    0,    0,    0],
        [   0,   16,    6,  431,   24,    5, 1078,    2,    0,    0,    0,    0,
            0,    0,    0,    0],
        [   0,    4,  255,   32,  146,  369,  285,    2,    0,    0,    0,    0,
            0,    0,    0,    0]], device='cuda:0')
----------------------------------------------------------[...]<jupyter_text>定义网络结构  定义模型 * 下面实现一个基于LSTM的网络架构* 输入数据经过一个Embedding层后输入给LSTM，对于LSTM每一个输出经过一个线性层作为输出* 模型`forward`过程中使用`pack_padded_sequence`和`pad_packed_sequence`方法处理变长输入<jupyter_code>class LSTMLM(nn.Module):
    """语言模型网络架构
    
    Args:
        n_words: 词表中的单词数目
        n_embed: 词向量维度
        n_hidden: LSTM隐含状态的维度
        dropout: Dropout概率
    """
    
    def __init__(self, n_words, n_embed=200, n_hidden=200, dropout=0.5):
        super(LSTMLM, self).__init__()
        self.drop = nn.Dropout(dropout)
        self.embed = nn.Embedding(n_words, n_embed)
        self.lstm = nn.LSTM(n_embed, n_hidden, batch_first=True)
        self.linear = nn.Linear(n_hidden, n_words)
        
    def forward(self, inputs, lengths):
        # inputs shape: (batch_size, max_length)
        # x_emb shape: (batch_size, max_length, embed_size)
        x_emb = self.drop(self.embed(inputs))
        
        packed_emb = nn.utils.rnn.pack_padded_sequence(
            x_emb, lengths, batch_first=True
        )
        # 这里LSTM的h_0,c_0使用全0的默认初始化，LSTM层经过后丢弃
        packed_out, _ = self.lstm(packed_emb)
        # x_out shape: (batch_size, max_length, hidden_size)
        x_out, _ = nn.utils.rnn.pad_packed_sequence(
            packed_out, batch_first=True
        )
        
        # outputs shape: (batch, max_length, vocab_size)
        return self.linear(self.drop(x_out))
        
model = LSTMLM(len(corpus.vocab), 200, 200)
model.to(device)<jupyter_output><empty_output><jupyter_text>**简单测试**<jupyter_code>inputs, targets, lengths, mask = next(iter(test_loader))
outputs = model(inputs, lengths)
print('模型输入Shape：', inputs.shape)
print('模型输出Shape：', outputs.shape)<jupyter_output>模型输入Shape： torch.Size([8, 16])
模型输出Shape： torch.Size([8, 16, 1498])<jupyter_text>定义损失函数 * 我们batch中存在padding不能直接使用`CrossEntropyLoss`* 这里需要在原有loss基础上乘以mask矩阵<jupyter_code>class MaskCrossEntropyLoss(nn.Module):
    def __init__(self):
        super(MaskCrossEntropyLoss, self).__init__()
        self.celoss = nn.CrossEntropyLoss(reduction='none')
    
    def forward(self, outputs, targets, mask):
        # outputs shape: (batch_size * max_len, vocab_size)
        outputs = outputs.view(-1, outputs.size(2))
        # targets shape: (batch_size * max_len)
        targets = targets.view(-1)
        # mask shape: (batch_size * max_len)
        mask = mask.view(-1)
        loss = self.celoss(outputs, targets) * mask
        return torch.sum(loss) / torch.sum(mask)<jupyter_output><empty_output><jupyter_text>**简单测试**<jupyter_code>inputs, targets, lengths, mask = next(iter(test_loader))
outputs = model(inputs, lengths)
criterion = MaskCrossEntropyLoss().to(device)
loss = criterion(outputs, targets, mask)
print('损失值：', loss)<jupyter_output>损失值： tensor(7.3313, device='cuda:0', grad_fn=<DivBackward0>)<jupyter_text>模型训练与预测  定义语言模型学习器 * 数据与模型都已经定义好了，接下来实现`LMLearner`类完成模型训练<jupyter_code>class LMLearner:
    def __init__(self, corpus, n_embed=200, n_hidden=200, dropout=0.5, 
                 batch_size=128, early_stopping_round=5):
        self.corpus = corpus
        self.batch_size = batch_size
        self.early_stopping_round = early_stopping_round
        self.model = LSTMLM(len(corpus.vocab), n_embed, n_hidden, dropout).to(device)
        self.criterion = MaskCrossEntropyLoss().to(device)
        self.optimizer = torch.optim.Adam(self.model.parameters())
        self.history = defaultdict(list)
        
    def fit(self, num_epochs):
        # 定义训练集dataloader
        train_set = BobSueLMDataSet(self.corpus.train_data)
        train_loader = torch.utils.data.DataLoader(
            dataset=train_set,
            batch_size=self.batch_size,
            shuffle=True,
            collate_fn=lm_collate_fn
        )
        
        # 定义验证集dataloader
        valid_set = BobSueLMDataSet(self.corpus.valid_data)
        valid_loader = torch.utils.data.DataLoader(
            dataset=valid_set,
            batch_size=self.batch_size,
            shuffle=True,
            collate_fn=lm_collate_fn
        )
        
        # 记录验证集没有提高的轮数，用于EarlyStopping
        no_improve_round = 0
        
        for epoch in range(num_epochs):            
            train_loss, train_acc, train_words = self._make_train_step(train_loader)
            if (epoch + 1) % 10 == 0:
                print(f'Epoch {epoch+1}:')
                print('Train Step --> Loss: {:.3f}, Acc: {:.3f}, Words: {}'.format(
                    train_loss, train_acc, train_words))
            # 记录训练信息
            self.history['train_loss'].append(train_loss)
            self.history['train_acc'].append(train_acc)
        
            valid_loss, valid_acc, valid_words = self._make_valid_step(valid_loader)
            if (epoch + 1) % 10 == 0:
                print('Valid Step --> Loss: {:.3f}, Acc: {:.3f}, Words: {}'.format(
                    valid_loss, valid_acc, valid_words))
            self.history['valid_loss'].append(valid_loss)
            self.history['valid_acc'].append(valid_acc)
            
            # 根据验证集的准确率进行EarlyStopping
            if self.history['valid_acc'][-1] < max(self.history['valid_acc']):
                no_improve_round += 1
            else:
                no_improve_round = 0
            if no_improve_round == self.early_stopping_round:
                print(f'Early Stopping at Epoch {epoch+1}')
                break
            
        
    def predict(self):
        test_set = BobSueLMDataSet(self.corpus.test_data)
        # 这里注意，为了方便之后分析不要shuffle，batch_size设置为1
        test_loader = torch.utils.data.DataLoader(
            dataset=test_set,
            batch_size=1,
            shuffle=False,
            collate_fn=lm_collate_fn
        )
        
        # 验证模式
        self.model.eval()
        
        # 总损失
        total_loss = 0.0
        # 正确预测的数目，单词总数
        total_correct, total_words = 0, 0
        # 预测结果字典，包含preds和targets
        test_result = defaultdict(list) 
        
        with torch.no_grad():
            for inputs, targets, lengths, mask in test_loader:
                # 计算模型输出
                outputs = self.model(inputs, lengths)
                
                # 统计当前预测正确的数目
                total_correct += (outputs.argmax(-1) == targets).sum().item()
                # 统计当前总预测单词数
                total_words += torch.sum(lengths).item()
                
                # 记录结果
                test_result['preds'].append(outputs.argmax(-1).data.cpu().numpy()[0])
                test_result['targets'].append(targets.data.cpu().numpy()[0])
                
                # 计算模型Mask交叉熵损失
                loss = self.criterion(outputs, targets, mask)
                # 统计总损失
                total_loss += loss.item() * torch.sum(mask).item()
        return total_loss / total_words, total_correct / total_words, total_words, test_result
        
    def _make_train_step(self, train_loader):
        # 训练模式
        self.model.train()
        
        # 总损失
        total_loss = 0.0
        # 正确预测的数目，单词总数
        total_correct, total_words = 0, 0
        
        for inputs, targets, lengths, mask in train_loader:
            # 计算模型输出
            outputs = self.model(inputs, lengths)
            
            # 统计当前预测正确的数目
            total_correct += (outputs.argmax(-1) == targets).sum().item()
            # 统计当前总预测单词数
            total_words += torch.sum(lengths).item()
            
            # 计算模型Mask交叉熵损失
            loss = self.criterion(outputs, targets, mask)
            # 统计总损失
            total_loss += loss.item() * torch.sum(mask).item()
                        
            # 反向传播
            self.optimizer.zero_grad()
            loss.backward()
            self.optimizer.step()
        return total_loss / total_words, total_correct / total_words, total_words
    
    def _make_valid_step(self, valid_loader):
        # 验证模式
        self.model.eval()
        
        # 总损失
        total_loss = 0.0
        # 正确预测的数目，单词总数
        total_correct, total_words = 0, 0
        
        with torch.no_grad():
            for inputs, targets, lengths, mask in valid_loader:
                # 计算模型输出
                outputs = self.model(inputs, lengths)
                
                # 统计当前预测正确的数目
                total_correct += (outputs.argmax(-1) == targets).sum().item()
                # 统计当前总预测单词数
                total_words += torch.sum(lengths).item()
                
                # 计算模型Mask交叉熵损失
                loss = self.criterion(outputs, targets, mask)
                # 统计总损失
                total_loss += loss.item() * torch.sum(mask).item()
        return total_loss / total_words, total_correct / total_words, total_words<jupyter_output><empty_output><jupyter_text>模型训练 * 这里设定好参数就可以开始训练了<jupyter_code>torch.cuda.empty_cache()
learner = LMLearner(corpus, n_embed=200, n_hidden=200, dropout=0.5, batch_size=128)
learner.fit(1000)<jupyter_output>Epoch 10:
Train Step --> Loss: 3.729, Acc: 0.288, Words: 65331
Valid Step --> Loss: 3.627, Acc: 0.304, Words: 7957
Epoch 20:
Train Step --> Loss: 3.433, Acc: 0.309, Words: 65331
Valid Step --> Loss: 3.442, Acc: 0.327, Words: 7957
Epoch 30:
Train Step --> Loss: 3.263, Acc: 0.324, Words: 65331
Valid Step --> Loss: 3.378, Acc: 0.333, Words: 7957
Epoch 40:
Train Step --> Loss: 3.144, Acc: 0.331, Words: 65331
Valid Step --> Loss: 3.352, Acc: 0.338, Words: 7957
Epoch 50:
Train Step --> Loss: 3.059, Acc: 0.338, Words: 65331
Valid Step --> Loss: 3.336, Acc: 0.335, Words: 7957
Early Stopping at Epoch 54<jupyter_text>模型预测 * 我们现在在测试集上运行模型，可以看到准确率大概在34%左右* 之后我们可以查看针对测试集上的预测结果<jupyter_code>test_loss, test_acc, test_words, test_result = learner.predict()
print('测试集上的结果 --> Loss: {:.3f}, Acc: {:.3f}, Words: {}'.format(
    test_loss, test_acc, test_words))
print('预测句子数量：', len(test_result['preds']))
print('-' * 60)

sample_index = 10
print('结果样例：')
print('预测值\t', test_result['preds'][sample_index])
print('实际值\t', test_result['targets'][sample_index])
print('预测句子\t', corpus.index_to_sentence(test_result['preds'][sample_index]))
print('实际句子\t', corpus.index_to_sentence(test_result['targets'][sample_index]))<jupyter_output>预测句子数量： 750
------------------------------------------------------------
结果样例：
预测值	 [ 12  83 407  14  13   6 125   6   3 180  52   2   1]
实际值	 [ 272    5  587   14    4  539   13   15    8  286 1126    2    1]
预测句子	 He first end , he was how was to great time . </s>
实际句子	 At the dentist , Bob learned he had a bad tooth . </s><jupyter_text>错误分析 * 这里我们统计了一下常见的35个预测错误（实际值，预测值）<jupyter_code>mistake_counter = Counter()
for i in range(len(test_result['targets'])):
    for j in range(len(test_result['targets'][i])):
        pred, target = test_result['preds'][i][j], test_result['targets'][i][j]
        if pred != target:
            pred, target = corpus.vocab.itos[pred], corpus.vocab.itos[target]
            mistake_counter[(target, pred)] += 1
mistake_counter.most_common(35)<jupyter_output><empty_output><jupyter_text>* 从上面的运行结果中，可以将主要错误类型分为以下几类：    1. **句首错误：**对于句子开头的单词模型会简单的预测为He，这主要是因为当句子开始时我们实际上只有简单的开始标记的信息，在没有其他上文信息的情况下，模型只能简单的预测训练集中常见的第一个词He    2. **介词错误：**对于句子中的介词(to,and,in etc)，模型倾向于预测为句号'.'，这部分应该是因为语言模型训练过程中没有办法看到文本后面的信息，模型在读入一部分单词之后很难判断后续是有介词连接的部分还是句子已经结束    3. **Be动词错误：**模型在一些情况下会将动词预测为'was'，这是由于训练语言模型时，单词从左向右输入，仅仅看到了主语很难判断主语之后的行为* * ** 总结一下，对于第一类错误可以通过添加上文信息来增加准确性，对于二、三类错误就不是简单语言模型能够解决的了，这两类错误需要后文信息来提高准确率，比如考虑使用BiLSTM之类的模型<jupyter_code>correct_counter = Counter()
for i in range(len(test_result['targets'])):
    for j in range(len(test_result['targets'][i])):
        pred, target = test_result['preds'][i][j], test_result['targets'][i][j]
        if pred == target:
            pred, target = corpus.vocab.itos[pred], corpus.vocab.itos[target]
            correct_counter[(target, pred)] += 1
correct_counter.most_common(35)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1592717914502216, 'fraction_numerical': 0.0387933088690845, 'mean_word_length': 2.5149078726968175, 'pattern_counts': {'":': 0, '<': 88, '<?xml version=': 0, '>': 106, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '523091', 'license': 'MIT', 'n_tokens_mistral': 9120, 'n_tokens_neox': 9078, 'n_words': 1397, 'path': 'notebooks/homework/PyTorch_LanguageModel_Part1.ipynb', 'repo_name': 'RegiusQuant/nlp-practice', 'size': '20983'}