<jupyter_start><jupyter_text>一、数据读入与处理  1. 数据读入<jupyter_code>train_data_df = pd.read_json(train_data_path)
dev_data_df = pd.read_json(dev_data_path)
train_data_df = train_data_df.rename(columns={'entities': 'label'})
dev_data_df = dev_data_df.rename(columns={'entities': 'label'})
train_data_df = train_data_df.loc[:,['text', 'label']]
train_data_df['label'] = train_data_df['label'].apply(lambda x: str(x))
dev_data_df = dev_data_df.loc[:,['text', 'label']]
dev_data_df['label'] = dev_data_df['label'].apply(lambda x: str(x))
ner_train_dataset = Dataset(train_data_df)
ner_dev_dataset = Dataset(dev_data_df)<jupyter_output><empty_output><jupyter_text>2. 词典创建和生成分词器<jupyter_code>tokenizer = Tokenizer(vocab='nghuyong/ernie-1.0', max_seq_len=50)<jupyter_output><empty_output><jupyter_text>4. ID化<jupyter_code>ner_train_dataset.convert_to_ids(tokenizer)
ner_dev_dataset.convert_to_ids(tokenizer)<jupyter_output><empty_output><jupyter_text>二、模型构建  1. 模型参数设置<jupyter_code>config = SpanBertConfig.from_pretrained('nghuyong/ernie-1.0', 
                                         num_labels=len(ner_train_dataset.cat2id))<jupyter_output><empty_output><jupyter_text>2. 模型创建<jupyter_code>torch.cuda.empty_cache()
dl_module = SpanBert.from_pretrained('nghuyong/ernie-1.0', 
                                         config=config)<jupyter_output>Some weights of the model checkpoint at nghuyong/ernie-1.0 were not used when initializing SpanIndependenceBert: ['cls.predictions.decoder.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.bias', 'cls.predictions.decoder.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.bias']
- This IS expected if you are initializing SpanIndependenceBert from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing SpanIndependenceBert from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of SpanIndependenceBert were not initialized from the model checkpoint at nghuyong/ernie-1.0 and are newly initialized: ['classifier.wei[...]<jupyter_text>三、任务构建  1. 任务参数和必要部件设定<jupyter_code># 设置运行次数
num_epoches = 1
batch_size = 32
optimizer = get_default_model_optimizer(dl_module)<jupyter_output><empty_output><jupyter_text>2. 任务创建<jupyter_code>model = Task(dl_module, optimizer, 'ce', cuda_device=0)<jupyter_output><empty_output><jupyter_text>3. 训练<jupyter_code>model.fit(ner_train_dataset, 
          ner_dev_dataset,
          lr=3e-5,
          epochs=5, 
          batch_size=batch_size
         )<jupyter_output><empty_output><jupyter_text>四、模型验证与保存  1. 模型验证<jupyter_code>from ark_nlp.model.ner.span_bert import Predictor
ner_predictor_instance = Predictor(model.module, tokenizer, ner_train_dataset.cat2id)
ner_predictor_instance.predict_one_sample('（5）房室结消融和起搏器植入作为反复发作或难治性心房内折返性心动过速的替代疗法。')<jupyter_output><empty_output><jupyter_text>2. 多样本验证<jupyter_code>test_data_df = pd.read_json(test_data_path)
test_data_df = test_data_df.loc[:,['text']]

record_ = []
for text_ in test_data_df['text'].to_list():
    record_.append([text_, ner_predictor_instance.predict_one_sample(text_)])<jupyter_output><empty_output><jupyter_text>3. Batch模型验证<jupyter_code>test_data_df = pd.read_json(test_data_path)
test_data_df = test_data_df.loc[:,['text']]

ner_test_dataset = Dataset(test_data_df, categories=ner_train_dataset.categories, is_test=True)<jupyter_output><empty_output><jupyter_text>五、模型测试报告<jupyter_code>1. 基本功能测试 通过
2. one sample predict 通过
3. batch predict  无batch方法
4. 多样本验证  通过<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11215932914046121, 'fraction_numerical': 0.00969601677148847, 'mean_word_length': 7.052742616033755, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 48, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '296646', 'license': 'Apache-2.0', 'n_tokens_mistral': 1498, 'n_tokens_neox': 1517, 'n_words': 205, 'path': 'test/span_bert.ipynb', 'repo_name': 'Zrealshadow/ark-nlp', 'size': '3816'}