<jupyter_start><jupyter_text>基于 gensim 的模型训练举例  概述您可以使用自己的数据和模型参数来训练和使用自己的模型。  导入模块<jupyter_code>import json
from tqdm import tqdm
from EduNLP.Pretrain import GensimWordTokenizer, train_vector
from EduNLP.Vector import D2V, W2V
from EduNLP.SIF.segment import seg
from EduNLP.SIF.tokenization import tokenize
import time<jupyter_output><empty_output><jupyter_text>准备模型训练数据<jupyter_code>test_items = [{'ques_content':'有公式$\\FormFigureID{wrong1?}$和公式$\\FormFigureBase64{wrong2?}$，如图$\\FigureID{088f15ea-8b7c-11eb-897e-b46bfc50aa29}$,若$x,y$满足约束条件$\\SIFSep$，则$z=x+7 y$的最大值为$\\SIFBlank$'},
             {"ques_content":"Human machine interface for lab abc computer applications"},
             {"ques_content": "A survey of user opinion of computer system response time"},
             {"ques_content": "The EPS user interface management system"},
             {"ques_content": "System and human system engineering testing of EPS"},
             {"ques_content": "Relation of user perceived response time to error measurement"},
             {"ques_content": "The generation of random binary unordered trees"},
             {"ques_content": "The intersection graph of paths in trees"},
             {"ques_content": "Graph minors IV Widths of trees and well quasi ordering"},
             {"ques_content": "Graph minors A survey"}
             ]

def load_items():
    for line in test_items:
        yield line


def data2Token():
    # 线性分词
    tokenization_params = {
        "formula_params": {
            "method": "linear",
        }
    }
    
    token_items = []
    count = 1
    for item in tqdm(load_items(), "sifing"):
        count = count + 1
        # -------------------------------------------- # 
        # """除文本、公式外，其他转化为特殊标记"""
        tokenizer = GensimWordTokenizer(symbol="gmas", general=True)
        token_item = tokenizer(item["ques_content"])

        # -------------------------------------------- # 
        if token_item:
            token_items.append(token_item.tokens)
    print("[data2Token] finish ========================> num = ",len(token_items))
    return token_items

token_items = data2Token()
print(token_items[0])
len(token_items[0])<jupyter_output><empty_output><jupyter_text>也可从文件导入数据例如：<jupyter_code>from EduData import get_data

# 导入项目提供的数据，存放路径："../../data/"
get_data("open-luna", "../../data/")


def load_items():
    with open("../../../data/OpenLUNA.json", encoding="utf-8") as f:
        for line in f:
            yield json.loads(line)
tokenizer = GensimWordTokenizer(symbol="gm")
sif_items = []
for item in tqdm(load_items(), "sifing"):
    sif_item = tokenizer(
        item["stem"]
    )
    if sif_item:
        sif_items.append(sif_item.tokens)

sif_items[0]<jupyter_output><empty_output><jupyter_text>EduNLP.Vector.D2V 模块的训练方法  1. 基于 bow 训练模型<jupyter_code>train_vector(token_items, "../../../data/d2v/gensim_luna_stem_tf_", method="bow")<jupyter_output>EduNLP, INFO model is saved to ../../../data/d2v/gensim_luna_stem_tf_bow.bin<jupyter_text>- 模型测试<jupyter_code>d2v = D2V("../../../data/d2v/gensim_luna_stem_tf_bow.bin", method = "bow")
print(d2v(token_items[1]))<jupyter_output>[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]<jupyter_text>2. 基于 tfidf 训练模型<jupyter_code>train_vector(token_items, "../../../data/d2v/gensim_luna_stem_tf_", method="tfidf")<jupyter_output>EduNLP, INFO model is saved to ../../../data/d2v/gensim_luna_stem_tf_bow.bin
EduNLP, INFO model is saved to ../../../data/d2v/gensim_luna_stem_tf_tfidf.bin<jupyter_text>- 模型测试<jupyter_code>d2v = D2V("../../../data/d2v/gensim_luna_stem_tf_tfidf.bin", method = "tfidf")
vec_size = d2v.vector_size
print("vec_size = ", vec_size)
print(d2v(token_items[1]))<jupyter_output>vec_size =  63
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.37858374396389033, 0.37858374396389033, 0.37858374396389033, 0.2646186811599866, 0.37858374396389033, 0.2646186811599866, 0.37858374396389033, 0.37858374396389033, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]<jupyter_text>3. 基于 Doc2Vec 训练模型<jupyter_code># 10 dimension with doc2vec method
train_vector(token_items, "../../../data/w2v/gensim_luna_stem_tf_", 10, method="d2v")

d2v = D2V("../../../data/w2v/gensim_luna_stem_tf_d2v_10.bin", method="d2v")
vec_size = d2v.vector_size
print("vec_size = ", vec_size)
print(d2v(token_items[1]))<jupyter_output>vec_size =  10
[-0.00211227  0.00167636  0.02313529 -0.04260717 -0.01389424 -0.03898989
  0.01181044  0.01069339 -0.03934718  0.00038158]<jupyter_text>EduNLP.Vector.W2V 模块支持的训练方法  1. 基于 FastText 训练模型<jupyter_code># 10 dimension with fasstext method
train_vector(token_items, "../../../data/w2v/gensim_luna_stem_t_",
             10, method="fasttext")<jupyter_output>EduNLP, INFO Epoch #0: loss-0.0000 
EduNLP, INFO Epoch #1: loss-0.0000 
EduNLP, INFO Epoch #2: loss-0.0000 
EduNLP, INFO Epoch #3: loss-0.0000 
EduNLP, INFO Epoch #4: loss-0.0000 
EduNLP, INFO model is saved to ../../../data/w2v/gensim_luna_stem_t_fasttext_10.bin<jupyter_text>- 模型测试<jupyter_code>w2v = W2V("../../../data/w2v/gensim_luna_stem_t_fasttext_10.bin", method="fasttext")
w2v["[FORMULA]"]<jupyter_output><empty_output><jupyter_text>2. 基于 cbow 训练模型<jupyter_code>train_vector(token_items, "../../../data/w2v/gensim_luna_stem_t_", 10, method="cbow")<jupyter_output>EduNLP, INFO Epoch #0: loss-0.0000 
EduNLP, INFO Epoch #1: loss-0.0000 
EduNLP, INFO Epoch #2: loss-0.0000 
EduNLP, INFO Epoch #3: loss-0.0000 
EduNLP, INFO Epoch #4: loss-0.0000 
EduNLP, INFO model is saved to ../../../data/w2v/gensim_luna_stem_t_cbow_10.kv<jupyter_text>- 模型测试<jupyter_code>w2v = W2V("../../../data/w2v/gensim_luna_stem_t_cbow_10.kv",
          method="fasttext")
w2v["[FORMULA]"]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17740841248303935, 'fraction_numerical': 0.08734735413839892, 'mean_word_length': 3.5962587685113014, 'pattern_counts': {'":': 11, '<': 42, '<?xml version=': 0, '>': 43, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '292110', 'license': 'Apache-2.0', 'n_tokens_mistral': 2845, 'n_tokens_neox': 2550, 'n_words': 480, 'path': 'examples/pretrain/gensim/d2v_bow_tfidf.ipynb', 'repo_name': 'nnnyt/EduNLP', 'size': '5896'}