<jupyter_start><jupyter_text>基于TFIDF的检索方法---<jupyter_code>import os
import sys
import pprint
sys.path.insert(0, "/home/team55/notespace/zengbin")

from jddc.utils import write_file, read_file, save_to_pkl, read_from_pkl, create_logger
from jddc.tfidf import *
from jddc.datasets import *<jupyter_output><empty_output><jupyter_text>1 - 构造训练数据集---<jupyter_code>sessions = read_from_pkl(conf.pkl_sessions)
questions, answers = create_dataset05(sessions, 30000)<jupyter_output>sample head and tail qa num is  30932
created qa pairs num is  154660<jupyter_text>2 - 构建tfidf相似性模型---<jupyter_code>create_tfidf_ir_model(questions, answers)<jupyter_output>cut_questions:   3%|█                                      | 4021/154660 [00:00<00:03, 40197.68it/s]<jupyter_text>3 - 测试检索效果---<jupyter_code>conf.top = 10
run_prediction(conf.file_test_q, "/home/team55/notespace/zengbin/answers/tfidf_answers011.txt")<jupyter_output><TfidfConfig | model:/home/team55/notespace/data/tfidf/tfidf_ir.model.middle.pkl; n:3; top:10
{'fraction_non_alphanumeric': 0.11940298507462686, 'fraction_numerical': 0.061691542288557215, 'mean_word_length': 5.618421052631579, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 14, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '398702', 'license': 'MIT', 'n_tokens_mistral': 470, 'n_tokens_neox': 438, 'n_words': 55, 'path': 'final/ipynb/test_tfidf_ir.ipynb', 'repo_name': 'caoxu915683474/jddc_solution_4th', 'size': '1005'}