<jupyter_start><jupyter_text>Датасет, размеченный вручную<jupyter_code>dtrain = pd.read_csv('./datasetmaker/dataset.csv')
print(len(dtrain))
dtrain.head()<jupyter_output><empty_output><jupyter_text>Classification and save<jupyter_code>df = pd.read_csv('./docs/csv/train.csv', sep=";")
print(len(df))
df.head()<jupyter_output>7756<jupyter_text>Загрузка модуля с классификатором, чтобы разметить данные<jupyter_code>from class_creator.class_creator import classPredictor

clcr = classPredictor('./class_creator/df1.txt', './class_creator/df2.txt', './class_creator/df3.txt', './class_creator/df4.txt', './class_creator/df5.txt')
# clcr.predictClass(msg)
df['Class'] = pd.Series(0, index=df.index)
df.head()
for i in range(len(df)):
#     print(df.Class[i])
    df.Class[i] = clcr.predictClass(df.Shrt_Desc[i])
#     print(df.Class[0])
df.Class.unique()
df.to_csv('Pred_class.csv', sep=',', encoding='utf-8')
x = len(df)
y = len(df[df['Class'] == 6])
print(x, y, 'Errors: ', y*100/x)
# df[df['Class'] == 6]
df[df['Class'] != 6].to_csv('predict_cl.csv', sep=',', encoding='utf-8')
df = pd.read_csv('./docs/csv/test.csv', sep=";")
print(len(df))
df.head()<jupyter_output>862<jupyter_text>Загрузка модуля с классификатором, чтобы разметить данные<jupyter_code>from class_creator.class_creator import classPredictor

clcr = classPredictor('./class_creator/df1.txt', './class_creator/df2.txt', './class_creator/df3.txt', './class_creator/df4.txt', './class_creator/df5.txt')
# clcr.predictClass(msg)
df['Class'] = pd.Series(0, index=df.index)
df.head()
for i in range(len(df)):
#     print(df.Class[i])
    df.Class[i] = clcr.predictClass(df.Shrt_Desc[i])
#     print(df.Class[0])
df.to_csv('Pred_extra_1.csv', sep=',', encoding='utf-8')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.20181921546333143, 'fraction_numerical': 0.01762364980102331, 'mean_word_length': 6.48936170212766, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '573754', 'license': 'MIT', 'n_tokens_mistral': 750, 'n_tokens_neox': 753, 'n_words': 95, 'path': '2/.ipynb_checkpoints/Dataframe-checkpoint.ipynb', 'repo_name': 'Dmitriy1594/DataScienceHackDays', 'size': '1759'}