<jupyter_start><jupyter_text>Fuzzy cognitive diagnosis framework (FuzzyCDF)This notebook will show you how to train and use the FuzzyCDF.First, we will show how to get the data (here we use Math1 from math2015 as the dataset).Then we will show how to train a FuzzyCDF and perform the parameters persistence.At last, we will show how to load the parameters from the file and evaluate on the test dataset.The script version could be found in [FuzzyCDF.py](FuzzyCDF.ipynb)  Data PreparationBefore we process the data, we need to first acquire the dataset which is shown in [prepare_dataset.ipynb](prepare_dataset.ipynb)<jupyter_code># Data preprocessing, split train/valid/test data

import json
import numpy as np
import random

def read_probdesc(filepath):
    prob_type = np.loadtxt(filepath, dtype=int, delimiter='\t',
                           converters={0: lambda x: int(x) - 1, 1: lambda s: s == b'Obj'}, skiprows=1)
    obj_prob_index, sub_prob_index = prob_type[prob_type[:, 1] == 1][:, 0], prob_type[prob_type[:, 1] == 0][:, 0]
    return prob_type, obj_prob_index, sub_prob_index

train_ratio = 0.8
valid_ratio = 0

# type of problems
problems, obj_prob_index, sub_prob_index = read_probdesc("../../data/math2015/Math1/problemdesc.txt")
np.savetxt("../../data/math2015/Math1/obj_prob_index.csv", obj_prob_index, delimiter=',', fmt='%d')
np.savetxt("../../data/math2015/Math1/sub_prob_index.csv", sub_prob_index, delimiter=',', fmt='%d')

# Q matrix
np.savetxt("../../data/math2015/Math1/q_m.csv", np.loadtxt("../../data/math2015/Math1/q.txt", dtype=int), delimiter=',', fmt='%d')

# response matrix, split dataset
R = np.loadtxt("../../data/math2015/Math1/data.txt")
stu_num, prob_num = R.shape[0], R.shape[1]
train_logs, valid_logs, test_logs = [], [], []
for stu in range(stu_num):
    stu_logs = []
    for prob in range(prob_num):
        log = {'user_id': int(stu), 'item_id': int(prob), 'score': R[stu][prob]}
        stu_logs.append(log)
    random.shuffle(stu_logs)
    train_logs += stu_logs[: int(train_ratio * prob_num)]
    valid_logs += stu_logs[int(train_ratio * prob_num): int(train_ratio * prob_num) + int(valid_ratio * prob_num)]
    test_logs += stu_logs[int(train_ratio * prob_num) + int(valid_ratio * prob_num):]

with open("../../data/math2015/Math1/train_data.json", 'w', encoding='utf8') as file:
    json.dump(train_logs, file, indent=4, ensure_ascii=False)
with open("../../data/math2015/Math1/valid_data.json", 'w', encoding='utf8') as file:
    json.dump(valid_logs, file, indent=4, ensure_ascii=False)
with open("../../data/math2015/Math1/test_data.json", 'w', encoding='utf8') as file:
    json.dump(test_logs, file, indent=4, ensure_ascii=False)

print(train_logs[0], test_logs[0])
# Load the data from files

# type of problems
obj_prob_index = np.loadtxt("../../data/math2015/Math1/obj_prob_index.csv", delimiter=',', dtype=int)
sub_prob_index = np.loadtxt("../../data/math2015/Math1/sub_prob_index.csv", delimiter=',', dtype=int)
# Q matrix
q_m = np.loadtxt("../../data/math2015/Math1/q_m.csv", dtype=int, delimiter=',')
prob_num, know_num = q_m.shape[0], q_m.shape[1]

# training data
with open("../../data/math2015/Math1/train_data.json", encoding='utf-8') as file:
    train_set = json.load(file)
stu_num = max([x['user_id'] for x in train_set]) + 1
R = -1 * np.ones(shape=(stu_num, prob_num))
for log in train_set:
    R[log['user_id'], log['item_id']] = log['score']

# testing data
with open("../../data/math2015/Math1/test_data.json", encoding='utf-8') as file:
    test_set = json.load(file)
len(train_set), len(test_set)<jupyter_output><empty_output><jupyter_text>Training and Persistence<jupyter_code>import logging
logging.getLogger().setLevel(logging.INFO)
from EduCDM import FuzzyCDF

cdm = FuzzyCDF(R, q_m, stu_num, prob_num, know_num, obj_prob_index, sub_prob_index, skip_value=-1)

cdm.train(epoch=10, burnin=5)
cdm.save("fuzzycdf.params")<jupyter_output>INFO:root:save parameters to fuzzycdf.params<jupyter_text>Loading and Testing<jupyter_code>cdm.load("fuzzycdf.params")
rmse, mae = cdm.eval(test_set)
print("RMSE: %.6f, MAE: %.6f" % (rmse, mae))<jupyter_output>INFO:root:load parameters from fuzzycdf.params
evaluating: 100%|█████████████████████████████████████████████████████████████| 16836/16836 [00:00<00:00, 91552.55it/s]<jupyter_text>Incremental Training<jupyter_code>new_data = [{'user_id': 0, 'item_id': 2, 'score': 0.0}, {'user_id': 1, 'item_id': 1, 'score': 1.0}]
cdm.inc_train(new_data, epoch=10, burnin=5)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17534914653070272, 'fraction_numerical': 0.033030370206162715, 'mean_word_length': 5.214876033057851, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '593334', 'license': 'Apache-2.0', 'n_tokens_mistral': 1825, 'n_tokens_neox': 1684, 'n_words': 360, 'path': 'examples/FuzzyCDF/FuzzyCDF.ipynb', 'repo_name': 'LegionKing/EduCDM', 'size': '4511'}