<jupyter_start><jupyter_text>Deterministic Inputs, Noisy “And” gate model (DINA)This notebook will show you how to train and use the GDDINA.First, we will show how to get the data (here we use a0910 as the dataset).Then we will show how to train a DINA and perform the parameters persistence.At last, we will show how to load the parameters from the file and evaluate on the test dataset.The script version could be found in [DINA.py](DINA.ipynb)  Data PreparationBefore we process the data, we need to first acquire the dataset which is shown in [prepare_dataset.ipynb](prepare_dataset.ipynb)<jupyter_code>import pandas as pd

train_data = pd.read_csv("../../../data/a0910/train.csv")
valid_data = pd.read_csv("../../../data/a0910/valid.csv")
test_data = pd.read_csv("../../../data/a0910/test.csv")
item_data = pd.read_csv("../../../data/a0910/item.csv")

knowledge_num = 123


def code2vector(x):
    vector = [0] * knowledge_num
    for k in eval(x):
        vector[k - 1] = 1
    return vector


item_data["knowledge"] = item_data["knowledge_code"].apply(code2vector)
item_data.drop(columns=["knowledge_code"], inplace=True)

train_data = pd.merge(train_data, item_data, on="item_id")
valid_data = pd.merge(valid_data, item_data, on="item_id")
test_data = pd.merge(test_data, item_data, on="item_id")

train_data.head(5)

len(train_data), len(valid_data), len(test_data)
# Transform data to torch Dataloader (i.e., batchify)
# batch_size is set to 256
import torch
from torch.utils.data import TensorDataset, DataLoader

batch_size = 32

def transform(x, y, z, k, batch_size, **params):
    dataset = TensorDataset(
        torch.tensor(x, dtype=torch.int64),
        torch.tensor(y, dtype=torch.int64),
        torch.tensor(k, dtype=torch.float32),
        torch.tensor(z, dtype=torch.float32)
    )
    return DataLoader(dataset, batch_size=batch_size, **params)


train, valid, test = [
    transform(data["user_id"], data["item_id"], data["score"], data["knowledge"], batch_size)
    for data in [train_data, valid_data, test_data]
]
train, valid, test<jupyter_output><empty_output><jupyter_text>Training and Persistence<jupyter_code>import logging
logging.getLogger().setLevel(logging.INFO)
from EduCDM import GDDINA

cdm = GDDINA(4164, 17747, knowledge_num)

cdm.train(train, valid, epoch=2)
cdm.save("dina.params")<jupyter_output>Epoch 0: 100%|██████████| 7534/7534 [00:54<00:00, 139.51it/s]
evaluating: 100%|██████████| 1036/1036 [00:00<00:00, 1287.18it/s]
Epoch 1: 100%|██████████| 7534/7534 [01:02<00:00, 120.28it/s]
evaluating: 100%|██████████| 1036/1036 [00:00<00:00, 1318.61it/s]
INFO:root:save parameters to dina.params<jupyter_text>Loading and Testing<jupyter_code>cdm.load("dina.params")
auc, accuracy = cdm.eval(test)
print("auc: %.6f, accuracy: %.6f" % (auc, accuracy))<jupyter_output>INFO:root:load parameters from dina.params
evaluating: 100%|██████████| 2248/2248 [00:01<00:00, 1301.36it/s]
{'fraction_non_alphanumeric': 0.15677530017152658, 'fraction_numerical': 0.061406518010291594, 'mean_word_length': 5.151898734177215, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '135200', 'license': 'Apache-2.0', 'n_tokens_mistral': 1217, 'n_tokens_neox': 1065, 'n_words': 255, 'path': 'examples/DINA/GD/DINA.ipynb', 'repo_name': 'ViviHong200709/EduCDM', 'size': '2915'}