<jupyter_start><jupyter_text>An explanation of the fieldsThis is taken from [the ASSISTments data website](https://sites.google.com/site/assistmentsdata/how-to-interpret).* `order_id`: These id's are chronological, and refer to the id of the original problem log.* `assignment_id`: Each time a teacher assigns a problem set that assignment gets a separate number. This is that number. * `user_id`: The ID of the student doing the problem.* `assistment_id`: Similar to problem_id. The ID of a problem one will see in the builder. If a problem has multiple main problems and/or scaffolding, everything relating to one problem is called an assistment and has the same assistment_id. If you see problem logs with the same assistment number, they are multiple main problems(or scaffolding problems) that are part of the same overarching problem.* `problem_id`: The ID of the problem. If a problem has multiple main problems, each multiple main problem will have a different problem_id.* `original`:   * 1 = Main problem  * 0 = Scaffolding problem      If a problem has scaffolding and the student answers incorrectly or asks for the problem to be broken into steps, a new problem will be created called a scaffolding problem. This creates a separate problem log row in the file with the variable original = 0.* `correct`:   * 1 = Correct on first attempt  * 0 = Incorrect on first attempt, or asked for help  This column is often the target for prediction. (Minor note: <NAME> notes that while this is true most of the time, we also have Essay questions that teachers can grade.  Neil thinks that if this value is say .25 that means the teacher gave it a 1 our of 4.)* `attempt_count`: Number of attempts(number of times a student entered an answer)* `ms_first_response`: Time between start time and first student action(asking for hint or entering an answer) (in milliseconds)* `tutor_mode`: Tutor, test mode, pre-test, or post-test* `answer_type`: Unknown* `sequence_id`: ASSISTments is very confusing in how they use "Problem Set" and "Sequence".  The same object that is called a "Sequence" in the database is exposed to teachers as a "Problem Set".  If you have a sequence ID, you can use the converter [here](http://users.wpi.edu/~xxiong/app.html) to get the corresponding problem set number to use ASSISTments and see exactly what the content looked like.* `student_class_id`: The ID of the class, the same for all students in the same class.  If you want to heiricharchila liearn modeling you can use this for the class ID.  We can also give you a teacher ID.  You might also want to look at section ID* `position`: The placement of the assignment within the teacher's assignment page (i.e., 5 means the 5th problem set assigned)* `type`: Determined by assignment type id. Usually ClassAssignment, but sometimes ARRS or remedial.* `base_sequence_id`: This is to account for if a sequence has been copied. This will point to the original copy, or be the same as sequence_id if it hasn't been copied.* `skill_id`: ID of the skill associated with the problem. For the skill builder dataset, different skills for the same data record are in different rows. This means if a student answers a multi skill question, this record is duplicated several times, and each duplication is tagged with one of the multi skills. For the non skill builder dataset, different skills for the same data record are in the same row, separated with comma.* `skill_name`: Skill name associated with the problem.* `teacher_id`: Unknown* `school_id`: ID number for the school* `hint_count`: Number of hints a student asked for during the duration of the problem.* `hint_total`: Number of possible hints on this problem.  We tell you the total number of hints so you can compute something like a % of hints used.  Not all problems have all the same number of hints. * `overlap_time`: Number of possible hints on this problem.  We tell you the total number of hints so you can compute something like a % of hints used.  Not all problems have all the same number of hints. This field is often computed incorrectly. Many data sets display overlap time the same as the first response time. You could compute overlap time using other fields, like using the state time of two problems. * `template_id`: The template ID of the ASSISTment. ASSISTments with the same template ID have similar questions.* `answer_id`: Only exists for multiple choice or choose all that apply questions.   * A number =  the answer the student put in corresponds with one of the answers for that problem  * 0 or empty = the student put an answer not corresponding with one of the answers for that problem* `answer_text`: The answer the student entered. Or the value the student selected in a multiple choice or choose all that apply problem.* `first_action`:   * 0 = attempt  * 1 = hint  * 2 = scaffolding  * empty = student clicked on the problem but did nothing else* `bottom_hint`:   * 1 = The student asked for the bottom out hint  * 0 = The student did not ask for the bottom out hint.  * If this is blank it means the student did not ask for a hint.  Remember that for scaffolding questions they can not get a hint.  * The bottom out hint is the last hint for a problem and will generally contain the problem’s answer.* `opportunity`: The number of opportunities the student has to practice on this skill. For the skill builder dataset, opportunities for different skills of the same data record are in different rows. This means if a student answers a multi skill question, this record is duplicated several times, and each duplication is tagged with one of the multi skills and the corresponding opportunity count.* `opportunity_original`: The number of opportunities the student has to practice on this skill counting only original problems. For the skill builder dataset, original opportunities for different skills of the same data record are in different rows. This means if a student answers a multi skill question, this record is duplicated several times, and each duplication is tagged with one of the multi skills and the corresponding original opportunity count.<jupyter_code>overlap_time = df['overlap_time']
correct = df['correct']
median_overlap = df['overlap_time'].median()
overlap = [0 if x < median_overlap else 1 for x in overlap_time]
np.unique(overlap)
from scipy.stats import pointbiserialr
from sklearn.metrics import matthews_corrcoef
matthews_corrcoef(overlap, correct)
groups = list(df.groupby(['user_id', 'skill_id']))
len(groups)
list(groups[9][1]['correct'])
from WebApp.server.BKT import BKT
model = BKT(np.array(groups[9][1]['correct']).reshape(1, -1))
model.fit()
A, pi, B = model.get_model_params()
# Start probabilities
print(A)
# Transition probabilities
print(pi)
# Emission probabilities
print(B)<jupyter_output>[[1.00000000e+000 2.70814246e-013]
 [1.58431238e-102 1.00000000e+000]]
{'fraction_non_alphanumeric': 0.059388646288209605, 'fraction_numerical': 0.010625909752547307, 'mean_word_length': 4.702074688796681, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 5, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '417385', 'license': 'MIT', 'n_tokens_mistral': 1806, 'n_tokens_neox': 1667, 'n_words': 1028, 'path': 'notebooks/.ipynb_checkpoints/01 - Initial investigation of the data-checkpoint.ipynb', 'repo_name': 'aks96/student-modeling-project', 'size': '6878'}