We propose a new application of embedding techniques to problem retrieval in adaptive tutoring. The objective is to retrieve problems similar in mathematical concepts. There are two challenges: First, like sentences, problems helpful to tutoring are never exactly the same in terms of the underlying concepts. Instead, good problems mix concepts in innovative ways, while still displaying continuity in their relationships. Second, it is difficult for humans to determine a similarity score consistent across a large enough training set. We propose a hierarchical problem embedding algorithm, called Prob2Vec, that consists of an abstraction and an embedding step. Prob2Vec achieves 96.88\% accuracy on a problem similarity test, in contrast to 75\% from directly applying state-of-the-art sentence embedding methods. It is surprising that Prob2Vec is able to distinguish very fine-grained differences among problems, an ability humans need time and effort to acquire. In addition, the sub-problem of concept labeling with imbalanced training data set is interesting in its own right. It is a multi-label problem suffering from dimensionality explosion, which we propose ways to ameliorate. We propose the novel negative pre-training algorithm that dramatically reduces false negative and positive ratios for classification, using an imbalanced training data set. The traditional teaching methods that are widely used at universities for science, technology, engineering, and mathematic (STEM) courses do not take different abilities of learners into account. Instead, they provide learners with a fixed set of textbooks and homework problems. This ignorance of learners' prior background knowledge, pace of learning, various preferences, and learning goals in current education system can cause tremendous pain and discouragement for those who do not keep pace with this inefficient system BID6 ; BID5 ; BID7 ; BID18 ; BID43 . Hence, e-learning methods are given considerable attention in an effort to personalize the learning process by providing learners with optimal and adaptive curriculum sequences. Over the years, many web-based tools have emerged to adaptively recommend problems to learners based on courseware difficulty. These tools tune the difficulty level of the recommended problems for learners and push them to learn by gradually increasing the difficulty level of recommended problems on a specific concept. The downside of such methods is that they do not take the concept continuity and mixture of concepts into account, but focus on the difficulty level of single concepts. Note that a learner who knows every individual concept does not necessarily have the ability to bring all of them together for solving a realistic problem on a mixture of concepts. As a result, the recommender system needs to know similarity/dissimilarity of problems with mixture of concepts to respond to learners' performance more effectively as described in the next paragraph, which is something that is missing in the literature and needs more attention.Since it is difficult for humans to determine a similarity score consistent across a large enough training set, it is not feasible to simply apply supervised methods to learn a similarity score for problems. In order to take difficulty, continuity, and mixture of concepts into account for similarity score used in a personalized problem recommender system in an adaptive practice, we propose to use a proper numerical representation of problems on mixture of concepts equipped with a similarity measure. By virtue of vector representations for a set of problems on both single and mixture of concepts (problem embedding) that capture similarity of problems, learners' performance on a problem can be projected onto other problems. As we see in this paper, creating a proper problem representation that captures mathematical similarity of problems is a challenging task, where baseline text representation methods and their refined versions fail to work. Although the state-of-the-art methods for phrase/sentence/paragraph representation are doing a great job for general purposes, their shortcoming in our application is that they take lexical and semantic similarity of words into account, which is totally invalid when dealing with text related to math or any other special topic. The words or even subject-related keywords of problems are not completely informative and cannot contribute to embedding of math problems on their own; as a result, the similarity of two problems is not highly correlated with the wording of the problems. Hence, baseline methods perform poorly on the problem similarity detection test in problem recommender application.We find that instead of words or even subject-related keywords, conceptual ideas behind the problems determine their identity. The conceptual particles (concepts) of problems are mostly not directly mentioned in problem wording, but there can be footprints of them in problems. Since problem wording does not capture the similarity of problems, we propose an alternative hierarchical approach called Prob2Vec consisting of an abstraction and an embedding step. The abstraction step projects a problem to a set of concepts. The idea is that there exists a concept space with a reasonable dimension N , with N ranging from tens to a hundred, that can represent a much larger variety of problems of order O(2 N ). Each variety can be sparsely inhabited, with some concept combination having only one problem. This is because making problems is itself a creative process: The more innovative a problem is, the less likely it has exactly the same concept combination as another problem. The explicit representation of problems using concepts also enables state-dependent similarity computation, which we will explore in future work. The embedding step constructs a vector representation of the problems based on concept cooccurrence. Like sentence embedding, not only does it capture the common concepts between problems, but also the continuity among concepts. The proposed Prob2Vec algorithm achieves 96.88% accuracy on a problem similarity test, where human experts are asked to label the relative similarity among each triplet of problems. In contrast, the best of the existing methods, which directly applies sentence embedding, achieves 75% accuracy. It is surprising that Prob2Vec is able to distinguish very fine-grained differences among problems, as the problems in some triplets are highly similar to each other, and only humans with extensive training in the subject are able to identify their relative order of similarity. The problem embedding obtained from Prob2Vec is being used in the recommender system of an e-learning tool for an undergraduate probability course for four semesters with successful results on hundreds of students, specially benefiting minorities who tend to be more isolated in the current education system.In addition, the sub-problem of concept labeling in the abstraction step is interesting in its own right. It is a multi-label problem suffering from dimensionality explosion, as there can be as many as 2 N problem types. This results in two challenges: First, there are very few problems for some types, hence a direct classification on 2 N classes suffers from a severe lack of data. Second, per-concept classification suffers from imbalance of training samples and needs a very small per-concept false positive in order to achieve a reasonable per-problem false positive. We propose pre-training of the neural network with negative samples (negative pre-training) that beats a similar idea to oneshot learning BID15 , where the neural network is pre-trained on classification of other concepts to have a warm start on classification of the concept of interest (transfer learning). A hierarchical embedding method called Prob2Vec for subject specific text is proposed in this paper. Prob2Vec is empirically proved to outperform baselines by more than 20% in a properly validated similarity detection test on triplets of problems. The Prob2Vec embedding vectors for problems are being used in the recommender system of an e-learning tool for an undergraduate probability course for four semesters. We also propose negative pre-training for training with imbalanced data sets to decrease false negatives and positives. As future work, we plan on using graphical models along with problem embedding vectors to more precisely evaluate the strengths and weaknesses of students on single and mixture of concepts to do problem recommendation in a more effective way.one of popular methods, E w : w ∈ W , where we use Glove, problem embedding for P i that is denoted by E i is computed as follows: DISPLAYFORM0 where u is the first principle component of E i : 1 ≤ i ≤ M and a is a hyper-parameter which is claimed to result in best performance when a = 10 −3 to a = 10 −4 . We tried different values for a inside this interval and out of it and found a = 10 −5 and a = 10 −3 to best work for our data set when using all words and a = 2 × 10 −2 to best work for when using selected words. (iii) 3-SVD: using the same hierarchical approach as Prob2Vec, concept embedding in the second step can be done with an SVD-based method instead of the Skip-gram method as follows.Recall that the concept dictionary is denoted by {C 1 , C 2 , · · · , C N }, where each problem is labeled with a subset of these concepts. Let N c (C i , C j ) for i = j denote number of cooccurrences of concepts C i and C j in problems of data set; i.e. there are N c (C i , C j ) number of problems that are labeled with both C i and C j . The co-occurrence matrix is formed as follows: The SVD decomposition of the P P M I matrix is as P P M I = U SV , where U, S, V ∈ R N ×N , and S is a diagonal matrix. Denote embedding size of concepts by d ≤ N , and let U d be the first d columns of matrix U , S d be a diagonal matrix with the first d diagonal elements of diagonal matrix S, and V d be the first d rows of matrix V . The followings are different variants of SVD-based concept embedding BID21 : DISPLAYFORM1 • eig: embedding of N concepts are given by N rows of matrix U d that are of embedding length d. • sub: N rows of U d S d are embedding of N concepts.• shifted: the P P M I matrix is defined in a slightly different way in this variant as follows: Note that the P P M I matrix is not necessarily symmetric in this case. By deriving U d and S d matrices as before, embedding of N concepts are given by N rows of U d S d .
We propose the Prob2Vec method for problem embedding used in a personalized e-learning tool in addition to a data level classification method, called negative pre-training, for cases where the training data set is imbalanced.
{"src": "scitldr-AIC_train", "paper_id": "SJl8gnAqtX"}