Given the size of our dataset, our setting falls within the limited labeled data regime [1]}, [2]}. To investigate whether our dataset is sufficient for training models that generalize, we considered a simple task: classifying problems by their concepts. This is a multi-label\(28.9\%\)  of the problems in our dataset span \(> 1\)  concepts. classification task. We used a pretrained transformer, BERT [3]}, to obtain contextualised word embeddings, and then trainedTraining setup details are in the appendix. various models. The results are shown in Table REF . For the MaxEnt model (a logistic regression classifier), and the multi-layer perceptron, MLP, problem representations were obtained by average pooling of the word embeddings. From initial experiments we found that using the CLS token representation, instead of average pooling, degrades performance.
A Convolutional Neural Network (CNN) text classifier, kernel sizes: 3,4,5, 6, each with 192 kernels, has the strongest performance on the full problem, F1 \(\mathbf {0.73}\)  on test data. We also implemented Prototypical Networks [4]}, an approach designed for limited labeled data settings. We used a CNN (same settings as above) as the base model for learning prototypical vector representations.
We evaluated the Prototypical Network on problems that are about a single concept, \(72.7\%\)  of dev, and \(75.8\%\)  of test data, and it produced strong results, F1 \(\mathbf {0.78}\) . Visualizing the learned prototypes and prototypical problems, Figure REF  shows that the learned representations are meaningful. For instance, prototypical problems of type Poisson Distribution are closest to the prototypical vector of that concept. This is not trivially expected because there are many prototypical vectors for a concept, which are generated from different support vectors (refer to [4]} for details), and we randomly picked one prototypical vector for each concept.
