A Mixed-Methods Study of Novice Programmer Interaction with Python Error Messages
The ability to interpret error messages in order to find and fix bugs is an essential skill for novice programmers. Unfortunately, the technical language of most error messages can hinder the progress of CS1 students and can lead to feelings of confusion and frustration. A potential intervention for CS educators is the use of enhanced error messages that utilise natural-language geared towards novice learners, but the current discourse in CS Education regarding benefits of such messages is inconclusive. In this paper, we describe a planned, semi-controlled experiment running parallel to a CS1 course at the University of Toronto Scarborough. The study aims to build upon existing work in quantifying the effects of enhanced error messages by incorporating additional quantitative metrics, usability surveys, student feedback, and semi-structured interviews. The additional methods serve to measure not only the effects on error recovery, but student satisfaction, sense of frustration and overall attitude towards error messages and debugging.
Error Message Readability and Novice Debugging Performance
It is well known that programming error messages can be notoriously difficult for novices to understand, hampering progress and leading to frustration. In response, researchers have explored various approaches for enhancing such messages, yet results from this active strand of research are currently mixed. Direct comparisons of results between studies is challenging as these typically investigate different kinds of message enhancements and report results using different metrics. In addition, many prior studies have involved code writing tasks. In such cases, not all students encounter the same errors and messages, and it is difficult to isolate the time spent interpreting messages and resolving errors from the time spent writing code. In this research, we explore the effects of presenting novices with compiler error messages designed using the most recent collection of published guidelines - specifically, more easily readable, short, positive messages containing resolution hints. To accurately determine the time and effort required to read and respond to the messages, we utilise a debugging task where all students are presented the same code and therefore encounter the same errors. We present results of a randomised controlled experiment (n > 700) which shows that, compared to standard error messages, the messages we tested resulted in significantly shorter debugging times and higher self-reported scores of message usefulness for students in the very early stages of learning a new language.
CAP
Context-aware mobile application (App) usage prediction benefits a variety of applications such as precise bandwidth allocation, App launch acceleration, etc. Prior works have explored this topic through individual data profiles and contextual information. However, it is still a challenging problem because of the following three aspects: i. App usage behavior is usually influenced by multiple factors, especially temporal and spatial factors. ii. It is difficult to describe individuals' preferences, which are usually time-variant. iii. A single user's data is sparse on the spatial domain and only covers a limited number of locations. Prediction becomes more difficult when the user appears at a new location. This paper presents CAP, a context-aware App usage prediction algorithm that takes both contextual information (location & time) and attribution (App with type information) into consideration. We find that the relationships between App-location, App-time, and App-App type are essential to prediction and propose a heterogeneous graph embedding algorithm to map them into the common comparable latent space. In addition, we create a user profile for each user with App usage and trajectory history to describe the individual dynamic preference for personalized prediction. We evaluate the performance of our proposed CAP with two large-scale real-world datasets. Extensive evaluations demonstrate that CAP achieves 30% higher accuracy than a state-of-the-art method Personalized Ranking Metric Embedding (PRME) in terms of Accuracy@5. In terms of mean reciprocal rank (MRR), CAP achieves 1.5× higher than the straightforward baseline Sta and 2× higher than PRME. Our investigation enables a range of applications to benefit from such timely predictions, including network operators, service providers, and etc.