In this paper we promote introducing software verification and control flow graph similarity measurement in automated evaluation of students' programs. We present a new grading framework that merges results obtained by combination of these two approaches with results obtained by automated testing, leading to improved quality and precision of automated grading. These two approaches are also useful in providing a comprehensible feedback that can help students to improve the quality of their programs We also present our corresponding tools that are publicly available and open source. The tools are based on LLVM low-level intermediate code representation, so they could be applied to a number of programming languages. Experimental evaluation of the proposed grading framework is performed on a corpus of university students' programs written in programming language C. Results of the experiments show that automatically generated grades are highly correlated with manually determined grades suggesting that the presented tools can find real-world applications in studying and grading.
Apart of assignment grading, regression techniques were also used for final grade forecasting with good results. For this purpose, used data from learning management system and identified variables most useful for the prediction, e.g., number of assessments completed and number of discussion and mail messages sent @cite_14 . Kotsiantis performed successful forecasting based on demographic characteristics of students, results of several written assignments, and class attendance @cite_28 .
@cite_28 @cite_14
2049868504 2153378020
Earlier studies have suggested that higher education institutions could harness the predictive power of Learning Management System (LMS) data to develop reporting tools that identify at-risk students and allow for more timely pedagogical interventions. This paper confirms and extends this proposition by providing data from an international research project investigating which student online activities accurately predict academic achievement. Analysis of LMS tracking data from a Blackboard Vista-supported course identified 15 variables demonstrating a significant simple correlation with student final grade. Regression modelling generated a best-fit predictive model for this course which incorporates key variables such as total number of discussion messages posted, total number of mail messages sent, and total number of assessments completed and which explains more than 30 of the variation in student final grade. Logistic modelling demonstrated the predictive power of this model, which correctly identified 81 of students who achieved a failing grade. Moreover, network analysis of course discussion forums afforded insight into the development of the student learning community by identifying disconnected students, patterns of student-to-student communication, and instructor positioning within the network. This study affirms that pedagogically meaningful information can be extracted from LMS-generated student tracking data, and discusses how these findings are informing the development of a customizable dashboard-like reporting tool for educators that will extract and visualize real-time data on student engagement and likelihood of success. Use of machine learning techniques for educational proposes (or educational data mining) is an emerging field aimed at developing methods of exploring data from computational educational settings and discovering meaningful patterns. The stored data (virtual courses, e-learning log file, demographic and academic data of students, admissions registration info, and so on) can be useful for machine learning algorithms. In this article, we cite the most current articles that use machine learning techniques for educational proposes and we present a case study for predicting students' marks. Students' key demographic characteristics and their marks in a small number of written assignments can constitute the training set for a regression method in order to predict the student's performance. Finally, a prototype version of software support tool for tutors has been constructed. Pearson product–moment correlations found significant relationships between students' grades in the online class and their GPA, attendance at a class orientation session, the number of previous course withdrawals, ASSET reading scores, the number of previous online courses, age, and ACT English scores. Regression analysis found that two variables serve as the best predictors: attendance at an orientation session, and the student's grade point average. Peer and self-assessment offer an opportunity to scale both assessment and learning to global classrooms. This article reports our experiences with two iterations of the first large online class to use peer and self-assessment. In this class, peer grades correlated highly with staff-assigned grades. The second iteration had 42.9p of students’ grades within 5p of the staff grade, and 65.5p within 10p. On average, students assessed their work 7p higher than staff did. Students also rated peers’ work from their own country 3.6p higher than those from elsewhere. We performed three experiments to improve grading accuracy. We found that giving students feedback about their grading bias increased subsequent accuracy. We introduce short, customizable feedback snippets that cover common issues with assignments, providing students more qualitative peer feedback. Finally, we introduce a data-driven approach that highlights high-variance items for improvement. We find that rubrics that use a parallel sentence structure, unambiguous wording, and well-specified dimensions have lower variance. After revising rubrics, median grading error decreased from 12.4p to 9.9p. To help solve the ongoing problem of student retention, new expected performance-prediction techniques are needed to facilitate degree planning and determine who might be at risk of failing or dropping a class. Personalized multiregression and matrix factorization approaches based on recommender systems, initially developed for e-commerce applications, accurately forecast students' grades in future courses as well as on in-class assessments. Crowdsourcing offers a practical method for ranking and scoring large amounts of items. To investigate the algorithms and incentives that can be used in crowdsourcing quality evaluations, we built CrowdGrader, a tool that lets students submit and collaboratively grade solutions to homework assignments. We present the algorithms and techniques used in CrowdGrader, and we describe our results and experience in using the tool for several computer-science assignments. CrowdGrader combines the student-provided grades into a consensus grade for each submission using a novel crowdsourcing algorithm that relies on a reputation system. The algorithm iterativerly refines inter-dependent estimates of the consensus grades, and of the grading accuracy of each student. On synthetic data, the algorithm performs better than alternatives not based on reputation. On our preliminary experimental data, the performance seems dependent on the nature of review errors, with errors that can be ascribed to the reviewer being more tractable than those arising from random external events. To provide an incentive for reviewers, the grade each student receives in an assignment is a combination of the consensus grade received by their submissions, and of a reviewing grade capturing their reviewing effort and accuracy. This incentive worked well in practice. The need for automated grading tools for essay writing and open-ended assignments has received increasing attention due to the unprecedented scale of Massive Online Courses (MOOCs) and the fact that more and more students are relying on computers to complete and submit their school work. In this paper, we propose an efficient memory networks-powered automated grading model. The idea of our model stems from the philosophy that with enough graded samples for each score in the rubric, such samples can be used to grade future work that is found to be similar. For each possible score in the rubric, a student response graded with the same score is collected. These selected responses represent the grading criteria specified in the rubric and are stored in the memory component. Our model learns to predict a score for an ungraded response by computing the relevance between the ungraded response and each selected response in memory. The evaluation was conducted on the Kaggle Automated Student Assessment Prize (ASAP) dataset. The results show that our model achieves state-of-the-art performance in 7 out of 8 essay sets. We have implemented a peer grading system for review of student assignments over the World Wide Web and used it in approximately eight courses in computing and electrical engineering. Students prepare their assignments and submit them to our Peer Grader (PG) system. Other students are then assigned to review and grade the assignments. The system allows authors and reviewers to communicate, with authors being able to update their submissions. Unique features of our approach include the ability to submit arbitrary sets of Web pages for review, and mechanisms for encouraging careful review of submissions. Electronic peer review facilitates collaborative learning in several ways. First, there is the obvious fact that students can learn from their reviewers' comments. Second, students help each other to improve their communication skills. Third, in team projects, peer review allows team members to be assessed by each other. Finally, peer review makes it possible to break up a large project into small chunks. In fact, new releases of PG are being developed in exactly this way. A system for automatic grading of programming assignments is described here. This grading system consists of a suite of Perl and Java programs, linked by a database, and driven by an interactive, user-grader controlled grading process. The rationale for the process, the process itself-and the process's interaction-with the programs, the database, and the user himself are discussed. This grading system has the ability to discover and accommodate unanticipated solutions by means of a grading process that is highly interactive with its user-the grader. It also has the ability to automatically accommodate a wide range of simple student errors, which can easily befuddle a more naive grading system. This system has been used since the Spring 2000 semester at Rutgers University, as a core part of the students' performance evaluation activities, in the Introduction to computer science course, where it routinely grades some 300-600 weekly assignments. The present study examined the associations of sleep patterns with multiple measures of academic achievement of undergraduate university students and tested whether sleep variables emerged as significant predictors of subsequent academic performance when other potential predictors, such as class attendance, time devoted to study, and substance use are considered. A sample of 1654 (55 female) full-time undergraduates 17 to 25 yrs of age responded to a self-response questionnaire on sleep, academics, lifestyle, and well-being that was administered at the middle of the semester. In addition to self-reported measures of academic performance, a final grade for each student was collected at the end of the semester. Univariate analyses found that sleep phase, morningness eveningness preference, sleep deprivation, sleep quality, and sleep irregularity were significantly associated with at least two academic performance measures. Among 15 potential predictors, stepwise multiple regression analysis identified 5 significant predictors of end-of-semester marks: previous academic achievement, class attendance, sufficient sleep, night outings, and sleep quality (R 2 = 0.14 and adjusted R 2 = 0.14, F(5, 1234) = 40.99, p< .0001). Associations between academic achievement and the remaining sleep variables as well as the academic, well-being, and lifestyle variables lost significance in stepwise regression. Together with class attendance, night outings, and previous academic achievement, self-reported sleep quality and self-reported frequency of sufficient sleep were among the main predictors of academic performance, adding an independent and significant contribution, regardless of academic variables and lifestyles of the students. (Author correspondence: ana.allen@ua.pt) The present study investigates whether the circadian preferences of students are related to their academic achievements. This study explores whether different class times affect students' achievement and examines the performance of students on final exams administered at 09:30 h for differences according to chronotype. A total of 1471 university students between 18 and 25 yrs of age responded to a morningness-eveningness questionnaire (MEQ), and data on their cumulative grade point averages (CGPA) were also collected from their transcripts. Some of the students in the sample attended classes during the first teaching period, which started at 08:00 h and ended at 14:50 h, and the remaining students followed the second schedule, which started at 15:00 h and ended at 21:50 h. MEQ scores were found to differ by sex. MEQ scores partially predicted academic success and that students' academic achievements differed according to the time of the teaching period. Moreover, final exam (administered at 09:30 h) score... This article describes a model and implementation of a system for automatically testing, evaluating, grading and providing critical feedback for submitted programming assignments. Complete automation of the evaluation process, with proper attention towards monitoring studentâ€™s progress and performing a finegrained analysis is addressed. The tool provides online support to both the instructors and students and is designed for service-oriented integration with a course management system using Web services. An automatic grading system is presented for grading assignments in scientific computing. A student can interactively use this system to check the correctness of his program assignments. The grade for a correct solution is automatically recorded. This paper also considers the security problems with such an automatic grading system. Abstract Automated Essay Grading (AEG) is defined as a computer technology that evaluates and scores written prose. A number of AEG systems have been developed since the 1960s and in most of them, an ad-hoc or generalized approach is used to grade spelling even though it is an important element of an essay-scoring rubric. Existing approaches do not therefore give an accurate representation or measure of spelling in essays. According to the rubric-based scoring method used in the National Assessment Program “Literacy and Numeracy (NAPLAN) in Australia, spelling is marked in three steps” first, by identifying the correct and incorrect words in the essay; second, by categorizing each word based on the difficulty level into one of four classes: Simple, Common, Difficult or Challenging, and counting the number of correct and incorrect words in each category; finally, by using the pre-defined NAPLAN rubric scale to assign the mark. Only a small number of existing AEG systems can be used for rubric-based scoring, and none can be used to grade spelling according to the NAPLAN rubric. In this paper, we address this shortcoming in the existing literature and present an innovative approach to automatically mark spelling using rubric based scoring. We develop two algorithms based on the rules and heuristics of the English language to formulize the rubric for spelling and then implement these algorithms in Java language and perform a series of evaluations of our system using an essay dataset. Our results are very promising, even though it is the first system of this kind. CrowdGrader is a system that lets students submit and collaboratively review and grade homework. We describe the techniques and ideas used in CrowdGrader, and report on the experience of using CrowdGrader in disciplines ranging from Computer Science to Economics, Writing, and Technology. In CrowdGrader, students receive an overall crowd-grade that reflects both the quality of their homework, and the quality of their work as reviewers. This creates an incentive for students to provide accurate grades and helpful reviews of other students' work. Instructors can use the crowd-grades as final grades, or fine-tune the grades according to their wishes. Our results on seven classes show that students actively participate in the grading and write reviews that are generally helpful to the submissions' authors. The results also show that grades computed by CrowdGrader are sufficiently precise to be used as the homework component of class grades. Students report that the main benefits in using CrowdGrader are the quality of the reviews they receive, and the ability to learn from reviewing their peers' work. Instructors can leverage peer learning in their classes, and easily handle homework evaluation in large classes. In massive open online courses (MOOCs), peer grading serves as a critical tool for scaling the grading of complex, open-ended assignments to courses with tens or hundreds of thousands of students. But despite promising initial trials, it does not always deliver accurate results compared to human experts. In this paper, we develop algorithms for estimating and correcting for grader biases and reliabilities, showing significant improvement in peer grading accuracy on real data with 63,199 peer grades from Coursera's HCI course offerings --- the largest peer grading networks analysed to date. We relate grader biases and reliabilities to other student factors such as student engagement, performance as well as commenting style. We also show that our model can lead to more intelligent assignment of graders to gradees. Curricula for Computer Science degrees are characterized by the strong occupational orientation of the discipline. In the BSc degree structure, with clearly separated CS core studies, learning skills between these and other required studies may vary a lot, showing in student’s overall performance. To analyze such a situation, we apply nonstandard educational data mining techniques on a preprocessed log file of the passed courses. The joint variation of the course grades are studied through correlation analysis while the intrinsic groups of students are created and analyzed using a special clustering technique. Since not all students have attended all the courses, there is a nonstructured sparsity pattern to cope with. Finally, multilayer perceptron neural network with cross-validation based generalization assurance is trained and analyzed using analytic mean sensitivity to explain the nonlinear regression model constructed. Local (within-methods) and global (between-methods) triangulation of different analysis methods is argued to improve the technical soundness of the presented approaches, giving more confidence on our final conclusion that general learning capabilities predict the success of students better than specific IT skills obtained as part of the core studies. Automatic assessment of programming exercises has become an important method for grading students' exercises and giving feedback for them in mass courses. We describe a system called Scheme-robo, which has been designed for assessing programming exercises written in the functional programming language Scheme. The system assesses individual procedures instead of complete programs. In addition to checking the correctness of students' solutions the system provides many different tools for analysing other things in the program like its structure and running time, and possible plagiarism. The system has been in production use on our introductory programming course with some 300 students for two years with good results. The state of Wisconsin has one of the highest four year graduation rates in the nation, but deep disparities among student subgroups remain. To address this the state has created the Wisconsin Dropout Early Warning System (DEWS), a predictive model of student dropout risk for students in grades six through nine. The Wisconsin DEWS is in use statewide and currently provides predictions on the likelihood of graduation for over 225,000 students. DEWS represents a novel statistical learning based approach to the challenge of assessing the risk of non-graduation for students and provides highly accurate predictions for students in the middle grades without expanding beyond mandated administrative data collections. Similar dropout early warning systems are in place in many jurisdictions across the country. Prior research has shown that in many cases the indicators used by such systems do a poor job of balancing the trade off between correct classification of likely dropouts and false-alarm (, 2013). Building on this work, DEWS uses the receiver-operating characteristic (ROC) metric to identify the best possible set of statistical models for making predictions about individual students. This paper describes the DEWS approach and the software behind it, which leverages the open source statistical language R (R Core Team, 2013). As a result DEWS is a flexible series of software modules that can adapt to new data, new algorithms, and new outcome variables to not only predict dropout, but also impute key predictors as well. The design and implementation of each of these modules is described in detail as well as the open-source R package, EWStools, that serves as the core of DEWS (Knowles, 2014). While computer and communication technologies have provided effective means to scale up many aspects of education, the submission and grading of assessments such as homework assignments and tests remains a weak link. In this paper, we study the problem of automatically grading the kinds of open response mathematical questions that figure prominently in STEM (science, technology, engineering, and mathematics) courses. Our data-driven framework for mathematical language processing (MLP) leverages solution data from a large number of learners to evaluate the correctness of their solutions, assign partial-credit scores, and provide feedback to each learner on the likely locations of any errors. MLP takes inspiration from the success of natural language processing for text data and comprises three main steps. First, we convert each solution to an open response mathematical question into a series of numerical features. Second, we cluster the features from several solutions to uncover the structures of correct, partially correct, and incorrect solutions. We develop two different clustering approaches, one that leverages generic clustering algorithms and one based on Bayesian nonparametrics. Third, we automatically grade the remaining (potentially large number of) solutions based on their assigned cluster and one instructor-provided grade per cluster. As a bonus, we can track the cluster assignment of each step of a multistep solution and determine when it departs from a cluster of correct solutions, which enables us to indicate the likely locations of errors to learners. We test and validate MLP on real-world MOOC data to demonstrate how it can substantially reduce the human effort required in large-scale educational platforms.