Massive Open Online Courses (MOOCs) use peer assessment to grade open ended questions at scale, allowing students to provide feedback. Relative to teacher based grading, peer assessment on MOOCs traditionally delivers lower quality feedback and fewer learner interactions. We present the identified peer review (IPR) framework, which provides non-blind peer assessment and incentives driving high quality feedback. We show that, compared to traditional peer assessment methods, IPR leads to significantly longer and more useful feedback as well as more discussion between peers.
However, with the potential benefits, peer grading still faces challenges. Providing accurate grading where the performance of a novice is being judged by other novices is problematic. Students mistrust peer grades and anonymity lead students to be unscrupulous. Reviews are often short and not well considered, which is especially problematic when students work hard on an assignment and receive comments they can't learn from @cite_19 .
@cite_19
2122588206 2106189547 2088700588 2506952740
In massive open online courses (MOOCs), peer grading serves as a critical tool for scaling the grading of complex, open-ended assignments to courses with tens or hundreds of thousands of students. But despite promising initial trials, it does not always deliver accurate results compared to human experts. In this paper, we develop algorithms for estimating and correcting for grader biases and reliabilities, showing significant improvement in peer grading accuracy on real data with 63,199 peer grades from Coursera's HCI course offerings --- the largest peer grading networks analysed to date. We relate grader biases and reliabilities to other student factors such as student engagement, performance as well as commenting style. We also show that our model can lead to more intelligent assignment of graders to gradees. We have implemented a peer grading system for review of student assignments over the World Wide Web and used it in approximately eight courses in computing and electrical engineering. Students prepare their assignments and submit them to our Peer Grader (PG) system. Other students are then assigned to review and grade the assignments. The system allows authors and reviewers to communicate, with authors being able to update their submissions. Unique features of our approach include the ability to submit arbitrary sets of Web pages for review, and mechanisms for encouraging careful review of submissions. Electronic peer review facilitates collaborative learning in several ways. First, there is the obvious fact that students can learn from their reviewers' comments. Second, students help each other to improve their communication skills. Third, in team projects, peer review allows team members to be assessed by each other. Finally, peer review makes it possible to break up a large project into small chunks. In fact, new releases of PG are being developed in exactly this way. Peer and self-assessment offer an opportunity to scale both assessment and learning to global classrooms. This article reports our experiences with two iterations of the first large online class to use peer and self-assessment. In this class, peer grades correlated highly with staff-assigned grades. The second iteration had 42.9p of students’ grades within 5p of the staff grade, and 65.5p within 10p. On average, students assessed their work 7p higher than staff did. Students also rated peers’ work from their own country 3.6p higher than those from elsewhere. We performed three experiments to improve grading accuracy. We found that giving students feedback about their grading bias increased subsequent accuracy. We introduce short, customizable feedback snippets that cover common issues with assignments, providing students more qualitative peer feedback. Finally, we introduce a data-driven approach that highlights high-variance items for improvement. We find that rubrics that use a parallel sentence structure, unambiguous wording, and well-specified dimensions have lower variance. After revising rubrics, median grading error decreased from 12.4p to 9.9p. Peer assessment is the most common approach to evaluating scientific work, and it is also gaining popularity for scaling evaluation of student work in large and distributed classes. The key idea is that each peer reviewer or grader rates a relatively small subset of the items, and that some method of manual, semi-automatic, or fully-automatic aggregation of all assessments defines the eventual rating of all items – the grade in peer grading, or whether to accept or reject a scientific manuscript. In this paper, we explore in how far a Bayesian Ordinal Peer Assessment (BOPA) method can provide additional decision support when making acceptance rejection decisions for a scientific conference. Using data from the 2015 ACM Conference on Knowledge Discovery and Data Mining (KDD), where this system was deployed, we discuss the potential merit of the BOPA approach compared to conventional decision support offered by the Microsoft Conference Management System (CMT).