cpr was used in two neuroscience courses : an introductory neuroscience course ( 50 students , 86% first - years and sophomores ) and an upper level sensation and perception course ( 40 students , 95% juniors and seniors ) . in the introductory course , students wrote one summary essay based on a scholarly review and one argumentative essay based on a primary research article . in the upper level course , students composed two summary and synthesis essays and one argumentative essay , all of which were based on primary research articles of increasing sophistication .
information on the sample groups , student performance , and instructor grading demands is summarized in table 1 . before the first cpr assignment
, students completed an anonymous survey about their previous experience with peer review . at the end of the semester
, students completed another anonymous survey that asked for qualitative and quantitative feedback on their experiences with the cpr program .
specifically , students were asked to state what they liked most and least about the cpr program , and to rate on a 15 likert scale how much they felt that their writing and peer review skills improved over the semester , how often they provided and received helpful peer reviews , to what degree they felt that they were evaluated fairly by the cpr program , and how much work they put into cpr assignments , compared to writing assignments of equal length .
instructors can opt to author new assignments , or select assignments from an online library that includes at least 24 assignments on neuroscience related topics ( i.e. , cognitive neuroscience , psychopharmacology , ethics , cellular and molecular neuroscience , visual neuroscience , signal transduction , clinical neuroscience , and neuroethology ) . using a simple , online interface ,
1 ) , and links to relevant source materials ( e.g. , lecture notes , primary research articles , and scholarly search engines ) .
next , instructors compose high , low , and mid - quality calibration essays ( fig .
2 ) , set the evaluation parameters ( ten style and content questions that can be answered in either yes / no or none / one / some categorizations ; fig .
finally , instructors set the deadlines for text entry and peer review completion , and establish the grading criteria for the calibrations ( the number of specific questions a student must answer correctly , and the acceptable deviation range for the holistic evaluation of the essays ) .
excellent technical advice is available through email and a list - serve , and formal workshops on authoring successful assignments are offered on a regular basis .
before students begin any assignment , a mandatory , online interactive tutorial familiarizes students with using the cpr interface to complete each stage of the assignment .
completing cpr assignments requires three phases of student participation : 1 ) composing an essay , 2 ) calibrating reviews against standards , and 3 ) reviewing self and others .
students first compose and electronically enter their essays by a certain deadline ; this can be done from any internet - ready computer .
second , students evaluate three instructor - provided calibration essays holistically by numerically rating the essay from 110 , and specifically by answering content and style questions set by the instructor ( fig .
students learn how their calibration evaluations compare with the instructor s evaluation ( i.e. , the number of style and content questions answered correctly , and the deviation from the instructor s holistic score for each calibration essay . )
if students do not  pass  a calibration evaluation , they have one more opportunity to revise their assessments before receiving their official scores on the calibration section .
this second feedback in the calibration phase includes extensive comments written by the instructor to clarify student understanding of the judgment criteria .
thus , all students in the class receive the same extensive training in peer review prior to evaluating others essays .
students evaluate the essays according to the same criteria as the calibration essays and provide written justifications for each answer . if a student does not perform satisfactorily on the calibrated reviews , then his or her peer reviews do not factor into the calculation of the peer - reviewed student text ratings .
finally , after reading six essays on the same topic , students evaluate their own essays using the evaluation parameters . immediately after the deadline for the completion of the peer and self reviews , students are able to access their peers narrative comments and receive a comprehensive score for the assignment based on four factors : essay quality , calibration review accuracy , peer review accuracy , and self review accuracy .
instructors determine weight of grading components ; for both classes i weighted text 15% ( 5% each essay ) , the accuracy of three peer reviews quality 60% , the accuracy of the three calibration reviews 15% ( 5% each essay ) , and the accuracy of the self review 10% .
a scoring algorithm flags the instructor to any potential errors in analysis ( e.g. , essays that are reviewed by fewer than three peers , peer reviews that vary significantly in holistic evaluation , or essays that are reviewed by students who have done poorly in the calibration stage ) . throughout all stages of the assignment ,
the instructor has access to all student work , including text entries , calibration and peer reviews .
instructors also know which students reviewed each essay , and are able to change essay scores and edit assignment grades .
the instructor can provide written commentary about grade changes , and has the option of making this logic visible to the students who peer reviewed the essay in question , or just to the author of the essay .
these results are permanently stored in a user - friendly database on a ucla server , and can be accessed by the instructor at any time .
the mean text ratings ( generated by the weighted average of three anonymous peer reviews ) and the mean deviations on calibrations , peer reviews , and self reviews are listed in table 1 .
the cpr - calculated text ratings were within 0.5 points ( on a ten - point scale ) of the instructor s assessment 75% of the time .
when the essays were rescored , the changes usually improved the students text ratings , consistent with the finding that peer assessment of writing tends to be slightly lower than an instructor or teaching assistant s evaluation ( stefani , 1994 ) . in both classes ,
the average deviation from the actual text rating was greatest in the calibration phase and least in the peer review phase ( table 1 ) , suggesting that the calibration process did indeed function successfully as a training mechanism .
however , the mean deviation for the self reviews , performed after the peer reviews , was greater than the mean deviation in peer review stage ; students usually rated their own essays higher than their peers assessment of the text .
similar to the findings of falchikov and goldfinch ( 2000 ) , advanced students in the 300-level class did not perform any better at assessments than the beginner students in the introductory class , suggesting that peer review is not necessarily a skill served in the normal course of undergraduate academic training .
student quantitative and qualitative assessments of the cpr program are summarized in figure 5 and table 2 .
students self - reported that both their writing and peer review skills improved  somewhat  over the course of the semester ( fig .
usually  provided helpful comments in peer reviews , whereas they only  sometimes  received helpful comments , and only  sometimes  felt like their essays were being evaluated fairly ( fig . 5 ) . however , most all students agreed that , compared to other writing assignments of equal length , they spent much more time on cpr assignments .
not surprisingly , when asked to name their primary dislike of the cpr program , students cited  too much work  more often than any other answer ( table 2 ) .
the students , by virtue of reviewing the essays in a double - blind anonymous format , lacked the requisite information to 
students occasionally did poor jobs at evaluating essays , but in these instances , their peer reviews were usually automatically discounted by the cpr program because these students also tended to fail the calibration section .
the added motivation of being graded on the reliability of one s peer review further encouraged consistency and thoughtfulness in the peer review process .
nonetheless , students did cite  grade - scheming and attempts to  second guess the scoring algorithm as a major disadvantage of the program .
for example , when the  acceptable  deviation for a peer review was 2 , some students tended to rate a very good ( 9 or 10 quality ) essay as an 8 in order to maximize their acceptable deviation range .
importantly , students voiced very few negative comments about cpr being inaccessible , hard to understand , or technically challenging .
although more students stated that the online format was a deterrent ( 5 ) than an advantage ( 2 ) , technical difficulty with the program itself was never mentioned as a concern . by far , most student confusion stemmed from ambiguities in the criteria used to evaluate essays ( fig .
2 ) ; specifically , students struggled to answer the evaluation questions with either yes or no answers . to address this concern , i offered full credit for a calibration review if a student holistically scored the essay within the acceptable range , but answered more of the content or style questions incorrectly than was permitted , only if the student was able to justify his or her answers with examples from the text .
another strategy to minimize confusion about the evaluation parameters is to spend time in class collectively evaluating a sample essay before students begin the first assignment , as peer assessments have been found to resemble instructor assessments most closely when judgments are based on well understood criteria ( falchikov and goldfinch , 2000 ) . despite student critiques of the program ,
i still recommend the cpr program as an excellent and free resource for incorporating more writing , peer review , and critical thinking into an undergraduate neuroscience curriculum .
cpr s online data management system also has many practical advantages to the traditional paper collection method of peer review .
first , the random double - blind distribution of peer essays simplifies a very difficult organizational feat and guarantees student anonymity .
second , the saved database of peer reviews is a great source of information for writing recommendation letters and for calculating participation grades in large classes .
third , the online submissions process eliminates problems with lost essays or broken printers , and the password - protected website for the class allows students to print off specific articles for the assignments without the burden of going to the library reserves , or worrying about copyright violations .
although i did not specifically measure whether learning was improved in topics covered by cpr assignments , many students cited a deepened understanding of the material as their primary like of the cpr program ( fig . 5 ) .
other studies have shown quantitative learning gains in topics presented in cpr assignments , as compared to those topics covered in didactic lectures or in active learning formats ( pelaez , 2002 ) .
perhaps one of the greatest advantages of cpr is the availability of immediate feedback on assignments .
learning is generally improved by detailed and timely feedback on student work ( brown et al . , 1995 ) , and using cpr , students received three detailed peer assessments of their work within five minutes of the deadline for completing the peer and self reviews . furthermore
, a quick online review of the essays allowed me to identify and address common content misconceptions in class and common errors in writing in the next set of calibration essays .
my two primary critiques of the cpr program are that 1 ) there is no process for revision built into the cpr program , and 2 ) the peer review component of the grade is determined by the quality of the students quantitative , but not qualitative , feedback .
thus , students can receive full credit for reviews that are sloppy , pithy , or cruel in content , but within the allotted margin of error . to address the first issue , i offered students an opportunity to revise their essays for extra credit ( a maximum of 40% of the difference between their grade and a perfect score ) .
sixty- three percent of students in the introductory class opted to revise at least one essay , whereas only 40% of students in the upper level class chose to revise an essay ( table 1 ) . this added revision process allowed me to provide students with more personal attention , and conversations with students about what peer advice to adopt and what suggestions to ignore were generally very fruitful . to address the second issue , i advised students that the quality of their written peer review comments would factor heavily into their class participation grade , 5% of their overall grade for the course .
although cpr was designed as a timesaving device for instructors , the workload associated with introducing the program and discussing the evaluation parameters using example essays ( 1.5 class periods ) , creating the assignments and the calibration essays ( 810 hours ) , and re - evaluating text ratings and contested calibration and peer reviews ( 48 hours ) is considerable .
of course , repeating old assignments , using an assignment from the cpr library , or hiring a qualified student assistant to compose the calibration essays would significantly expedite this process .
nonetheless , cpr still requires less time than critically responding to 30 + student essays by hand , and provides far more direct experience for the students in abstract reasoning and peer review .
results of the student evaluations indicate that cpr fostered a multi - dimensional comprehension of the course material while teaching traditionally underserved academic skills : science writing and peer review .
over the course of the semester , i observed that students purpose for writing shifted from writing for the professor , to writing a clear argument for a general audience .
indeed , one of the downfalls of traditional instructor review is that comments by instructors detract students attention from their own intentions in writing , and focus that attention instead on  the teachers purpose in commenting 
cpr certainly requires more student engagement and autonomy in the writing and review process than traditional assignments .
cpr was used in two neuroscience courses : an introductory neuroscience course ( 50 students , 86% first - years and sophomores ) and an upper level sensation and perception course ( 40 students , 95% juniors and seniors ) . in the introductory course , students wrote one summary essay based on a scholarly review and one argumentative essay based on a primary research article . in the upper level course , students composed two summary and synthesis essays and one argumentative essay , all of which were based on primary research articles of increasing sophistication .
information on the sample groups , student performance , and instructor grading demands is summarized in table 1 . before the first cpr assignment
, students completed an anonymous survey about their previous experience with peer review . at the end of the semester
, students completed another anonymous survey that asked for qualitative and quantitative feedback on their experiences with the cpr program .
specifically , students were asked to state what they liked most and least about the cpr program , and to rate on a 15 likert scale how much they felt that their writing and peer review skills improved over the semester , how often they provided and received helpful peer reviews , to what degree they felt that they were evaluated fairly by the cpr program , and how much work they put into cpr assignments , compared to writing assignments of equal length .
instructors can opt to author new assignments , or select assignments from an online library that includes at least 24 assignments on neuroscience related topics ( i.e. , cognitive neuroscience , psychopharmacology , ethics , cellular and molecular neuroscience , visual neuroscience , signal transduction , clinical neuroscience , and neuroethology ) . using a simple , online interface ,
1 ) , and links to relevant source materials ( e.g. , lecture notes , primary research articles , and scholarly search engines ) .
next , instructors compose high , low , and mid - quality calibration essays ( fig .
2 ) , set the evaluation parameters ( ten style and content questions that can be answered in either yes / no or none / one / some categorizations ; fig .
finally , instructors set the deadlines for text entry and peer review completion , and establish the grading criteria for the calibrations ( the number of specific questions a student must answer correctly , and the acceptable deviation range for the holistic evaluation of the essays ) .
excellent technical advice is available through email and a list - serve , and formal workshops on authoring successful assignments are offered on a regular basis .
before students begin any assignment , a mandatory , online interactive tutorial familiarizes students with using the cpr interface to complete each stage of the assignment .
completing cpr assignments requires three phases of student participation : 1 ) composing an essay , 2 ) calibrating reviews against standards , and 3 ) reviewing self and others .
students first compose and electronically enter their essays by a certain deadline ; this can be done from any internet - ready computer .
second , students evaluate three instructor - provided calibration essays holistically by numerically rating the essay from 110 , and specifically by answering content and style questions set by the instructor ( fig .
students learn how their calibration evaluations compare with the instructor s evaluation ( i.e. , the number of style and content questions answered correctly , and the deviation from the instructor s holistic score for each calibration essay . ) if students do not  pass  a calibration evaluation , they have one more opportunity to revise their assessments before receiving their official scores on the calibration section .
this second feedback in the calibration phase includes extensive comments written by the instructor to clarify student understanding of the judgment criteria .
thus , all students in the class receive the same extensive training in peer review prior to evaluating others essays .
students evaluate the essays according to the same criteria as the calibration essays and provide written justifications for each answer . if a student does not perform satisfactorily on the calibrated reviews , then his or her peer reviews do not factor into the calculation of the peer - reviewed student text ratings .
finally , after reading six essays on the same topic , students evaluate their own essays using the evaluation parameters .
immediately after the deadline for the completion of the peer and self reviews , students are able to access their peers narrative comments and receive a comprehensive score for the assignment based on four factors : essay quality , calibration review accuracy , peer review accuracy , and self review accuracy . instructors determine weight of grading components ; for both classes i weighted text 15% ( 5% each essay ) , the accuracy of three peer reviews quality 60% , the accuracy of the three calibration reviews 15% ( 5% each essay ) , and the accuracy of the self review 10% .
a scoring algorithm flags the instructor to any potential errors in analysis ( e.g. , essays that are reviewed by fewer than three peers , peer reviews that vary significantly in holistic evaluation , or essays that are reviewed by students who have done poorly in the calibration stage ) . throughout all stages of the assignment ,
the instructor has access to all student work , including text entries , calibration and peer reviews .
instructors also know which students reviewed each essay , and are able to change essay scores and edit assignment grades
. the instructor can provide written commentary about grade changes , and has the option of making this logic visible to the students who peer reviewed the essay in question , or just to the author of the essay .
these results are permanently stored in a user - friendly database on a ucla server , and can be accessed by the instructor at any time .
the mean text ratings ( generated by the weighted average of three anonymous peer reviews ) and the mean deviations on calibrations , peer reviews , and self reviews are listed in table 1 .
the cpr - calculated text ratings were within 0.5 points ( on a ten - point scale ) of the instructor s assessment 75% of the time .
when the essays were rescored , the changes usually improved the students text ratings , consistent with the finding that peer assessment of writing tends to be slightly lower than an instructor or teaching assistant s evaluation ( stefani , 1994 ) . in both classes ,
the average deviation from the actual text rating was greatest in the calibration phase and least in the peer review phase ( table 1 ) , suggesting that the calibration process did indeed function successfully as a training mechanism .
however , the mean deviation for the self reviews , performed after the peer reviews , was greater than the mean deviation in peer review stage ; students usually rated their own essays higher than their peers assessment of the text .
similar to the findings of falchikov and goldfinch ( 2000 ) , advanced students in the 300-level class did not perform any better at assessments than the beginner students in the introductory class , suggesting that peer review is not necessarily a skill served in the normal course of undergraduate academic training .
student quantitative and qualitative assessments of the cpr program are summarized in figure 5 and table 2 .
students self - reported that both their writing and peer review skills improved  somewhat  over the course of the semester ( fig .
usually  provided helpful comments in peer reviews , whereas they only  sometimes  received helpful comments , and only  sometimes  felt like their essays were being evaluated fairly ( fig . 5 ) . however , most all students agreed that , compared to other writing assignments of equal length , they spent much more time on cpr assignments .
not surprisingly , when asked to name their primary dislike of the cpr program , students cited  too much work  more often than any other answer ( table 2 ) .
however , students did report an increased sense of empathy for instructor workload . the second most common complaint was the sense of being graded 
the students , by virtue of reviewing the essays in a double - blind anonymous format , lacked the requisite information to 
students occasionally did poor jobs at evaluating essays , but in these instances , their peer reviews were usually automatically discounted by the cpr program because these students also tended to fail the calibration section .
the added motivation of being graded on the reliability of one s peer review further encouraged consistency and thoughtfulness in the peer review process .
nonetheless , students did cite  grade - scheming and attempts to  second guess the scoring algorithm as a major disadvantage of the program .
for example , when the  acceptable  deviation for a peer review was 2 , some students tended to rate a very good ( 9 or 10 quality ) essay as an 8 in order to maximize their acceptable deviation range .
importantly , students voiced very few negative comments about cpr being inaccessible , hard to understand , or technically challenging .
although more students stated that the online format was a deterrent ( 5 ) than an advantage ( 2 ) , technical difficulty with the program itself was never mentioned as a concern . by far , most student confusion stemmed from ambiguities in the criteria used to evaluate essays ( fig .
2 ) ; specifically , students struggled to answer the evaluation questions with either yes or no answers . to address this concern , i offered full credit for a calibration review if a student holistically scored the essay within the acceptable range , but answered more of the content or style questions incorrectly than was permitted , only if the student was able to justify his or her answers with examples from the text .
another strategy to minimize confusion about the evaluation parameters is to spend time in class collectively evaluating a sample essay before students begin the first assignment , as peer assessments have been found to resemble instructor assessments most closely when judgments are based on well understood criteria ( falchikov and goldfinch , 2000 ) .
despite student critiques of the program , i still recommend the cpr program as an excellent and free resource for incorporating more writing , peer review , and critical thinking into an undergraduate neuroscience curriculum .
cpr s online data management system also has many practical advantages to the traditional paper collection method of peer review .
first , the random double - blind distribution of peer essays simplifies a very difficult organizational feat and guarantees student anonymity .
second , the saved database of peer reviews is a great source of information for writing recommendation letters and for calculating participation grades in large classes .
third , the online submissions process eliminates problems with lost essays or broken printers , and the password - protected website for the class allows students to print off specific articles for the assignments without the burden of going to the library reserves , or worrying about copyright violations .
although i did not specifically measure whether learning was improved in topics covered by cpr assignments , many students cited a deepened understanding of the material as their primary like of the cpr program ( fig . 5 ) .
other studies have shown quantitative learning gains in topics presented in cpr assignments , as compared to those topics covered in didactic lectures or in active learning formats ( pelaez , 2002 ) .
perhaps one of the greatest advantages of cpr is the availability of immediate feedback on assignments .
learning is generally improved by detailed and timely feedback on student work ( brown et al . , 1995 ) , and using cpr , students received three detailed peer assessments of their work within five minutes of the deadline for completing the peer and self reviews .
furthermore , a quick online review of the essays allowed me to identify and address common content misconceptions in class and common errors in writing in the next set of calibration essays .
my two primary critiques of the cpr program are that 1 ) there is no process for revision built into the cpr program , and 2 ) the peer review component of the grade is determined by the quality of the students quantitative , but not qualitative , feedback .
thus , students can receive full credit for reviews that are sloppy , pithy , or cruel in content , but within the allotted margin of error . to address the first issue , i offered students an opportunity to revise their essays for extra credit ( a maximum of 40% of the difference between their grade and a perfect score ) .
sixty- three percent of students in the introductory class opted to revise at least one essay , whereas only 40% of students in the upper level class chose to revise an essay ( table 1 ) .
this added revision process allowed me to provide students with more personal attention , and conversations with students about what peer advice to adopt and what suggestions to ignore were generally very fruitful . to address the second issue , i advised students that the quality of their written peer review comments would factor heavily into their class participation grade , 5% of their overall grade for the course .
although cpr was designed as a timesaving device for instructors , the workload associated with introducing the program and discussing the evaluation parameters using example essays ( 1.5 class periods ) , creating the assignments and the calibration essays ( 810 hours ) , and re - evaluating text ratings and contested calibration and peer reviews ( 48 hours ) is considerable .
of course , repeating old assignments , using an assignment from the cpr library , or hiring a qualified student assistant to compose the calibration essays would significantly expedite this process .
nonetheless , cpr still requires less time than critically responding to 30 + student essays by hand , and provides far more direct experience for the students in abstract reasoning and peer review .
results of the student evaluations indicate that cpr fostered a multi - dimensional comprehension of the course material while teaching traditionally underserved academic skills : science writing and peer review .
over the course of the semester , i observed that students purpose for writing shifted from writing for the professor , to writing a clear argument for a general audience .
indeed , one of the downfalls of traditional instructor review is that comments by instructors detract students attention from their own intentions in writing , and focus that attention instead on  the teachers purpose in commenting 
cpr certainly requires more student engagement and autonomy in the writing and review process than traditional assignments .
 although the majority of scientific information is communicated in written form , and peer review is the primary process by which it is validated , undergraduate students may receive little direct training in science writing or peer review . 
 here , i describe the use of calibrated peer review ( cpr ) , a free , web - based writing and peer review program designed to alleviate instructor workload , in two undergraduate neuroscience courses : an upper- level sensation and perception course ( 41 students , three assignments ) and an introductory neuroscience course ( 50 students ; two assignments ) . using cpr online , students reviewed primary research articles on assigned  hot topics , wrote short essays in response to specific guiding questions , reviewed standard  calibration essays , and provided anonymous quantitative and qualitative peer reviews . 
 an automated grading system calculated the final scores based on a student s essay quality ( as determined by the average of three peer reviews ) and his or her accuracy in evaluating 1 ) three standard calibration essays , 2 ) three anonymous peer reviews , and 3 ) his or her self review . 
 thus , students were assessed not only on their skill at constructing logical , evidence - based arguments , but also on their ability to accurately evaluate their peers writing . according to both student self - reports and instructor observation , students writing and peer review skills improved over the course of the semester . 
 student evaluation of the cpr program was mixed ; while some students felt like the peer review process enhanced their understanding of the material and improved their writing , others felt as though the process was biased and required too much time . despite student critiques of the program 
 , i still recommend the cpr program as an excellent and free resource for incorporating more writing , peer review , and critical thinking into an undergraduate neuroscience curriculum . 