The present invention is related to the field of standardized and assessment testing, and specifically to essay evaluation and scoring.
The use of standardized testing has increased steadily over the past several decades. Standardized tests are now an essential part of almost every educational system. These tests are used for in-grade competency and achievement evaluation, as teaching aides, as measurements of student performance, and for admission and qualification procedures.
These tests were often designed to be scored by machine methods. As a result, multiple choice tests were prevalent in the past. However, there has arisen an ever increasing trend away from multiple choice tests in favor of tests that require open-ended responses, such as essay tests. The assessment and scoring of essay responses to essay test questions has created more labor intensive evaluation systems and systems with increased numbers of decision making parameters.
Generally in the past, essay evaluation systems have operated in the context of a particular test administration. The systems were used to collect, distribute and grade actual test responses for a particular test. Groups of human scorers were engaged to read and score tests working from computer work stations.
Early efforts were directed to essay responses which were “fill-in-the-blank”. This fill-in method of essay testing response is less structured and more open ended than multiple choice response, but is an extension of multiple choice answer scoring. Possible correct answers are withheld from the user/test taker. Optical character recognition (OCR) was utilized to read the one or two word answer. The user/test taker answer was compared against a preferred answer and a list of less preferred answers. A score was assigned according to the degree of closeness to the preferred answer.
As students have became more interested in practicing their essay testing skills, and as teachers have began to use essay testing as a teaching aid, there has arisen a need for systems which provided the capability of practicing essay test taking skills, demonstrating content mastery, and providing feedback to the use/essay tester.
A writing evaluation system has been developed by Educational Testing Service, Princeton, N.J. (ETS), which provides on-line evaluation of a constructed essay response to an examination question. A test taker submits an essay response via the Internet. The test response is assigned to a human scorer at a workstation and enters his scoring queue. The scorer evaluates the essay according to assigned calibrated grading guidelines to select an overall evaluation. The scorer can also select from a list of pre-defined generalized comments which characterize the user/test takers submission. A selected comment accompanies the score provided to the user/test taker. A given scorer's performance on a particular essay answer may be evaluated by a scorer leader or supervisor.
The pre-selected comments vary from such overall characterizations as: “your essay is well-developed”, to “You offer an analysis of how literary devices are used . . . The essay would be stronger if the analysis were more penetrating”.
Although ETS's system is a variable method of evaluating essays through a human process, it is too labor intensive in the scoring and evaluating stage to be an economical offering when evaluating essays for instructional purposes. Moreover, the type of feedback comments offered are intended to be generalized and to provide an overall characterization of the essay response, i.e., on a macro-level, and not to be focused on the mechanics of essay drafting and the mechanics of organization and presentation on a micro-level.
A desired approach and objective of the present invention, which departs from the prior art, is an automated tutoring system which is capable of: (1) scoring a user/test taker's essay response, and of real-time, on-line, micro-evaluations of essay mechanics and organization; (2) providing essay feedback while the user is drafting an essay, on an extemporaneous basis; (3) factoring an open-ended essay response from a user into a plurality of separate conceptual areas of inquiry, for scoring with feedback; and (4) providing for a modification of operation and feedback to a writing prompt from one of a plurality of essay genre.