third quatrain and replaces “This Sea” (with a capital “S”) with “the sea.” Plus, the actual final line of the sonnet is significantly different; these surface-level problems caught my attention even before submitting it to GPTZero.
The chat bot also tends to elide in-text citations (though, again, this may change as the technology adapts). And, perhaps most notably for contemporary writing instructors, the essay reiterates only basic ideas or the most accepted interpretations similar to those on Sparknotes, Cliff Notes, and Wikipedia. I.e., it offers the most accepted reading and the main points easily found elsewhere, and, in terms of argumentation, the close reading is not an original or unique interpretation. The sample essay probably would not score an “A” based on expectations for college-level students to go beyond the commonplace reading, balance multiple interpretations at once, or to “read against the grain.”
Formulaic organization, simple interpretations, and basic errors when identifying the lines of a poem are not definitive proof that cheating has occurred, however. A student (or parent) might easily point out that the issues above are reasonable for a student writer.
GPTZero detects AI-generated text based on “perplexity” and “burstiness.” Perplexity refers to a probabilistic detection of unique words within natural language, including randomness and complexity. It also measures “burstiness,” or what rhetoricians commonly refer to as sentence variation (including random variations between complex, compound, simple, and compound complex sentences). GPTZero detects patterns in word choice and sentence length based on massive databases of stored information. In sum, human writers are often more random than a chat bot, though, obviously ChatGPT is not a static technology – it “learns” over time based on inputs.
GPTZero is currently free to use and generates a descriptive report that not only evaluates a sample text but also explains terminology.
In the sample text, GPTZero detected that the close reading of Wordsworth’s poem had a comparatively low rate of perplexity in several instances (see Figure 1) – an indication that certain parts of the text were not written by a human.
Repeated use of GPTZero with the same text found the same results:
- Perplexity: 16
- Perplexity (across sentences): 46.1
- The line with the highest perplexity: 151
- GPT Zero wrongly concluded that the text was human-generated (see Figure 2)
The last two bullet points are related. The most unique line in the essay came from a direct quote of the poem (it is actually two sentences, but it is missing punctuation). In other words, the AI-generated text included several quotes from a Romantic poet whose work is highly distinct (see Figure 3). When summarizing the results, this led to a false-negative report. The tool was fooled into believing all of the text was written by a human because several parts were in fact written by Wordsworth.
Submitting a Text without Direct Quotes
I submitted a close reading of the poem without direct quotations. This involved deleting quotes while still maintaining the integrity of the syntax in the original essay. The results were promising:
- Perplexity: 10
- Perplexity (across sentences): 32.8.
- The line with the highest perplexity: 58.
- GPT Zero correctly concluded that the text was AI-generated.
This indicates that GPTZero is an effective tool for recognizing when text is AI-generated. However, when using the tool, instructors need to be aware that false-negatives are highly possible in a sample text that quotes other material.
Finally, GPTZero users should acknowledge two important factors. First, GPTZero was “trained” on a previous iteration of ChatGPT and will likely become less effective over time. Second, GPTZero uses a limited, probabilistic method and cannot confirm its results with 100-percent accuracy. Students accused of using ChatGPT could easily point out that GPTZero is subject to errors in several cases.
Other Ways of Addressing AI-Generated Text
Instructors should also recognize that the