Sebastian Williams, PhD
Instructors can use several methods to detect whether a text has been authored by a chat bot such as ChatGPT, including both heuristic approaches and GPTZero. Additionally, instructors can limit the use of ChatGPT and other chat bots by modifying assignment prompts, asking for pre-writing (e.g., outlines) and early drafts, or by requiring more handwritten work.
Ultimately, however, faculty should try to integrate these technologies into the learning process. ChatGPT does not fully eliminate the need for problem solving or navigating complex issues, and, in many cases, it frees up writers to worry less about grammar or style than higher order issues such as developing original claims or expressing an individual position.
Heuristic Methods for Detection
ChatGPT can be detected by many writing instructors based on organization, word choice, and the originality of ideas.
For example, when asked to write a close reading of William Wordsworth’s “The World is Too Much with Us,” ChatGPT authored a relatively formulaic response that moved from the start of the poem to the end (i.e., chronologically) and was unable to produce a coherent argument when asked to organize topically instead. (See Appendix for the sample text created by Chat GPT.)
This reflects another tendency in the current iteration of ChatGPT to organize short essays using a tripartite structure. For example, ChatGPT now tends to write “First, X occurred [. . .] Then, Y happened [. . .] Finally, Z was present.” Though machine-learning programs are able to adapt over time, some writing experts have noted a simple, formulaic organization method (“X, Y, Z”). This may only be true if a user does not modify an original response (e.g., by asking ChatGPT to write more).
Other basic issues indicated that the text was automated: the AI-generated text misquotes the final line of Wordsworth’s poem as “the sea that bares her bosom to the moon; / The winds that will be howling at all hours.” The bot includes two lines (not a single line) from the third quatrain and replaces “This Sea” (with a capital “S”) with “the sea.” Plus, the actual final line of the sonnet is significantly different; these surface-level problems caught my attention even before submitting it to GPTZero.
The chat bot also tends to elide in-text citations (though, again, this may change as the technology adapts). And, perhaps most notably for contemporary writing instructors, the essay reiterates only basic ideas or the most accepted interpretations similar to those on Sparknotes, Cliff Notes, and Wikipedia. I.e., it offers the most accepted reading and the main points easily found elsewhere, and, in terms of argumentation, the close reading is not an original or unique interpretation. The sample essay probably would not score an “A” based on expectations for college-level students to go beyond the commonplace reading, balance multiple interpretations at once, or to “read against the grain.”
Formulaic organization, simple interpretations, and basic errors when identifying the lines of a poem are not definitive proof that cheating has occurred, however. A student (or parent) might easily point out that the issues above are reasonable for a student writer.
GPTZero detects AI-generated text based on “perplexity” and “burstiness.” Perplexity refers to a probabilistic detection of unique words within natural language, including randomness and complexity. It also measures “burstiness,” or what rhetoricians commonly refer to as sentence variation (including random variations between complex, compound, simple, and compound complex sentences). GPTZero detects patterns in word choice and sentence length based on massive databases of stored information. In sum, human writers are often more random than a chat bot, though, obviously ChatGPT is not a static technology – it “learns” over time based on inputs.
GPTZero is currently free to use and generates a descriptive report that not only evaluates a sample text but also explains terminology.
In the sample text, GPTZero detected that the close readi