ng of Wordsworth’s poem had a comparatively low rate of perplexity in several instances (see Figure 1) – an indication that certain parts of the text were not written by a human.
Repeated use of GPTZero with the same text found the same results:
- Perplexity: 16
- Perplexity (across sentences): 46.1
- The line with the highest perplexity: 151
- GPT Zero wrongly concluded that the text was human-generated (see Figure 2)
The last two bullet points are related. The most unique line in the essay came from a direct quote of the poem (it is actually two sentences, but it is missing punctuation). In other words, the AI-generated text included several quotes from a Romantic poet whose work is highly distinct (see Figure 3). When summarizing the results, this led to a false-negative report. The tool was fooled into believing all of the text was written by a human because several parts were in fact written by Wordsworth.
Submitting a Text without Direct Quotes
I submitted a close reading of the poem without direct quotations. This involved deleting quotes while still maintaining the integrity of the syntax in the original essay. The results were promising:
- Perplexity: 10
- Perplexity (across sentences): 32.8.
- The line with the highest perplexity: 58.
- GPT Zero correctly concluded that the text was AI-generated.
This indicates that GPTZero is an effective tool for recognizing when text is AI-generated. However, when using the tool, instructors need to be aware that false-negatives are highly possible in a sample text that quotes other material.
Finally, GPTZero users should acknowledge two important factors. First, GPTZero was “trained” on a previous iteration of ChatGPT and will likely become less effective over time. Second, GPTZero uses a limited, probabilistic method and cannot confirm its results with 100-percent accuracy. Students accused of using ChatGPT could easily point out that GPTZero is subject to errors in several cases.
Other Ways of Addressing AI-Generated Text
Instructors should also recognize that they may need to modify their assignment prompts, ask for pre‑writing or drafts, or require more handwritten work to diminish use of ChatGPT.
For example, ChatGPT is less effective when writing about events, texts, or issues within the last few years (post-2020). So, asking a student to analyze a historical document and then to connect it to current events may impact their ability to use a chat bot in place of original work.
Alternatively, asking students to write assignments by hand or to turn in pre-writing and drafts will likely offset the use of ChatGPT. But students can obviously generate an automated text and work backward from it, either copying it by hand or creating an outline after the fact (which is something many students admit to doing in composition classes anyway).
In short, these are not fool-proof methods.
Conclusion: ChatGPT in the Classroom
Perhaps the best method to address automated text in classrooms is to change the mindset that instructors have when approaching the issue. Rather than viewing ChatGPT as yet another tool for plagiarism, instructors can adopt methods for integrating it into classwork.
For example, in a previous writing course, I asked students to generate poetry using Google Verse, which is a machine-learning chat bot similar to ChatGPT that is designed to write poetry specifically. One assignment required students to generate a poem using AI and then to analyze what is omitted in the process. The assignment recognizes that these technologies are highly effective, yet it shows students the value of individual thought, critical analysis, and creativity.
In a literary studies class, a similar assignment might ask students to read a ChatGPT-generated close reading of Wordsworth’s poetry. Then, students could write about what the chat bot has omitted (I have used a similar assignment with SparkNotes and Wikipedia entries). This acknowledges the value of balancing multiple, often contradictory, interpretations of a literary text while still finding meaning in imaginative writing. Mos