region:us
---
pretty_name: AG News Tweets
---


\subsection{Motivation}
AG News is a four-way topic classification task introduced in \cite{Zhang2015CharacterlevelCN}. In this setup, a task model must classify whether a given news article is about world events (\textbf{\textit{World}}), sports and athletics (\textbf{\textit{Sports}}), business and economics (\textbf{\textit{Business}}), and scientific developments (\textbf{\textit{Sci/Tech}}). The test set on HuggingFace (\url{huggingface.co/datasets/ag_news}) is composed of 7,600 examples equally balanced across the four classes. 

News topic classification presents a promising opportunity for largely isolating the effect of writing style shifts. Existing deep learning methods also perform well on this dataset with accuracy reaching higher than 90\% (\url{paperswithcode.com/sota/text-classification-on-ag-news}). 

Another motivation for this particular task is the common risk of data augmentation inadvertently flipping the label/semantics of the text \cite{Bayer2021ASO}. Unlike other tasks such as sentiment classification or subtle hate speech, the topic of a news article is unlikely to change during augmentation, thus preserving the original label.

\subsection{Creation}

We used GPT-3.5 Turbo \cite{brown2020language} (6/7/23 version) for style transfer. We did an initial pass through all 7,600 examples using a conservative "V1" prompt and greedy decoding. Calls were made using the OpenAI Python SDK with top\_p and temperature set to zero. The data was then lightly preprocessed to reduce the number of examples that began with \textbf{BREAKING NEWS} flanked my emojis.

512 of the initial model responses did not result in satisfactory generations. These were typical cases where the generated text was almost indiscernible from the original text or the generation was entirely emojis. We called GPT-3.5 Turbo again with an updated prompt and hyperparameters (temperature=0.7, top\_p=0.9, frequency\_penalty=0.5, presence\_penalty=0.5) for these examples. Whereas all the first-pass generations did not have any instructions to the model as to the sentiment/mood of the hypothetical post author, we purposefully instructed the model to "\textit{Add some flare with humor, anger, or sarcasm.}" in the generation.

It's important to note that we did not enforce Twitter's character limit. These sequences should be considered as more broadly inspired by social media posts rather than following the exact specifications of Twitter posts. We also did not manually review every sequence in the dataset to confirm that the original label was preserved. GPT 3.5 Turbo also hallucinates facts, such as adding the hashtag \#Olympics2021 even though the original dataset was created in 2015.