Unraveling the Mystery of Artifacts in Machine Generated Text

As neural Text Generation Models (TGM) have become more and more capable of generating text indistinguishable from human-written ones, the misuse of text generation technologies can have serious ramifications. Although a neural classifier often achieves high detection accuracy, the reason for it is not well studied. Most previous work revolves around studying the impact of model structure and the decoding strategy on ease of detection, but little work has been done to analyze the forms of artifacts left by the TGM. We propose to systematically study the forms and scopes of artifacts by corrupting texts, replacing them with linguistic or statistical features, and applying the interpretable method of Integrated Gradients. Comprehensive experiments show artifacts a) primarily relate to token co-occurrence, b) feature more heavily at the head of vocabulary, c) appear more in content word than stopwords, d) are sometimes detrimental in the form of number of token occurrences, e) are less likely to exist in high-level semantics or syntaxes, f) manifest in low concreteness values for higher-order n-grams.


Introduction
In recent years, neural Text Generation Models (TGM) have grown by leaps and bounds with the support of big data and escalating computing power (Brown et al., 2020). State-of-the-art models can generate fluent and genuine-looking text (Gehrmann et al., 2019;Dou et al., 2021), thus we need to be especially wary of the abuse of text generation techniques, such as producing fake news, impersonating others in email, phishing, etc. (Jawahar et al., 2020). Recently, Humans are worse at detecting synthetic text than Machines (Gehrmann et al., 2019;Solaiman et al., 2019) because Humans cannot properly perceive the difference in data distribution, but rely more on semantic errors or logical contradictions . This phenomenon has spawned much work investigating the use of neural classifiers to detect synthetic text, most of which analyze the impact of variables such as the structure of TGM, the decoding strategy, and the sentence length (Tay et al., 2020;Ippolito et al., 2020;Solaiman et al., 2019;Munir et al., 2021). Besides, a small amount of work studies TGM's property through its left behind artifacts (detectable signatures that originate from the TGM). (Tay et al., 2020) reveals that TGMs of different decoding configurations leave distinguishable artifacts when generating text, proving artifacts are not correlated with word order. Another work  claims artifacts exist in over-sampled high-likelihood words when TGM adopts decoding strategies of top-k sampling. Despite these sporadic findings, there is a lack of systematic analysis of the forms in which artifacts exist. To fill this void, we conduct an empirical study on three datasets to reveal characteristics and possible manifestations of artifacts. Specifically, we corrupt text in different ways or replace them with high-level linguis-* Corresponding author tic/statistical features, retaining or corrupting certain information of the text; we locate artifacts mainly based on the performance variation of the classifier against different corruptions. Besides, to track artifacts in the details, we compute the contribution of tokens by means of interpretable methods. We summarize our contribution as follows: -We open-sourced a Chinese Novel Dataset crAwled fRom mIxed online sOurces (CnDARIO) for artifacts discovery in the Chinese Literature domain.
-We offer empirical findings for artifacts: a) token cooccurrence is the predominant form in which artifacts exist, b) artifacts exist mainly at the head of the vocabulary, c) content words contain more artifacts than stopwords, d) the number of occurrences of tokens offers limited or even detrimental artifacts, e) high-level semantic/syntactic features contain much fewer artifacts than shallow features, f) some artifacts are present in higher-order n-grams in the form of low concreteness values, g) pre-training is extremely helpful for effective use of artifacts 1 .

Task Formulation
Though neural classifiers achieve extremely high accuracy on discerning synthetic text, the reason for it is not fully-studied. Our work attempts to explain this phenomenon by unveiling the forms in which artifacts exist. To this end, we adopt several text corruption/replacement operations, which may retain or corrupt certain linguistic or statistical information, as surrogates for unraveling artifacts. Formally, given a classification dataset D = (E 1 , . . . , E n ) containing both Human and Machine excerpts 2 , we corrupt and replace the original text in D with various operations. For each operation CO, we train the neural classifier f θ on the corrupted training set D co train , regarding the accuracy variation of f θ on D co test as a sensitivity indicator for the presence or the absence of certain linguistic/statistical features associated with CO. The high absolute value or the drastic change of accuracy helps us to attribute artifacts to specific linguistic/statistical features. This process can be comprehend as sensitivity analysis (Saltelli, 2002;Saltelli et al., 2008) of corruption/replacement operations (input) on the generalisation performance of f θ (output). As a practical example, suppose we apply the operation of 'token shuffle' to D, and f θ still manages to achieve high accuracy, we can conclude token co-occurrences in the corrupted D co contains enough distinguishable artifacts. Conversely, if there is a significant drop in accuracy, it is evident that the token order has detectable artifacts and there are relatively small differences in the distribution of word co-occurrence between genuine and synthetic text. Furthermore, we use interpretable methods based on Influence Functions (Koh and Liang, 2017) to directly analyze the connection between the classifier's decisions and the properties of the n-grams.

Text Corruption Operations
We define the corruption/replacement operation as CO : E → E co , where E co is a corrupted excerpt; we keep the length variable fixed, adding [MASK] token to E co when necessary. All text corruptions operate within the scope of E, with the token being the basic operating unit. Operations in the following sections are selected to cover as many forms in which artifacts may exist as possible.

Corruptions in the text form
Deduplicate tokens Given the original excerpt E, the deduplicate operation only keeps the first occurrence of token t, replacing all subsequent occurrences of t with the [MASK] token.
Shuffle tokens Given the original novel excerpt E, the shuffle operation produces a new excerpt E co with random token orders, i.e., ∀t ∈ E, t has a different position in E co .
Retain only (non)-stopwords Stopwords 3 are the most common words, and they are considered to retain abstract author signatures for Human authorship attribution (Rajkumar et al., 2009). We are curious whether the inclusion or exclusion of stopwords helps to attribute a Machine author-the TGM, too.
3 For Chinese stopwords, we adopt the Baidu stopword list https://github.com/goto456/stopwords, while for English, we combine the Baidu stopword list with NLTK stopword list https://gist.github.com/se bleier/554280. Retain tokens in high/low frequency regions To investigate the differences in token frequency between Human and Machine text, we design the frequency gap score as , where F req(t, D) is the frequency of a token t in the corpus D and Rank(t, D) is the frequency rank of t in descending order. D h and D m are subsets of D, containing Human written and Machine generated text respectively. Figure 1 shows that the frequency distri- bution between Human text and Machine text differs in each of the three datasets. Human and Machine use of high-frequency tokens is broadly similar on CnDARIO but Humans use low-frequency tokens more frequently, while the Writing Prompt dataset shows an opposite trend. On the CNN/Daily Mail dataset, the difference of token usage between Humans and Machines is consistently small across all frequency ranks. We introduce two corruption operations to further study the effect of retaining tokens within a certain frequency range. The operation to constrain tokens of E co in the high-frequency region satisfies where t i and t co i are the i-th token in E and E co , ρ ∈ N is the rank threshold to split the high and Origin 1 It is believed Congress is going to finish its authorization of NASA's budget. Replace text with likelihood ranks One of the characteristics of Human language is that it intermittently dips in and out of low probability zones . Inspired by this, we propose to replace text with likelihood ranks to verify whether the classifier can exploit this distinctiveness. The likelihood rank of an excerpt E can be obtained by querying a TGM. Concretely, at each position i, the TGM produces a probability distribution P (t i | E 1...i−1 ) of token t i , from which we obtain the likelihood rank r i of t i , constituting E co = (r 1 , . . . , r i , . . . ). For Out-Of-Vocabulary tokens, we use the rank of [UNK] for substitution. Replace text with specific linguistic features To test whether Machine-generated text differs significantly from Human text in certain linguistic features, we replace the original text form with part-of-speech, Dependency Trees and Constituent Trees. In experiments, Linguistic features of POS and Dependency Trees are obtained by spaCy 4 , while the Constituent Tree of text is parsed by Berkeley Neural Parser (benepar) (Kitaev et al., 2019). We adopt specific models of en_core_web_sm and zh_core_web_sm in spaCy, and benepar_en3 and benepar_zh2 in benepar.
• Part of Speech A part-of-speech (POS) is a category of words that have similar grammatical properties. It has been shown the distribution of POS n-grams is a useful feature to represent the textual style (Roemmele • Dependency Trees Dependency parsing is a syntactical parsing technique used to identify semantic relations between words in a sentence (Kübler et al., 2009). Such relations form a tree, with each word having exactly one head. We parse the original text input T into a list of dependency relations, with each followed by the position of its head word and tail word. The list of Dependency Trees is defined as E co = (dp 1 , head 1 , tail 1 , . . . , dp i , head i , tail i . . . ), dp i ∈ V dp , where dp i is a dependency relation, head i and tail i are the positions of head and tail word of dp i . The size of dependency relation Vocabulary V dp is 45.
• Constituent Trees Constituency parsing is another type of syntactical parsing, based on the formalism of context-free grammars. In the constituency parse tree, a sentence is divided into constituents of a specific category in the grammar, such as Verb phrase (VP) and Noun phrase (NP) (Jurafsky, 2000). We transform the original text input E into a string-form constituency parse tree E co , where only constituent categories and the tree structure are available. The vocabulary size of constituents is 136.
• Named Entities Named entities often include person names, organizations, locations, and etc. (Sekine and Ranchhod, 2009). We extract all named entities from the text with corresponding span locations. Given the original text input E, we extract named entities and form a NE list E co = (ne 1 , sp l1 , sp r 1 , . . . , ne i , sp li , sp r i . . . ), ne i ∈ V ne , where ne i is a named entity, sp li and sp r i are the left and right position index of the span of entity ne i . The total number of named entities V ne is 18.

Dataset
We collect and generate datasets of paired samples of Human-written and Machine-generated text from three sources. The first source is the CNN/Daily Mail  Table 2: Basic data analysis. Abbreviations of N. Sample, N. Unigram, and Avg.Len. stands for the number of samples, the number of unique unigram tokens and the average excerpt length in a dataset respectively. Distinct-1/2/3 is the Distinct value (Li et al., 2015) for unigram, bigram and trigram.
dataset (Nallapati et al., 2016), from which we randomly select the headline and the three subsequent sentences as the prefix. With the prefix, we use Grover-Mega (Zellers et al., 2019) to generate an excerpt by nucleus sampling (p = 0.95, t = 1.0) , pairing it with the original subsequent text. Similarly, we collected the readily available synthetic and Human excerpts from the second source -the Writing Prompt dataset (Fan et al., 2018), with synthetic text generated by the Fusion Model (Fan et al., 2018). The third source is CnDARIO dataset, the size of which is 30 gigabytes. We first solidly train the GPT-2 (Radford et al., 2019) on this dataset, with a vocabulary size of 13,762 and a context length of 1,024. Afterward, we query the pre-trained GPT-2 to generate synthetic novel excerpts. The length of the prefix and continuation is randomly chosen between 3 and 5. For the construction of all datasets, we include a post-processing step to ensure the genuine and the synthetic excerpt of the same prefix contains roughly the same amount of tokens, with a length difference of 10 tokens or less. We also provide the likelihood rank of the TGM when decoding each token for the CNN/Daily Mail and Cn-DARIO dataset.
In summary, we provide carefully constructed datasets of paired samples from three different domains: Online Forum, News, and Literature 5 . We believe these readily available datasets are friendly for students and labs with fewer computing resources. It is also worth noting that we provide an alternative form of data-likelihood ranks-that corresponds to the token, which allows researchers to easily investigate the effect of using likelihood rank to distinguish synthetic text.

Experimental setup
We adopt Roberta-base (Liu et al., 2019) as the classifier 6 because Roberta has been proved to be highly effective in detecting synthetic text and author attribu-tion (Uchendu et al., 2021;Uchendu et al., 2020). We choose subword (Byte-Pair-Encoding (Gage, 1994)) and character tokenization for English and Chinese text respectively. We split all datasets into three parts, 80% for training, 10% for validation, and the rest 10% for testing. We determine the best model based on its accuracy on the validation set, then report its performance on the test set. For every corruption or replacement operation, we repeat the training and evaluation process 15 times with different random seeds, with each seed controlling the data split, the shuffle order, the initialized parameters of the non-pre-trained model, etc.

Effect of corruptions and replacements
We define the abbreviations of corruptions and replacements as follows: (1)

Effect of text corruption operations
From Table 3, we summarise the following key findings. Erasing information on the number of occurrences of token hardly hurts the performance and even greatly improves the accuracy of the non-pretrained classifier. For all datasets, the difference in accuracy between classifiers trained on Dedup. and origin data is minimal. Interestingly, when the classifier is not pre-trained, the accuracy of Dedup. is much higher than the accuracy of origin on the Cn-DARIO and CNN/Daily Mail dataset. We speculate that Dedup. operation simplifies the text and reduces the interference of the number of token occurrences, making it easier for the classifier to concentrate on token co-occurrences.
The Boolean BOW features are sufficient for the classifier to detect synthetic text. The shuffle operation disrupts the semantic and syntactic structure of   the text, but even so, the performance loss of the classifier on two datasets is negligible. Decreased value of accuracy on the CnDARIO and Writing prompt dataset is merely 0.003 and 0.002 respectively, while the accuracy loss of 0.1 is significant on the CNN/Daily Mail dataset. The insensitivity of token order has been mentioned in several works (Pham et al., 2020;Sinha et al., 2021;Wang et al., 2018). Moreover, when we apply shuf. and Dedup. operations simultaneously, this loss of accuracy is still minimal (CnDARIO, Writing prompt) or acceptable (CNN/Daily Mail). The results indicate that the current TGM-generated text can easily be recognized by co-occurrence of tokens without using much advanced semantic and syntactic knowledge.
The content word contains more artifacts that can be learned than the stopwords. As the CNN/Daily Mail data has a large vocabulary, stopwords only account for a small percentage. When only retaining stopwords, the effective length of the text is very short, due to this, the classifier's performance degrades greatly. On the other two datasets, training the classifier with merely stopword or content words both yield acceptable accuracy, but the accuracy of in.stop is always lower than that of ex.stop. Figure 2 is generalized as follows: sufficiently accurate differentiation between Human and Machine-written text requires only a fraction of the vocabulary. Furthermore, on the CnDARIO and Writing Prompt dataset, we find the classifier trained with E co of highfrequency tokens is comparable with or better than that trained with E co of low-frequency tokens, even when the effective length of the former is much shorter. In contrast, the classifier shows the opposite pattern on the CNN/Daily mail dataset: the accuracy of the highfrequency region is slightly lower than that of the lowfrequency region when the effective length of the highfrequency region is longer (ρ = 400 and ρ = 800). Despite the inconsistencies, all classifiers achieve satisfactory accuracy (close to or significantly above 0.8) when only a small portion of the vocabulary-0.0014, 0.00135, 0.0012 for CnDARIO, CNN/Daily mail, and Writing Prompt respectively-is available.

Effect of text replacement operations
Based on the statistics in Table 2 and trends in Figure 1, we rank the quality and the complexity of text on Writing Prompt, CnDARIO, and CNN/Daily Mail from easy to difficult. Based on this quality ranking and the results in Figure 4, we conclude the following pattern: retaining only high-level semantic or syntactic features reduces the classifier's ability to recognize synthetic text, with the decrease of accuracy proportional to the text quality generated by TGM. Training with pure dependency trees and constituent trees obtain comparable accuracy while training with POS has mixed results on three datasets. Replacement with Named Entities achieve the worst result probably because the effective length of E co is too short, with only a few tokens in the text marked as Named Entities. Notably, the like.rank. operation has opposite results on CnDARIO and CNN/Daily Mail. When replacing text with likelihood ranks on CNN/Daily Mail data, the accuracy of the classifier is almost identical to that of the classifier trained from the original data, while replacing text with likelihood ranks on CnDARIO reduces the accuracy of the classifier by about 0.2. The low accuracy on CnDARIO is in better agreement with previous works Diwan et al., 2021), where results show that likelihood rank is a poor feature for neural classifiers to identify Machine-generated text.

Pre-training matters a lot
Model pre-training has become the preceding step for almost all NLP tasks and this paradigm indeed proves effective (Devlin et al., 2019;Liu et al., 2019;Qiu et al., 2020). Our results in Table 2 once again confirm the validity of pre-training. It is extremely helpful in improving the accuracy of detecting synthetic text, and the classifier without pre-training is even worse than the pre-trained one by more than 0.2 on accuracy (e.g. applying shuf. on CNN/Daily Mail data). (Aghajanyan et al., 2020) provides a theoretical analysis of why pretraining is so effective -attributing the benefits of pretraining to minimized intrinsic dimension.

Interpretability analysis
In addition to exploring the existence of artifact forms in a holistic manner, we also seek to find artifacts by cutting through the more localized details. We employ the Integrated Gradients (IG) 7 (Sundararajan et al., 2017) approach to assign an interpretable attribution score for each token t in an excerpt E, rendering a 7 We adopt the implementation of Captum (Kokhlikyan et al., 2020) and set the step of approximation to 500.
list of attribution scores of (a 1 , . . . , a i , . . . ), where a i is the sum of gradients of the classifier's prediction output to its embedding layer of t i . The higher the value of a i , the greater the contribution of t i to the classifier's prediction of a particular class.

The relationship between term frequency
and attribution score From Figure 2 we find that the classifier achieves high performance with only a portion of high-frequency tokens, therefore we would like to further verify whether classifiers assign higher attribution scores to highfrequency tokens. For each dataset, we randomly sampled 10, 000 excerpts and apply IG towards them. From Figure 3, we can see that the classifier does not deliver higher attribution scores to high-frequency tokens, instead, some of the tokens in the low-frequency region receive larger absolute values of the attribution score, probably because the number of tokens in the low-frequency region is much larger, leading to high variance. In Figure 3(a), low-frequency tokens have a positive attribution mean (around 0.1) for the Human label; Human text of CnDARIO containing more lowfrequency words (Figure 1) may contribute to this particular result. In summary, except for CnDARIO's Human label, attribution scores of unigram tokens basically maintain a normal distribution with zero as the mean regardless of the frequency variation. The results also demonstrate that the classifier does not pay special attention to a specific frequency band or certain high frequency tokens, which further supports the view that token co-occurrence is the most critical artifact.

Concreteness of highly attributed n-grams
We design experiments to explore whether n-grams tokens with high attribution values differ in concreteness. The concreteness of a word is defined as 'the degree : Univariate joint distribution of unigram token frequency (X-axis) and IG attribution score (Y-axis). The probability density is estimated by kernel density (KDE) (Terrell and Scott, 1992).
to which the concept denoted by a word refers to a perceptible entity' (Brysbaert et al., 2014;See et al., 2019). Words such as 'soybean' and 'liquid' have a high concreteness of 4.82 and 4.72 while word such as 'acceptableness' has a low concreteness of 1.28. For English, we adopt the concreteness ratings of 40,000 common English lemmas by (Brysbaert et al., 2014), with words' rating scaling from 1 to 5.14. For Chinese, we adopt concreteness ratings of 9,877 two-character Chinese words 8 . We use these ratings to measure the total concreteness of an n-grams by means of sliding windows (concreteness of tokens that has no rating is set to zero). We employ IG to calculate the attribution scores for all n-grams in E, averaged over the entire training set, and calculated the concreteness values for top attributed ngrams in both classes (Human, Machine). In Figure 4, we present the relationship between n-gram attribution ranks and their concreteness values, finding little difference in the mean values of concreteness between Human and Machine text, especially for the CNN/Daily Mail dataset. Moreover, we find an intriguing pattern on the CnDARIO and CNN/Daily Mail datasets: for top attributed n-grams, as n increases, the concrete-ness value of Machine text becomes lower and lower with respect to the mean value, and the classifier increasingly prefers lower concreteness values as cues (artifacts) for Machine-generated text. However, on the Writing Prompt dataset, we find that the concreteness value is always around the mean, regardless of the variation in n. We hypothesize that Human and Machine text on this dataset are already very different in terms of shallow features, leading to the classifier not learning more implicit features like concreteness.

Related work
The contrast of Humans' inability of detecting synthetic text and models' impressive performance has attracted many researchers.  using divergence frontiers. Shifting the analysis target, (Dugan et al., 2020) shows that Human annotators mainly use logic inconsistency, token repetition, etc. as cues for identifying synthetic text. The most relevant work to us is the work of (Tay et al., 2020); it exclude its presence of artifacts in word order but leaves other presence forms of artifacts under-explored.

Limitations and Future Work
Most of the experiments we have designed revolve around uncovering the form in which the artifacts of synthetic text appear, but do not provide a good answer as to why TGM generates certain forms of artifacts. In addition, some of our findings are not agreed across all datasets (e.g. replace with like.rank) and more detailed experiments are needed for generalizable conclusions. We study the attribution scores of continuous n-grams, but there is still no practical solution for attributing skipped n-grams through existing interpretable methods of Influence Functions 9 . As the experimental results indicate the co-occurrences of tokens suffices to make a distinction, the explanation of how the classifier works can start from pinpointing the decisive token set. Besides, we call for new metrics for scoring synthetic 9 The number of combinations grows exponentially with n, for instance, the total number of combinations of selecting 3 tokens from a sequence of length 256 is as high as 2, 763, 520.
text (e.g. the degree to which synthetic text can be easily detected by simple structure models and BOW features), new design of artifacts-inspired systems to aid Human judgment (e.g. prompt the user when a large number of n-grams of significantly lower concreteness appear) and greater use of simple classifiers that rely only on shallow features but are still effective-for detecting synthetic text-to minimize carbon emissions.

Conclusion
Currently, neural classifiers are far more capable than Humans at recognizing synthetic text. Most of the work addressing this phenomenon has been around the impact of factors such as TGMs' structure and decoding strategies on the ease of detection. Instead, we systematically investigate the presence of artifacts directly from the generated text, which helps gain insight into the TGMs. Detailed experiments on three datasets show that artifacts exist mostly in the form of surfacelevel semantics, and appear more in high-frequency tokens, but rarely present in the form of high-level semantics or syntax. In addition, we unexpectedly found artifacts in the form of concreteness in higher-order ngrams using an interpretable approach. For the dataset, we have open-sourced a high-quality paired Chinese novel dataset and also supplemented the CNN/Daily dataset with a new form of data, likelihood ranks that stem from the Grover-Mega model.