A Deep Generative Framework for Paraphrase Generation

Paraphrase generation is an important problem in NLP, especially in question answering, information retrieval, information extraction, conversation systems, to name a few. In this paper, we address the problem of generating paraphrases automatically. Our proposed method is based on a combination of deep generative models (VAE) with sequence-to-sequence models (LSTM) to generate paraphrases, given an input sentence. Traditional VAEs when combined with recurrent neural networks can generate free text but they are not suitable for paraphrase generation for a given sentence. We address this problem by conditioning the both, encoder and decoder sides of VAE, on the original sentence, so that it can generate the given sentence's paraphrases. Unlike most existing models, our model is simple, modular and can generate multiple paraphrases, for a given sentence. Quantitative evaluation of the proposed method on a benchmark paraphrase dataset demonstrates its efficacy, and its performance improvement over the state-of-the-art methods by a significant margin, whereas qualitative human evaluation indicate that the generated paraphrases are well-formed, grammatically correct, and are relevant to the input sentence. Furthermore, we evaluate our method on a newly released question paraphrase dataset, and establish a new baseline for future research.

Introduction
Paraphrase generation is an important problem in many NLP applications such as question answering, information retrieval, information extraction, and summarization. QA systems are often susceptible to the way questions are asked; in fact, for knowledge-based (KB) QA systems, question paraphrasing is crucial for bridging the gap between questions asked by users and knowledge based assertions (Fader, Zettlemoyer, and Etzioni 2014;Yin et al. 2015). In an open QA system pipeline, question analysis and paraphrasing is a critical first step, in which a given question is reformulated by expanding it with its various paraphrases with the intention of improvement in recall, an important metric in the early stage of the pipeline. Similarly paraphrasing finds applications in information retrieval by generating query variants, and in machine translation or summarization by generating variants for automatic evaluation.
Copyright c 2018, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.
In addition to being directly useful in QA systems, paraphrase generation is also important for generating training data for various learning tasks, such as question type classification, paraphrase detection, etc., that are useful in other applications. Question type classification has application in conversation systems, while paraphrase detection is an important problem for translation, summarization, social QA (finding closest question to FAQs/already asked question) (Figueroa and Neumann 2013). Due to the nature and complexity of the task, all of these problems suffer from lack of training data, a problem that can readily benefit from the paraphrase generation task.
Despite the importance of the paraphrase generation problem, there has been relatively little prior work in the literature, though much larger amount of work exists on paraphrase detection problem. Traditionally, paraphrase generation has been addressed using rule-based approaches (McKeown 1983a;Zhao et al. 2009), primarily due to the inherent difficulty of the underlying natural language generation problem. However, recent advances in deep learning, in particular generative models (Bowman et al. 2015;Chung et al. 2015), have led to powerful, data-driven approaches to text generation.
In this paper, we present a deep generative framework for automatically generating paraphrases, given a sentence. Our framework combines the power of sequenceto-sequence models, specifically the long short-term memory (LSTM) (Hochreiter and Schmidhuber 1997), and deep generative models, specifically the variational autoencoder (VAE) (Kingma and Welling 2013;Rezende, Mohamed, and Wierstra 2014), to develop a novel, end-to-end deep learning architecture for the task of paraphrase generation.
In contrast to the recent usage of VAE for sentence generation (Bowman et al. 2015), a key differentiating aspect of our proposed VAE based architecture is that it needs to generate paraphrases, given an original sentence as input. That is, the generated paraphrased version of the sentence should capture the essence of the original sentence. Therefore, unconditional sentence generation models, such as (Bowman et al. 2015), are not suited for this task. To address this limitation, we present a mechanism to condition our VAE model on the original sentence for which we wish to generate the paraphrases. In the past, conditional generative models (Sohn, Lee, and Yan 2015;) have been applied in computer vision to generate images conditioned on the given class label. Unlike these methods where number of classes are finite, and do not require any intermediate representation, our method conditions both the sides (i.e. encoder and decoder) of VAE on the intermediate representation of the input question obtained through LSTM.
One potential approach to solve the paraphrase generation problem could be to use existing sequence-to-sequence models (Sutskever, Vinyals, and Le 2014), in fact, one variation of sequence-to-sequence model using stacked residual LSTM (Prakash et al. 2016) is the current state of the art for this task. However, most of the existing models for this task including stacked residual LSTM, despite having sophisticated model architectures, lack a principled generative framework. In contrast, our deep generative model enjoys a simple, modular architecture, and can generate not just a single but multiple, semantically sensible, paraphrases for any given sentence.
It is worth noting that existing models such as sequenceto-sequence models, when applied using beam search, are not able to produce multiple paraphrases in a principled way. Although one can choose top k variations from the ranked results returned by beam-search, k th variation will be qualitatively worse (by the nature of beam-search) than the first variation. This is in contrast to the proposed method where all variations will be of relatively better quality since they are the top beam-search result, generated based on different z sampled from a latent space. We compare our framework with various sophisticated sequence-to-sequence models including the state-of-the-art stacked residual model (Prakash et al. 2016) for paraphrase generation, and show its efficacy on benchmark datasets, on which it outperforms the stateof-the-art by significant margins. Due to the importance of the paraphrase generation task in QA system, we perform a comprehensive evaluation of our proposed model on the recently released Quora questions dataset 1 , and demonstrates its effectiveness for the task of question paraphrase generation through both quantitative metrics, as well as qualitative analysis. Human evaluation indicate that the paraphrases generated by our system are well-formed, and grammatically correct for the most part, and are able to capture new concepts related to the input sentence.

Methodology
Our framework uses a variational autoencoder (VAE) as a generative model for paraphrase generation. In contrast to the standard VAE, however, we additionally condition the encoder and decoder modules of the VAE on the original sentence. This enables us to generate paraphrase(s) specific to an input sentence at test time. In this section, we first provide a brief overview of VAE, and then describe our framework.

Variational Autoencoder (VAE)
The VAE (Kingma and Welling 2014; Rezende, Mohamed, and Wierstra 2014) is a deep generative latent variable model 1 https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs Figure 1: A macro-view of our model: the paraphrase generation model is also conditioned on the original sentence that allows learning rich, nonlinear representations for highdimensional inputs. The VAE does so by learning a latent representation or "code" z ∈ R K for an input x ∈ R D such that the original input x can be well reconstructed from the latent code z. In contrast to the standard autoencoder (Goodfellow, Bengio, and Courville 2016) which learns, for any input x, a deterministic latent code z via a deterministic encoder function q φ , the VAE encoder is actually a posterior distribution q φ (z|x) (also known as the recognition model) over the latent code z. The posterior q φ (z|x) is usually assumed to be a Gaussian distribution N (µ(x), diag(σ 2 (x))), and the parameters φ = {µ(x), σ 2 (x)} are nonlinear transformations of the input x and are the outputs of feedforward neural networks that take x as input. The VAE also encourages its posterior distribution q φ (z|x) to be close to the prior p(z), which is typically taken as a standard normal distribution N (0, I).
The VAE also consists of a decoder model, which is another distribution p θ (x|z) that takes as input a random latent code z and produces an observation x. The parameters of the decoder distribution θ are defined by the outputs of another feedforward neural networks, akin to the VAE encoder model.
The parameters defining the VAE are learned by maximizing the following objective: Here KL stands for the KL divergence. Eq. 1 provides a lower bound on the model evidence p(x|θ, φ) and the VAE parameters are learned by maximizing this lower bound Rezende, Mohamed, and Wierstra 2014).
Endowing the latent code z with a distribution "prepares" the VAE decoder for producing realistic looking inputs even when z is a random latent code not representing the encoding of any of the previously seen inputs. This makes VAE very attractive for generative models for complex data, such as images and text data such as sentences.
In particular, (Bowman et al. 2015) presented a textgeneration model in which the encoder and decoder were modeled by long short-term memory (LSTM) networks. Moreover, training tricks such as KL-term annealing and dropout of inputs of the decoder were employed to circumvent the problems encountered when using the standard VAE for the task of modeling text data. Our work is in a We describe our VAE-LSTM architecture in more detail in the next section.

Model Architecture
Our training data is provided in form of N pairs {s Mn } denote the set of L n words from the original sentence and M n words from its paraphrase, respectively. In the following description, we will omit explicitly using the pair index n; e.g., we will denote a pair of original sentence and its paraphrase simply by s (o) and s (p) , respectively. We will also use x (o) and x (p) to denote the vector space representations of the original sentence and its paraphrase, respectively. These representations will be learned using LSTM networks, whose parameters will be learned in an end-to-end fashion, with the rest of the model. Fig. 1 shows a macro view (without the LSTM) of our proposed model architecture, which is essentially a VAE based generative model for each paraphrase's vector representation x (p) , which in turn is generated by a latent code z and the original sentence x o . In addition, unlike the standard VAE, note that our VAE decoder model p θ (x (p) |z, x (o) ) is also conditioned on the vector representation x (o) of the original sentence. In particular, as Fig. 1 shows, the VAE encoder as well as decoder are conditioned on the original sentence.
A detailed zoomed-in view of our model architecture is shown in Fig. 2, where we show all the components, including the LSTM encoders and decoders. In particular, our model consists of three LSTM encoders and one LSTM decoder (thus a total of four LSTMs), which are employed by our VAE based architecture as follows: • VAE Input (Encoder) Side: As shown in Fig. 2, two of the LSTM encoders are used on the VAE's input side. The first one converts the original sentence s (o) into its vector representation x (o) , which is fed, along with the paraphrase version s (p) of this sentence, to the next LSTM encoder. The output of this LSTM encoder (x (p) ) is passed through a feedforward neural network to produce the mean and variance parameters i.e., φ, of the VAE encoder.
• VAE Output (Decoder) Side: As shown in Fig. 2, the VAE's output side uses an LSTM decoder which takes as input (1) the latent code z, and (2) vector representation x (o) (produced by the third LSTM encoder) of the original sentence. The vector representation x o is used to initialize the LSTM decoder by feeding it to the first stage of the decoder, in contrast to the latent code z which is fed to each stage of the LSTM decoder (after being concatenated with the output of previous LSTM stage). Thus both z and x o are used to reconstruct the paraphrased sentence s (p) .
Similar to the VAE, the variational lower-bound of the proposed model is given by: Maximizing the above lower bound trades off the expected reconstruction of the paraphrased sentence's representation x (p) (given x (o) ), while ensuring that the posterior of z is close to the prior. We train our model following the same training procedure as employed in (Bowman et al. 2015).

Related Work
The task of generating paraphrases of a given sentence has been dealt in great depth and in various different types of approaches. Work has been done to use Statistical Machine Translation based models for generating paraphrases. (Quirk, Brockett, and Dolan 2004) apply SMT tools, trained on large volumes of sentence pairs from news articles. (Zhao et al. 2008) proposed a model that uses multiple resources to improve SMT based paraphrasing, paraphrase table and feature function which are then combined in a log-linear SMT model. Some old methods use data-driven methods and hard coded rules such as (Madnani and Dorr 2010), (McKeown 1983b). (Hassan et al. 2007) proposes a system for lexical substitution using thesaurus methods. (Kozlowski, McCoy, and Vijay-Shanker 2003) pairs elementary semantic structures with their syntactic realization and generate paraphrases from predicate/argument structure. As mentioned in (Prakash et al. 2016), the application of deep learning models to paraphrase generation has not been explored rigorously yet. This is one of the first major works that used deep architecture for paraphrase generation and introduce the residual recurrent neural networks. Finally, our work is also similar in spirit to other generative models for text, e.g. controllable text generation (Hu et al. 2017), which combines VAE and explicit constraints on independent attribute controls. Other prior works on VAE for text generation include (Bowman et al. 2015;Semeniuta, Severyn, and Barth 2017) which used VAEs to model holistic properties of sentences such as style, topic and various other syntactic features.

Experiments
In this section, we describe the datasets, experimental setup, evaluation metrics and the results of our experiments.

Datasets
We evaluate our framework on two datasets, one of which (MSCOCO) is for the task of standard paraphrase generation and the other (Quora) is a newer dataset for the specific problem of question paraphrase generation.
MSCOCO (Lin et al. 2014): This dataset, also used previously to evaluate paraphrase generation methods (Prakash et al. 2016), contains human annotated captions of over 120K images. Each image contains five captions from five different annotators. This dataset is a standard benchmark dataset for image caption generation task. In majority of the cases, annotators describe the most prominent object/action in an image, which makes this dataset suitable for the paraphrase generation task. The dataset has separate division for training and validation. Train 2014 contains over 82K images and Val 2014 contains over 40K images. From the five captions accompanying each image, we randomly omit one caption, and use the other four as training instances (by creating two source-reference pairs). Because of the free form nature of the caption generation task , some captions were very long. We reduced those captions to the size of 15 words (by removing the words beyond the first 15) in order to reduce the training complexity of the models, and also to compare our results with previous work (Prakash et al. 2016). Some examples of input sentence and their generated paraphrases can be found in Table 4.
Quora: Quora released a new dataset in January 2017. The dataset consists of over 400K lines of potential question duplicate pairs. Each line contains IDs for each question in the pair, the full text for each question, and a binary value that indicates whether the questions in the pair are truly a duplicate of each-other. 2 . Wherever the binary value is 1, the question in the pair are not identical; they are rather paraphrases of each-other. So, for our study, we choose all such question pairs with binary value 1. There are a total of 155K such questions. In our experiments, we evaluate our model on 50K, 100K and 150K training dataset sizes. For testing, we use 4K pairs of paraphrases. Some examples of question and their generated paraphrases can be found in Table 5.

Baselines
We consider several state-of-the-art baselines for our experiments. These are described in Table 1. For MSCOCO, we report results from four baselines, with the most important of them being by (Prakash et al. 2016) using residual LSTM. Residual LSTM is also the current state-of-the-art on the MSCOCO dataset. For the Quora dataset, there were no known baseline results, so we compare our model with (1) standard VAE model i.e., the unsupervised version, and (2) a "supervised" variant VAE-S of the unsupervised model. In the unsupervised version, the VAE generator reconstructs multiple variants of the input sentence using the VAE generative model trained only using the original sentence (without their paraphrases); in VAE-S, the VAE generator generates the paraphrase conditioned on the original sentence, just like in the proposed model. This VAE-S model can be thought of as a variation of the proposed model where we remove the encoder LSTM related to the paraphrase sentence from the encoder side. Alternatively, it is akin to a variation of VAE where decoder is made supervised by making it to generate "paraphrases" (instead of the reconstructing original sentence as in VAE) by conditioning the decoder on the input sentence.

Experimental Setup
Our framework primarily uses the following experimental setup. These settings are directly borrowed from an exist- ing implementation 3 of the paper (Bowman et al. 2015), and were not fine tuned for any of the datasets. In our setup, we do not use any external word embeddings such as Glove; rather we train these as part of the model-training. The dimension of the embedding vector is set to 300, the dimension of both encoder and decoder is 600, and the latent space dimension is 1100. The number of layers in the encoder is 1 and in decoder 2. Models are trained with stochastic gradient descent with learning rate fixed at a value of 5 × 10 −5 with dropout rate of 30%. Batch size is kept at 32. Models are trained for a predefined number of iterations, rather than a fixed number of epochs. In each iteration, we sequentially pick the next batch. A fixed number of iterations makes sure that we do not increase the training time with the amount of data. When the amount of data is increased, we run fewer passes over the data as opposed to the case when there is less data. Number of units in LSTM are set to be the maximum length of the sequence in the training data.

Evaluation
Quantitative Evaluation Metrics For quantitative evaluation, we use the well-known automatic evaluation metrics 4 in machine translation domain : BLEU (Papineni et al. 2002), METEOR (Lavie and Agarwal 2007), and Translation Error Rate (TER) (Snover et al. 2006). Previous work has shown that these metrics can perform well for the paraphrase recognition task (Madnani, Tetreault, and Chodorow 2012) and correlate well with human judgments in evalu-ating generated paraphrases (Wubben, Van Den Bosch, and Krahmer 2010). BLEU considers exact match between reference paraphrase(s) and system generated paraphrase(s) using the concept of modified n-gram precision and brevity penalty. METEOR also uses stemming and synonyms (using WordNet) while calculating the score and is based on a combination of unigram-precision and unigram-recall with the reference paraphrase(s). TER is based on the number of edits (insertions, deletions, substitutions, shifts) required for a human to convert the system output into one of the reference paraphrases.

Qualitative Evaluation Metrics
To quantify the aspects that are not addressed by automatic evaluation metrics, human evaluation becomes necessary for our problem. We collect human judgments on 100 random input sentences from both MSCOCO and Quora dataset. Two aspects are verified in human evaluation : Relevance of generated paraphrase with the input sentence and Readability of generated paraphrase. Six Human evaluators (3 for each dataset) assign a score on a continuous scale of 1-5 for each aspect per generated paraphrase, where 1 is worse and 5 is best.

Model Variations
In addition to the model proposed in Methodology Section, we also experiment with another variation of this model. In this variation, we make the encoder of original sentence same on both sides i.e. encoder side and the decoder side. We call this model VAE-SVG-eq (SVG stands for sentence variant generation). The motivation for this variation is that having same encoder reduces the number of model parameters, and hopefully helps in learning.  Group of mixed vegetables sitting on a counter top in a kitchen .

Generated
Several plates of fruits sitting on a table top in a kitchen Assortment of fruits and vegetables on a wooden table in a kitchen . Several types of fruit sitting on a counter top in a kitchen .

Source
Large motorcycle sitting on a grassy area in a line .

Reference
Older motorcycle displayed on grass along with several old cars .

Generated
Black motorcycle parked on the roadside next to a house . Black motorcycle rider on dirt next to a group of people . Green motorcycle parked on display outside with several old buses .

Results
We perform experiments on the above mentioned datasets, and report, both qualitative and quantitative results of our approach. The qualitative results for MSCOCO and Quora datasets are given in Tables 4 and 5 respectively. In these tables, Red and Blue colors denote interesting phrases which are different in the ground truth and generated variations respectively w.r.t. the input sentence. From both the tables, we see that variations contain many interesting phrases such as in front of an airport, busy street, wooden table, recover, Tech training etc. which were not encountered in input sentences. Furthermore, the paraphrases generated by our system are well-formed, semantically sensible, and grammatically correct for the most part. For example, for the MSCOCO dataset, for the input sentence A man with luggage on wheels standing next to a white van., one of the variants A young man standing in front of an airport. is able to figure out that the situation pertains to "waiting in front of an airport", probably from the phrases standing and luggage on wheels. Similarly, for the Quora dataset, for the question What is my old Gmail account?, one of the variants is Is there any way to recover my Gmail account? which is very similar -but not the same-to the paraphrase available in the ground truth. It is further able to figure out that the input sentence is talking about recovering the account. Another variant How can I get the old Gmail account password? tells us that accounts are related to the password, and recovering the account might mean recovering the password as well.
In Tables 2 and 3, we report the quantitative results from various models for the MSCOCO and Quora datasets respectively. Since our models generate multiple variants of the input sentence, one can compute multiple metrics with respect to each of the variants. In our tables, we report average and best of these metrics. For average, we compute the metric between each of the generated variants and the ground truth, and then take the average. For computing the best variant, while one can use the same strategy, that is, compute the metric between each of the generated variants and the ground truth, and instead of taking average find the best value but that would be unfair. Note that in this case, we would be using the ground truth to compute the best which is not available at test time. Since we cannot use the ground truth to find the best value, we instead use the metric between the input sentence and the variant to get the best variant, and then report the metric between the best variant and the ground truth. Those numbers are reported in the Measure column with row best-BLEU/best-METEOR.  Table 2, we report the results for MSCOCO dataset. For this dataset, we compare the results of our approach with existing approaches. As we can see, we have a significant improvement w.r.t.the baselines. Both variations of our supervised model i.e., VAE-SVG and VAE-SVG-eq perform better than the state-of-the-art with VAE-SVG performing slightly better than VAE-SVG-eq. We also evaluate with respect to best variant. The best variant is computed using different metrics, that is, BLEU and METEOR, however the best variant is not always guaranteed to perform better than average since best variant is computed with respect to the input question not based on the ground truth. When using the best variant, we get improvement in all three metrics in the case of non-beam search, however when experimented with generating paraphrases through beam-search, we get further improvement for METEOR and TER however these improvement are not as significant as for the Quora dataset, as you will see below. This could be because MSCOCO is an image captioning dataset which means that dataset does not contain fully formed grammatical sentences, as one can see from the examples in Table 4. In such cases, beam search is not able to capture the structure of the sentence construction. When comparing our results with the state-of-the-art baseline, the average metric of the VAE-SVG model is able to give a 10% absolute point performance improvement for the TER metric, a significant number with respect to the difference between the best and second best baseline which only stands at 2% absolute point. For the BLEU and METEOR, our best results are 4.7% and 4% absolute point improvement over the state-of-the-art.
In Table 3, we report results for the Quora dataset. As we can see, both variations of our model perform significantly better than unsupervised VAE and VAE-S, which is not surprising. We also report the results on different training sizes, and as expected, as we increase the training data size, results improve. Comparing the results across different variants of supervised model, VAE-SVG-eq performs the best. This is primarily due to the fact that in VAE-SVG-eq, the parameters of the input question encoder are shared by the encoding side and the decoding side. We also experimented with generating paraphrases through beam-search, and, unlike MSCOCO, it turns out that beam search improves the results significantly. This is primarily because beam-search is able to filter out the paraphrases which had only few common terms with the input question. When comparing the best variant of our model with unsupervised model (VAE), we are able to get more than 27% absolute point (more than 3 times) boost in BLEU score, and more than 19% absolute point (more than 2 times) boost in METEOR; and when comparing with VAE-S, we are able to get a boost of almost 19% absolute points in BLEU (2 times) and more than 10% absolute points in METEOR (1.5 times). The results of the qualitative human evaluation are shown in Table 6. From the Table, we see that our method produces results which are close to the ground truth for both metrics Readability and Relevance. Note that Relevance of the MSCOCO dataset is 3.38 which is far from a perfect score of 5 because unlike Quora, MSCOCO dataset is an image caption dataset, and therefore allows for a larger variation in the human annotations.
Note that one can use the metric between the variant and the input question to provide filtering in the case of multiple variants, or even to decide if a variant needs to be reported or not. So in order to make the system more practical (a high precision system), we choose to report the variant only when the confidence in the variant is more than a threshold. We use the metric between input question and the variant to compute this confidence. Naturally this thresholding reduces the recall of the system. In Figure 3, we plot the recall for Quora dataset, after thresholding the confidence (computed using the BLEU between the variant and the input question), and the average metrics for those candidates that pass the threshold. Interestingly, we can increase the BLEU score of the system as much as up to 55% at the recall of 10%. Plots generated using other metrics such as METEOR and TER showed a similar trend.

Conclusion
In this paper we have proposed a deep generative framework, in particular, a Variational Autoencoders based architecture, augmented with sequence-to-sequence models, for generating paraphrases. Unlike traditional VAE and unconditional sentence generation model, our model conditions the encoder and decoder sides of the VAE on the input sentence, and therefore can generate multiple paraphrases for a given sentence in a principled way. We evaluate the proposed method on a general paraphrase generation dataset, and show that it outperforms the state-of-the-art by a significant margin, without any hyper-parameter tuning. We also evaluate our approach on a recently released question paraphrase dataset, and demonstrate its remarkable performance. The generated paraphrases are not just semantically similar to the original input sentence, but also able to capture new concepts related to the original sentence.


extfieldsofstudy:
Medicine
Computer Science

text:
Fast and slow errors: Logistic regression to identify patterns in accuracy–response time relationships

Understanding error and response time patterns is essential for making inferences in several domains of cognitive psychology. Crucial insights on cognitive performance and typical behavioral patterns are disclosed by using distributional analyses such as conditional accuracy functions (CAFs) instead of mean statistics. Several common behavioral error patterns revealed by CAFs are frequently described in the literature: response capture (associated with relatively fast errors), time pressure or urgency paradigms (slow errors), or cue-induced speed–accuracy trade-off (evenly distributed errors). Unfortunately, the standard way of computing CAFs is problematic, because accuracy is averaged in RT bins. Here we present a novel way of analyzing accuracy–RT relationships on the basis of nonlinear logistic regression, to handle these problematic aspects of RT binning. First we evaluate the parametric robustness of the logistic regression CAF through parameter recovery. Second, we apply the function to three existing data sets showing that specific parametric changes in the logistic regression CAF can consistently describe common behavioral patterns (such as response capture, time pressure, and speed–accuracy trade-off). Finally, we discuss potential modifications for future research.

In many domains of cognitive psychology, understanding the different types of error and response time (RT) patterns is necessary for appropriate inferences. Consider, for example, one of the hallmark experimental paradigms of the cognitive control literature, the Simon task (Van Campen, Keuken, Van den Wildenberg, & Ridderinkhof, 2014;Van Maanen, Turner, & Forstmann, 2015). In this task, participants are asked to respond to some relevant stimulus feature with either their left or their right hand. Crucially, the stimulus is placed on the left or right side of a computer screen, creating a congruency or incongruency between the stimulus location and the response hand. On average, incongruent stimulus-response mappings result in relatively slower and more incorrect responses than do congruent stimulus-response mappings, suggesting a relatively simple mechanism. Interestingly, however, analyzing the full RT distributions of congruent and incongruent mappings for RT and error patterns resulted in a large body of research acknowledging the existence of a more complex picture of different underlying processes within the Simon task (Burle, Possamaï, Vidal, Bonnet, & Hasbroucq, 2002;De Jong, Liang, & Lauber, 1994;Forstmann et al., 2008;Hommel, 1993Hommel, , 1994Proctor, Miles, & Baroni, 2011;Ridderinkhof, 2002;Stürmer, Leuthold, Soetens, Schröter, & Sommer, 2002;Tagliabue, Zorzi, Umiltà, & Bassignani, 2000;Van Campen, Kunert, Van den Wildenberg, & Ridderinkhof, 2018;Van Campen et al., 2014;Van den Wildenberg et al., 2010).
One popular analysis tool that jointly considers the accuracy of responses and the distribution of their RTs is the conditional accuracy function (CAF; Gratton, Coles, Sirevaag, Eriksen, & Donchin, 1988;Heitz, 2014;Lappin & Disch, 1972;Proctor et al., 2011;Ratcliff, 1979;Ridderinkhof, 2002). A CAF expresses how the accuracy of responses depends on the speed of responses, by formulating how the probability of a correct response depends on the RT.
Analyzing CAFs is appealing because it allows one to investigate the categorical relationships between responses (typically, correct or incorrect) and RTs. The first type of categorical error-RT relationship that is often observed is that the RTs The original version of this paper was revised. of incorrect responses are relatively fast. This is for example the case in the previously introduced Simon paradigm, in which a higher proportion of these fast errors for incongruent stimulus-response mappings than congruent stimulus-response mappings is interpreted as "response capture," the automatic activation of the (incorrect) response hand invoked by the stimulus location (Forstmann et al., 2008;Ridderinkhof, 2002;Ulrich, Schroter, Leuthold, & Birngruber, 2015;Van Campen et al., 2014;Van Campen et al., 2018;Van den Wildenberg et al., 2010;Van Wouwe et al., 2016).
A second common categorical error-RT relationship entails response times of incorrect responses that are slower than average. In certain experimental settings, this is sometimes interpreted as an indication of time pressure or urgency on behavior. Time pressure would result in a higher proportion of relatively late incorrect responses, representing that as participants "feel" the time pressure, they start to make errors (Hanks, Kiani, & Shadlen, 2014;Murphy, Boonstra, & Nieuwenhuis, 2016;Thura & Cisek, 2016).
A third relationship between accuracy and RTs is that erroneous responses are distributed evenly across the RT distribution, and there is in fact no dependence of the correctness of the response on RT (Donkin & Van Maanen, 2014;Mulder & Van Maanen, 2013;Van Ede, de Lange, & Maris, 2012). However, the proportion of errors might still depend on some experimental manipulation, such as a cue-induced speed-accuracy trade-off task. In such an experiment, participants are instructed to focus either on accurately responding (ignoring response speed) or on response speed. The focus on response speed typically comes at the cost of making more errors, which are evenly distributed across the RT distribution (Heitz, 2014;Schouten & Bekker, 1967;Van Maanen et al., 2011;Wickelgren, 1977). Theoretical models of this kind of speed-accuracy trade-off behavior propose that people confronted with such a trade-off entertain a critical confidence value (threshold setting in sequential-sampling models; e.g., Bogacz, Wagenmakers, Forstmann, & Nieuwenhuis, 2010), which triggers a response as soon as that critical value is surpassed. This typically results in an equal distribution of errors across the RT distribution, even though accuracy and RT are both affected by changing the critical confidence value.
To draw inferences such as the ones sketched out above, CAFs have been extensively used. The standard method for quantifying CAFs is to indicate a set of RT bins, and then to compute the average proportion of correct responses per RT bin. However, there is not a generally accepted method for computing RT bins. Hyndman and Fan (1996) discussed nine different methods that are used in various statistical packages. In addition, the number of RT bins is also a matter of judgment. In the cognitive control literature, often three or four bins are used (Forstmann et al., 2008;Van den Wildenberg et al., 2010), but in many domains, dividing the RT distribution into five or six bins is much more common (Brown & Heathcote, 2008;Heathcote, Brown, & Mewhort, 2002).
The choice of the number of bins may have important consequences for the potential conclusions drawn from a particular data set. This is illustrated in Fig. 1, in which we present simulated data from a task in which fast errors occur (Fig. 1a) and from a task in which slow errors occur (Fig. 1b). The black dots in Fig. 1, with an accuracy of 0 or 1, represent the responses in a fictional experiment. The locations on the x-axis indicate the associated RTs. In frequency, on the right side of the x-axis (as well as by the lower frequency of correct responses-accuracy of 1-on the right side). Each panel shows three CAFs (red, blue, and green), computed by averaging the error proportion for a particular RT bin. The numbers of bins are 10 (red), 7 (blue), and 4 (green). If errors are predominantly associated with the fastest responses (Fig. 1a), then the proportion of errors in the first RT bin depends on the width of that bin. If the bin is wider, reflecting a larger part of the RT distribution, then the error proportion will decrease accordingly. In Fig.  1a, this is illustrated as the thicker part of each CAF. If errors are predominantly associated with slower responses, then the number of bins influences the measurement of the change in error proportion. In Fig. 1b, we have highlighted this in the last segment of each CAF. Also, because there are fewer responses per RT bin, the uncertainty in these measurements increases.
In summary, the effect of the chosen number of bins-and related to this, the width of the bin, as illustrated in Fig. 1-is the first potential pitfall for applying CAFs. The reliability of the estimate of the error rate within a bin depends critically on the chosen number of bins and presents a potential threat for statistical interferences.
The second potential pitfall is related to the way the size of the RT bins is determined. The typical approach is to set the boundaries of each RT bin on the basis of the frequency of responses per bin (i.e., using the quantiles of the RT distribution; sometimes the range of the RT distribution is used, to ensure that the ranges of the RT bins remain equal instead of the frequency of observations). If one wants to draw a conclusion about a difference between conditions, however, the RT bins can be determined using the quantiles of each condition separately or in combination with all others. Which method is chosen may considerably impact the conclusion. For example, if the RT distributions of two conditions are shifted relative to each other, but the bin sizes are determined according to the combined RT distribution, then the proportions of fast errors may seem to differ just because the number of trials from each condition differs per RT bin.
Finally, the third pitfall is that averaging accuracy over RT bins raises statistical problems with respect to hypothesis testing. The response variable is treated as continuous on the interval [0, 1], rather than categorical with two levels (correct or incorrect response, often coded as 0 or 1). Treating categorical variables as continuous is problematic from a statistical viewpoint (Jaeger, 2008). For example, the confidence intervals on the expected proportion of correct responses for a particular RT bin could exceed the levels of the categorical variable (e.g., a confidence interval of CI = [0.9, 1.03], where the categorical levels are 0 and 1, exceeds the real maximum possible score). Furthermore, the general linear model assumes equal variances across groups. As Jaeger pointed out, this is not the case for categorical variables with two levels that are binomially distributed, because by definition in binomial distributions, a change in the variable's mean induces a change in its variance. This means that testing for accuracy differences per RT bin using an analysis of variance (an instance of the general linear model) may yield incorrect conclusions, because the assumptions of the statistical test(s) are not met.
In the present article, we will introduce a novel method of analyzing the accuracy-RT relationship, based on nonlinear logistic regression. The new method does not suffer from the issues discussed above, yet it has the flexibility to account for the most common patterns in accuracy-RT data. It captures the shape of the accuracy-RT relationship without the specific potential pitfalls described above. The method is model-free in the sense that it does not commit to a specific cognitive modeling framework, yet the parameters can be interpreted in light of a specific hypothesis about the accuracy-RT relationship in one's experimental data.

Methods and results
We highlight another approach for estimating the dependence of accuracy on RTs: nonlinear logistic regression. Logistic regression models aim to predict a categorical response variable using a continuous predictor variable. In the present case, this is binary accuracy (i.e., a correct or incorrect response), which is predicted by RT. Thus, the logistic regression model estimates the probability of a correct response based on a particular RT. Although a common assumption of logistic regression is the linear dependence of the response variable (correct or not) on the predictors, we propose a nonlinear transformation of the predictors to account for the typical patterns observed in the CAF literature. A first requirement of the nonlinear transformation is that it is nonmonotonic, meaning that the probability of a correct response can both decrease and increase. A second requirement is that the shape of the function is flexible enough that it can account for asymmetric response patterns. That is, the increase in the probability of a correct response with fast RTs is typically faster than the decrease with slower RTs. The shape of the patterns of faster errors is not necessarily the same as the shape on the decrease of slower errors (unlike in, e.g., a parabola). A third requirement is that the function can be parameterized in such a way that each parameter captures a specific qualitative property of the shape of a CAF, relating the behavioral phenomena expressed in CAFs to parameter-specific changes.
Following these requirements, we propose to model the probability of a correct response at a certain RT using Eq. 1: Equation 1 is an instance of the logistic function parametrized by four parameters that together account for the full range of data patterns observed in the accuracy-RT relationship. It defines the probability of a correct response, p(correct | t), as a function of the RT t. Parameter a defines an asymptote that captures the maximum accuracy that is obtained (Fig. 2a). A lower a gives lower maximum accuracy. Parameter b defines the downward slope of the second segment of the curve (Fig. 2b). If b = 0, there is no downward section, but for all b > 0 the parameter value determines how steep the function declines after its peak value. Larger b results in a steeper slope and an increase in the number of slow errors. Parameter c defines the location of the peak value or bend point (Fig. 2c). The d parameter defines a shift of the curve over the x-axis (Fig. 2d). A positive value of d indicates that the curve is not defined for RTs smaller than d, which can be interpreted as a lower bound on the observed RTs: the lower limit of the accuracy-RT shape.
If b = 0 and c = 0, Eq. 1 reduces to p correctjt ð Þ¼ e a e a þ1 . In this case, if a approaches infinity, the probability of a correct response approaches 1. If a = 0, the probability of a correct response is .5, with negative values for a yielding lower probabilities of a correct response. In this specific case, the probability of a correct response is also not predicted to change for different RTs, since the probability does not depend on t anymore.
If c = 0 but b is positive, the probability of a correct response is determined by a if td = 0 (i.e., at the fastest response), but declines for increasing RTs t. If c is positive as well, then the probability of a correct response first rises with increasing RTs, but then declines. In practical situations, this initial rise and eventual drop is the most typical pattern, and therefore we suggest constraining all parameters to the positive range when estimating their values on the basis of empirical data (see the Application section below).

Parameter recovery
The goal of this analysis tool is to reach better conclusions about the accuracy-RT relationship, which will help support or reject hypotheses that researchers have about cognitive processes. For this purpose, it is crucial to study whether it is possible to identify the data-generating parameters in a sample of synthetic data (Anders, Alario, & Van Maanen, 2016;Miletić, Turner, Forstmann, & Van Maanen, 2017). If the data-generating parameters can be recovered reliably, then the parameters estimated from the data can be interpreted in support of a specific hypothesis.
To study this, we performed a parameter recovery study. The setup of the parameter recovery was a follows: 1. We defined the parameter space from which we sampled. 2. Data were generated according to one parameter vector. 3. We estimated the parameters of the logistic regression model for this new data set. 4. Steps 2 and 3 were repeated 1,000 times, to sample the full parameter space.
Parameter space To make sure that our parameter recovery involves reasonable parameter ranges, we estimated parameters for experimental data from ten experiments with multiple participants and conditions, for a total of 750 data sets (Table 1). The experiments included perceptual judgments, memorybased choice tasks, and a Simon experiment. Care was taken to include data sets that we hypothesized would affect every parameter, to ensure that critical cases were also included in the parameter space. In addition, the parameter space was constrained to positive values. The resulting distributions of parameters from which we sampled are shown in Fig. 3.
Data generation On each of 1,000 iterations, we randomly sampled a set of parameters estimated from one of the 750 data sets. We computed the predicted accuracy for the range [50 ms-5 s], discretized by steps of 5 ms, and generated binary response data by performing exactly one Bernoulli trial for each discrete time step in the range, with the probability of a positive outcome set at the predicted probability of the logistic regression model (for this set of parameters). This procedure resulted in 1,000 simulated data sets consisting of response-RT pairs that could be subsequently used to estimate parameters.
Parameter estimation The parameters of the so-generated data sets were estimated using SIMPLEX optimization of the squared residuals (Nelder & Mead, 1965). The optimizer minimized the weighted sum of the squared errors using the ordinary least squares procedure. As starting values of the SIMPLEX search, we used the mean values of the parameters in the parameter space. This procedure resulted in the set of four parameters for the simulated data that had the lowest squared error. Figure 4 displays that the parameters were reliably recovered. The values on the y-axes indicate the parameters that were used to generate the data sets, and the values on the xaxis indicate the estimated parameters The panels on the diagonal display how well the parameters were recovered. A perfect recovery would mean that all true estimated pairs would lie on the dashed line. The blue regression lines indicate the actual relationship between true and estimated parameters, which closely follows the perfect line, indicating that on average the logistic regression model nicely recovered the true parameters. The diagonal panels also display the correlations between the true and estimated parameters, as well as the root mean squared deviations (RMSD) between the true and estimated parameters. These measures indicate that the parameter recovery of the logistic regression model was satisfactory.
Off the diagonal, the relation of the parameter estimates to the other true parameters is displayed, providing a measure of trade-off between the parameter estimates. The correlations between the parameter estimates and the other true parameters are clearly below the diagonal, although they are quite high for the a and b parameters. The interpretation of these correlations is that effects in the data that are explained by the b parameter are also partially explained by the a parameter. This is potentially a consequence of a quite high correlation in the parameter spaces between the a and b parameters of r = .59: Intuitively, this makes sense: A higher asymptote indeed allows for a greater negative slope in the second half of the accuracy-RT data.
Recovery in smaller data sets The first parameter recovery studies had a highly idealized scenario with many observations equally spaced across the full RT range. However, in practice, data typically have fewer observations and are not evenly distributed. To show that our method is resilient to more realistic data patterns, we performed an additional parameter recovery, in which we included only 200 responses per simulated data set. The simulated RT data were generated according to a shifted Wald distribution function, which is a common description of RT distributions (cf. Anders et al., 2016). The mean and shape parameters of the shifted Wald Table 1 Brief description of the experimental data sets fitted to obtain a sensible parameter space  Table 1. function were uniformly sampled across simulations (between 0.5 and 2 and between 2 and 4, respectively). This parameter recovery revealed that, although slightly less accurate, the model still showed strong correlations between the true and estimated parameters (Table 2). This means that for experiments with around 200 trials, the parameters of the logistic regression model can be reliably interpreted to study the accuracy-RT relationship.

Application to standard data sets
In this section, we illustrate how conclusions can be drawn by the application of the non-linear logistic function to the accuracy-RT relationship. In Example 1 the model was fit to data of a lexical decision task in which participants were cued to apply a speed-accuracy trade-off (Wagenmakers, Ratcliff, Gomez, & McKoon, 2008). Here we expected the a parameter to vary across conditions (for reasons that will be specified below). Example 2 pertains to an experiment in which time pressure was experimentally manipulated (Van Maanen, Fontanesi, Hawkins, & Forstmann, 2016), and we expected differences in the b parameter. In Example 3 the model was fit  to data from a Simon task (Van Campen et al., 2014), and we expected the c parameter to vary across conditions. Because the d parameter is comparable to a nondecision time parameter in sequential-sampling models and is often considered uninformative and only used to better fit the data, we will not provide an additional application example for this parameter.
Cue-induced speed accuracy trade-off Cue-induced speedaccuracy trade-off refers to the often observed result that when participants are cued to respond fast, they can only increase their speed of responding at the expense of their response accuracy (Bogacz et al., 2010;Heitz, 2014;Wickelgren, 1977). A common interpretation of this finding is that in many situations, participants contemplate their actions until they reach a specific level of certainty about the planned action, and then commit to that action. When they are pressed for speed, this level of certainty is hypothesized to be less. One very prominent class of mathematical models (a simple diffusion decision model that assumes no betweentrial variability in the rate of evidence accumulation; Ratcliff, 1978) suggests that although participants indeed trade response accuracy for response speed in this scenario, there is no dependence between the probability of a correct response and response speed within each condition. This suggests that fitting the CAF should result in a lower a parameter under cue-induced speed stress (see also Fig. 2a), but not differences between conditions for the remaining parameters. In the lexical decision task of Wagenmakers et al. (2008), participants were asked to indicate with a button press whether or not a letter string presented on a computer screen was a valid English language word. Each of 17 participants contributed 960 responses while it was stressed that being accurate was more important than being fast, and 960 responses while the opposite instruction was provided: The speed of responding was more important than accuracy. In half of the trials, valid words were presented, and in the remaining trials items were presented that resembled words but in which one or a few characters were adjusted to create an invalid word (for details about the experimental design, we refer the reader to Wagenmakers et al., 2008). The word and nonword trials are collapsed for the purposes of the present analysis.
We fit the CAF separately to the individual speed-accuracy conditions and participants. The parameters of the CAF were optimized using SIMPLEX optimization (Nelder & Mead, 1965), with reasonable starting points (a = 3, b = 0.1, c = 0.01, d = 0.1) that did not differ across participants and conditions. All parameters were bound within the (0, Inf) range (i.e., parameter estimates could not become negative).
Because of floor effects on the possible parameter estimates, we log-transformed the b and c parameters before performing statistical analysis. Paired t tests indicated a significant difference in the estimates of a [t(16) = 3.16, p = .006], as well as a significant difference in the logtransformed estimates of c [t(16) = -2.65, p = .017]. The other two parameters did not differ significantly [t values < 1.1; a test on the nontransformed b estimate also did not reach significance: t(16) = 1.4, p = .16].
Thus, as predicted, the cue-induced speed-accuracy instruction provided in this experiment was reflected in the a parameter, yielding CAFs that asymptoted at different levels (Fig. 5). In addition, it seems that in this data set, part of the speed-accuracy trade-off behavior can be explained by a higher proportion of fast guesses, as indicated by the higher estimate of c for speed-instructed trials (Dutilh, Wagenmakers, Visser, & Van der Maas, 2011;Schneider & Anderson, 2012; but see Van Maanen, 2016).
Urgency Recent work has suggested that an error rate that increases with RT is a signature of time pressure or urgency (Hanks et al., 2014;Murphy et al., 2016;Thura & Cisek, 2016). That is, if participants have to make decisions under time pressure (e.g., before a certain time limit has been reached, but not if they are simply cued to be fast, as above; cf. Katsimpokis et al., 2018), then they might make more mistakes the longer the decision process takes, resulting in a negative slope of the CAF, which is expressed by the b parameter (see also Fig. 2b). An increase in b therefore may Acc Spd (2019) 51:2378-2389 Behav Res reflect a decision boundary that decreases over time (Frazier & Yu, 2008;Malhotra, Leslie, Ludwig, & Bogacz, 2017).
Van Maanen and colleagues ) explored this line of reasoning using an expanded judgment paradigm, in which participants were asked to make a choice about a noisy stimulus that slowly built up on the screen. The researchers manipulated the speed of the buildup, thereby inducing time pressure effects without experimentally manipulating the choice difficulty. Specifically, the interval between consecutive updates of the stimulus was either 200 or 400 ms, meaning that the same amount of information about the stimulus was presented in half the time in one condition as compared to the other condition (on average, across trials). Van Maanen et al. (2016) found that when the stimulus updated quickly, participants felt pressed for time and made decisions based on less information on the screen. Moreover, a negative relation was found between RT and the amount of information, similarly indicating that participants felt time pressure during a trial. A nonnegative b parameter in the present analysis would be conceptually similar to these findings. Here we present the data from their Experiment 2, which was conducted while participants were in an MRI scanner (for details of the experimental design, we refer the reader to Van Maanen et al., 2016). Figure 6 presents the distributions of the estimated parameters. Because of floor effects on the possible parameter estimates, we again log-transformed the c parameter before performing statistical analysis. In this case, we judged this additional step unnecessary for the b parameter, because the median RTs were sufficiently above the floor value of 0. Although the parameter estimates did not differ significantly [at an alpha level of .05, t(19) = 1.90, p = .07], we believe that it is interesting that the direction of the effect was consistent with the previous results and with the previous analyses of these data . Also note that no other parameter showed an effect [a: t(19) = 1.28, p = .21; with respect to the c and d parameters, t values < 1].
Response capture In certain experimental paradigms, the stimuli are such that they seem to engage an automatic process, resulting in very fast but error-prone responses. This process, sometimes referred to as response capture, occurs in the Simon task. In this task, participants are asked to indicate with a left or right button press whether a stimulus has a specific color. Because the stimuli appear on the left or right side of the screen, this often elicits a fast response with the hand on the same side as the stimulus. If the stimulus location is congruent with the (correct) response hand, indicating that an automatic button press yields a correct outcome, there is no decrease in accuracy for fast responses. For incongruent trials, however, this leads to a large proportion of errors, decreasing the overall accuracy for fast responses (Forstmann et al., 2008;Ridderinkhof, 2002;Van Campen et al., 2014;Van Campen et al., 2018;Van den Wildenberg et al., 2010). Although the standard method in the field has been to compute the proportion of errors in the first RT bin, as we outlined above, a novel methodology has recently become available that addresses the specific question of response capture (Servant, Gajdos, & Davranche, 2018).
We predict that response capture should be visible in the c parameter of the CAF function, since that parameter decreases the accuracy in the initial segment of the curve for incongruent trials, which would be consistent with the higher fast-error rate associated with response capture (see also Fig. 2c).
Van Campen et al. (2014) did a fairly standard version of the Simon task, in which ten participants had to respond with a right or left button press (counterbalanced across participants) to whether a circle that appeared on the screen was green or blue. During the experiment, single-pulse transcranial magnetic stimulation (TMS) was administered in order to obtain indices of corticospinal excitability. Crucially for the present purpose, the single-pulse TMS did not disrupt behavior in any way (for details of the experimental design, we refer the reader to Van Campen et al., 2014). Figure 7 displays the distributions of parameter estimates, as well as the typical CAFs for one representative participant. Because of floor effects on the possible parameter estimates, we again used log-transformation of the b and c parameters before performing the statistical analyses. Paired t tests resulted in a significant difference in the c parameter [t(9) = 5.9, p < .001], suggesting that participants indeed had more fast incorrect responses in the incongruent than in the congruent condition. In addition, the a parameter differed between conditions [t(9) = 2.7, p = .02], indicating that the asymptote of the CAF was higher for the incongruent than for the congruent condition. The b and d parameters did not differ between conditions (t values < 1.6).

Another inferential method: Model comparison
Another method of inference is to compare how well models balance their goodness of fit and their flexibility to account for data (Pitt & Myung, 2002). That is, if a model has many parameters, in many cases it can also account for many different patterns in the data. Thus, a model that is overly flexible in this way might overfit the data, and so not generalize to other data sets. In those cases, a simpler model might account for the data almost as well, by explaining the true effects in the data but not the noise that is inherent in a specific data set.
Many methods exist to formally compare the goodness of fit of two models with different levels of flexibility. One typical method for this model comparison is the Akaike information criterion (AIC; Akaike, 1974), which corrects the goodness of fit for the number of free parameters of each model. We illustrate this method here on the data of Van Maanen et al. (2016), to illustrate its use in the context of CAFs and show that there is evidence that the c parameter is effectively zero in this data set, as was already suggested by Fig. 6. In this case, we wanted to test whether a nonzero c parameter explained a significant proportion of the explained variance. Therefore, we compared the full model discussed above, in which all parameters were estimated, with a reduced model in which we forced the c parameter to be c = 0. Consequently, the reduced model had fewer free parameters, and one could compare the balance of the flexibility of the models with respect to their goodness of fit. If the reduced model fit almost as well as the more complex full model, even though it was less flexible, it would be considered the better model.
Because we applied least-squares parameter estimation, it was possible to compute AIC values through the residual sum of squares of the models (RSS; Burnham & Anderson, 2002). For each participant and condition, the AIC was computed according to the formula AIC = 2k − n ln(RSS), with n being the number of observations per cell, and k the number of free parameters, which was k = 8 for the full model and k = 6 for the reduced model. Comparison of the AIC values obtained in this way revealed that in 92.5% of the cases, the reduced model was preferred over the full model (for 90% of the participants in the 200-ms condition, and 95% of the participants in the 400-ms condition). Akaike weights (Wagenmakers & Farrell, 2004) showed that the reduced model was about twice as likely to be correct as the full model [averaged AIC w (Full) = .32; averaged AIC w (Reduced) = .68]. Thus, although the difference between the AIC values was not big (as revealed by the Akaike weights), the reduced model was consistently preferred over the full model, leading to the inference that there were no specifically fast incorrect responses in this data set. This conclusion thus also corroborates our previous analysis that showed no significant difference between the c-parameters in the Van Maanen et al. (2016) data, but with the stronger claim that these parameters were in fact equal to 0.

Discussion
There are many researcher degrees of freedom in the typical methods for identifying relationships between accuracy and RTs, and potential pitfalls exist in interpretation due to the chosen strategy. For this reason, we introduced a novel approach to computing CAFs, which is based on nonlinear logistic regression and solves the problem of choosing the exact amount of RT bins. The logistic regression model assumes a flexible functional form that accounts for the typical patterns in the data, and using least squares regression the best fitting parameters can be estimated. In simulation and in example applications, we showed that the estimated parameters are accurate and that meaningful inferences can be made. It is pertinent to finish the discussion of this approach by stressing its limitations. In our view, three important limitations warrant discussion. First, it is worth mentioning that the logistic regression model is only applicable to binary outcomes (typically correct/incorrect). For most application domains, however, this is not problematic.
Second, although it is an improvement over current practices, the new method still allows for some researcher degrees of freedom. Specifically, as with any mathematical model, there are multiple ways of drawing inferences. We have focused here on performing statistical tests on the estimated parameters. Another method that we discussed is to compare the fit quality and the flexibility of two (or more) models that implement different theoretical assumptions about the data. We illustrated this approach with a model that did not allow for specifically fast incorrect responses, and one that did, and compared these using AICs. Other methods are also available, potentially leading to different inferential outcomes (Dutilh et al., 2018).
A related issue is that the standard statistical tests that we chose to perform on the parameters have relatively strong assumptions. Violating these assumptions might increase the chance of an incorrect inference if this were not properly controlled for. In the case of the logistic regression model, the lower bound on the parameter space may result in non-Gaussian distributions of the parameters. Here we chose to apply a logarithmic transformation of the data to obtain more Gaussian-like distributions when this was the case, but this is a degree of freedom that is allowed to the researcher, as well (Gelman & Loken, 2014).
A third important limitation is that the present setup of the model does not allow for the inclusion of random effects, which may be pertinent in application domains where stimuli differ from trial to trial (Anders, Oravecz, & Alario, 2018). Here we chose to estimate the model parameters independently for each condition and to do inference on the group level in a second stage, but in situations with small sample sizes or large item effects, an analysis that included (crossed) random effects might increase power (Baayen, Davidson, & Bates, 2008).

Conclusion
The present article has introduced a new method for analyzing conditional accuracy in a principled, model-free way. The method alleviates some of the problems associated with RT binning to obtain different accuracy levels per RT bin. Specifically, the researcher degrees of freedom are reduced. Using this method, we can reliably identify the presence of fast or slow errors, which may be beneficial in many domains of cognitive psychology in which the relationship between responses and RTs is of theoretical importance.


Access to European Statistical System Microdata

The chapter presents the European microdata access system. This system allows eligible researchers to analyse detailed data transmitted to Eurostat by national statistical offices in the European Union. Eurostat is a single entry point of access to such data. Individual data collected by national statistical offices to produce official statistics are strictly confidential. The data are anonymised and further processed before they can be made available for scientific purposes. Statistical offices are legally obliged to protect information received from individual respondents. They use this information solely to produce official statistics. The entities collecting data for other purposes (e.g. administrative, commercial or health) fall into the scope of personal data protection legislation. Statistical confidentiality measures are stricter than those resulting from personal data protection measures.

The European Statistical System and European Statistics
The European Statistical System (ESS) is a partnership between Eurostat and the national statistical institutes (NSIs) and other national authorities responsible in each Member State for the development, production and dissemination of European statistics. National statistical authority (NSA) is a generic term for NSIs and other national data providers (e.g. regional statistical offices, ministries providing administrative data, etc.); a list of NSAs is available on the Eurostat website. 1 European official statistics are important for EU. They are produced and disseminated by Eurostat in partnership with NSAs. Usually, national official statistics are based on microdata, collected or accessed by NSAs. Microdata are then aggregated, transmitted to Eurostat and published. Where necessary for the production of European statistics, NSAs also transmit microdata to Eurostat (see Fig. 1). Whenever microdata are transmitted, Eurostat may consider granting access to these for scientific purposes. In this way, almost all microdata received by Eurostat are released for scientific purposes.

Microdata Access Terms and Concepts
Microdata are a form of data where sets of records contain information on individual persons, households or business entities. Traditionally, statistical offices use microdata only to produce aggregated information such as tables. Publication of individual information (microdata) is generally not allowed because it may easily lead to identification of the data subject (person, household or business entity) and therefore to a breach of statistical confidentiality.
Statistical confidentiality is one of the fundamental principles of official statistics. It is the obligation of the statistical offices to protect confidential data. 2 In the context of European statistics, confidential data are data that allow the identification of statistical units (individual persons, households or business entities), thereby disclosing individual information. The statistical unit may be identified in the different forms of statistical output, e.g. the contribution of largest companies may be approximated in business statistics. To prevent this, statistical offices check each output from the point of view of statistical confidentiality. This check is called statistical disclosure control (SDC).
The SDC methodology helps to identify confidential data in these various output forms and to hide such data, taking into account relationships between the data (e.g. additivity of the tables).
In general, official statistics are available in the form of tables where confidential data are not visible and the data are highly aggregated. But many statistical offices also make available their data in the form of microdata, namely as (see Fig. 2): • Public-use files accessible to everybody (sometimes upon registration or licence signature) • Confidential microdata files accessible to researchers satisfying specific access conditions Confidential microdata files are invaluable for the research community as they allow deep analysis of relationships in the data, i.e. causalities, dependencies, convergences, etc. Microdata access systems were developed by statistical institutes to allow legitimate access to confidential data for scientific purposes.  Tables  Microdata   Confidential  microdata Public use files

Elements of the Generic Microdata Access System
Microdata access systems define under which conditions access to confidential microdata can be granted for external persons, such as researchers. These conditions are normally outlined in legal acts. In the European Statistical System, access to microdata may be granted to researchers carrying out statistical analysis for scientific purposes. 3 Microdata files may have different levels of detail. The more detailed the data, the easier it is to identify individuals. Original statistical records can be easily identifiable as they contain unique direct identifiers such as names, address, social security number or identification number (ID number). These confidential records with direct identifiers are available to the statistical offices only under strict confidentiality protocols.
Microdata without direct identifiers are called 'de-identified' or 'pseudonymised' microdata (if direct identifiers are replaced by pseudo-identifiers: unique codes replacing all direct identifiers). De-identified microdata with pseudoidentifiers are more and more important for the production of official statistics, as they allow linking data collected from different sources, thus fostering the use of, for example, administrative sources and derivation of further results on the basis of already collected data. Pseudo-identifiers also allow the creation of longitudinal files, following individuals over time. These microdata are still confidential, as the combination of some rare characteristics may lead to identification of unique statistical units.
De-identification is a subprocess of anonymisation. In general, anonymisation is the process of making the data anonymous. However, approaches to this process differ between countries. In some countries, making the data anonymous is defined as removal of names, i.e. de-identification. In the European law, anonymisation is defined as the process aiming at complete protection of microdata, such that the records are no longer identifiable (the records cannot be linked to any 'real' person, household or business entity). The different stages of microdata anonymisation/protection are (see Fig. 3): • De-identification or pseudoanonymisation: process of removing direct identifiers (such as name, ID number and address) from the confidential data, and replacing them with pseudo-identifiers. Pseudo-identifiers can be used to link datasets. • Partial anonymisation: application of a set of SDC methods to microdata in order to reduce the risk of identification of the statistical unit. Scientific-use files are the result of partial anonymisation. • Complete anonymisation: application of SDC methods that completely eliminate the risk of identification of the statistical unit (directly or indirectly). Public-use files contain completely anonymised records. Table 1 compares all basic types of microdata files and access conditions. The terms secure-use files and scientific-use files are specific to the European microdata access system. In the EU countries, there exist similar files but with different names, e.g. scientific-use files are often called 'microdata files for research'. The basic characteristics of these files remain the same: • Secure-use files are files to which no further methods of statistical disclosure control have been applied. Researchers access these files in the secure environment provided by NSAs (local or remote access). The final results of the work of researchers are checked by NSAs to ensure that they do not reveal confidential data. Each output is checked separately. • Scientific-use files are files to which methods of statistical disclosure control have been applied to reduce (not to eliminate!) the risk of identification to an appropriate level (partial anonymisation). Researchers have access to such files outside the controlled NSA environment. There are usually no ex post controls by NSAs; researchers need to follow the confidentiality instructions and are responsible for making the published results non-confidential.
Secure use files are the richest form of microdata for research. However, the services related to provision of access are usually expensive for statistical offices. This is because of infrastructure (dedicated environment for on-site or remote access) and operational costs related to output checking.
For statistical offices, scientific-use files seem to be more efficient in terms of cost-benefit ratio. For researchers, the advantage is that they can be used without having to travel to the premises of the statistical offices (or without logging in to a remote, secure system).
Scientific-use files may be standard or tailor made, i.e. adapted to the particular needs of the research project. The risk of a breach confidentiality is smaller if standard files are released than if specific files are produced on request. For researchers, however, the standard files are often not sufficiently detailed (e.g. the researcher may not need regional details but is interested in the exact age of individuals, whereas the standard files usually provide a medium level of regional details and age in bands). The scientific-use files released by Eurostat are standard, i.e. they are prepared once for all access requests. Production of tailor-made files would be too burdensome, as the SDC protection measures must be always agreed with the NSAs.
Example of partial anonymisation methods for EU Labour Force Survey (LFS) scientific-use files: AGE-by 5-year bands NATIONALITY/COUNTRY OF BIRTH-up to 15 predefined groups NACE (economic activity)-at 1-digit level ISCO (occupation)-at 3-digit level INCOME-provided only as (national) deciles and from 2009 HHNUM-household numbers are randomised per dataset, so that respondents cannot be tracked across time The most common SDC methods to anonymise (partially or completely) the microdata files are: • Recoding: provision of information at the more general level (e.g. age bands instead of exact age). • Micro-aggregation: replacement of the original value of the variable (e.g. income) with the average of some (usually 3-5) similar units. • Record swapping: swapping of, for example, persons between similar households. Swapping adds uncertainty about the identity of the unit in a microdata file. • Rounding: replacement of original value with rounded figure. • (Local) suppression: removal of identifying variables in the record or the entire record (e.g. a very large household). • Sampling: provision of sampled microdata to increase uncertainly about identification as a record referring to particular individual may but does not have to be included in the sample.
The modes of access to secure-use files and scientific-use files are presented in Table 2.
The modes of access listed in Table 2 are complementary and some NSAs provide all options. As the operational costs may be high, the NSA services are sometimes payable.

Table 2
Modes of access to confidential data and respective protection measures Microdata type used Mode of access

Confidentiality protection
Secure-use files -On site (separate room, usually in the premises of the NSA where access is provided; researchers can see the data but have no internet access and cannot download or copy the data) The final results of data analysis are checked for confidentiality (output checking). Each output is checked separately. In some systems, only a sample of output is checked; in others, researchers do it themselves -Remote access (same functionalities as for on-site access but facilities provided online, no need to travel to the NSA) -Remote execution (authorised users submit codes that are executed on the data, but users do not see the data) The results are checked automatically ('on the fly') or manually -There might be also combinations of the above modes of access Scientific-use files Files are sent to authorised researchers and used in the premises of research entities The data are protected (partial anonymisation) before being sent to researchers. Researchers must ensure that the published results do not contain confidential data

Use Case: Access to European Statistical System Microdata (European Microdata)
How does the microdata access system work in practice? Eurostat applies a twostep procedure to grant access to microdata for research purposes. In the first step, organisations interested in accessing European microdata submit an application for recognition to Eurostat. In the second step, researchers from recognised research entities submit their concrete research proposals. 4

Step 1 Recognition as a Research Entity
The recognition of research entities aims at identifying those organisations (or specific departments of the organisations) that carry out research and can be entrusted with confidential data. The assessment criteria refer to the purpose of the entity, its available list of publications and scientific independence. The entities must also describe security measures in place for microdata protection. The content of the application is evaluated by Eurostat. Upon positive assessment, the head of a recognised research entity signs the commitment that the microdata will be used and protected according to the terms agreed. Eurostat publishes the list of recognised research entities on its website. 5 To date (2017) more than 700 research entities were recognised. The majority of them are universities and research organisations (see Fig. 4).
Recognition of research entities was introduced by Eurostat to provide a contractual link with the legal entities, rather than with individual researchers. 6 Step 2 Submission of Research Proposal In the second step, researchers from recognised entities submit their concrete research proposals to Eurostat. Eurostat then consults all national statistical authorities that provided the data. If an NSA refuses the access, the data of that country are removed from the microdata file.
To be eligible, the research proposal must specify the scientific purpose of the research in sufficient detail, justify the need to use microdata and present the expected outcomes of the research. The results of the research must be made public. Each researcher named in the research proposal as a potential user of the microdata signs an individual confidentiality declaration, in which he or she commits to respect the specific terms of use of confidential data.
In the research proposal, researchers choose the microdata collections they are interested in. In 2017 Eurostat granted access to microdata to 12 data collections (see 4 The legal basis for access to ESS microdata is Commission Regulation (EU) No. 557/2013 on access to confidential data for scientific purposes. The Regulation defines criteria for eligible research entities and research proposals. It also describes how the microdata shall be made available to researchers (modes of access). 5 http://ec.europa.eu/eurostat/documents/203647/771732/Recognised-research-entities.pdf. 6 However, in some national systems, only individual researchers are 'recognised'. Annex 1). Most of the European microdatasets are released as scientific-use files. 7 The datasets most frequently demanded by researchers are EU Statistics on Income and Living Conditions (EU-SILC) and Labour Force Survey (LFS). Together they account for more than 70% of all access requests. When the research proposal is accepted, the data are made available to the researchers. Researchers may access the data for the period specified in the research proposal. If so requested, researchers receive new releases of the approved microdatasets.

61%
Once the project is finalised, researchers send Eurostat the resulting publications, which are made available on the dedicated website. 8 Researchers must also destroy the confidential data received.
Eurostat receives around 350 applications for access to microdata per year.

Conclusions
The ESS microdata access system is specific as it creates a single entry point of access to European microdata owned by the NSAs. NSAs agree on the general access conditions (Regulation 557/2013) and are directly involved in decisions on the release of particular datasets in particular ways (anonymisation method and mode of access), and for particular projects (all NSAs are consulted about each access request).
For Eurostat, access to microdata has become a well-established process. Recently, Eurostat worked on modernising the microdata access system, e.g. launching online forms for microdata access applications and piloting online transmission of scientific-use files. The future plans aim to develop remote execution and to publish more public-use files. 9 Closer collaboration with organisations such as CESSDA (Consortium of European Social Science Data Archives) should contribute to the improvement of microdata access services provided by Eurostat. On site (safe centre in Eurostat) and off site a The years covered by the MMD datasets vary from one country to another and are subject mainly to the availability of the Community Innovation Survey and Survey on ICT Usage and e-Commerce in Enterprises data Aleksandra Bujnowska is a Statistical Officer in Unit B1 'Methodology and Corporate Architecture' at Eurostat. She is leading a team 'Statistical confidentiality and access to microdata'. For many years, she has been contributing to the development of the European microdata access system and has made several interventions on this subject at various events. She has also coordinated numerous European projects aiming at wider access to confidential data for scientific purposes and at efficient way of micro-and tabular data protection.

Annex 1: European Microdatasets Available for Scientific Purposes
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.