QAConv: Question Answering on Informative Conversations

This paper introduces QAConv, a new question answering (QA) dataset that uses conversations as a knowledge source. We focus on informative conversations, including business emails, panel discussions, and work channels. Unlike open-domain and task-oriented dialogues, these conversations are usually long, complex, asynchronous, and involve strong domain knowledge. In total, we collect 34,608 QA pairs from 10,259 selected conversations with both human-written and machine-generated questions. We use a question generator and a dialogue summarizer as auxiliary tools to collect and recommend questions. The dataset has two testing scenarios: chunk mode and full mode, depending on whether the grounded partial conversation is provided or retrieved. Experimental results show that state-of-the-art pretrained QA systems have limited zero-shot performance and tend to predict our questions as unanswerable. Our dataset provides a new training and evaluation testbed to facilitate QA on conversations research.


Introduction
Having conversations is one of the most common ways to share knowledge and exchange information. Recently, many communication tools and platforms are heavily used with the increasing volume of remote working, and how to effectively retrieve information and answer questions based on past conversations becomes more and more important. In this paper, we focus on QA on conversations such as business emails (e.g., Gmail), panel discussions (e.g., Zoom), and work channels (e.g., Slack). Different from daily chit-chat (Li et al., 2017) and task-oriented dialogues (Budzianowski et al., 2018), these conversations are usually long, complex, asynchronous, multi-party, and involve strong domain knowledge. We refer to them as informative conversations and an example is shown in Figure 1.
However, QA research mainly focuses on document understanding (e.g., Wikipedia) not dialogue understanding, and dialogues have significant differences with documents in terms of data format and wording style, and important information is scattered in multiple speakers and turns (Wolf et al., 2019b;Wu et al., 2020). Moreover, existing work related to QA and conversational AI focuses on conversational QA (Reddy et al., 2019;Choi et al., 2018) instead of QA on conversations. Conversational QA has sequential dialogue-like QA pairs that are grounded on a short document paragraph, but what we are more interested in is to have QA pairs grounded on conversations, treating past dialogues as a knowledge source.
QA on conversation has several unique challenges: 1) information is distributed across multiple speakers and scattered among dialogue turns; 2) Harder coreference resolution problem of speakers and entities, and 3) missing supervision as no training data in such format is available. The most related work to ours is the FriendsQA dataset (Yang and Choi, 2019) and the Molweni dataset (Li et al., 2020). However, the former is built on chit-chat transcripts of TV shows with only one thousand dialogues, and the latter has short conversations in a specific domain (i.e., Ubuntu). The dataset comparison is shown in Table 1.
Therefore, we introduce QAConv dataset, sampling 10,259 conversations from email, panel, and channel data. The longest dialogue sample in our data has 19,917 words (or 32 speakers), coming from a long panel discussion. We segment long conversations into shorter conversational chunks to collect human-written (HW) QA pairs or to modify machine-generated (MG) QA pairs from Amazon Mechanical Turk (AMT). We train a multi-hop question generator and a dialogue summarizer to generate QA pairs. We use QA models to identify uncertain samples and conduct an additional human verification stage. The data collection flow is shown in Figure 1. In total, we collect 34,608 QA pairs.
We construct two testing scenarios: 1) In the chunk mode, a conversational chunk is provided to answer questions, similar to the SQuAD dataset (Rajpurkar et al., 2016); 2) In the full mode, a conversational-retrieval stage is required before answering questions, similar to the open-domain QA dataset (Chen and Yih, 2020). We explore several state-of-the-art QA models such as the span extraction RoBERTa-Large model (Liu et al., 2019) trained on SQuAD 2.0 dataset, and the generative UnifiedQA model (Khashabi et al., 2020) trained on eight different QA datasets and showed its generalization ability to 12 unseen QA corpora. We investigate the statistic-based BM25 (Robertson et al., 1994) retriever and the neural-based dense passage retriever (Karpukhin et al., 2020) trained on Wikipedia (DPR-wiki). We show zero-shot and finetuning performances in both modes and conduct improvement study and error analysis.
The main contributions of our paper are threefold: 1) QAConv provides a new testbed for QA on informative conversations including emails, panel discussions, and work channels. We show the po-tential of treating long conversations as a knowledge source, and point out a performance gap between QA on documents and QA on conversations; 2) We incorporate question generation (QG) model into the QA data collection, and we show the effectiveness of such approach in human evaluation. 3) We introduce chunk mode and full mode settings for QA on conversations, and our training data enables existing QA models to perform better on dialogue understanding.

QAConv Dataset
Our dataset is collected in four stages: 1) selecting and segmenting informative conversations, 2) generating question candidates by QG models, 3) crowdsourcing question-answer pairs on those conversations/questions, and 4) conducting quality verification and data splits.

Selection and Segmentation
Full data statistics are shown in Table 2. First, we use the British Columbia conversation corpora (BC3) (Ulrich et al., 2008) and the Enron Corpus (Klimt and Yang, 2004) to represent business email use cases. The BC3 is a subset of the World Wide Web Consortium's (W3C) sites that are less technical. We sample threaded Enron emails   One of the main challenges in our dataset collection is the length of input conversations and thus resulting in very inefficient for crowd workers to work on. For example, on average there are 13,143 words per dialogue in the Court dataset, and there is no clear boundary annotation in a long conversation of a Slack channel. Therefore, we segment long dialogues into short chunks by a turn-based buffer to assure that the maximum number of tokens in each chunk is lower than a fixed threshold, i.e., 512. For the Slack channels, we use the disentanglement script from (Chatterjee et al., 2020) to split channel messages into separated conversa-tional threads, then we either segment long threads or combine short threads to obtain the final conversational chunks.

Question Generation
Synthetic dataset construction has been shown to improve robustness (Gupta et al., 2021) and improve the complexity of test sets (Feng et al., 2021). We leverage a question generator and a dialogue summarizer to generate and recommend some questions to workers. We train a T5-Base (Raffel et al., 2019) model on HotpotQA (Yang et al., 2018), which is a QA dataset featuring natural and multihop questions, to generate questions for our conversational chunks. By the second hypothesis, we first train a BART (Lewis et al., 2020) summarizer on News (Narayan et al., 2018) and dialogue summarization corpora (Gliwa et al., 2019) and run QG models on top of the generated summaries.
We filter out generated questions that a QA model can predict the same answers we used in our QG model, which we hypothesize that these questions could be easy questions that we would like to avoid. Note that our QG model has grounded answers since it is trained to generate questions by giving a text context and an extracted entity. We hypothesize that these questions are trivial questions in which answers can be easily found, and thus not interesting for our dataset. Examples of   our generated multi-hop questions are shown in the Appendix (Table 18).

Crowdsourcing QA Pairs
We use two strategies to collect QA pairs, human writer and machine generator. We first ask crowd workers to read partial conversations, and then we randomly assign two settings: 1) writing QA pairs themselves or 2) selecting one recommended machine-generated question to answer. We apply several on-the-fly constraints to control the quality of the collected QA pairs: 1) questions should have more than 6 words with a question mark in the end; 2) questions and answers cannot contain firstperson and second-person pronouns (e.g., I, you, etc.); 3) answers have to be less than 20 words , and 4) all words have to appear in source conversations. We randomly select four MG questions from our question pool and ask crowd workers to answer one of them, without providing any potential answers. They are allowed to modify questions if necessary. To collect unanswerable questions, we ask crowd workers to write questions with at least three entities mentioned in the given conversations but they are not answerable. We pay crowd workers roughly $8-10 per hour, and the average time to read and write one QA pair is approximately 4 minutes.

Quality Verification and Data Splits
We design a filter mechanism based on different potential answers: human writer's answers, answer from existing QA models, and QG answers. If all the answers have a pairwise fuzzy matching ratio (FZ-R) scores 2 lower than 75%, we then run another crowdsourcing round and ask crowd workers to select one of the following options: A) the QA pair looks good, B) the question is not answerable, C) the question has a wrong answer, and D) the question has a right answer but I prefer another answer. We run this step on around 40% samples which are uncertain. We filter the questions of the (C) option and add answers of the (D) option into the ground truth. In questions marked with option (B), we combine them with the unanswerable questions that we have collected. In addition, we include 1% random questions (questions that are sampled from other conversations) to the same batch of data collection as a qualification test. We filter crowd workers' results if they fail to indicate such a question as an option (B). Finally, we split the data into 27,287 training samples, 3,660 validation samples, and 3,661 testing samples. There are 4.7%, 5.1%, 4.8% unanswerable questions in train, validation, and test split, respectively.

QA Analysis
In this section, we analyze our collected questions and answers. We first investigate question type distribution and we compare human-written questions and machine-generated questions. We then analyze answers by an existing named-entity recognition (NER) model and a constituent parser.

Question Analysis
Question Type. We show the question type tree map in Figure 2 and the detailed comparison with other datasets in Table 3. In QAConv, the top 5 question types are what-question (29%), which-question (27%), how-question (12%), whoquestion (10%), and when-question (6%). Comparing to SQuAD 2.0 (49% what-question), our dataset have a more balanced question distribution. The question distribution of unanswerable questions is different from the overall distribution. The top 5 unanswerable question types are what-question (45%), why-question (15%), howquestion (12%), which-question (10%), and whenquestion (8%). Human Writer v.s. Machine Generator. As shown in Table 4, there are 41.7% questions which are machine-generated questions. Since we still give crowd workers the freedom to modify questions if necessary, we cannot guarantee these questions are unchanged. We find that 33.56% of our recommended questions have not been changed (100% fuzzy matching score) and 19.92% of them are slightly modified (81%-99% fuzzy matching score). To dive into the characteristics and differences of these two question sources, we further conduct the human evaluation by sampling 200 conversation chunks randomly. We select chunks that have QG questions unchanged (i.e., sampling from the 33.56% QG questions). We ask three annotators to first write an answer to the given question and conversation, then label fluency (how fluent  and grammatically correct the question is, from 0 to 2), complexity (how hard to find an answer, from 0 to 2), and confidence (whether they are confident with their answer, 0 or 1). More details of each evaluation dimension (Section A.4) and performance difference (Table 12) are shown in the Appendix. The results in Table 4 indicate that QG questions are longer, more fluent, more complex, and crowd workers are less confident that they are providing the right answers. This observation further confirmed our hypothesis that the question generation strategy is effective to collect harder QA examples.

Answer Analysis
Following Rajpurkar et al. (2016), we used Part-Of-Speech (POS) (Kitaev and Klein, 2018) and Spacy NER taggers to study answers diversity. Firstly, we use the NER tagger to assign an entity type to the answers. However, since our answers are not necessary to be an entity, those answers without entity tags are then pass to the POS tagger, to extract the corresponding phrases tag. In Table 5, we can see that Noun phrases make up 30.4% of the data; followed by People, Organization, Dates, other numeric, and Countries; and the remaining are made up of clauses and other types. Full category distribution is shown in the Appendix ( Figure 3). Note that there are around 1% of answers in our dataset are coming from multiple source text spans (examples are shown in Appendix Table 17).

Chunk Mode and Full Mode
The main difference between the two modes is whether the conversational chunk we used to collect QA pairs is provided or not. In the chunk mode, our task is more like a traditional machine reading comprehension task that answers can be found (or cannot be found) in a short paragraph, usually less than 500 words. In the full mode, on the other hand, we usually need an information retrieval stage before the QA stage. For example, in the Natural Question dataset (Kwiatkowski et al., 2019), they split Wikipedia into millions of passages and retrieve the most relevant one to answer.  We define our full mode task with the following assumptions: 1) for the email and panel data, we assume to know which dialogue a question is corresponding to, that is, we only search chunks within the dialogue instead of all the possible conversations. This is simpler and more reasonable because each conversation is independent; 2) for slack data, we assume that we only know which channel a question belongs to but not the corresponding thread, so the retrieval part has to be done in the whole channel. Although chunk mode may be a better way to evaluate the ability of machine reading comprehension, the full mode is more practical as it is close to our setup in the real world.

State-of-the-art Baselines
There are two categories of question answering models: span-based extractive models which predict answers' start and end positions, and free-form text generation models which directly generate answers token by token. All the state-of-the-art models are based on large-scale language models, which are first pretrained on the general text and then finetuned on other QA tasks. We evaluate all of them on both zero-shot and finetuned settings (further finetuned on the QAConv training set), and both chunk mode and full mode with retrievers. In addition, we run these models on the Molweni (Li et al., 2020) dataset for comparison and find out our baselines outperform the best-reported model, DADgraph (Li et al., 2021a) model, which used expensive discourse annotation on graph neural network. We show the Molweni results in the Appendix (Table 11).

Span-based Models
We use several models finetuned on the SQuAD 2.0 dataset as span extractive baselines. We use uploaded models from huggingface (Wolf et al., 2019a) library. DistilBERT (Sanh et al., 2019) is a knowledge-distilled version with 40% size reduction from the BERT model, and it is widely used in mobile devices. The BERT-Base and RoBERTa-Base (Liu et al., 2019) models are evaluated as the most commonly used in the research community. We also run the BERT-Large and RoBERTa-Large models as stronger baselines. We use the wholeword masking version of BERT-Large instead of the token masking one from the original paper since it performs better.

Free-form Models
We run several versions of UnifiedQA models (Khashabi et al., 2020) as strong generative QA baselines. UnifiedQA is based on T5 model (Raffel et al., 2019), a language model that has been pretrained on 750GB C4 text corpus. UnifiedQA further finetuned T5 models on eight existing QA corpora spanning four diverse formats, including extractive, abstractive, multiple-choice, and yes/no questions. It has achieved state-of-the-art results on 10 factoid and commonsense QA datasets. We finetune UnifiedQA on our datasets with T5-Base, T5-Large size, and T5-3B. We report T5-11B size for the zero-shot performance.

Retrieval Models
Two retrieval baselines are investigated in this paper: BM25 and DPR-wiki (Karpukhin et al., 2020). The BM25 retriever is a bag-of-words retrieval function weighted by term frequency and inverse document frequency. The DPR-wiki model is a BERT-based dense retriever model trained for opendomain QA tasks, learning to retrieve the most relevant Wikipedia passage.

Computational Details
We train most of our experiments on 2 V100 NVIDIA GPUs with a batch size that maximizes their memory usage, except T5-3B we train on four A100 NVIDIA GPUs with batch size 1 with several parallel tricks, such as fp16, sharded_ddp and deepseep library. We train 10 epochs for all T5 models and 5 epochs for all BERT-based models. We release hyper-parameter setting and trained models to help reproduce baseline results.

Evaluation Metrics
We follow the standard evaluation metrics in the QA community: exact match (EM) and F1 scores. The EM score is a strict score that predicted answers have to be the same as the ground truth    answers. The F1 score is calculated by tokens overlapping between predicted answers and ground truth answers. In addition, we also report the FZ-R scores, which used the Levenshtein distance to calculate the differences between sequences. We follow Rajpurkar et al. (2016) to normalize the answers in several ways: remove stop-words, remove punctuation, and lowercase each character. We add one step with the num2words and word2number libraries to avoid prediction difference such as "2" and "two".

Chunk Mode
As the chunk mode results on the test set shown in Table 6, UnifiedQA T5 models, in general, outperform BERT/RoBERTa models in the zero-shot setting, and the performance increases as the size of the model increases. This observation matches the recent trend that large-scale pretrained language model finetuned on aggregated datasets of a specific downstream task (e.g., QA tasks (Khashabi et al., 2020) or dialogue task (Wu et al., 2020)) can show state-of-the-art performance by knowledge transfer. Due to the space limit, all the development set results are shown in the Appendix.
We observe a big improvement from all the baselines after finetuning on our training set, suggesting the effectiveness of our data to improve dialogue understanding. Those span-based models, meanwhile, achieve similar performance to Uni-fiedQA T5 models with smaller model sizes. BERT-Base model has the largest improvement gain by 22.6 EM score after finetuning. We find that the UnifiedQA T5 model with 11B parameters cannot achieve performance as good as the 3B model, we guess that the released checkpoint has not been optimized well by Khashabi et al. (2020). In addition, we estimate human performance by asking crowd workers to answer around 10% QA pairs in test set. We collect two answers for each question and select one that has a higher FZ-R score. We observe an EM score at around 80% and an F1 score at 90%, which still shows a considerable gap with existing  models.

Full Mode
The retriever results are shown in Table 8, in which we find that BM25 outperforms DPR-wiki by a large margin in our dataset on the recall@k measure, where we report k = 1, 3, 5, 10. The two possible reasons are that 1) the difference in data distribution between Wikipedia and conversation is large and DPR is not able to properly transfer to unseen documents, and 2) questions in QAConv are more specific to those mentioned entities, which makes the BM25 method more reliable. We show the full mode results in Table 7 using BM25 (DPRwiki results in the Appendix Table 16). We use the top one retrieved conversational chunk as input to feed the trained QA models. As a result, the performance of UnifiedQA (T5-3B) drops by 18.2% EM score in the zero-shot setting, and the finetuned results of RoBERTa-Large drop by 22.2% EM score as well, suggesting a serious error propagation issue in the full mode that requires further investigation in the future work.

Error Analysis
We further check the results difference between answerable and unanswerable questions in Table 9. The UnifiedQA T5 models outperform span-based models among the answerable questions, however, they are not able to answer any unanswerable questions and keep predicting some "answers". More interestingly, we observe that those span-based models perform poorly on answerable questions, as they can achieve a high recall but a low F1 score on unanswerable questions with a binary setting (predict answerable or unanswerable). This implies that existing span-based models tend to predict our task as unanswerable, revealing their weakness of dialogue understanding ability.
Then we check what kinds of QA samples in the test set are improved the most while finetuning on our training data using RoBERTa-Large. We find that 75% of such samples are incorrectly predicted to be unanswerable, which is consistent with the results in Table 9. We also analyze the error prediction after finetuning. We find that 35.5% are what-question errors, 18.2% are which-question errors, 12.1% are how-question errors, and 10.3% are who-question errors.
In addition, we sample 100 QA pairs from the errors which have an FZ-R score lower than 50% and manually check and categorize these predicted answers. We find out that 20% of such examples are somehow reasonable and may be able to count as correct answers (e.g., UCLA v.s. University of California, Jay Sonneburg v.s. Jay), 31% are predicted wrong answers but with correct entity type (e.g., Eurasia v.s. China, Susan Flynn v.s. Sara Shackleton), 38% are wrong answers with different entity types (e.g., prison v.s. drug test, Thanksgiving v.s., fourth quarter), and 11% are classified as unanswerable questions wrongly. This finding reveals the weakness of current evaluation metrics that they cannot measure semantic distances between two different answers.

Related Work
QA datasets can be categorized into four groups. The first one is cloze-style QA where a model has to fill in the blanks. For example, the Children's Book Test (Hill et al., 2015)  In general, our task is also related to conversations as a knowledge source. The dialogue state tracking task in task-oriented dialogue systems can be viewed as one specific branch of this goal as well, where tracking slots and values can be reframed as a QA task (McCann et al., 2018;Li et al., 2021b). Moreover, extracting user attributes from open-domain conversations (Wu et al., 2019), getting to know the user through conversations, can be marked as one of the potential applications. The very recently proposed query-based meeting summarization dataset, QMSum (Zhong et al., 2021), can be viewed as one application of treating conversations as databases and conduct an abstractive question answering task.

Conclusion
QAConv is a new dataset that conducts QA on informative conversations such as emails, panels, and channels. We show the unique challenges of our tasks in both chunk mode with oracle partial conversations and full mode with a retrieval stage. We find that state-of-the-art QA models have limited dialogue understanding and tend to predict our answerable QA pairs as unanswerable. We provide a new testbed for QA on conversation tasks to facilitate future research.

Ethical Considerations
The QAConv benchmark proposed in this work could be helpful in creation of more powerful conversation retrieval and QA on conversations. However, QAConv benchmark only covers a few domains as background conversations. Furthermore, even with our best efforts to ensure high quality and accuracy, the dataset might still contain incorrect labels and biases in some instances, which could be the inherent mistakes from the original dialogue datasets. This could pose a risk if models that are evaluated or built using this benchmark are used in domains not covered by the dataset or if they leverage evidence from unreliable or biased dialogues. Thus, the proposed benchmark should not be treated as a universal tool for all domains and scenarios. We have used only the publicly available transcripts data and adhere to their guideline, for example, the Media data is for research-purpose only and cannot be used for commercial purpose. As conversations may have biased views, for example, specific political opinions from speakers, the transcripts and QA pairs will likely contain them. The content of the transcripts and summaries only reflect the views of the speakers, not the authors' point-of-views. We would like to remind our dataset users that there could have potential bias, toxicity, and subjective opinions in the selected conversations which may impact model training. Please view the content and data usage with discretion.

A.1 Dataset documentation and intended uses
We follow datasheets for datasets guideline to document the followings.

A.1.1 Motivation
• For what purpose was the dataset created? Was there a specific task in mind? Was there a specific gap that needed to be filled?
-QAConv is created to test understanding of informative conversations such as business emails, panel discussions, and work channels. It is designed for QA on informative conversations to fill the gap of common Wikipedia-based QA tasks.
• Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?
-Salesforce AI Research team and HKUST CAiRE team work together to create this dataset.
• Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grantor and the grant name and number.
-Salesforce AI research team funded the creation of the dataset.

A.1.2 Composition
• What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description.
-QAConv has conversations (text) among speakers (people) and a set of corresponding QA pairs (text).
• How many instances are there in total (of each type, if appropriate)?
-QAConv has 34,608 QA pairs and 10,259 conversations. Each conversation has 568.8 words in average and the longest one has 19,917 words.
• Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable).
-The conversations in QAConv are randomly sampled from several conversational datasets, including BC3, Enron, Court, Media, and Slack, and the number of samples is decided based on related work and the budget.
• What data does each instance consist of? "Raw" data (e.g., unprocessed text or images) or features? In either case, please provide a description.
-Each sample has raw text of conversations, speaker names, and QA pairs.
• Is there a label or target associated with each instance? If so, please provide a description.
-Each answerable sample has at least one possible answer in a list format.
• Is any information missing from individual instances? If so, please provide a description, explaining why this information is missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text.
-We do not include the crowd worker information due to the potential privacy issue.
• Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? If so, please describe how these relationships are made explicit. -N/A • Are there recommended data splits (e.g., training, development/validation, testing)? If so, please provide a description of these splits, explaining the rationale behind them.
-We provide official training, development, and testing splits.
• Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide a description.
-There could have some potential noise of question or answer annotation.
• Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? If it links to or relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); c) are there any restrictions] (e.g., licenses, fees) associated with any of the external resources that might apply to a future user? Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, as appropriate.
• Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctorpatient confidentiality, data that includes the content of individuals' nonpublic communications)? If so, please provide a description.
-No, all the samples in QAConv is public available.
• Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If so, please describe why. -No • Does the dataset relate to people? If not, you may skip the remaining questions in this section. -Yes • Does the dataset identify any subpopulations (e.g., by age, gender)? If so, please describe how these subpopulations are identified and provide a description of their respective distributions within the dataset.
-QAConv contains different speakers with their names. Some samples have their role information, e.g., petitioner.
• Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset? If so, please describe how.
-Yes, because some of the conversations are coming from public forums, therefore, people may be able to find the original speaker if they find the original media source.
• Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals racial or ethnic origins, sexual. orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? If so, please provide a description. -N/A.

A.1.3 Collection Process
• How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data (e.g., part-ofspeech tags, model-based guesses for age or language)? If data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verified? If so, please describe how.
-The QA data is collected by Amazon Mechanical Turk. The data is directly observable.
• What mechanisms or procedures were used to collect the data (e.g., hardware apparatus or sensor, manual human curation, software program, software API)? How were these mechanisms or procedures validated? If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)?
-The QA data is collected by Amazon Mechanical Turk, we design a user interface with instructions on the top and then given partial conversation as context.
• Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)?
-Crowdworkers. We paid them roughly $8-10 per hour, calculated by the average time to read and wriite one QA pair is approximately 4 minutes.
• Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created.
-The data was collected during Feb 2021 to March 2021.
• Were any ethical review processes conducted (e.g., by an institutional review board)? If so, please provide a description of these review processes, including the outcomes, as well as a link or other access point to any supporting documentation.
-We have conduct an internal ethical review process by Salesforce ethical AI team, https://einstein.ai/ethics.
• Does the dataset relate to people? If not, you may skip the remainder of the questions in this section. -Yes.
• Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)?
-We obtain the data through AMT website.
• Were the individuals in question notified about the data collection? If so, please describe (or show with screenshots or other information) how notice was provided, and provide a link or other access point to, or otherwise reproduce, the exact language of the notification itself.
-Yes, the turkers know the data collect procedure. Screenshots are shown Figure 4, Figure 5, Figure 6 in the Appendix.
• Did the individuals in question consent to the collection and use of their data? If so, please describe (or show with screenshots or other information) how consent was requested and provided, and provide a link or other access point to, or otherwise reproduce, the exact language to which the individuals consented.
• If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses? If so, please provide a description, as well as a link or other access point to the mechanism (if appropriate).
• Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted? If so, please provide a description of this analysis, including the outcomes, as well as a link or other access point to any supporting documentation. -N/A

A.1.4 Preprocessing/cleaning/labeling
• Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description. If not, you may skip the. remainder of the questions in this section.
-We conduct data cleaning such as removing code snippets before asking the crowd workers to provide corresponding QA pairs. Thus, no additional cleaning or preprocessing is done for the released dataset, only the reading scripts used to change the format for model reading are used.
• Was the "raw" data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)? If so, please provide a link or other access point to the "raw" data.
-Yes, in the same link.
• Is the software used to preprocess/clean/label the instances available? If so, please provide a link or other access point.

A.1.5 Uses
• Has the dataset been used for any tasks already? If so, please provide a description.
-It is proposed to use for QA on conversations task.
• Is there a repository that links to any or all papers or systems that use the dataset? If so, please provide a link or other access point.
-It is a new dataset. We run existing state-ofthe-art models and release the code.
• What (other) tasks could the dataset be used for?
-Many conversational AI related tasks can be applied or transferred, for examples, conversational retrieval and conversational machine reading.
• Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? For example, is there anything that a future user might need to know to avoid uses that could result in unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) or other undesirable harms (e.g., financial harms, legal risks) If so, please provide a description. Is there anything a future user could do to mitigate these undesirable harms?
-Different ways to disentangle conversations could impact the overall performance. In our current setting, we use and release the buffer-based chunking mechanism.
• Are there tasks for which the dataset should not be used? If so, please provide a description.
-Conversations from Media corpus should not be used for commercial usage.

A.1.6 Distribution
• Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? If so, please provide a description. -No.
• How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have a digital object identifier (DOI)?
• When will the dataset be distributed? -Released.
• Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? If so, please describe this license and/or ToU, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions.
• Have any third parties imposed IP-based or other restrictions on the data associated with the instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms, as well as any fees associated with these restrictions. -No.
• Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any supporting documentation.
-Media dataset is restricted their conversations to be research-only usage.
-Salesforce AI Research team. Chien-Sheng (Jason) Wu is the corresponding author.
• How can the owner/curator/manager of the dataset be contacted (e.g., email address)?
-Create an open issue on our Github repository or contact the authors.
• Is there an erratum? If so, please provide a link or other access point. -No.
• Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? If so, please describe how often, by whom, and how updates will be communicated to users (e.g., mailing list, GitHub)?
-No. If we plan to update in the future, we will indicate the information on our Github repository.
• If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were individuals in question told that their data would be retained for a fixed period of time and then deleted)? If so, please describe these limits and explain how they will be enforced. -No.
• Will older versions of the dataset continue to be supported/hosted/maintained? If so, please describe how. If not, please describe how its obsolescence will be communicated to users.
-Yes. If we plan to update the data, we will keep the original version available and then release the follow-up version, for example, QAConv-2.0 • If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? If so, please provide a description. Will these contributions be validated/verified? If so, please describe how. If not, why not? Is there a process for communicating/distributing these contributions to other users? If so, please provide a description.
-Yes, they can submit a Github pull request or contact us privately.

A.2 Test Data Additional Verification
After random split, we run an additional verification step on the dev and test set. If the new collected answer is very similar with the original answer (FZR score > 90), we keep the original answer. If the new answer is similar within a margin (90 > FZR score > 75), we keep both answers. If the new answer is very different from the original answer (75 > FZR score), we will run one more verification step to get the 3rd answers. We pick the most similar two answers as the gold answers if their FZR score is > 75, otherwise, we manually looked into those controversial QA pairs and made the final judgement.
• Enron: Creative Commons Attribution 3.0 United States license.
• Court: This material is based upon work supported in part by the National Science Foundation under grant IIS-0910664. Any opinions, findings, and conclusions or recommendations expressed above are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
• Media: Only the publicly available transcripts data from the media sources are included.
• Slack: Numerous public Slack chat channels (https://slack.com/) have recently become available that are focused on specific software engineering-related discussion topics.
A.4 Human evaluation description of human-written and machine-generated questions.
Rate [Fluency of the question]: • (A) The question is fluent and has good grammar. I can understand clearly.
• (B) The question is somewhat fluent with some minor grammar errors. But it does not influence my reading.
• (C) The question is not fluent and has serious grammar error. I can hardly understand it.
Rate [Complexity of the question]: • (A) The answer to the question is hard to find. I have to read the whole conversation backand-forth more than one time.
• (B) The answer to the question is not that hard to find. I can find the answer by reading several sentences once.
• (C) The answer to the question is easy to find. I can find the answer by only reading only one sentence.
Rate [Confidence of the answer]: • (A) I am confident that my answer is correct.
• (B) I am not confident that my answer is correct.        Steve Duffy: ..., but I don't know if Enron would even consider this. Studdert might have\nthe best feel for this. Separately, the defendant group will get back to us\non any offer they might be willing to make to settle just the Montana case,\nbut it appears that their real interest would be in a \"global\" deal. Any\ncomments? SWD Michael Burke: Steve, Stan and I have discussed this and we agree that Mike Moran should\ntake the lead and explore all aspects of an Enron Global deal. I know that\nyou will assist Mike in this endeavor. thanks, mike MR. FREEDMAN (RESPONDENT): . . . They both deserve the death penalty. They -they were -the prosecutors were aware that the -the death penalty is what stirs the pot here, and so they were urging somebody to be the shooter to get the death penalty. If this wasn't a death penalty case, I don't think they -it would have mattered who killed who. And so they were urging -JUSTICE KENNEDY: Well, I think there's quite a difference in -in case A where you say our position is that Stumpf was the shooter, pure and simple. That's it. In case B, they say we think Stumpf was the shooter. We're not 100 percent sure, but he should get the death penalty. The alternative is before the sentencer and the sentencer can make that determination.

Question
Which person was mentioned as the shooter in case A and B?