We introduce a new method for selecting questions.•The method identifies the survey questions that best correspond to answers given in a qualitative interview about the concept.•We field many survey questions and use machine learning (ML) tools to find the questions that best predict the coded qualitative data.•We apply the method to create a five-question survey module to measure women’s agency in north India.•The questions chosen are similar across three ML algorithms, and the resulting survey measure of women’s agency performs well in diagnostics. Open-ended interview questions elicit rich information about people’s lives, but in large-scale surveys, social scientists often need to measure complex concepts using only a few close-ended questions. We propose a new method to design a short survey measure for such cases by combining mixed-methods data collection and machine learning. We identify the best survey questions based on how well they predict a benchmark measure of the concept derived from qualitative interviews. We apply the method to create a survey module and index for women’s agency. We measure agency for 209 married women in Haryana, India, first, through a semi-structured interview and, second, through a large set of close-ended questions. We use qualitative coding methods to score each woman’s agency based on the interview, which we use as a benchmark measure of agency. To determine the close-ended questions most predictive of the benchmark, we apply statistical algorithms that build on LASSO and random forest but constrain how many variables are selected for the model (five in our case). The resulting five-question index is as strongly correlated with the coded qualitative interview as is an index that uses all of the candidate questions. This approach of selecting survey questions based on their statistical correspondence to coded qualitative interviews could be used to design short survey modules for many other latent constructs. An examAgency is not directly observable, and it is multi-faceted: It encompasses the many domains of a person’s life including reproductive health, employment, and household finances, and it is defined as having both instrumental and intrinsic value to a person.In this study, we develop a new method to select a few survey questions to measure a latent construct. The method delivers a set of survey questions of a desired length and an index . We apply the method to create a five-question module for women’s agency using data collected from married women in rural Haryana, India.How well a psychometric captures the concept it is trying to measure is called its validity . Our metWe refer to this new approach to survey module design as MASI, for MAchine learning and Semi-structured Interviews.N studies. We collect the richer measure for a relatively small sample to serve as a benchmark in a one-time exercise. The short survey module developed using this small sample can then be incorporated into large-N surveys, with some confidence that it provides an accurate measure of the construct, despite its brevity.If a richer measure based on semi-structured interviews exists, then why not always use it? Because the measure is time- and skill-intensive, and thus expensive, to collect, making it infeasible for large-To implement this approach, we collected data on women’s agency in multiple ways. First, trained qualitative researchers conducted semi-structured interviews. These interviews provide nuanced data but require highly skilled staff to conduct and code them. Second, we collected another candidate benchmark measure of agency using a lab game. In the game, which we adopt from The third way we measure women’s agency is through close-ended survey questions. We ask a long list of questions, drawing on existing survey instruments. Our objective at the data collection stage was to be comprehensive and agnostic about which were the best questions, and then to later use a data-driven approach to select the best ones. There is nothing special about five questions, but this length seems appropriate for survey designers seeking a short module on agency.The goal of our statistical analysis is to identify the best close-ended questions to field from among many candidates. The algorithms we use build on standard supervised machine learning techniques, adding a constraint on the number of survey questions that are selected. This type of problem is referred to as feature selection. We apply three feature selection algorithms. Our preferred algorithm is LASSO stability selection, in which the top questions are those selected most frequently when LASSO is repeatedly run on subsamples of the data . This meTurning to our results, when we use the qualitative interviews as the benchmark measure, all three of the statistical algorithms produce an index of women’s agency that is quite strongly correlated with the interview score. There is considerable overlap in the top questions selected by each algorithm. In addition, the five-question indices are considerably more correlated with the benchmark than if we had chosen the subset of questions randomly. They also perform better than indices constructed from all 63 candidate questions, either their first principal component or a standardized index that averages them. Interestingly, the algorithm-selected questions are quite specific ones about decision-making in particular situations, rather than questions that ask women about their power in general.In the lab game, the premise is that a woman with less agency will more often choose money for herself because she would not have a say in how money given to her husband is spent. We do see this behavior, but we also see an opposing force: some women with very low agency never want money for themselves because they view money as men’s domain or are fearful of their husband finding out and becoming angry. Even when we take into account this bimodal behavior – women with low agency either have very high or very low demand to receive money themselves – the survey index obtained when we apply the statistical techniques is only weakly correlated with the lab game behavior. We conclude that only the semi-structured interviews can be considered a good benchmark measure of agency in our setting. Another advantage of the qualitative interviews is that they cover many domains of agency, not just financial agency.The primary contribution of our study is methodological: We introduce a new mixed-methods way to develop a survey measure. Using qualitative methods in the design of measurement scales is not new . For exaA second contribution of our study is the new short survey module and index for women’s agency that we develop. Our study thus adds to the literature proposing measures of women’s agency or empowerment, which we review in the next section. We created a module optimized for use in north India. One direction for future research is to replicate the study elsewhere to create short modules appropriate for other contexts and to assess the extent to which the same questions are or are not selected elsewhere. One could also apply our method to design a “universal” module based on how robustly it predicts qualitative interview scores across multiple contexts.22.1Agency is one aspect of women’s empowerment. Empowerment as defined by Agency specifically refers to the ability to make decisions and act on one’s goals. It is often defined in a way that captures both an intrinsic characteristic and something with external, instrumental value. To do this, many definitions of agency reflect both an internal feeling of agency  and the external actions of pursuing goals, which is the instrumental aspect of agency .Scholars have also highlighted that the conceptualization of women’s agency depends on the context, for example differing in more coercive settings. Individual actions must be viewed within social, economic, and cultural contexts, and there are multiplicities and hidden forms of women’s agency .2.2There is an array of research on how to measure women’s empowerment and agency. Recent proposed measurement tools include the Women’s Empowerment in Agriculture Index (WEAI)  and PRO-Another measure is the Survey-based Women’s Empowerment Index (SWPER), which was developed by analyzing responses to Demographic and Health Survey questions among partnered women in 34 African countries . SWPER iA different strand of the literature assesses current practices for measuring women’s agency. 33.1We selected Kurukshetra district in the Indian state of Haryana as the study site based on several considerations. We chose north India because of our knowledge of the context and because women’s agency is an important topic of study there. To match our team’s language skills, we restricted attention to Hindi-speaking areas. Within this narrowed set of possible sites, we chose Kurukshetra for practical reasons. First, we could draw on a pool of female surveyors who had worked on earlier studies conducted by J-PAL South Asia, the research organization through which our fieldwork was conducted. Second, the main town was large enough that we could recruit two lead research assistants from New Delhi who would be willing to be based there for several months. Third, Kurukshetra was within a few hours of New Delhi by car or train, which facilitated site visits by the principal investigators.We focused on the rural population and worked backwards from our target sample size of 210 semi-structured interviews to determine how many villages within Kurukshetra to include in our sample. We were able to recruit two qualitative interviewers, and about 100 interviews each was the most they could conduct within the three months we had planned for the data collection. We wanted to complete data collection in each village within two or three days so that there would not be discussion among women about our study that might prime their answers. We expected each interviewer to conduct two to three interviews per day, which implied that our team should conduct about 10 qualitative interviews per village. We thus included 21 villages in our sample in order to complete roughly 210 interviews.We had a separate, larger team of surveyors that conducted the quantitative surveys and lab game. The quantitative team spent about the same number of days in each village, collecting data from twice as many women. The final sample size for that team was 443 women, of whom the 209 semi-structured interviewees are a subset.We chose a random sample of villages for the study that were representative of Kurukshetra, with the selection stratified by village population, distance from the district headquarters, and the ratio of male to female literacy.3.2We used the ASHA lists to choose a preliminary random sample of eligible women in each village. Our eligibility criterion was that a participant was a married woman with a child under the age of 10; we wanted the sample to be homogeneous in this way so that we could ask everyone similar questions, for example about their relationships with their husbands and about decisions over children’s health. The ASHA data included a household roster but not relationships among household members, so we chose households with a child under age 10 and a woman at least 15 years older than that child, who was feasibly the child’s mother. We aimed to enroll 20 women per village (with no more than one enrolled woman per household) in the study, and we randomly chose 50% of them for the semi-structured interview.We collected the data between February and May 2019. We varied whether the qualitative or quantitative data collection came first. The quantitative team started fieldwork in a random half of villages, and the qualitative team started in the other half; halfway through the data collection, they switched villages. 8The first step when the first team visited a household was to verify the woman’s eligibility for the study, which also required that she speak Hindi.44.1We administered a 45-min survey that asked close-ended questions to the full sample of 443 study participants. It was conducted by female enumerators.After asking a few questions on demographic characteristics such as age and religion, the questionnaire focused on measures of women’s agency within her household. We asked a long list of such questions, aiming to be exhaustive. We drew on existing questions to measure instrumental and intrinsic agency from other surveys. These included questions from the Demographic and Health Surveys, Relative Autonomy Index , a J-PALConcatenating all of the existing modules would introduce a lot of redundancy, resulting in a long and repetitive survey from the respondent’s point of view, so we made judgment calls in removing questions that overlapped. In total, we asked 63 questions measuring agency. The question order was not randomized. (The list of questions is provided in Appendix B.).Some of the agency questions were about the woman’s say in specific decisions, such as, “If money is available, who in your household decides whether to pay school fees for a relative from your side of the family?” and “Can you go unescorted to the next village?” Other questions were more general, asking the woman about her overall impression of her agency. An example is, “This is a ten step ladder, where on the bottom, the first step, people who are completely coerced or powerless stand, and on the highest step, the tenth step, stand those with the most ability to advance goals that they value in their own homes and in the world. On which step are you today?”.We convert each of the survey responses to a single numerical variable. Some of the responses have a natural numerical unit  or are binary. For questions asked on a Likert scale, we treat the categorical response as a cardinal variable. In a handful of cases where the numerical mapping is less clear, we make judgment calls. For example, in questions asked about whether women make decisions alone, jointly with their husband, or not at all, we code those responses as 2, 1, and 0. Note that we code all of the variables so that a higher value corresponds to more agency.10It is also possible to include multiple variables, or recodings, per survey question; the important constraint is the number of survey questions at the data collection stage, not the number of variables. For the ladder question mentioned above, we could construct variables for the response being 4.2The semi-structured interviews were on average 45 min long. They were conducted primarily by two female interviewers who had prior experience with in-depth interviewing. A third interviewer conducted a few of the interviews. As part of their training, one of the authors (MB) observed each interviewer conducting pilot interviews and provided feedback to improve their interview skills. The interviewers and MB met weekly to discuss substantive and methodological issues that arose, with learnings fed back into subsequent interviews.The interviews, which were recorded, followed an interview guide (see Appendix C) that was refined through piloting. The initial guide covered five domains of agency within the household: the respondent’s decision-making around her children’s education and health, household expenditures, and her own fertility and mobility. In pilot interviews, employment emerged as another theme and was added as a sixth domain to probe in the interview.The interviewers were trained to follow the interview guide and cover all six domains but to use their judgment to phrase questions differently, ask follow-up questions, or otherwise diverge from the guide if they felt that doing so would elicit better information from the respondent. The open-endedness of the interviews and the multiple domains allowed women to discuss direct and hidden strategies and the meanings behind their actions, including “bargaining and negotiation, deception and manipulation, subversion and resistance, and more intangible, cognitive processes of reflection and analysis” .To ensure privacy during the interviews, we paired each interviewer with someone initially recruited for our quantitative surveyor team who acted as a “distractor.” The distractor would have a discussion with other family members in a separate room so that the qualitative interviewer and study participant could have an uninterrupted private conversation.The interviews were transcribed, and two people, the same two who conducted the interviews, coded them using Dedoose software. We randomly assigned which interviews each person coded, so in about half the cases, it was an interview she had conducted.We used a two-step approach to coding, following The analytic codes were used to arrive at ranks  and ranking definitions for each index code. The use of qualitative data to arrive at numeric scores has been widely used in participatory rural appraisal (PRA) methods . MB and The ranks ranged from 1 for a woman with the lowest level of agency to a 4 for a woman with the highest level of agency.The one domain not initially coded on a 1 to 4 scale is fertility. Many women had discordant levels of agency across the four sub-domains of number of children, birth spacing, reversible birth control, and sterilization, so we coded a woman separately in each of the sub-domains and then averaged these scores. This fertility score was then re-scaled to also range from 1 to 4. Every domain had multiple questions, but when defining ranks, only the questions in the fertility domain required that sub-questions be mapped individually. In the example above on mobility, there were multiple questions on specific locations, but it was not necessary to map responses to locations separately because what mattered is whether women needed permission and needed accompaniment in general. We then calculate an overall agency score for the woman as the average across the six domains.4.3We also used a lab-in-the-field game to measure women’s agency over household income. The game was conducted during the same visit and by the same surveyor as the quantitative survey. It took place in private at the end of the survey and took on average 15 min.The measure uses real-stakes choices the woman makes, specifically her willingness to pay (WTP) to be the recipient of money given to the household. This measure was developed by In the game, the woman is offered choices between ₹ 300 (4 USD) for herself and different amounts of money to be given to her husband.As This reasoning implies women should have a positive WTP to control the money, with perhaps some highly empowered women have a WTP of zero. However, in our context, many women always preferred that their husband get the money even when it was less than ₹ 300 and thus they had a negative WTP. We debriefed with women who had a negative WTP to understand their behavior in the game . This re5The goal of our data analysis is to choose the best five survey questions to measure women’s agency. We do so by selecting those that are the best predictors of a benchmark measure of agency.An intuitive approach to finding the best subset of survey questions would be to try every possible combination of five questions and use the set that yields the highest Standard supervised machine learning techniques like LASSO and random forest share our goal of out-of-sample prediction.Below we first describe LASSO stability selection, which is our preferred approach; it strikes a balance between simplicity and robustness. The second algorithm builds on random forest and is more complex, while the third algorithm, backward sequential selection, is the simplest one. At the end of this section, we compare the algorithms in more detail.5.1In the LASSO stability selection algorithm, the best questions are those most commonly selected when LASSO is repeatedly run on subsamples of the data. 151.Draw a 50% subsample of observations without replacement.2.Run a LASSO regression of the benchmark measure of the outcome on all of the survey variables, keeping track of which predictors are selected, i.e., have coefficients not shrunk to 0.173.Complete 1000 iterations of steps 1 and 2.The proposed survey module consists of the five survey questions chosen most frequently by LASSO across the iterations. We then combine them into an index by normalizing each of the variables to have a standard deviation of 1 and mean of 0 and averaging the standardized variables. We refer to this type of aggregation as a standardized index. Using (regular or LASSO) regression coefficients as weights to create a weighted index is another natural way to combine the variables. We opt for just an average of the standardized variables for simplicity and to make the aggregation less dependent on the estimates.We use 50% subsamples and run LASSO 1000 times:16Unlike in some prediction exercises, there is a “correct” sign of each regression coefficient in our case. The premise of our criterion validation exercise is that we are regressing one measure of agency on another, so the sign of the coefficients should be positive. Nothing in the statistical procedure constrains the coefficients to be positive. Thus, one diagnostic for how well the procedure works is whether any of the coefficients are wrong-signed.5.2The second algorithm we use is 19This algorithm is considerably more complicated than the other two we implement. A reader who is not interested in the technical details can skip the rest of this subsection.1.Build 100 random forests using all of the available predictors. Calculate the average across the forests of each variable’s variable importance (VI), which is a measure of the improvement in model prediction when one includes the variable.212.Build 100 random forests using the most important variable from step 1, then 100 random forests using the two most important variables, and continue up to 100 random forests using all variables retained in step 1. From among these models (where each model is an average of 100 forests), retain the smallest one  among those with an out-of-bag (OOB) error less than a threshold.223.Build another set of random forest models, sequentially introducing the variables retained after step 2, in the order of VI from step 1. Build and average 100 random forests that include the introduced variable. Keep the variable in the model if it decreases OOB error, relative to the model thus far, by more than a threshold amount.23We tune the threshold in the final step of the algorithm so that the desired number of variables (five) are selected.24The algorithm proceeds as follows:5.3The third algorithm we use is a simplified version of a backward sequential selection technique using linear regression . The genk variables left, for all combinations of We start with the full set of survey questions and iteratively remove the one that adds the least predictive power (for predicting the benchmark measure), stopping when the target number of questions  are left.1.k variables into an index .Discard one of the available variables and combine the remaining 2.Calculate the correlation coefficient between the benchmark measure of agency and the index.3.Repeat steps 1 and 2 for all remaining variables.4.k in the set.Drop from the set the variable that led to the smallest decrease (or largest increase) in the correlation coefficient, relative to including all 5.Repeat steps 1 to 4 until the desired number of variables for the index is reached.The last five questions that remain comprise the proposed survey module, and the standardized index based on them is the proposed measure of women’s agency.The first step is to combine all the candidate survey variables on agency into an index. Then we iteratively remove variables as follows:Note that we do not include any cross-validation in the algorithm, although in principle one could.5.4Our rationale for using three different algorithms was to better understand how sensitive the general approach — combining machine learning and qualitative interviews for survey design — is to the specific statistical algorithm used.LASSO stability selection and random forest selection both address over-fitting in each iteration or decision tree. An advantage of the LASSO approach is the final model’s transparency or interpretability. The model prediction is a parsimonious five-term linear equation. For random forest, the model prediction is an average across many trees of many interaction and non-linear terms. Moreover, the “wrapper algorithm” around LASSO used in LASSO stability selection is simple iteration, while the VSURF (random forest) wrapper algorithm is more complex. Thus, LASSO stability selection’s attractiveness relative to random forest selection is the transparency of the algorithm and the resulting model.Backward sequential selection’s disadvantage is that, in our implementation of it without cross-validation, it does not address over-fitting. Its advantage is its simplicity: It uses a standard linear regression in each iteration.For each of the algorithms, we propose to combine the five variables into a standardized index. The algorithms differ in how restrictive this method of aggregation is. Backward sequential selection optimizes the predictive power of the top five questions when they are combined in this way; there is no mismatch between the predictive model and how the selected questions are then aggregated. LASSO stability selection collapses each question to a linear variable, which matches how the questions are then aggregated. However, the top variables are chosen without their aggregate predictive power taken into consideration. Two highly ranked variables could be collinear and thus redundant, with each chosen in different LASSO iterations. (This does not occur in practice in our application). Aggregating via a standardized index is the least appropriate for random forest. The advantage of random forest is that it allows for non-linearities and interaction terms, but the aggregation then discards this information. Thus, when we present the results, we also consider the predicted value from the model as an alternative index. For random forest, this alternative index has a much stronger correlation with the benchmark measure.Putting this all together, we favor LASSO stability selection among the algorithms because it addresses over-fitting yet is transparent and intuitive. Backward sequential selection is a potentially useful alternative because it involves nothing more than a loop over ordinary linear regressions. Random forest can extract more information from five variables, so it might be the first choice of researchers who are undeterred by a more complex algorithm and index.66.1We report the best set of survey questions to measure agency, as determined by the MASI method, in Interestingly, none of the general questions that ask a woman to assess her overall agency or perception of her power are among the top questions. The top three questions ask about her role in specific purchase decisions: large household purchases, clothing for herself, and items in the market. The other two questions pertain to her physical mobility (whether she can visit women in her neighborhood without permission) and to decisions about her children’s health care. The mobility question highlights that the best five-question module is likely to differ by context; restrictions on women’s travel within their village are more common in north India than many other places .All five of the selected variables are predictive in the correct direction; with the variables coded such that a higher value theoretically represents more agency, the raw correlation with the qualitative score is always positive. Appendix r) between the qualitative score and the resulting index is shown at the bottom of The proposed way to combine the survey questions into one measure is to average the five variables: We code each survey question as a continuous variable, make them comparable by normalizing each to have a standard deviation of 1, and then average them. The correlation coefficient , and likewise with the husband-wife education gap. A first step is to check the correlation between these factors and the qualitative score itself. As reported in Appendix 6.7The fact that the interview is conducted by a specific individual and one person coded the interview adds subjectivity to the qualitative agency score. To assess how much the algorithm-selected survey questions depend on the identity of the interviewer and coder, we repeated the analyses using only the data from one qualitative interviewer or one coder.Appendix 30We view this degree of correspondence across interviewers and coders to be high. Because we are analyzing subsamples, there is more sampling error than when using the full sample (as discussed below). Thus, even if there were no true interviewer or coder effects on the qualitative score, the overlap in selected questions in this analysis would be imperfect.6.8A sample size of 209 qualitative interviews might be impractically large in some applications, due to time or budget constraints. To understand how well MASI would work with a smaller sample size, we drew random subsamples of 100 observations (48% subsamples) and repeated the variable selection process, focusing on the LASSO stability selection algorithm. We repeated this 100 times and assessed how well the 100 resulting indices performed and the degree to which the selected questions overlapped with those chosen with the full sample.The top full-sample question, about the woman’s say in large household purchases, is among the top 5 selected questions 73% of the time when we use 100-observation subsets of the data. On average, 2.4 questions from the full-sample set of five questions were selected using the smaller samples. Another metric for assessing performance is the correlation between the resulting indices and the qualitative score. The average correlation using the smaller subsamples is 0.48; the correlation is 0.54 for the index created using the full sample.To summarize, there is some instability in the specific questions chosen if one uses a smaller sample size. However, much of the value of MASI seems to derive from identifying the best one or two questions plus the next six to ten very good questions. A smaller sample size seems to suffice for these purposes.6.9Given the problems with the lab game discussed in Section 7In this study, we developed a new five-question survey module for women’s agency from a starting set of 63 questions, using a data-driven approach. This short module could be useful for those seeking an off-the-shelf way to measure agency in north India and perhaps elsewhere. The module was created using data from married women with children in one part of India, so a valuable direction for future research is to replicate the study in other populations. Indeed, the fact that some of the selected questions pertain to women’s physical mobility, a dimension of agency particularly salient in India, highlights the context-specificity of women’s agency and its measurement .Another finding that highlights the importance of context is that behavior in a lab game that has been used in Macedonia, Zambia, and Tanzania mapped to agency in too messy of a way in our study to serve as a good benchmark measure. Specifically, the game uses high demand for agency as a proxy for having low agency, but many women with low agency did not want more agency. We conclude that using semi-structured interviews to obtain a benchmark measure of agency is advantageous in large part because such interviews are intrinsically context-specific, with the flow of the conversation adapting to the woman’s responses.The primary contribution of the study is to introduce a new method for developing validated measures of constructs by combining machine learning and semi-structured interviews (MASI). Based on the principle of criterion validation, the method vets quantitative measures of a construct by benchmarking them against semi-structured interviews. Specifically, we use supervised machine learning techniques to select the best survey questions based on how well they predict the measure of agency obtained through in-depth but time- and skill-intensive qualitative interviews.MASI has many other potential applications. For example, the best questions to measure changes in a woman’s agency, such as those caused by policy interventions, might differ from the best ones to measure a woman’s current agency (our focus). One could carry out a similar study to create a survey module optimized for measuring changes, with the data collection carried out at two points in time, and the statistical analysis centered around changes in responses. More broadly, combining machine learning and semi-structured interviews to develop short survey measures of complex constructs has many promising applications beyond women’s agency.Seema Jayachandran: Conceptualization, Methodology, Formal analysis, Writing - original draft, Funding acquisition. Monica Biradavolu: Methodology, Investigation, Formal analysis, Writing - review & editing. Jan Cooper: Methodology, Investigation, Writing - review & editing.The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.


Nanostructured composite particles of nano- and submicron sizes were synthesized by a combination of sol–gel and sonochemical techniques. Their graphene content was 0.8–0.9 wt%. These layered particles consisted of graphene sheets in which zirconia nanocrystals were discretely incorporated. The synthesized powders were characterized using XRD, TEM, HRTEM, diffusion aerosol spectrometry and elemental analysis. A comparison of the compressibility modulus, limit values of linear section deformation and compressibility factor shows that the compressibility of the composite is difficult to achieve compared to that of pure zirconia, apparently, due to the low elasticity of graphene sheets. Nanostructured composite particles of nano- and submicron sizes were synthesized by a combination of sol–gel and sonochemical techniques. One of the main problems in obtaining graphene-based composites is related to the fact that nanoparticles are not evenly distributed over graphene sheets. Homogenization processes involving graphene reinforcement are complicated due to the requirement of providing an even distribution while avoiding the introduction of oxygen-containing defects into graphene.8 Uniform dispersion of components in the material was obtained by dispersing graphene and the ceramic powder separately in isopropanol media and then mixing them in an ultrasonic bath under continuous mechanical stirring.9 Isopropyl alcohol, N-methylpyrrolidone and N,N-dimethylformamide are solvents that can prevent agglomeration in processes in colloidal systems.10The use of nanostructured powders allows obtaining materials with high density and electrical conductivity. They are essential for the development of small-sized high-speed electronic device production with a broad set of functions and at the same time with low energy consumption due to the presence of specific interactions within complex nanostructures and small energy losses to the environment.11–14 Hybrid structures consisting of nanocrystals wrapped in graphene sheets are recognized as the most effective for many relevant applications: fuel cells , solar cells, supercapacitors, electronic devices, Fischer–Tropsch synthesis catalysts, hydro processes, ammonia synthesis/decomposition, photocatalysts for environmental protection (decomposition of organic dyes and solvents), thermal protective coatings, transistors, and biocompatible matrices for immobilization of proteins.15,16 Developing a reliable method for the synthesis of a hybrid with controlled structure and properties was challenging to put into practice.Composites consisting of graphene sheets and zirconia are promising materials for various industrial fields due to their physicochemical characteristics . These features of the composites are due to the presence of graphene.17 It was shown that the addition of not more than 1 vol% graphene to Zr-ceramics makes it possible to obtain a composite material with increased density, hardness, and wear resistance.18 The inclusion of ZrO2 nanoparticles in graphene sheets leads to the production of promising materials for the anodes of lithium-ion batteries.19 This graphene/zirconia composite shows high specific capacity, high coulomb efficiency, excellent rate performance and cycling stability. Due to the sandwich-like stable structure of zirconia, its inert pillar supports graphene, which offers more active sites for Li+ insertion.Only a few publications are available about graphene/zirconia composites and their applications in lithium-ion batteries, supercapacitors, or detectors of organic phosphorus agents. Compared with unoxidized graphene, graphene-oxide and reduced graphene-oxide showed a minor enhancement of fracture toughness due to the degraded mechanical strength of reduced graphene-oxide and the structural defects of graphene-oxide.20 so the importance of structural design is emphasized.21 In 2 reaches several tens of percents.It was found that graphene with a nano laminated structure can effectively improve the strength and hardness of composites without reducing their plasticity,2 with a mesoporous structure was proposed in the article.23 Epichlorohydrin and dimethylformamide were used to reduce oxidized graphene in the synthesis. Du et al. have mentioned the preparation, characterization, and electrochemical properties of graphene/ZrO2 nanocomposites and their application in the enrichment and detection of methyl parathion.24 However, none of the mentioned composites have been in the condensed state of aerogel, and thus have limited applications in many fields due to their quite low specific surface area.An inexpensive and straightforward method for the synthesis of composite aerogels consisting of reduced oxidized graphene and ZrO2 deposition on graphene oxide.25–27A few years ago, some methods were developed for ZrO28,29 on strengthening polymer materials with graphene oxide have been published, they clearly show the positive effect of an additive of 1–2 wt% graphene oxide on the elastic modulus and fracture toughness, which increases the strength of the composite material reinforced with graphene. The strength characteristics of the composite increased by 10–40%, the composite material successfully withstood dynamic loads at higher voltages than the initial one before graphene was dispersed in it.Several studiesetc. In via oxidative exfoliation of graphite and subsequent chemical reduction, have been selected as anode materials.In 2 nanocomposite, highly reduced graphene oxide–ZrO2, was considered using benzyl alcohol as a solvent and stabilising ligand. Electronic interactions between ZrO2 and highly reduced graphene oxide were studied by comparing their electrochemical properties with those of pure ZrO2 and highly reduced graphene oxide using cyclic voltammetry.In 2/graphene oxide mass ratio above 1/2 leads to a decrease in the specific capacitance of the composite. It was suggested that the improvement in the electrochemical properties is due to the nanoscale components and the high electronic conductivity of graphene oxide. Both of these factors contribute to the enhanced charge-transfer-reaction pseudocapacitance of the nanocomposite owing to rapid and reversible redox reactions at its surface.In 2–graphene oxide composites, there is still no reliable and technologically promising method for the production of composites based on oxygen-free graphene and ZrO2. The difficulties in synthesising an electrode material with a stable structure have not yet been overcome. A challenge associated with the problem of obtaining high-quality graphene, free of oxygen-containing functional groups and with a low agglomeration degree, also remains unresolved. We proposed a method for the synthesis of composite nanostructured powder, which combines sol–gel and sonochemical methods. This approach ensures an uniform distribution of both components, graphene sheets and zirconia nanocrystals, in the composite due to the fact that the mixing of the components is carried out at the molecular level.Despite a large number of publications on ZrO34–36 This method ensures an uniform distribution of components in the nanopowders, almost every nanocrystal of metal oxide is wrapped with a graphene sheet. Such nanostructured powders as raw products are very interesting and promising for the synthesis of new types of ceramics with improved physicochemical properties. At the same time, the rheological properties of such powders need to be specifically investigated. It was shown that the presence of sheets of 0.05 wt% oxidized graphene in cement leads to an increase in compressive strength and flexural strength of the material by 15–33 and 41–59%, respectively.37–39 However, it is not known yet if there is any study of nanocarbon species inclusion effect on the rheological properties of nano- and submicron powders. We carry out comparative research of the nanopowders of a graphene-containing composite and pure zirconia obtained from the same sol as a composite. Therefore, we combined physicochemical characterization and the rheological test results of the powders.We have developed a method for producing hybrid nanostructures based on oxygen-free graphene and metal oxides with a nanocrystal size of 3 nm. The uniqueness of this method lies in the combination of sol–gel synthesis and ultrasonic exfoliation of graphene sheets from the surface of synthetic graphite in an organo–inorganic medium and in the synthesis of sol and in the exfoliation of graphene in the same organic amine.22.1.For the synthesis of hybrid nanostructures, the as-prepared Zr-containing sol and a graphene suspension after sedimentation and decanting were used.3)2·2H2O . The concentration of the salt stock solution in deionized water was 0.05 M. Acetylacetone  and N,N-dimethyloctylamine  were used for the complexation and stabilization of sol. The weight ratio of AcAcH/DMOA was 1.5. In the synthesis of Zr-containing sol, the molar ratios of DMOA/Zr and AcAcH/Zr were 2.0 and 4.8, respectively. For the formation of a mesophase, a water–alcohol mixture was used with a water/ethanol volume ratio of 2. The sol was synthesized in a flask with a reflux condenser on a magnetic stirrer at a temperature of 80–90 °C.The Zr-containing sol was synthesized using zirconyl nitrate dehydrate, ZrO had a particle size of 600–800 microns and a purity of 99.99%, while the impurities presented as sulfur (less than 10 ppm) and chloride ions (10 ppm). The obtained graphene sheets were passed to the acidic substrate, where their stabilization took place through the fixation at the aqua-DMOA boundary. Then the obtained graphene suspension was used to synthesize hybrid nanoparticles after sedimentation and decanting.For the obtaining of a mixed colloid, the as-prepared Zr-containing sol and graphene suspension were mixed under stirring (500 rpm) and heating to 60–65 °C on a magnetic stirrer for 20–25 minutes. Then it was evaporated at 95–98 °C and stirred on a magnetic stirrer (500 rpm) until a gel was formed, which was transferred to a porcelain cup and placed in an oven. The subsequent heat treatment was carried out in air at 500 °C for 1 h, resulting in the formation of a fine light beige powder.2.2.40,41 The morphology of the obtained powders was investigated by transmission electron microscopy (TEM) with use of a LEO-912 AB OMEGA instrument at 100 kV. The powder particle size distribution was investigated using an automated diffusion aerosol spectrometer, Model 2702, Aeronanotech (DAS). The weighed portion of the studied powder was 0.001 g; the ultrasonic treatment duration of the sample was 3 min. The relative measurement error was 5%. The HRTEM data and the EDS spectra were obtained on a JEM 2010 instrument (JEOL Ltd.) equipped with an energy dispersive spectrometer . The thickness and the number of layers in the sheets were calculated based on 5–10 measurements on each of 5–6 TEM and HRTEM microphotos for each composite sample. The carbon content of the powders was determined using an elemental analyzer . The bulk density of the synthesized powders after shaking was determined by the Scott method in accordance with international standard ASTM B329-18  and using the JCPDS card index and interpretation software for XRD, PHAN and PHAN%. B329-18  using th2.3.−1. A sample of the test sample weighing 0.15 g was placed in a steel mould. At least 5 measurements were carried out for each sample, then the values were averaged , and calculations were carried out using the formulas below. The average value of the standard deviation of deformation in the entire interval of the experiment was σ1 = 0.0744 and σ2 = 0.0476 for graphene–zirconia composite and pure zirconia powders, respectively. The schematic loading diagram is presented in The rheological properties were tested on a mechanical testing machine . The plunger velocity was 1 mm minh is the change in the height of the bulk layer over time; h0 is the initial height of the bulk layer.The deformation was calculated by the equation:p′ is the limiting change in the linear section pressure value; ε* is the final value of the linear section deformation. The linear section was selected based on the condition of the deviation of the experimental data from the linear curve by more than 2%.The compressibility modulus is numerically equal to the tangent of the linear section slope angle of the diagram strain against deformation, and is calculated by the equation:kc) characterises the reversible decrease in the sample height (volume) under the pressure conditions and is quantitatively determined by the equation:h′ and Δp′ are the limiting values of the height and the linear section pressure variation and h0 is the initial height of the backfill.The compressibility coefficient  wt%. It was incorporated into the composite structure as nanometer thickness graphene sheets  and 31.5 (111) correspond to monoclinic ZrO2 (JCPDS card no. 78-078-1807); the reflections at 2θ c.a. 30.3 (111), 35.2 (200), 50.5 (220), and 60.1 (311) correspond to tetragonal ZrO2 (JCPDS card no. 01-080-2155); the reflections at 2θ c.a. 30.4 (111), 35.3 (200), and 50.6 (220) correspond to tetragonal ZrO1.99 (PCPDF card # 802155). The XRD pattern shows that in the composite, zirconia is present as monoclinic and tetragonal crystallographic ZrO2 modifications, and in the nanocrystalline non-stoichiometric ZrO1.99 phase. The same set of peaks can be seen in the diffractogram of the pure zirconia nanopowder obtained from the same Zr-containing as-synthesized sol. However, the peak ratio is different from that of the composite . The sizes of zirconia crystallites incorporated in graphene are mainly 3–10 nm and do not exceed 15 nm, which corresponds to the TEM and XRD data  of the synthesized powders was 1.30 (±0.05) and 1.50 (±0.06) g cmSince the developed powdery composites are intended for ceramics, a comparative study of the compressibility of the nanostructured graphene–zirconia composite and pure zirconia powders was carried out. As can be seen, at low pressures, the curves increased monotonically linearly at the initial stage, which corresponds to elastic deformation, when the stress–strain relation obeys Hooke's law. Then the curves become bent due to the transition of the systems to the plastic deformation stage due to the generation of dislocations.2 nanopowder.The tangent angle of the linear portions of both curves is equal to the compressibility modulus; therefore, the compressibility modulus of the hybrid nanopowders is much higher than that of pure zirconia. A comparison of the curves shows that the composite goes into the region of plastic deformation at a lower pressure than the pure zirconia nanopowder. It can also be observed that in order to achieve the same degree of deformation, higher pressure is required for the pure ZrO4i.e. in the region of elastic–plastic deformation, when optimization of the particle distribution in the sample volume and then compaction due to accommodation and deformation of the powder particles occur, respectively. As a result, it was found that the introduction of even 1 wt% oxygen-free graphene into the zirconia nanopowder significantly affects its rheological properties, reducing the compressibility factor up to two times due to the elasticity of the incorporated graphene sheets.A composite based on zirconia and oxygen-free graphene was synthesized by a combination of sol–gel and sonochemical techniques. During its synthesis, crystallization was carried out from a liquid substrate in the presence of the as-prepared Zr-containing sol. As a result, layered composite particles of nano- and submicron sizes were formed. In the composite, the zirconia crystallites with sizes less than 15 nm were incorporated into graphene sheets of 2–3 nm thickness. In the composite and in the pure zirconia nanopowder, the dispersion, phase composition, and lattice parameters of zirconia are different, despite the fact that the pure zirconia was obtained from the same sol as the composite. Apparently, graphene sheets have a significant impact on the zirconia crystallization, despite the fact that the graphene content in the composite does not exceed 1 wt%. The structural and phase differences of the powders lead to a difference in their density and compressibility. A comparison of the compressibility modulus, limit values of linear section deformation, and compressibility factor shows that the compressibility of the composite is difficult to achieve compared to that of pure zirconia due to the low elasticity of graphene sheets. It shows up in the first and second stages of pressing, There are no conflicts to declare.


Furthermore, droplet-based hydrovoltaic electricity generator devices based on directly grown graphene were found to exhibit robust voltage output and long cyclic stability, in stark contrast to their counterparts based on transferred graphene, demonstrating the potential for emerging energy harvesting applications. The work presented here offers a promising solution to organize the metal catalytic booster toward transfer-free synthesis of high-quality graphene and enable smart energy generation.Direct synthesis of high-quality graphene on dielectric substrates without a transfer process is of vital importance for a variety of applications. Current strategies for boosting high-quality graphene growth, such as remote metal catalyzation, are limited by poor performance with respect to the release of metal catalysts and hence suffer from a problem with metal residues. Herein, we report an effective approach that utilizes a metal-containing species, copper acetate, to continuously supply copper clusters in a gaseous form to aid transfer-free growth of graphene over a wafer scale. The thus-derived graphene films were found to show reduced multilayer density and improved electrical performance and exhibited a carrier mobility of 8500 cm An effective approach by utilizing metal-containing species, copper acetate, to catalyze the transfer-free graphene growth over dielectric substrates is put forward. Droplet-based hydrovoltaic electricity generator devices based on thus-grown graphene manifest robust voltage output and long cyclic stability. Chemical vapor deposition (CVD) has been recognized as one of the most promising methodologies for the controllable preparation of high-quality graphene. It predominantly uses metals such as Cu as growth supports . The traTo address these issues, strategies that use metals for the transfer-free synthesis of graphene have attracted particular interest. Among various strategies aimed toward this purpose, the use of sacrificial metal layers  has garnered considerable attention . However2-facilitated growth of transfer-free graphene on wafer-scale insulators. Cu(OAc)2 was subjected to volatilization using an independent heating system to enable the delivery of Cu clusters. The derived Cu clusters enable effective decomposition of the CH4 precursor by reducing its activation energy, as exemplified by density functional theory (DFT) calculations. Thus-grown graphene was found to be of high quality, multilayer-deficient and free of Cu residues. In addition, the yielded transfer-free films were found to exhibit favorable electrical performance, harvesting a carrier mobility of 8500 cm2 V–1 s–1 at room temperature; this value is comparable to that of graphene grown on Cu [Taking the above considerations into account, we herein report the Cu(OAc)wn on Cu . As a pr2 are schematically illustrated in Fig. 4 herein) decomposes into active carbon fragments at elevated growth temperatures (∼1080°C). These active carbon fragments are subsequently adsorbed onto the substrate surface and initiate the formation of graphene domains [2 using an independent heating system. The temperature of the heater (150°C herein) must be maintained below the pyrolysis point of Cu(OAc)2 to ensure the in situ generation of Cu clusters in the growth zone . The thus-derived Cu clusters enable effective CH4 decomposition by reducing the activation energy, and further induce the facile linkage of sp2 carbon networks and suppress the formation of amorphous carbon [2 showed a higher percentage of single-layer coverage (black) and a lower percentage of few/multilayer regions (dark blue), whereas graphene without Cu(OAc)2 possessed ample multilayer flakes originating from amorphous carbon impurities [2 was found to reach ∼97% (at 550 nm), with dual-side polished quartz or sapphire as the growth substrate. Figure 2-facilitated route, which showed high transparency and uniformity. Such graphene wafers could be synthesized in a batch-to-batch manner, allowing the production of 10 pieces of 2-inch product per day.The growth processes of graphene films without/with the presence of Cu(OAc) domains . The dec domains . By conts carbon . Notablypurities . This waratures ∼080°C. Th2 was analyzed using DFT calculations [4 involves four steps. In the absence of Cu(OAc)2, the activation energy for each step reaction was calculated to be 2.49 eV for step I, 2.74 eV for step II, 2.65 eV for step III and 1.44 eV for step IV , the threshold barrier for CH4 decomposition into CH3 was calculated to be 0.71 eV, which is markedly lower than that without Cu(OAc)2. Evidently, CH4 decomposes more completely with the assistance of Cu clusters provided by Cu(OAc)2; this is ultimately reflected in the subsequent steps II, III and IV ulations . Typical2-facilitated growth. Figure 2/Si support ; typical graphene signals including the D peak (∼1350 cm−1), G peak (∼1580 cm−1) and 2D peak (2700 cm−1) can be clearly observed. The intensity ratio of the D and G peaks (ID/IG) reflects the defect density of graphene, while I2D/IG reflects the film thickness [ID/IG of graphene prepared by the Cu(OAc)2-facilitated process is 0.41, which is markedly lower than that (0.85) of graphene prepared without Cu(OAc)2, indicating a reduction in defect density and improvement in film quality and purity. Meanwhile, I2D/IG varies from 1.13 to 1.45 in the presence of Cu(OAc)2, suggesting the decreased layer thickness of transfer-free graphene. Raman mapping was further performed to evaluate the uniformity of the graphene film . As plotted in Fig. 2 reflect a significant reduction in ID/IG and a conspicuous increase in I2D/IG with the Cu(OAc)2 facilitated route. Samples prior to the transfer process showed similar results . In addition, this method presents its universality on insulator substrates . Notably, similar trends can be observed by synthesizing transfer-free graphene at different growth temperatures, ranging from 1020°C through 1050°C to 1080°C  of the grown graphene (deducting the substrate background) with Cu(OAc)2 was improved from 96% to 97% hickness . As suchsp2 and sp3 carbon signals, respectively [2-facilitated graphene possesses a weaker sp3 signal, which originates from sufficient decomposition of CH4 to inhibit the generation of disordered carbons [3/2 peak region . These results are in good agreement with the Raman investigation findings and clearly demonstrate the catalytic merits of the introduced Cu(OAc)2. Transmission electron microscopy (TEM) combined with selected area electron diffraction (SAED) was performed to examine the crystal quality of graphene  analysis was performed to probe the surface states during growth. Figure ectively . A broad carbons . Neither2 during synthesis, respectively. The corresponding histograms of the statistical analysis are displayed in the inset. Maintaining an identical optical transmittance of ∼96% at 550 nm, the graphene sample grown with Cu(OAc)2 exhibits a narrow distribution of sheet resistance, affording an average value of 1.24 kΩ sq−1, which is markedly lower than that without Cu(OAc)2 (2.70 kΩ sq−1). Such a sharp reduction in sheet resistance can be ascribed to the reduced defect density and superior crystal quality, which results from the Cu(OAc)2 facilitated growth process. Perfect film uniformity was also validated by the homogeneous sheet resistance mapping results.A four-point probe method was applied to the as-grown samples to investigate their macroscopic conductivity. Figure 2-facilitated growth, reflecting our capacity to tailor the optical and electrical properties by comprehensively determining the growth conditions. In this respect, the performance of our graphene compares favorably with that reported in the literature, demonstrating that it is one of the best results reported so far for the transfer-free growth of graphene on insulators . The droplet could be drawn by the wafer at different velocities, thereby generating a corresponding voltage signal . When the droplet is dragged from one side to another side at a constant velocity of 2 cm s–1, a voltage of 0.5 mV can be responsively produced . Figure –1. In contrast, transferred graphene shows a much lower voltage value, where the curve of the voltage response gradually becomes quite flat until the induced voltage no longer increases at 10 cm s–1. This could be attributed to the unavoidable damage, corrugation and polymeric contamination caused by the transfer procedure, as well as the inferior adhesive force between the transferred graphene and the substrate. To further highlight the advance of our droplet-based electricity generator, long-term cyclic tests were carried out. As shown in Fig. In comparison with previously reported hydrovoltaic devices based on transferred graphene samples, our directly grown graphene device is free of transfer-related contaminants and possesses a robust contact between graphene and the substrate. As shown in Fig. −1 at a high optical transmittance of 96% and a carrier mobility reaching 8500 cm2 V−1 s−1, greatly surpassing their directly grown counterparts reported previously. The thus-derived hydrovoltaic electricity generator based on our high-quality directly grown graphene demonstrates impressive voltage output and advanced cyclic stability, which indicates great promise for smart energy harvesting with environmental friendliness and cost effectiveness.In summary, we have demonstrated a copper acetate-facilitated approach for the transfer-free synthesis of graphene films on dielectric substrates, which readily harnesses high quality, high purity and multilayer deficiency. The obtained graphene films exhibited favorable electrical performance with a sheet resistance reaching 1.24 kΩ sq2 blow. Subsequently, the substrate was loaded into a horizontal quartz tube (1- or 3-inch diameter) inside a furnace, with the temperature set to 1030–1080°C, while the Cu(OAc)2 powder (50 mg) was loaded with a quartz boat upstream the same horizontal quartz tube with another independent furnace, with the temperature set to 150°C. First, the substrate was heated to 1020–1080°C and annealed in air to eliminate surface contamination. Subsequently, Cu(OAc)2 was heated to 150°C to continuously supply metal-containing precursors. Ar was then introduced into the chamber to remove the air, and H2 and CH4 were introduced into the chamber to initiate the growth of graphene. The graphene films were synthesized using the atmospheric-pressure CVD (APCVD) method. As for the 1-inch system, typical graphene growth was performed with a gas mixture of Ar/H2/CH4 (200/100/10–25 sccm) for 2–5 hours. After the graphene growth process, the two heating systems were cut off and the sample was cooled to room temperature in an atmosphere of Ar and H2. For graphene growth without copper acetate assistance, the process was almost the same, but without the use of Cu(OAc)2.Copper acetate-facilitated CVD growth of graphene was performed using two independent heating systems. Prior to the CVD growth of graphene, the substrate  was washed in an ultrasonic bath with deionized water, acetone and isopropanol consecutively, and then dried with a Nab initio simulation package (VASP). The exchange-correlation functional was described by the Perdew-Burke-Ernzerhof version of the generalized gradient approximation and the core region by the projector augmented wave method with the cutoff of the plane-wave set as 400 eV. A 1 × 1 × 1 k-point mesh was used for structure relaxation and climbing image nudged elastic band (CI-NEB) calculations. The vacuum layer was larger than 10 Å to avoid interactions with the neighboring images. Using plane-wave-based total energy minimization, all structures were fully relaxed until the force on each atom was less than 0.01 eV/Å. The decomposition energy of CHx was calculated as the energy difference between CHx and the case that one H atom is moved far away from CHx−1. The Cu4 cluster was the relatively stable structure to participate in the decomposition of CHx.DFT calculations were performed using the Vienna For the fabrication of a hydrovoltaic device based on transferred graphene sample, graphene was grown on copper foil by the CVD method and then transferred onto a quartz substrate. Two terminals of the graphene samples were wired with carbon tape to form an ohmic contact, followed by silver epoxy insulation. The fabrication of a hydrovoltaic device based on the directly grown graphene followed identical procedures.2 SiO2/Si support connected to a variable-speed motor. The droplet can be drawn by the wafer at different velocities. The voltage signal between the two electrodes was recorded in real time using a Keithley DMM6500 multimeter.A droplet of 0.6 M NaCl aqueous solution (∼20 μL) was sandwiched between the graphene sample and a 3 × 3 mmnwab169_Supplemental_FileClick here for additional data file.


One of the effects of COVID-19 pandemic is a rapidly growing and changing stream of publications to inform clinicians, researchers, policy makers, and patients about the health, socio-economic, and cultural consequences of the pandemic. Managing this information stream manually is not feasible. Automatic Question Answering can quickly bring the most salient points to the user’s attention. Leveraging a collection of scientific articles, government websites, relevant news articles, curated social media posts, and questions asked by researchers, clinicians, and the general public, we developed a dataset to explore automatic Question Answering for multiple stakeholders. Analysis of questions asked by various stakeholders shows that while information needs of experts and the public may overlap, satisfactory answers to these questions often originate from different information sources or benefit from different approaches to answer generation. We believe that this dataset has the potential to support the development of question answering systems not only for epidemic questions, but for other domains with varying expertise such as legal or finance.  The COVID-19 pandemic has highlighted two major problems with access to reliable domain-specific information: (1) information providers are rapidly generating reams of potentially vital information that might never reach their intended audience, and (2) information seekers struggle to find answers to their specific and changing information needs in the rapidly growing and changing body of communications. These two problems are effectively two sides of the same coin, and making information easier to discover reduces the potential for vital information to remain overlooked by its intended audience. Information Retrieval (IR) systems  are now an integral part of information seeking and one of the main tools used to satisfy information needs. The pandemic exposed a major problem when relying on information retrieval systems under stressful and fast-changing circumstances: existing solutions retrieve hundreds or thousands of documents –- many of which may not be relevant to the user’s specific information need. This needlessly increases the time and mental effort required by information seekers who must read through these documents to find answers to their information needs. Question Answering is a research area aiming to alleviate the problem of finding pertinent information from thousands of documents by bringing the most salient points of relevant documents to focus immediately, saving time and highlighting information that may have been overlooked.Epidemic QA (EPIC-QA), intended to evaluate automatic approaches for answering ad-hoc questions about the disease COVID-19, its causal virus SARS-CoV-2, related coronaviruses, and the recommended response to the pandemic. While COVID-19 has been an impetus for a large body of emergent scientific research and inquiry, the response to COVID-19 raises questions for the general public. The rapid increase in coronavirus literature and evolving guidelines on community response creates a challenging burden not only for the scientific and medical communities but also the general public to stay up-to-date on the latest developments. Consequently, the goal of the dataset is to evaluate systems on their ability to provide timely and accurate expert-level answers as expected by the scientific and medical communities as well as answers in suitable language for the general public. The EPIC-QA dataset consists of (a) questions collected from various stakeholders who need information on COVID-19, (b) documents – including scientific research articles suited for experts as well as relevant news, social media, and pages from authoritative websites intended for the general public, and (c) answers to these questions extracted from documents in the dataset and systematically annotated and judged by experts.To this end, we propose a new question answering dataset: 1 is the most widely used document collection, containing documents from PubMed, research articles from the World Health Organization (WHO), pre-prints from bioRxiv, medRxiv, and arXiv, and SemanticScholar. CORD-19, however, does not provide questions or answers. In terms of COVID question answering, CovidQA2 provides 124 question–article–answer triplets from 85 articles; COVID-Q3 consists of 1,690 questions (without answers) annotated into 15 categories and 207 clusters; Medical Question Pairs (MQP)4 contains 3,048 pairs of medical questions annotated as being similar or dissimilar to COVID-19 FAQs; the Synergy Task of the 2021 BioASQ Challenge (http://www.bioasq.org/) included four rounds of binary, factoid, list, and summary questions based on the CORD-19 collection; and the 2021 TREC Health Misinformation Track  includes topics focused on COVID-19 with annotations indicating whether documents contradict the topic’s answer. For a detailed review of additional natural language processing resources for COVID-19, see Chen et al. (2021)5. By contrast, EPIC-QA, which incorporates both public-facing and expert-level documents, is the only dataset exploring the difference between answers suitable for the general public and those appropriate for experts. It is the only dataset designed to include a wide variety of unique possible answers for each question.Only a handful of publicly available health-related QA datasets exist. In terms of information retrieval, CORD-19How does the vaccine work? In the eyes of an expert, an answer should indicate the exact mechanism and pathways involved. By contrast, for a member of the general public, this information is too involved and may cause confusion; instead, a more appropriate answer would provide a more general overview of the role of the vaccine and a high-level description of its mechanism. In the context of the rapidly accelerating knowledge of COVID-19, managing this duality between expert- and general- level information is even more important. It is our hope that the EPIC-QA dataset will stimulate research in automatic question answering not only to support providing high-quality timely information about COVID-19, but also to support the development of generalizable approaches to meeting information needs in the face of varying levels of expertise for other domains, such as general healthcare, finance, or legal.While there is overlap in the types of questions asked by different stakeholders, the answers to such questions should vary based on the background knowledge of the user. For example, consider the simple question illustrated in Fig. information retrieval step, in which relevant documents are retrieved for a given question, and an answer extraction step, in which relevant passages or answers are identified within the retrieved documents and extracted as-is or used to synthesize an answer. Recently, end-to-end question answering systems have been developed in which (deep) neural networks are trained to jointly identify relevant passages and extract answers from those passages. Consequently, the EPIC-QA dataset includes two collections intended to facilitate research on (a) answer extraction or (b) end-to-end question answering, respectively. The questions in the answer-extraction dataset are adapted from the queries of the TREC-COVID retrieval shared task (https://ir.nist.gov/trec-covid), and are associated with the set of relevancy annotations from that task. For the end-to-end dataset, we introduce a novel set of questions, and provide relevant documents and answer annotations without associated retrieval relevance rankings. These two collections can be used separately or in concert to study the relationship between document retrieval and answer extraction. Both collections consist of three components:Two sets of manually-produced questions, one asked by the general public as well as another asked by experts pertaining to COVID-19;A collection of documents relevant to COVID-19 including published and pre-print biomedical research articles as well as documents obtained from government websites, news, and social media, automatically parsed into sentences; andA diverse list of manually-produced answers for each question as well as a set of relevant document excerpts with manual sentence-level annotations indicating which answers, if any, are evidenced by each sentence.Question answering systems traditionally involve two main steps: an To facilitate answer-extraction research, the questions in the answer-extraction dataset are additionally associated with a set of relevant documents as might be retrieved by an optimal search engine, while the questions in the end-to-end dataset are not. We detail the process by which each of these components was produced.https://ir.nist.gov/covidSubmit). The majority of these questions originated from users’ interactions with MedlinePlus. Additional scientific questions were developed based on group discussions from the National Institutes of Health (NIH) special interest group on COVID-19, questions asked by Oregon Health & Science University clinicians, and responses to a public call for questions. The TREC-COVID topics included expert-level background descriptions of the user’s information need; to facilitate public-facing question answering we produced public-friendly backgrounds for each question. We provide the document-level relevance judgments produced during the TREC-COVID evaluation for these questions. The answer extraction collection includes 45 expert questions and 42 general public questions with document-level relevance judgments. For the end-to-end collections, a new set of 30 expert and 30 general public questions were developed, none of which were evaluated in TREC-COVID; consequently, these questions do not come with document-level relevance judgments. Example questions are provided in Fig. Two sets of questions are included: one for expert-level questions and one for the general public. By design, many of the expert and consumer questions overlap in focus to facilitate comparison between both types of stakeholders. For the answer extraction dataset, we adapted the topics evaluated in the fourth round of TREC-COVID  and sentence boundaries. To support both levels of expertise, the document collections consists of two parts: scientific and medical research, and public-facing online documents.Each document in EPIC-QA includes explicitly pre-defined 6. The dataset was created by the Allen Institute for AI in partnership with the Chan Zuckerberg Initiative, Georgetown University’s Center for Security and Emerging Technology, Microsoft Research, and the National Library of Medicine – National Institutes of Health, in coordination with The White House Office of Science and Technology Policy. The CORD-19 collection includes a subset of articles in PubMed Central (PMC) as well as pre-prints from bioRxiv and medRxiv. Contexts in this collection correspond to automatically identified paragraphs in the articles’ abstracts, or main texts. We include two snapshots of CORD-19: a snapshot from June 19, 2020 to be used with the answer extraction collection, and a snapshot from October 22, 2020, to be used for the end-to-end collection.We adapt the collection of biomedical articles released for the COVID-19 Open Research Dataset Challenge (CORD-19)7. This collection includes authoritative articles from the Centers for Disease Control and Prevention (CDC); the Genetic and Rare Disease Information Center (GARD); the Genetics Home Reference (GHR); Medline Plus; the National Institute of Allergy and Infectious Diseases (NIAID); and the World Health Organization (WHO). Contexts in this collection correspond to paragraphs or sections as indicated by the HTML markup of the document. All articles were filtered for COVID-19 content using the terms in Fig. https://trec-health-misinfo.github.io/). To avoid misinformation, we only considered Reddit threats from the heavily-moderated /r/askscience community that were tagged as pertaining to COVID-19, Medicine, Biology, or the Human Body  service of the U.S. National Library of Medicine (NLM)Technical Validation for more details). These contexts were pooled at a depth of five for expert questions and a depth of eight for questions from the general public, resulting in roughly 45–60 contexts per question. Human assessors read these retrieved contexts and judged them. Specifically, 17 medical indexers with five to thirty years of experience at the National Library of Medicine analyzed the pooled contexts in two rounds: (1) an answer-key generation round, followed by (2) a sentence-level answer annotation round. Annotations were performed as part of the medical indexers’ official duties at the National Library of Medicine.When producing answers for the EPIC-QA dataset, our intent was to explore the landscape of answers asserted in the document collection. Thus, we consider any statement that answers the question as an “answer” regardless of whether or not the answer is factually accurate at the time the document was authored or given current information. The answers in this dataset are intended as an intermediary step where-in one would like to explore all answers provided by the document collection – both correct answers as well as incorrect answers that people may have discovered on their own. Answers were produced by applying multiple automatic question answering systems to the questions in EPIC-QA to identify contexts that may answer the question  nuggets that they feel should be included in the answer key based on their understanding of the question.In the first round, assessors had access to a pool of contexts identified for a given question as well as an ad-hoc search engine over the document collection. The role of the answer key generation round was for assessors to explore the contexts identified by automatic question answering systems as well as the document collection to determine a set of atomic “facts” that answer the question based on the question’s information need or background see Fig. . As in VCOVID trial, trial pause, lack of benefit, hydroxychloroquine, patient safety, and review of the data.In the second round, the answer key (list of nuggets) was fixed. Assessors were given the same set of pooled contexts used in round one. This time, they were asked to annotate all sentences in each context indicating which nugget(s) (if any) are addressed by each sentence. For example, for the question “Which COVID-19 vaccine trials were paused and what were the health safety concerns?” the sentence “[I]n the light of a recent publication...on the lack of safety and efficacy of HCQ in the treatment for COVID-19 patients, the Executive Group of the Solidarity Trial decided to implement a temporary pause of the HCQ arm within the trial as a precaution, while the safety data is being reviewed” was annotated as containing the expert nuggets: 12. The questions in the EPIC-QA dataset are provided in four files following the JSON format illustrated in Fig. ae/expert_questions.json contains 45 expert-level questions produced for the answer extraction collection; andae/general_questions.json contains 42 questions from the general public produced for the answer extraction collection.e2e/expert_questions.json contains 30 expert-level questions produced for the end-to-end collection; ande2e/general_questions.json contains 30 questions from the general public produced for the end-to-end collection.The Epidemic Question Answering (EPIC-QA) dataset is available through the Open Science Foundation (OSF) at 10.17605/OSF.IO/VNYK8ae/cord19/ contains 129,069 JSON files, each corresponding to a biomedical research article from a June 19, 2020 snapshot of CORD-19; andae/general/ contains 925 JSON files, each corresponding to a government or health agency website crawled on June 10, 2020.e2e/cord19/ contains 236,035 JSON files, each corresponding to a biomedical research article from an October 22, 2020 snapshot of CORD-19;e2e/ask_science/ is initially empty , however, it can be populated using the Python script scripts/populate_reddit.py to create 263 JSON files, each corresponding to a post made between December 01 2019 and October 29, 2020 to the /r/askscience community on Reddit tagged by community moderators as corresponding to COVID-19, health, medicine, or biology, which contained one of the COVID-19 keywords shown in Fig. 3e2e/ccns_trec/ contains 114,645 JSON files, each corresponding to an HTML-parsed website included in the CommonCrawl News Subset used by the TREC Misinformation Track containing one of the COVID-19 keywords shown in Fig. e2e/chqa contains 2,739 JSON files, each corresponding to an October 9, 2020 HTML-parsed snapshot of a web page affiliated with the NIH, CDC, or WHO containing one of the COVID-19 keywords shown in Fig. The documents included in the EPIC-QA dataset are provided separately for the answer extraction and end-to-end collections. For each collection, each document is provided as a separate JSON file, adhering to the JSON schema illustrated in Fig. https://github.com/usnistgov/trec_eval) format, as illustrated in Fig. Technical Validation for details), as well as an “ideal” ranking of answers based on our assessor’s answer annotations.Finally, answers are provided in four files, corresponding to the answer extraction and end-to-end collections of expert and public questions. For each question: each answer  is associated with the set of sentence-IDs that nugget was identified within, as shown by the JSON schema in Fig. passages for each question, where a passage was defined as a contiguous sequence of sentences from a single context. We first evaluated the answer extraction collection, and provided the judgments produced from that evaluation as additional data for participants of the later end-to-end evaluation.To validate the EPIC-QA dataset, we organized a community evaluation at the 2020 Text Analysis Conference (TAC), in which participants were provided with the dataset (without answers) and asked to submit ranked lists of up to 1,000 relevant 13 (NDCG) which we refer to as the Normalized Discount Novelty Score (NDNS). Importantly, while the cumulative gain in NDCG can be computed for a document independently of the other retrieved documents, the Novelty Score (NS) measures the information in a passage that has not been seen previously in the ranked list of passages. Formally, we define the novelty score, NS, of passage p as:novel answers in passage p and Exact, which prioritizes passages that contain as many novel nuggets in as few sentences as possible, such that Partial, in which the number of sentences used to express novel nuggets is not considered, i.e., Relaxed, in which the number of sentences used to express novel or previously seen nuggets is not considered, i.e., We encouraged teams to explore diversity in their answers rather than returning many passages providing the most common or obvious answer. To this end, we validated EPIC-QA using a modified version of Normalized Discounted Cumulative GainThe evaluation script is included with the dataset.14 fine-tuned on the SQuAD 2.015 general domain question answering data; their second and third runs additionally fine-tuned on QuAC16, a dialogue-based question answering dataset involving crowd workers and a teacher discussing a passage of text. Interestingly, their third and best-performing run relied on their own information retrieval system rather than the relevance labels produced during TREC-COVID. IBM’s three runs ignored the included relevance labels, using three different ensembles of elastic search with passage retrieval and/or neural re-ranking approaches. Their final runs were a combination of their ensemble information retrieval scores and scores from RoBERTa17 fine-tuned on Natural Questions18, a general domain question answering dataset derived from Google search queries and Wikipedia passages. The HLTRI runs relied on a BioBERT19-based neural passage re-ranker applied to sentence shingles (runs of multiple adjacent sentences) using the TREC-COVID relevance labels. Overall, while the relevance judgments did help with answering questions for the general public, their impact was less clear for expert-level questions, suggesting that documents may be relevant from an information retrieval perspective despite not having explicitly extraditable answers to the question.A total of 17 submissions were received from eight teams for the expert questions, and 10 submissions from six teams for the general public questions, as shown in Table 20, followed by a BERT-based21 re-ranker, and a final-step inspired by recognizing question entailment22. The first run from Yastil_R relied on an information retrieval step informed by BM25, DeepCT23, ColBERT24, and docTTTTTquery25, followed by an answer-extraction step relying on BERT fine-tuned on MS-MARCO26. When looking at the general public, the top performance was exhibited by h2loo, whose runs involved a BM25-based retrieval step followed by point-wise and pair-wise re-ranking using T5, and a final sentence-re-ranking step using maximal marginal relevance (MMR) to improve diversity.For the end-to-end questions, a total of 16 submissions were received from seven teams for expert questions, and 12 submissions from five teams were received for questions from the general public, as shown in Table When comparing between the answer extraction and end-to-end validation results, it is clear that teams generally performed better on end-to-end despite end-to-end being a more challenging problem. This is likely explained by the fact that the answer extraction phase of the TAC challenge was designed and promoted as a preliminary data-collection round, while the end-to-end questions were presented as the primary round for evaluation. It is our belief that the higher performance on the end-to-end evaluation suggests that teams spent more of their research effort on that task than on the answer extraction task.Overall, teams were able to produce satisfactory answers using a variety of approaches based on the EPIC-QA data. Interestingly, the changing in ranking between expert and public performance also suggests that one-size does not fit all, and that systems need to better account for varying stakeholders, such as experts and the general public.Fig. We presented the Epidemic Question Answering (EPIC-QA) dataset, designed to evaluate and draw attention to the important problem of answering questions about COVID-19 from emergent literature, as well as to explore the differences between answers expected by different stakeholders. The EPIC-QA dataset consists of two parts, intended to facilitate answer extraction or end-to-end question answering research, respectively, with each part including (a) two sets of questions collected from two types of stakeholders , (b) manually-produced answers to these questions extracted from documents in the dataset and systematically hand-annotated and judged by human experts, and (c) a document collection including scientific research articles suited for experts as well as relevant news, social media, and pages from authoritative websites intended for the general public. Technical validation accomplished through a community evaluation at the Text Analysis Conference (TAC) indicates that systems can discover useful answers in the collections, and that answers (and supporting passages) expected by different stakeholders can vary substantially. We believe that our technical validation demonstrates the importance of exploring the diverse landscape of answers available online for health questions and shows the importance of accounting for varying levels of understanding when identifying satisfactory answers to health questions, and hope that this dataset will be of value to researchers exploring answer diversity, multiple stakeholders, or open-ended healthcare question answering.https://tac.nist.gov.We have provided instructions for how to process this dataset in the README file provided with the dataset. Descriptions of automatic systems developed using this dataset are available at