WHAT IS RESEARCH RIGOR ? LESSONS FOR A TRANSDISCIPLINE

Aim/Purpose Use of the term “rigor” is ubiquitous in the research community. But do we actually know what it means, and how it applies to transdisciplinary research? Background Too often, rigor is presumed to mean following an established research protocol scrupulously. Unfortunately, that frequently leads to research with little or no impact. Methodology We identify a sample of 62 articles with “rigor” in the title and analyze their content in order to capture the range of perspectives on rigor. We then analyze how these findings might apply to informing science. Contribution This paper offers an approach to defining rigor that is theory based and appropriate for transdisciplinary research. Findings Rigor definitions tend to fall into one of two categories: criteria-based and compliance-based. Which is appropriate depends on the research context. Even more variation was found with respect to relevance, which is often used as a catch-all for research characteristics that aren’t associated with rigor. Recommendations for Practitioners Recognize that when researchers are referring to rigor and relevance, they often mean these to apply to other researchers rather than to practice. When funding research, it is important to understand who the rigor and relevance are directed towards. Recommendations for Researchers When using the term “rigor”, think carefully about which meaning is intended and be transparent about that meaning in your writing. Impact on Society A great deal of public money is invested in achieving research rigor. Society should be aware of what it is buying with that funding. What is Research Rigor? 48 Future Research Developing a better understanding of research fitness and the factors that contribute to it.


INTRODUCTION
One of the most insulting things you can say to a group of academics is that their research isn't rigorous. This is hardly surprising given the importance ascribed to rigor in our doctoral training and in the subsequent review and publication process. What is more surprising is that there is no clear consensus regarding what "rigor" means. Does the progress of our careers truly depend on a concept that is so fuzzy?
Understanding the precise nature of rigor is particularly challenging in transdisciplinary research, such as that emphasized by informing science. To the extent that any consensus on the definition of rigor exists, that consensus is most likely to exist within a discipline or subdiscipline. Given the large differences between how research is conducted in diverse fields-for example, design science research contrasted with finance research-we would not expect that a universal recipe for rigor is likely serve the needs of all research methods. Nevertheless, as a matter of definition, transdisciplinary research necessarily draws upon findings and approaches from multiple disciplines. How can the rigor of the resulting research product be assured? Especially if we aren't sure what rigor is.
In the present paper, we survey how research rigor is defined and assessed across a broad range of disciplines, drawn principally from the social sciences. Our goal is to develop a better understanding of the core aspects of rigor that can be used across disciplines, particularly in the context of interdisciplinary or transdisciplinary research. Our hope is that this analysis may help researchers better assess the role that rigor should play in their research.
Because usage of the term rigor in research is extremely widespread, we begin by describing the process through which 62 social science research articles dealing specifically with rigor were assembled. We then identify the subset of 11 papers where a definition for rigor was included and summarize the differences between definitions. We also attempt to discern different definitions of rigor from the context in which the term is used across the entire set of 62 papers. In doing so, we find considerable variation in where rigor is assumed to exist across the different stages of the research process (e.g., research design, execution, analysis).
The analysis of these articles produced several rigor-related issues that are then discussed. These include: • What are the research outcomes that achieving high levels of rigor are expected to produce? Research rigor does not come for free. Its costs can include economic costs, time costs, and can also extend to accepting constraints on methodologies and on the selection of appropriate topics. Given the various definitions of rigor, what benefits are expected to compensate for these costs? • What is the relationship between rigor and relevance? In nearly 60% of the papers examined, the relationship between rigor and relevance was considered. Interestingly, very different relationships were often posited. • How can existing informing science research contribute to our understanding of rigor? We relate rigor (and relevance) to certain concepts-such as diffusion, filters, and fitness-that have been examined in informing science research.
We conclude the paper by considering how rigor might be defined in a manner that supports transdisciplinary research in informing science.

METHOD
The term "rigor" is widely used in the research literature. For example, a simple Google Scholar search consisting of rigor OR rigour produced over one million records; for purpose of comparison, a similar search consisting solely of the term research produced just over eight million records. Because of this wide usage, an exhaustive study of how rigor is defined and used was not possible. Instead, the following protocol was used to identify a representative body of literature that could be examined: 1. We constrained the search to articles with rigor (or rigour) in the title. The rationale here was that papers that focused specifically on rigor seemed would be the best place to find examples of how it was defined and used. This reduced the number of records to 8220.
2. We limited our search to articles with a substantial number of citations. This appeared to be the best way to focus one widely held perspectives on rigor. We were concerned, however, that this approach would unnecessarily bias our search against recent articles (i.e., articles that had not been around long enough to accumulate many references. For this reason, we adjusted our cut points for inclusion as follows: a 3. We chose to focus on articles in the social sciences only. Our assumption was that research in the natural sciences, life sciences, and humanities was likely to be conducted in ways sufficiently different from most social science research and that including them would unrealistically expand the range of definitions that we would find. The criteria for elimination was based on the article title and publication outlet. The main impact here was the elimination of a surprising large body of literature from the field of nursing.

4.
We included only articles relating to rigor applied to research. As we conducted the search, it became clear that rigor could be applied in very different contexts. For example, it can refer to specific medical conditions (e.g., "rigor mortis") and can also be applied to teaching (e.g., "a rigorous program of study"). While the latter usage certainly bears some relationship to the usage in the context of research, we considered it outside of the scope of the current paper. Most of the selection process was accomplished through the article title and outlet. Five papers were subsequently eliminated based on content.
All articles meeting the four criteria for inclusion were downloaded, a total of 62 articles. For imaged documents, the OCR capability incorporated into Adobe's Acrobat™ product was used to create searchable text.
Each author then independently searched for all occurrences of the term rigor (including rigour, rigorous, and rigourous) within each document, looking for text in which the term was defined or where it was used in a manner that shed light on its intended meaning. In each case, the relevant text was copied into each author's own spreadsheet.
After a first pass, in which the two authors had an 81% level of agreement of whether rigor was defined in each article, we collaborated to reach a 100% consensus. Interestingly, the quotes we both collected independently were highly consistent. The initial disagreements largely arose from interpreting the text. The challenges included the following (bold emphasis added by the authors): • Use of conjunctions making it difficult to disambiguate rigor from other constructs. For example: Although effective writing and transparency are necessary, the rigor and quality of inductive papers rest on three fundamental criteria, as follows. First, and as with all strong theory, is the emergent theory internally coherent and parsimonious? ... Second, are the constructs or themes convincingly grounded in compelling data? ... Third, does the research provide rich and unexpected insights? (Eisenhardt et al, 2016(Eisenhardt et al, , pp. 1120(Eisenhardt et al, -1121 • Presence of multiple definitions that are not necessarily consistent with each other. For example: Rigor is described as demonstrating integrity and competence within a study. ... Schutz's first postulate of logical consistency is similar to the description by Horsfall, Byrne-Armstrong, and Higgs … of rigor in qualitative research, which involves in-depth planning, careful attention to the phenomenon under study, and productive, useful results. Descriptions of theoretical rigor involve sound reasoning and argument and a choice of methods appropriate to the research problem. (Fereday & Muir-Cochrane, 2006, p. 82) • Ambiguity as to whether a statement defines rigor or identifies characteristics that are likely to accompany, or arise as a consequence of, rigor. For example: We argue that rigorous PD [participatory design] work exhibits qualities that are coherent, e.g., it is based on an epistemology that accommodates the values that drive the effort, involves stakeholders in ways that reflect these foundations and accordingly defines and delivers its meaningful outcomes. (Frauenberger et al., 2015, p. 103) Where rigor could not be disambiguated from other concepts, we treated the article as not defining rigor. On the other hand, where multiple definitions were proposed, we classified that article as defining rigor. This was consistent with our expectation that multiple definitions would be expected in transdisciplinary research. Finally, we discussed each ambiguous usage of rigor to determine if a definition could reasonably be assumed from the context, coming to a consensus.
After identifying definitions of rigor and contexts where rigor's meaning was suggested by usage, we sought to classify different perceptions of rigor. After considerable discussion and analysis of the text, we chose a scheme based upon where (in the research process) rigor was emphasized. The process model we used is presented in Figure 1.

Figure 1: Stages of the research process
The stages were defined as follows: 1. Design: Rigor was specifically mentioned as being critical in the research design or in the choice of an appropriate research methodology. For example: This has motivated some postpositivist researchers to carefully design their studies, using quantitative methods to minimize "bias" or "subjectivity." Over time, these efforts have become standardized as criteria to ensure the rigor of the work. (Barusch et al., 2011, p. 13) 2. Execution: The importance of systematically and faithfully executing the chosen research methodology was indicated to be a key element of rigor. For example: the current trend in methodological rigor is to focus on rigor per se with the concomitant operationalization of concepts and systematic utilization of procedures that foster standardization. (Armour et al., 2009, p. 102) 3. Analysis: Quality of reasoning, application of logic or the use of appropriate statistical techniques was specifically mentioned as being critical to rigor. While we had originally considered this a subset of execution, certain methodologies-such as those involved with the rigor of theory-building-seemed better served with a separate category. For example: In conclusion, rigor in research is the strength of the chain of reasoning (Biggs & Büchler, 2007, p. 69) 4. Product: Rigor was specifically tied to the final product of the research, such as an article. We selected this category when articles referred to the rigor of the description or indicated that transparency of the research was a critical component rigor. For example: Similarly, rigor-often seen as one of the key weaknesses of case studies-often seems to lie in the eye of the beholder and may even involve ''persuading'' readers and reviewers … of the ''credibility'' of methodological procedures (Gibbert & Ruigrok, 2010, p. 710) 5. Outcome: Some articles suggested that rigor could only be achieved where a particular outcome of the research-beyond publication-is achieved. For example: Interpretive rigor goes beyond the sound application of method to focus on moral and aesthetic dimensions of scholarly quality. It pushes researchers to engage strategies that directly contribute to worthwhile human purposes, not only from their own perspective, but from the perspectives of other social actors involved in the research (Dodge et al., 2005, p. 297) Both authors classified all 62 articles independently, with nearly all the articles appearing in multiple research stage categories. The classifications were then compared. While the initial level of agreement was disappointingly low (70%), a discussion regarding when the design category was applicable eliminated about half the differences in classification. At that point, we went through each article and arrived at a consensus on the final classifications.
Later, during the analysis process, it became clear that one of the most interesting outcomes of the process was the high level of variation in how the relationship between rigor and relevance, a topic explored in 60% of the articles studied. At this point, the original process of gathering quotes and definitions was repeated for both the terms relevance and relevant. These results are referred to in the discussion section of the current paper.

RESULTS
In this section we report the results of applying the preceding methodology to the collection of articles. We begin by looking at the definitions, then examine the distribution of rigor considerations across the different stages of research.

DEFINITIONS
Of the 62 articles examined, 24% (15) contained statements meeting our criteria for being classified as definitions. Of the 15 articles, only 6 appeared to have chosen a single definition (i.e., Armour et al., 2009;Gulati, 2007;Hasson & Keeney, 2011;McAlister, 2016;Nunamaker et al., 2015;Ogawa & Malen, 1991). The list of references and associated quotes are presented in tabular form in the Appendix.
Figure 2 presents a word cloud and frequency table extracted from the Appendix definition text, constructed using words that appear more than once and removing words not related to the definition (e.g., research, rigor, relevance, etc.).

RIGOR BY RESEARCH STAGE
When the articles were classified according to the research stage where rigor was described, more than two thirds (42 out of 62) listed multiple categories. The distribution of research stages within the rigor articles is presented in Figure 3. In Table 1, key themes relating to how rigor is considered are summarized, organized by research stage.

X
Kieser & Nicolai's paper focuses on the rigour-relevance gap, particularly the communication processes between scientists and practitioners. Kincheloe argues that bricolage (interdisciplinarity) will naturally lead to greater rigor. Lu & Shulman profess that computerized qualitative data analysis is a great way to achieve higher rigor. Odwyer argues that there is great potential for papers that are both rigorous and relevant in the field of accounting, and that it is just those papers that will end up leading to meaningful changes. Onwueguzie's paper analyzes the rigor of various methods of focus group research, concluding transcriptbased analysis to be the most rigorous. Ross came to the conclusion that research must be rigorous in order for it to have relevance in the field of education technology. Sculley argues that technological advances must be matched by advances in rigor in machine learning research. Aguinis et al. (2010), Davenport et al. (1999), Kieser & Leiner (2009) X These papers share a commonality in their primary focus, viz., the rigor-relevance gap. Keiser and Leiner take a pessimistic view, asserting that the gap is unbridgeable as the relationship between researchers and practitioners currently stands. The other two papers are more optimistic. Davenport

X
The Gutierrez paper argues that relevance should be made a criterion of rigor. Hodgkinson and Rousseau argue that both rigor and relevance are important, but there is a limit to how much they can both be present in a single study. The Lee paper stresses not relevance but summative validity.

Cite Design Execution Analysis Product Outcome
Comments Applegate & King (1999), Snow (2015) X X While Snow believes that rigor is important, her article focuses more on putting what has been learned in education research into practice. Applegate and King present a case regarding an assistant professor who had submitted a paper which was praised for rigor but criticized for providing few new insights. These papers are different in that Snow's focuses more on finding a way to get practitioners to adopt new research finding whereas Applegate & King's highlights the struggle of one researcher to keep up with hot topics in research before they become oversaturated Argues that rigor and relevance are not mutually exclusive and that both are necessary to have a lasting impact. Believes that the availability of more sophisticated methods can be a double-edged sword. On the one hand these methods can potentially increase rigor, on the other hand this does not matter if they are only used to delve into the more mundane aspects of research. The core question must always be the main focus

X X
Dodge argues that researchers should not assume that rigor automatically begets relevance, as it can cause them to become complacent in the search for findings that are useful and actionable. Hasson & Keeney focus more on rigor than on relevance, highlighting the dependability rather than the utility of research outcomes Eisenhardt et al. (2016), Gasson (2004), Webster (1978) X X Webster's article attempts to help researchers reap the benefits of new models and the potential for greater rigor in industrial marketing research. Gasson's article focuses on qualitative research and its reputation for low rigor, arguing that there need be a different set of rules for rigor in qualitative research. Eisenhardt likewise suggests that the typical standards for rigor are ill suited to certain problems, viz., Grand Challenges: "highly significant yet potentially solvable problems such as urban poverty, insect-borne disease, and global hunger" (p. 1113), for which inductive methods could prove particularly useful.
Argues that entrepreneurship research leans more towards rigor than relevance because rigor is more in their comfort zone. They are comfortable assessing rigor whereas they are not comfortable assessing relevance from a practitioner standpoint. Their proposed solution entails creating more of a dialogue between the researchers and the stakeholders  (2002),

X X X
Barbour suggests some fixes (e.g., grounded theory, multiple coding) to make qualitative research papers more rigorous but concludes by asserting that true rigor can only be achieved through a general knowledge of data analysis and qualitative research design. Darke lays out some suggestions for making case study research more rigorous such as choosing the right data analysis technique. Dube & Pare's article also focuses on case research, laying out attributes for rigor in the areas of research design (e.g., clear research questions, theory of interest,) data collection (e.g., multiple methods, triangulation,) and data analysis (e.g., logical chain of evidence, empirical testing). Varadarajan calls for more consensus with regard to the definitions of constructs in marketing research as an aide to both rigor and relevance. White suggests ways to increase both rigor and relevance in Asian management research by aspiring to achieve a generalizable understand of underlying phenomena. Biggs & Buchler argue that regardless of the type of research, rigor always comes down to a strong chain of reasoning. Gulati argues that the belief that one has to choose between rigor and relevance is damaging to research overall, and gives recommendations for achieving both such as a greater focus on theory building and paying greater attention to managerial sensibilities. Nunamaker makes a similar argument, referring to researchers who vie to achieve both rigor and relevance as Last Mile Researchers. Gioia et al. (2013) X X X Makes a case for the potential for rigor in qualitative research, arguing that one of the reasons that qualitative research has a reputation for low rigor is that researchers limit their conception of rigor to the extension of old concepts, even when new ones are called for. X X X Poland focuses on transcription quality as means to greater rigor in research involving interviews. Rand & Rust suggest agent based models must be verified and validated before they can be considered rigorous. Stewart focuses on rigor in qualitative research, concluding that credibility and trustworthiness are essential to achieving it. Tushman suggests that it is possible to have both rigor and relevance and that aspiring to have both is what separates business schools from other academic institutions.
Vermeulen also argues that rigor and relevance are not mutually exclusive, but posits that they are rarely seen together because the people conducting rigorous research are not asking questions that are of interest to practitioners. Walt argues that greater rigor can be achieved in political science research by focusing more on testing theories and making them clearer and thus easier to find flaws with. The Ivarsson and Gorschek article focuses on reporting, i.e., giving readers enough details to assess the rigor of the study for themselves. This, they assert, is also necessary to achieving relevance. Ogawa and Malen attempt to layout guidelines for achieving rigor in literature reviews including clear definitions of constructs and the search for contrary findings.

Cite Design Execution Analysis Product Outcome
Comments Sovacool et al. (2018) X X X X Sovacool vies to help other researcher achieve both rigor and relevance in research, giving guidelines for both. He argues that researchers should have a broad awareness of the various research methods as this will both help them achieve rigor, and help them be more humble and open minded, viz., in a better state to seek out novel (relevant) ideas. Frauenberger et al. (2015) X X X X This paper focuses on achieving rigor in participatory design research. By way of aiding researchers on the path to rigor, it proposes four lenses through which to analyze a project: epistemology, values, outcomes, and stakeholders. The paper also asserts that rigor is not one size fits all, and that there may be projects that bear little resemblance to each other method wise, but are nonetheless each rigorous in their own way X X X X X The Fereday and Nowell articles both focus on rigor in qualitative research, both likewise calling for a balance between inductive and deductive methods. Shrivastava analyzes the presence of both rigor and usefulness in strategic management research, highlighting articles where both are present and suggesting that the field in general has great potential to make simultaneous strides in both.

DISCUSSION
Based on the articles surveyed, there appear to be two principal motivations for analyzing rigor:
2) To explore the relationship between rigor and relevance. Of the 62 articles studied, 47 (76%) referred to relevance and considered its relationship to rigor. Indeed, separating out the elements of research quality that were attributable to rigor vs. relevance proved to be a topic closely related to how rigor was defined.
In this section, we will examine these two topics. We will begin by looking at the challenges of defining rigor across a broad range of research methodologies and disciplines. We then consider the specifics of the rigor-relevance relationship-a relationship whose proposed nature varies considerably across the articles surveyed.

ALTERNATIVE PERSPECTIVES ON RIGOR
As noted in the results section, there was considerable variation in views on the nature of rigor. Of particular interest were two areas: 1) approaches to rigor and the 2) perceived impact of rigor.

Approaches to rigor
Most of the perspectives on rigor that we examined fell into one of two categories: compliance-based or criteria-based. We now consider the similarities and differences between the two.
Compliance-based perspectives. Many of the definitions and usages of rigor that we examined were based strictly upon degree to which the research method complied with an accepted methodology. Specifically, they focused on rigor as it related to (a) selection of an appropriate research methodology, (b) fidelity in executing the methodology (and associated analysis), and (c) transparency in documenting the use of the methodology. Of these three items, (b) was viewed as particularly critical, as indicated earlier by the large number of articles falling in either or both of the execution and analysis categories. An example of a definition fitting this category is: Rigor, therefore, can refer either or both to methodological thoroughness and precision or criteria used to judge the trustworthiness of the results. Methodological stringency and accuracy of the results are related because solidity in methods provides greater assurance that the findings are valid. (Armour et al., 2009, p. 102) Naturally, the nature of rigor varies according to the research methodology. For positivist, empirical research using a large sample it would likely involve issues such as the choice of sampling approach, the appropriateness of the statistical tests employed, whether the interpretation of the findings is consistent with the results, and so forth. For qualitative, interpretive research, rigor might involve considerations such selection of interviewees, approach used to capture and code interviews, use of triangulation to confirm findings, techniques employed to reduce researcher preconceptions and bias, and so forth.
The advance of compliance-based perspectives on rigor is that they can build upon previous applications of the same research method. Thus, over time, an implicit "rule book" for each method can evolve. These rules can be taught to researchers unfamiliar with the method, can guide researchers employing the method, and can provide standards that reviewers can follow.
The potential drawback of these approaches is that they focus nearly all attention on the method employed without stepping back and asking if the method is likely to produce useful results. For example, 16 of the articles examined specifically mention replication in conjunction with rigor. On the surface this makes sense; the ability to replicate a study in order to confirm its results is considered critical in the natural sciences. In the context of the social sciences, however, the preponderance of evidence suggests that expecting results to replicate across similar studies is optimistic. For example, attempts to replicate research across different business disciplines have yielded disappointing results (Gill, 2016b, pp. 134-135). Even attempts to replicate "textbook" psychological research conducted under far more controlled conditions have failed to live up to expectations (Open Science Collaboration, 2015). Thus, compliance-based rigor allows researchers to design and conduct research that corresponds as closely as possible to a previous study. If we have no reason to expect the results should be the same or similar, however, why bother?
Criteria-based perspectives. The issues associated with pure compliance-based approaches to rigor-particularly as they apply to replication-have been recognized in some of the articles we examined. For example: Two criteria appropriate for deductive research but NOT appropriate for inductive inquiry include: 1) Is there evidence that the causal factors, processes, nature, meaning, and/or significance of the phenomenon generalize to the broader population?
2) Are the findings able to be replicated in the sense that two researchers asking the same question would come to the same interpretation of the data?
These two criteria, held sacred as cornerstones of rigor in deductive inquiry, seem to cause the greatest amount of heartburn within the field of public management and its relationship to inductive qualitative inquiry. If it is not generalizable and it does not replicate, how is that possibly science? This results in on-going frustration among qualitative scholars as they attempt to respond to criticisms of their design by reviewers, colleagues, and advisors in terms of the lack of representative sampling and/or inter-rater reliability measures. (Nowell & Albrecht, 2018, p. 353) To address this issue, some authors argue that research rigor should be assessed based on more global attributes of the research, what we are referring to as criteria-based rigor. In their analysis of rigor in qualitative research, Nowell and Albrecht (2018) continue: If we cannot assess inductive studies in terms of generalizability and replication, what are valid criteria upon which they might be evaluated? In very global terms, rigorous inductive research in public management can be judged on two core criteria: 1) Does the research design and its execution generate new insight into the causal factors, processes, nature, meaning, and/or significance of a phenomenon of interest to the field? … and 2) Is the account of these causal factors, processes, nature, meaning, and/or significance within these cases trustworthy? … The trustworthiness and depth of insight of an inductive study is manifest in its research design, execution, reporting. (p. 354) The advantage of criteria-based assessments of rigor is that they can be readily adapted to very different research approaches. For example, although the interpretation of a property such as "trustworthiness" is likely to be very different for alternative methodologies, the concept itself is fairly universal. The obvious drawback to criteria-based approaches is along the same lines. Since they require interpretation for each research project considered, they will necessarily be somewhat subjective. As a consequence, the assessment of rigor for a given project will likely produce considerable disagreement across different reviewers as readers.

Impact of rigor
Just as the articles studied illustrate differences in how rigor is assessed, they also differ in the expected impact of ensuring rigor in research. Some of the perceived impacts include the following: • Credibility. Attention paid to rigor is seen to increase the credibility of research. Many of the characteristics of rigorous research, such as being thorough, careful, systematic, and logical contribute directly to this outcome. In business, this credibility is seen as one of the key factors that distinguishes academic research from other research sources, such as consulting. For example: Readers are likely to be attracted to academic research reports because of this perceived neutrality, knowing that our results are more trustworthy than vendors' claims and promotional materials. (Davenport et al., 1999, p. 14) • Consistency. Particularly where compliance-based rigor is enforced, within a given research context different researchers should come up with similar findings. This quality is also referred to as reliability and can be applied across a wide range of contexts (e.g., reliability of instruments, reliability of different investigator ratings, statistical reliability tests).
• Replicability. The ability to reproduce the research in a different context, also particularly relevant to compliance-based rigor. As noted previously, however, within the social sciences it is not clear that we should expect results to replicate. Nevertheless, we should be able to replicate the method applied. This quality is closely related to transparency-describing to the degree that the rationale for all research decisions is made clear to the reader-is specifically mentioned in 16 of the articles. For example: rigour and transparency in the process is a major contribution to knowledge in Design Research (Frauenberger et al., 2015, p. 99) • Validity. Two thirds of the articles surveyed specifically related research rigor to its validity. Internal validity refers to the logical consistency of relationships described by the research. Construct validity refers to the degree to which concepts or variables used or developed by the research can be supported in terms of both their independent contribution to understanding the phenomenon being studied and the degree to which they can be distinguished from other constructs. External validity refers to the degree to which research results are expected to apply outside of the research context and is closely related to generalizability.
• Quality. In many of the articles studied, rigor and research quality are closely associated. One interpretation is that rigor is the source of quality, although it might also be argued that rigor and quality are deemed to be synonymous. Since quality is not defined, it is difficult to discern which interpretation applies. Some examples illustrate the challenge: Although effective writing and transparency are necessary, the rigor and quality of inductive papers rest on three fundamental criteria (Eisenhardt, et al., 2016(Eisenhardt, et al., , p. 1120. As the main indicator of academic quality rigour can be more easily evaluated because researchers are trained in scientific methods and the identification of research gaps (Frank & Landström, 2016, p. 66).
In being asked for this paper to propose directions for improving the relevance (meaningfulness and utility) and quality (rigor and credibility) of research, we accept that many alternative viewpoints, each having its own compelling rationale, are likely to exist. (Ross et al., 2010, p. 30) A significant issue raised by some of the perceived impacts of rigor involves how they impact the usefulness of the construct. Where rigor's impact depends on the nature of the research product (e.g., transparency) or on the research outcome (e.g., quality), it will be hard to ensure rigor prior to knowing the results of the research. This presents a significant barrier to the research designer. Other expected impacts, such as credibility and validity, can be planned for through careful attention to design, executive and analysis.

THE RELATIONSHIP BETWEEN RIGOR AND RELEVANCE
As previously noted, more than three quarters of the articles surveyed addressed the question of the tradeoff between rigor and relevance. Understanding this tradeoff is important in clarifying what is meant by rigor, as relevance is frequently treated as those aspects of research that are not achieved through rigor; hence differences in the expected impact of rigor will likely be accompanied by complementary differences in the expected impact of relevance. Dodge et al. (2005, p. 287) voice the following complaint: "The traditional definition of 'rigor,' which assumes an automatic connection to relevance, is particularly problematic." Gulati (2007, p. 775) further notes that debates relating to rigor vs. relevance often start from the assumption that the two qualities are antithetical.

Alternative perspectives of relevance
While we have noted that the term rigor is frequently used without definition, the situation is much worse with regards to relevance. Indeed, of the 47 articles that addressed relevance, only 6 offered anything remotely resembling a definition. These definitions varied widely, as shown in Table 2.

Definition Source
Relevance refers to the potential of research (questions and findings) to enable practitioners "to make informed choices about important practical problems and to implement solutions to them effectively" (AOM 2004). It also refers to the extent to which research addresses the challenges that practitioners face in their work and whether the questions and findings resonate with practitioners' experience, shedding new light on existing problems in ways that are actionable. (p. 288) Dodge et al.
Per Webster's dictionary, relevance is "relation to the matter at hand" or "practical and especially social applicability" (p. 775) Gulati (2007) In an applied research field such as software engineering, the transfer and widespread use of research results in industry ultimately determine the relevance and success (p. 366) Ivarsson & Gorschek (2011) Relevance is the degree to which research contributes directly to improving outcomes of interest to practitioners in the field, that is, solves an important class of problems. (p. 41) Nunamaker et al. (2015) For educational technology research to help solve real-world educational problems, we advocate that studies increasingly reflect two qualities. One is to achieve balance between rigor (internal validity) and relevance (external validity) (p. 24). In being asked for this paper to propose directions for improving the relevance (meaningfulness and utility) and quality (rigor and credibility) of research, we accept that many alternative viewpoints, each having its own compelling rationale, are likely to exist. (p. 30) Ross et al. (2010) While conventional academic disciplines are typically about a quest for understanding (rigor) with little thought of use (relevance), business schools, and professional schools more generally, are about both -operating in Pasteur's Quadrant. …Consulting firms, unlike business schools, are focused on meeting clients' needs (relevance) but are less concerned with general theory building or carefully controlled research (rigor). (p. 347) Tushman et al. (2007) While all of these definitions suggest a relationship between relevance and practical usefulness, they leave two important areas of ambiguity: • Is the "usefulness" potential or must it be realized? In other words, can relevance be established prior to the research outcomes being known or must we know the impact of research before we deem it to be relevant? • What community determines usefulness? Research often has two distinct audience, the community of researchers within a discipline and the community of practitioners to which the research might apply. Depending upon which community we are talking about, the meaning of relevance would be very different. For example, relevant research to the business research community might involve identifying and filling a gap in the literature; for the business practice community, it might involve addressing an important question facing managers and providing actionable, practical recommendations.
For our purposes, we believe that both rigor and relevance are best defined in a manner that offers the researcher insights with respect to conceiving and designing research. Otherwise, they devolve into terms for classifying research outcomes, in which case we would need to define new terms (e.g., potential for rigor, potential for relevance) to guide design. We are particularly attracted to a definition associating relevance with the type of research questions being asked, i.e., Are the questions guiding the research of importance to the community of practice? To quote Vermeulen (2005): In any study, it is the research question that was asked in the first place that determines the usefulness of the study's findings. Thus, academic answers often lack practical meaning because the questions that were asked to start with lacked relevance. Asking questions that are of importance to reality, while not making concessions in terms of rigor in developing theory and empirical evidence, would provide most value. Relevance is then found in the question, rigor in the method applied to provide the answer. (p. 979)

The rigor-relevance relationship
While many of the articles studied suggest that rigor and relevance are treated as diametrically opposed in past research (e.g., Dodge et al. 2005, p. 287), we could not find any articles that actually took that position. We did, however, find a wide range of proposals regarding how rigor and relevance were related. We now consider some of these.
Prerequisite Relationships: The first type of relationship we observed assumes that either rigor or relevance is a prerequisite of the other. Both types of relationship have been proposed, as shown in Figure 4.

Figure 4: Proposed prerequisite relationships between rigor and relevance
For example, Van Weele and Van Raaij (2014) assert the following: When we speak about relevance, we do not view rigor and relevance as trade-offs, but we prefer to view methodological rigor in service of research relevance. Management research cannot be truly relevant, if it has not been executed rigorously. (p. 64) On the other hand, the opposite relationship has been proposed. For example: Making relevance to practice a key criterion of rigor is an important step toward more equitable and consequential research. Making relevance to practice a key criterion of rigor is an important step toward more equitable and consequential research. (Gutiérrez & Penuel, 2014, p. 22) Rigor and Relevance as Separate Dimensions. If rigor and relevance are related through prerequisite relationships, they are inseparable. Another proposed relationship between the two concepts is as separate and independent relationships. One example of this involves a slight variation on the graph used to visualize Pasteur's quadrant, as shown in Figure 5. In this depiction, "pure research" is treated as being highly rigorous, with little obvious immediate relevance. At the opposite extreme, applied research is conceived as being highly relevant with little attention being paid to formal rigor. The "ideal" research takes place in Pasteur's Quadrant, with highly useful questions being tackled with rigorous methods. The obvious question the Figure 5 depictions raises is the following: If use-inspired, opportunities to conduct rigorous research abound, why would we ever choose to operate in any other quadrant? This question is particularly applicable in fields like management, where a rapidly changing, complex environment make it unlikely that long standing principles will be found through a pure research process. For this reason, another category of relationships focus on tradeoffs between rigor and relevance.
Rigor-Relevance Tradeoffs. A couple of different approaches to characterizing the tradeoffs between rigor and relevance exist. Starting with the same rigor and relevance dimensions as Figure 5, the left-hand side of Figure 6, adapted from Robey and Markus (1998, p. 9), presents a curve showing how research can vary from academic-focused to practitioner-focused. They posit a middle ground, referred to as "consumable academic research" that would maintain substantial rigor while being presented in a form accessible to practitioners.

Figure 6: Research curve adapted from Robey & Markus (1998, p. 9) and their re-interpretation as microeconomic indifference curves
The shape of the Robey-Markus curve closely resembles that of indifference curves, familiar to most veterans of introductory microeconomics. Each curve, illustrated on the right-hand side of Figure 6, represents rigor-relevance combinations of equal utility-the decision maker does not care (i.e., is indifferent to) which rigor/relevance combination on a particular curve is selected.
Another view of a rigor-relevance tradeoff is proposed by Davenport et al. (1999), shown on the lefthand side of Figure 7. They propose a curve that represents a threshold between impactful and nonimpactful research. From a "big picture" perspective, the curve proposes that impactful research can arise from either very rigorous research or from very relevant research or from research that is a combination of the two.
We are somewhat suspicious of the convex shape that the authors used to draw their curve. The implication of that shape is that once you reach a very high level of rigor, relevance does not really matter; the same applies to relevance-at very high levels, rigor adds little or nothing to impact. On the other hand, their impact curve bears a striking resemblance to another familiar microeconomic curve: the production possibilities curve (shown on the right hand side of Figure 7). The production possibilities curve identifies possible combinations of two goods that can be produced with a given production capacity. Any combination on or within the curve is feasible to produce; any combination outside the curve requires additional capacity. The curve itself therefore represents the best tradeoffs between the two "goods" (in this case, rigor and relevance) that can be achieved. Again drawing on basic microeconomics, if we combine the utility curves and the production possibilities curve we can identify the "optimal" combination of the two goods. The optimum point occurs where the production curve is tangent to the utility curve; such a point will always exist where the production curve is convex and the indifference curves are concave. This is illustrated in Figure 8 for rigor and relevance.

Figure 8: Utility-maximizing balance between rigor and relevance
We are quick to point out that Figure 8 does not imply that there is an "optimal" balance between rigor and relevance, nor is such an optimum suggested by the literature. Each individual researcher or research consumer will have their own indifference curves. The shapes of these curves can vary substantially. As illustrated in Figure 9, very different utility-maximizing combinations can result. A strong preference for either rigor or relevance produces curve with a very sharp bend-becoming a right angle in the limit-such that only increases in the preferred dimension produce significant improvements in utility.

Sufficiency of rigor and relevance
Another question raised by the articles surveyed relates to the degree to which achieving rigor and relevance is sufficient to achieve research effectiveness-an intentionally vague term that we are using to include aspects of the research such as impact and quality. The earlier Figures 4 through 9 all seem to imply that research effectiveness can be achieved through rigor and relevance. Some of the articles, however, argue that other qualities are needed. For example, Sovacool et al. (2018) identify rigor, novelty and style as the source of research quality, with relevance included in the four core elements necessary to achieve quality, e.g., Although the later parts of this Review will explore how to improve aspects of novelty, rigor, and style, a useful starting point is to consider four core elements: 1) asking concise, interesting, socially relevant, and answerable research questions; 2) applying and testing theoretical constructs or conceptual frameworks; 3) clearly stating research objectives and intended contributions; and 4) developing an appropriate research design. (p. 13) Rajagopalan (2019, p.1) refers to rigor, relevance and resilience, the latter being a measure of the robustness of research. Aguinis et al. (2010, p. 512) refer to rigor relevance and practical impact. Barusch et al. (2011) similarly state: Careful attention to rigor is necessary but not sufficient to ensure high-quality research. Rigorous research is not necessarily "good" research. As one of our peer reviewers pointed out, research must also be evaluated on the basis of its relevance to the profession and its potential impact on social justice. (p. 18) We note that the distinction between impact and potential impact is an important one, since potential impact may be assessed (perhaps inaccurately) during the course of the research, whereas actual impact can only be known after the research has been disseminated.
We concede that the issue of rigor/relevance sufficiency remains muddy in our review. A major contributor to this is the fuzziness of the two constructs. The situation is particular troublesome in the (frequent) cases where relevance seems to be implicitly defined as everything that contributes to a desired research outcome that is not explicitly addressed by rigor. For example: Relevance is the degree to which research contributes directly to improving outcomes of interest to practitioners in the field (Nunamaker et al.,p. 41) Defined in this manner, there is no need for any constructs beyond rigor and relevance to achieve the desired research impact. Of course, such a definition also calls into question the need for "relevance," since it is essentially synonymous with "improved outcomes to practitioners in the field".
With concerns such as those relating to research effectiveness and sufficiency of rigor and relevance in mind, we now turn to looking at rigor as it is treated within the informing science transdiscipline.

RIGOR IN THE INFORMING SCIENCE TRANSDISCIPLINE
The previously mentioned challenges in applying the concept of rigor across a wide range of subject areas and methodologies are particularly pronounced in informing science. By its very transdisciplinary nature, the expectation is that ideas will be drawn from an extremely diverse collection of informing-related research. Can the process of integrating such research ever be considered truly rigorous?

DEFINING RIGOR IN INFORMING SCIENCE
The definition of rigor that has been proposed for informing science is as follows: In order for research to be rigorous it must: 3. Ask challenging questions: If you know the answers to your questions in advance, then there is little risk that your hypotheses will be proven incorrect. If there is no doubt of the outcome of your research before you conduct it, then whatever tests are performed cannot be considered particularly stringent. (Gill, 2016b, p. 122) This criteria-based definition was adapted from an earlier definition originally proposed in the field of nursing (see Allison & Rootman, 1996, p. 334). Like many of the definitions seeking to extend rigor to non-positivist research, it is built on general criteria for assessing rigor that can be adapted to a broad range of contexts. Its first two elements, systematic inquiry and appropriate design, seem quite consistent with the definitions we surveyed. Words such as method, methodology, appropriate, thorough, careful, exhaustive, and systematic all appear in the word frequency list.
Where the informing science definition diverges somewhat is with respect to its final criterion: asking challenging questions. That aspect of the definition was deemed appropriate for informing science for two reasons. First, difficulties encountered whenever interdisciplinary research is conducted would tend to limit its application to questions for which standard disciplinary research fails to provide answers. Such questions would, almost by necessity, be challenging. Second, rigor comes with a price, in time, effort, and constraints placed upon the researcher with respect to methodology and the types of questions being asked. What makes a research question challenging is most likely the fact that its possible answers are either unknown or that they stand a high likelihood of conflicting with what is already known (e.g., the conventional wisdom). In such cases, the cost of rigor is likely to be justified by the effort. We might even argue that if a research process ignores the cost of conducting the research entirely, then a failure of rigor in selecting an appropriate design is indicated.

RESEARCH FITNESS
In our earlier discussion of "research effectiveness", we noted that the term was intentionally vague. Within informing science, we are particularly interested in both the durability of ideas and the degree to which they spread throughout an informing system. For this reason, to concept of fitness-drawn from evolutionary biology-has been proposed as a particularly appropriate construct for judging research success (Gill, 2016a).
Briefly stated, fitness consists of two components: an entity's potential to survive and an entity's potential to reproduce (completely or in-part). In the context of design, the entity is an artifact that may also be broken up into a series of elements. Fitness would reflect the degree to which the artifact remains in use over time and the degree to which its elements are incorporated into subsequent designs (Gill & Hevner, 2013).
Generalizing the concept of fitness to research, the artifact is the research paper and its elements are the distinct ideas that the paper communicates. From an academic standpoint, a reasonable way of estimating an article's fitness is through metrics such as citation rate. Where we are concerned with a research artifact's fitness with respect to practice-recognizing that the research may not be rendered in the form of an article, but might instead be in the form of a book, a software application, or even a presentation-we base our fitness estimates on values such as the rate at which the various ideas diffuse to practice. This closely corresponds to the notion of a meme (Dawkins, 1976).
The advantage of using fitness as a measure of research effectiveness is that it can, at least in theory, be estimated objectively. Other plausible research outcomes-such as quality and impact-will necessarily be subjective. Any author or editor who has had to deal with conflicting peer reviews will be able to attest to that. The principal drawback of using fitness to assess research outcome is that objectively untrue or verifiably falsified research can, at least in the short term, exhibit high fitness. Some well-known examples include the study linking autism and vaccination and the twin studies conducted by Cyril Burt. It is precisely to avoid "high fitness" research products such as these that rigor needs to be an important contributor to fitness.

RIGOR AND RELEVANCE IN INFORMING SCIENCE
In the absence of a strong consensus, researchers will likely define rigor and relevance to be whatever is most convenient. The best we can hope for is that they specify how they are using the terms.

Informing-based definitions
Within the informing science transdiscipline we recommend using definitions that are grounded in existing informing science conceptual schemes. One example of such a scheme is the single client resonance model (Gill, 2016a), shown in Figure 10, based on an earlier bias filter model (Jamieson & Hyland, 2006). The model proposes that in order to be absorbed by a client, a message must pass through a series of filters-not necessarily in any particular order. These filters can, in turn, either transform the message, distort the message, or intercept it altogether. In any of these events, the intended meaning of the message is not faithfully conveyed.
What we propose is that to make the terms rigor and relevance more precise when we use them in informing science, we attempt to match them to specific filters. For relevance, the motivation filter would be an obvious candidate. Defined in this way, relevance would be achieved if the research addresses a question or problem that is important to the client. This is consistent with some of the existing definitions (e.g., Dodge et al., 2005) but is considerably narrower than others, particularly those that seem to define relevance as anything that is not rigor. Rigor presents a more challenging case. The cognitive filter deals with whether the incoming information seems to make sense and the risk/time filter addresses the client's reaction to uncertainty. Both of these are elements of credibility, and therefore clearly map to rigor. They are also consistent with the first two items in the informing science definition: appropriate design and systematic inquiry. Another possible filter is the information filter, which addresses whether the incoming message contains information that is already known (or appears, at first glance, to be known). We would argue that this can be treated as consistent with the existing definition's third element: challenging questions. The rationale is that in order to be considered rigorous, research should begin the process by focusing on questions for which the answer is not already known; in other words, research conducted specifically to confirm findings that are already well established is not rigorous. While this sounds obvious, when rigor is defined solely in terms of compliance to a well-accepted method, researchers may experience a strong temptation to trod well-worn paths in order to have a high likelihood of finding significant results.

A simple research fitness model
If we accept the mapping of rigor and relevance to the Figure 10 model just proposed, it becomes evident that the two constructs do not address all the proposed filters. Specifically: • The channel filter represents the potential of the medium itself to distort the message. For example, messages received though social media are likely to be absorbed differently from message presented in journal articles. • The attention filter specifies whether we are attending the channel through which a massage is being communicated. • The visceral filter addresses the impact of our emotional state on how we interpret the message (or choose to ignore it).
Within informing science, we have chosen to group these three characteristics together and refer to them as resonance. The concept is similar to stickiness (Gladwell, 2000), which is readily achieved when a message is simple, unexpected, concrete, credible, emotional and tells a story (Heath & Heath, 2007). The proposed model for research fitness is presented in Figure 11.

CONCLUSIONS
Research rigor means different things to different people. The review that we have presented here will not change this. Our hope, however, is that it will encourage researchers not to use the term offhandedly. Too often we have observed the term being used casually in the literature, seemingly assuming that the reader perceives it to mean the same as the author.
For a transdiscipline such as informing science, the challenge presented by rigor is particularly acute. Even within disciplines there are disputes about what is rigorous. Across disciplines the gap is huge. For this reason, we encourage the use of flexible, criteria-based definitions of rigor that can be interpreted across a wide range of research contexts. Definitions based on the degree to which accepted procedures are followed tend to be quite narrow with respect to where they can be applied. Moreover, they can easily focus the researcher's entire attention on following the procedure instead of asking whether or not the procedure, along with the question it is intended to address, is sensible.
For informing science, our research suggests that the existing proposed definition of rigor is plausible. It is a criteria-based definition that views research rigor in terms of three characteristics: 1. It is systematic in its inquiry. 2. It employs an appropriate design for the questions being asked. 3. It asks challenging questions.
Defined in this way, rigor is clearly only part of what makes for effective research. We caution the reader against grouping everything else needed to make research impactful into "relevance." While we observed that such an approach was common in the literature, we believe that it forces many unrelated aspects of research together. Instead, we advocate using a conceptual scheme, such as the single client resonance model (Gill, 2016a, p. 10) to develop theory-based distinctions. For the purposes of informing science, we propose: • Rigor: Is the research credible and is it challenging?
• Relevance: Are we studying a question or challenge that our clients are motivated to address?
• Resonance: Are we communicating the research in a manner that grabs the client's attention and engages the client emotionally using a channel appropriate to the content? Armour et al. (2009) Rigor is the degree to which researchers hold themselves to standards of inquiry that address challenges to the credibility of a study's findings. Rigor, therefore, can refer either or both to methodological thoroughness and precision or criteria used to judge the trustworthiness of the results. Methodological stringency and accuracy of the results are related because solidity in methods provides greater assurance that the findings are valid. (p. 102) Biggs & Büchler (2007) Rigor refers to the process of undertaking activities such as the literature search. It connotes a systematic and thorough search. As a result, the researcher can be confident that from a "null return" (i.e., when the researcher's knowledge and understanding has been identified as absent from the published body of knowledge and understanding in the field), it can be concluded that the researcher's knowledge and understanding is new knowledge and understanding. (p. 66) In conclusion, rigor in research is the strength of the chain of reasoning, and that has to be judged in the context of the question and the answer, for example, in the context of design as opposed to the context of physics or philosophy (p. 69) Dodge et al. (2005) As a starting point, rigor traditionally refers to the accurate and systematic application of theory and method (p. 288) Interpretive rigor goes beyond the sound application of method to focus on moral and aesthetic dimensions of scholarly quality. It pushes researchers to engage strategies that directly contribute to worthwhile human purposes, not only from their own perspective, but from the perspectives of other social actors involved in the research (p. 297) Fereday & Muir-Cochrane (2006) Rigor is described as demonstrating integrity and competence within a study. ... Schutz's first postulate of logical consistency is similar to the description by Horsfall, Byrne-Armstrong, and Higgs of rigor in qualitative research, which involves in-depth planning, careful attention to the phenomenon under study, and productive, useful results. Descriptions of theoretical rigor involve sound reasoning and argument and a choice of methods appropriate to the research problem. ... The step-by-step process of analysis that is outlined in this article is a method of demonstrating transparency of how the researcher formulated the overarching themes from the initial participant data. ... Interpretive rigor requires the researcher to demonstrate clearly how interpretations of the data have been achieved and to illustrate findings with quotations from, or access to, the raw data (p.82) Frauenberger et al. (2015) The notion of "rigour" is commonly associated with a strict positivistic view on science, emphasising universal truths validated by deductive reasoning or measured evidence (p. 94) We argue that rigorous PD [participatory design] work exhibits qualities that are coherent, e.g., it is based on an epistemology that accommodates the values that drive the effort, involves stakeholders in ways that reflect these foundations and accordingly defines and delivers its meaningful outcomes. (p. 103) Gulati (2007) Sometimes scholars define rigor as use of a narrow disciplinary paradigm involving a set of theories, methodologies, and data analyses that they themselves would use (p. 777)

Source
Definitional Quotes Hasson & Keeney (2011) The "holy grail" of research is establishing methodological rigour. This refers to a researcher's responsibility to ensure that procedures have been adhered to and confounding factors eliminated [where possible] to produce dependable results. (p. 1695) Houston (2019) Rigor is not just a buzzword-it is the basis for having confidence in research findings. In the words of Miriam-Webster's Dictionary, it is the quality of being extremely thorough, exhaustive, or accurate; the English Language Learner's Dictionary defines it as the quality or state of being very exact, careful, or strict. Applied to research, these definitions imply that rigor is more than the application of sophisticated and complicated quantitative methodologies. ( (2011) Rigor refers to both how an evaluation is performed and how it is reported. If the study is not adequately described, the rigor of the evaluation cannot be evaluated by reviewers and other researchers. (p. 367) Rigor in research often refers to the precision or exactness of the research method used; e.g. a controlled experiment often enables greater control over variables than a case study. ...This is one way to view rigor, the precision of the research approach utilized. Rigor can also mean the correct use of any method for its intended purpose ... implying that there is a context or application in which certain methods are appropriate or applicable. (p. 369) McAlister (2016) Rigorous (defined as extremely thorough, exhaustive, or accurate) does not imply newness or complicatedness. Rigorous explicitly includes accuracy, and it implies that one has chosen the right method for the problem. (p. 565) Melrose (2001) One term that was used at this conference by some presenters as synonymous with rigor was validity ... others used the term in a broader sense, referring to the whole process or to different parts of the process of research (e.g., the choice of method to suit the research question, the constitution and facilitation of collaborative research groups, and the dissemination of the data in appropriate ways to suit audiences) (p. 163) Another meaning for rigorous is "scrupulous" ... Here the idea of rigor is linked to telling the truth (as far as can be established by evidence) and behaving ethically. …One common definition of ethical behavior is behavior that does no one harm. (pp. 174-175) In addition to the meanings explored so far in this article, rigor can also mean "not bending or inflexible," "harsh or oppressive," and "very strict" (p. 177) Nowell & Albrecht (2018) Rigor, then, can be conceptualized as the appropriate execution of that method. Put simply, if quality is the what, rigor for our purposes becomes the how. (p. 352) rigorous analysis is based on 1) whether the interpretation is credible in light of the data, 2) whether it was the result of a robust and systematic analytical process designed to move beyond superficial findings and minimize and/or account for investigator bias, and 3) whether it is reported with sufficient attention to context so as to facilitate the potential relevance of insights to similar contexts (p. 357)