Against a backdrop of declining tobacco use, e-cigarette markets are growing. The UK now has a higher percentage of e-cigarette users than any other European country. These developments have prompted fierce discussions in scientific, advocacy and policy communities about how best to respond. This article is one of the first to examine the role of evidence in these debates.We analysed 121 submissions to two Scottish policy consultations on e-cigarettes (in 2014 and 2015) and undertook interviews with 26 key informants in 2015–2016, following up with a sub-set in 2019–2020. All data were thematically coded, and our analysis was informed by insights from policy studies and the sociology of science.First, we affirm previous research in suggesting that e-cigarettes appeared to have triggered a breakdown of old public health alliances. Second, we demonstrate that, amid concerns about research quality and quantity, actors are guided by normative outlooks (and/or economic interests) in their assessments of evidence. Third, we show that, despite describing e-cigarette debates as contentious and polarised, actors engaging in Scottish policy debates exhibit a spectrum of views, with most interviewees occupying an uncertain ‘middle ground’ that is responsive to new evidence. Fourth, we suggest that the perceived divisiveness of e-cigarette debates is attributed to recurrent media simplifications and tensions arising from the behaviours of some actors with settled positions working to promote particular policy responses . Fifth, we argue that the actions of these actors are potentially explained by the prospect that e-cigarettes could usher in a new tobacco ‘policy paradigm’. Finally, we show how scientific authority is employed as a tool within these debates.E-cigarette debates are likely to reconcile only if a clear majority of participants in the uncertain ‘middle ground’ settle on a more fixed position. Our results suggest that many participants in Scottish e-cigarette debates occupy this ‘middle ground’ and express concerns that can be empirically assessed, implying evidence has the potential to play a more important role in settling e-cigarette debates than previous research suggests.The online version contains supplementary material available at 10.1186/s12889-021-10396-6. Should e-cigarettes be banned (as consumer products that enable users to consume an addictive drug)?Alternatively, should e-cigarettes be promoted as a unique opportunity to support smokers who want to quit?If e-cigarettes are not banned, should they be regulated as tobacco products or medicines or something else and what does this mean, in practice?Will e-cigarettes contribute to a reduction in smoking?Will e-cigarettes sustain addictions and/or attract new (younger) consumers?As tobacco companies expand their market share of e-cigarettes, will they be able to regain policy credibility and influence or should Article 5.3 of the global Framework Convention on Tobacco Control apply in excluding them from e-cigarettes policy discussions?Electronic devices that mimic aspects of smoking  have been around since the late 1960s , 3. A 20In attempting to respond to these questions, public health researchers and policy actors are far from unified, with different policy bodies and states taking radically different approaches. For example, while Public Health England has promoted the relative safety of e-cigarettes , supporting calls for relaxed regulation e.g. , 6), the, the6]),It is not only policymakers who are divided: key charities involved in battling the tobacco epidemic have come down on different sides of e-cigarette regulatory debates ; media cSo far, the rapidly growing evidence-base does not appear to be resolving matters, at least in the UK. In Hawkins and Ettelt’s  assessmeIn analysing these debates, Hawkins and Ettelt  argue thAccounts of the research and policy debates surrounding e-cigarettes in the UK , 15 contThis paper draws on document and interview data to provide the first detailed empirical analysis of the role that evidence is playing in the contentious policy debates surrounding e-cigarettes in Scotland, a devolved UK nation with a particularly strong record on public health and tobacco control . TheoretWe conducted a thematic analysis of 121 publicly available documents and interviews with 26 key informants. The document analysis centred on written responses to the Scottish Government consultation on Electronic Cigarettes and Strengthening Tobacco Control in Scotland  and to qDifficulties in researching e-cigarettesResearch fundingSources of evidenceEmerging evidenceQuality of evidenceMissing evidenceUncertainty of evidenceInterpreting evidenceManipulating evidencePublishingResearch communication or knowledge exchange workDiscrediting or supressing discussion of evidenceSame evidence used to support opposing viewsFigure n = 47), via the project advisory board (n = 8) or via webchat and LinkedIn (n = 3). Up to three follow-up approaches were made. When participants agreed to take part, a suitable date and location for the interview was arranged. In total, we conducted 25 semi-structured interviews with 26 interviewees (one was a joint interview) between November 2016 and August 2017  developing a thematic coding frame in NVivo11, which was collectively discussed and agreed with the full team . We initially coded a sample of nine transcripts, all of which were cross-checked with a second team member, after which we discussed and refined the coding frame. The full set of transcripts were then systematically coded by TI using the revised coding frame, with all team members having access to the NVivo11 file. A second researcher cross-checked the coding on a sample of transcripts for consistency. Following data coding, the key themes of relevance to this paper were analysed by the lead author . The lead author then presented early analysis to the rest of the research team and to the project’s Advisory Board, developing this analysis further in response to feedback and discussion.Conscious of the time that had passed since our data collection, and the speed with which e-cigarette products, markets and debates are evolving, in 2019, we approached all of our original interviewees with a request to undertake a follow-up interview. Between August 2019 and March 2020, we conducted follow-up interviews with all those who were available , and on“The […] criticism that we get nowadays is actually from our colleagues, it’s from public health, and there are particular individuals, [gives named examples], who have been incredibly critical of the position that [we and others] have taken. They’ve been vicious. They’ve been insulting. […] So, it’s hugely divisive and I would say that relationships that used to be very positive – so I used to have a very positive relationship with [named example] and that’s not the case anymore – […] now it’s public health fighting amongst itself instead of what we used to do, which is we knew who the enemy was, the enemy was the tobacco industry and that has… the debate has shifted.”Interviewee 11 (academic researcher): “It’s been a terrible shame actually because having worked in tobacco control for the twenty years […], I have never known an issue to cause the divisiveness that this has caused. […] I mean, I got some feedback on something which was in an extremely unprofessional manner actually. I was absolutely horrified. And academics who I have worked with very closely and very well with, in the past.”Interviewee 12 : “We tried to do a joint statement with [a public health organisation] in the beginning but then we realised that, actually, we’re coming from two different places. […] Before e-cigarettes, we would’ve done a joint statement with [that organisation] very easily because we’re very much on the same page. We suddenly realised we’re actually on quite different pages on this, so we haven’t done anything.”Interviewee 14 (pharmacy): In Sabatier and Jenkins-Smith’s  accounts“The standard of some of the evidence, and to be fair on both sides, is not great quality...”Interviewee 19 (pro-e-cigarette campaign group): “The evidence is absolutely rubbish. There’s stuff published which is terrible, awful studies. People over-egging stuff. People, you know - it is a kind of Wild West with e-cigarettes, with people making all sorts of claims about it.”Interviewee 4 (NHS): In explaining how they were negotiating the new tobacco policy landscape, almost all our interviewees suggested that evidence had an important role to play. However, there was widespread agreement that available evidence was limited and not always of high quality:Interviewees generally attributed the limited quality of available evidence to the novelty of e-cigarettes, the diversity of products captured by the ‘e-cigarettes’ label and the dynamism of both the market and products, with several interviewees highlighting the inevitable absence of long-term evidence. For example:“The evidence that’s missing is about long-term use, and the consequence and risks associated with that. And the reality is you can’t have long-term evidence until the product’s been around for a long time.”Interviewee 19 (pro e-cigarette campaign group): “You just can’t generalise about this huge range of devices and then huge range of fluids as well. It’s not a thing, and that’s what I’m always trying to remind enquirers, it’s just, it’s not one thing (sigh). […] I can’t think of any product that’s quite as diverse.”Interviewee 2 : “I think some academics have been so keen to actually gather evidence which supports their view, rather than gathering the evidence and letting that form their view, and I have to confess, I’ve been a bit shocked at the extent to which I think I’ve seen that take place.”Interviewee 5 : “[I]f your basic premise is inconsistent with use of electronic cigarettes… because you believe that nicotine addiction is wrong or because you believe that they are a force of evil… driven by the tobacco industry, then you are likely to take an extreme view in interpreting evidence that supports those positions.”Interviewee 1 (academic researcher): Despite this consensus, most actors appeared to have preferred policy positions on key regulatory issues and most employed evidence to support these positions . In the following extracts, we see both interviewees attributing contrasting positions on e-cigarettes to pre-existing views and values:Indeed, some of our interviewees openly reflected that, having settled on a particular policy position, they had then sought evidence to support it. For example:“We’re all guilty of it to some degree, of using evidence to support an argument rather than basing arguments on evidence.”Interviewee 1 (academic researcher): “I was trying to find out more on that side of it, hoping for a few golden nuggets […] which would support…what we were trying to say.”Interviewee 14 (pharmacy sector): In sum, our data make clear how, in the face of uncertain evidence, actors draw on their beliefs and normative values to help them make sense of the evidence that is available. This finding is in line with Sabatier and Jenkins-Smith’s  claim th“There are some who want very high levels of regulation and some who feel that we shouldn’t have any regulation at all […] But […] those are extremes on a spectrum, and I think most people want proportionate regulation that protects consumers but makes sure the products stay available.”Interviewee 1 (academic researcher): So far, our analysis tells a very similar story to that of Hawkins and Ettelt : the emeAlthough this claim may, as another interviewee suggested, be partially explained by the strategic value of self-positioning away from the extremes of a debate, it nonetheless speaks to the fact that many of our interviewees were resistant to being positioned in either of the two, opposing ‘camps’ that Hawkins and Ettelt  identifiOverall, our analysis identified a range of views about e-cigarettes within our data, each of which was associated with distinct policy positions. As Fig. Beyond this, however, our data suggest few actors could be clearly positioned at either end of the spectrum. Rather, most fell somewhere in the middle, favouring some degree of regulation but varying in their perspective on specific regulatory proposals. For these interviewees, the position their organisation adopted on specific regulatory proposals in the consultation was often presented as evidence-informed and open to change.potential harm of the products, mainly in terms of the potential for e-cigarettes to act as a ‘gateway’ into smoking for non-smokers and youth but also in terms of renormalising smoking. The difference between Position 2 and Position 4 is that Position 2 was not informed by a belief that nicotine addiction is, in principle, undesirable but about concerns with e-cigarette products that could more easily be empirically assessed. In contrast, Position 4’s concern with nicotine addiction effectively ruled e-cigarettes as undesirable by virtue of the fact they are largely designed to deliver nicotine to users. Position 3 is placed closer to the right-hand side than Position 2 as it is underpinned by a principle of tobacco industry exclusion that suggests less openness to limited regulation than Position 2.Within this ‘middle ground’, we identified two dominant positions. The first (Position 2) involved caution about e-cigarettes on the basis of the Importantly, our data suggest that it was not always possible to place actors at specific points along the spectrum Fig. . Rather,All this suggests that, while positions within e-cigarette debates are shaped by normative outlooks and material interests, many actors’ positions are more fluid and intersecting than Hawkins and Ettelt  suggest,“Going back to where we started to look at this, one of the biggest concerns seemed to be about children and young people using these products as a gateway. And we’ve monitored the data very closely and talked to other groups […] and it’s very obvious that the data in the UK doesn’t suggest that’s a problem, and consistently so. […] So where I think in the past we were certainly talking about regulating [e-cigarettes] very strongly to ensure the gateway effect didn’t happen, we’re very clear now that the evidence and data just has… there’s probably enough years of it and it hasn’t shown that to be a big issue.”Interviewee 10 : Actors whose economic interests were intertwined with the success of e-cigarettes and those whose normative outlooks placed them firmly at either end of the spectrum in Fig. Theoretically, our analysis in this section suggests that the e-cigarettes market has introduced complex dynamics to the tobacco policy and advocacy field. As a consequence, Sabatier and Jenkins-Smith’s  widely e“I just do not want involvement with being perceived to be on one side or the other […] which makes it slightly difficult because where I would probably naturally want to involve academics in certain issues [… but] if you get involved with one academic, then the perception is, that’s your approach. When actually, it’s not.”Interviewee 12 (NHS): Since our analysis suggests that many of the actors engaging in Scottish policy debates about e-cigarettes occupy a fluid and uncertain ‘middle ground’ Fig. , this raAnother interviewee – an academic who said they tried to speak to ‘both sides’ - was concerned about how they might be represented in our research as a consequence of this.Interviewees attributed the difficulty in avoiding being positioned on one or the other ‘side’ to two sources: (i) mass media coverage of e-cigarette debates; and (ii) the behaviour of other actors. On the first of these, multiple interviewees noted the large appetite for stories about e-cigarettes in the UK media and said they felt stories  tried to position everyone as being in favour or against e-cigarettes . For ex“It’s at times been particularly divisive and not always a hundred percent clear why, and […] actually, sometimes the positions of organisations aren’t as far apart, when you look at the detail, as they might seem. When you have people looking for a high level kind of position or you get, again, media reports of your position, they tend to focus on the extreme aspects or the kind of overarching headlines, rather than the detail, and I understand that, but I certainly think it’s been a real issue on this area.”Interviewee 10 : “[I]n studies of children’s use and gateway progression […] the evidence published to date doesn’t support that that is happening to any significant degree. But that’s not what you would think, […] reading the press releases that go with papers.”Interviewee 1 (academic researcher): “[T]he examples of what I have seen […] and what I’ve heard in terms of reports of events and the way people have been treated if they don’t subscribe to a particular line. I’ve also had a look at, well I did and I haven’t gone back to look at it again, because I was so horrified at what I saw, but just some Twitter accounts of the latest research that has come out about things and […] just comments around those issues and, yeah, it’s shocked me.”Interviewee 12 (NHS): The second explanation interviewees gave for the difficulty they faced in avoiding being positioned at either end of the spectrum related to behaviours and actions of some of the other actors (including academics) engaging in e-cigarettes debates. As the opening section of our results make clear, some interviewees described other actors in ways that suggested they were working to attract support for their preferred position, while undermining opposing positions. Here, the interview data were at their most critical. While one interviewee described the situation as ‘healthy debate’, several others expressed frustration, surprise, exasperation and annoyance at the behaviours they had witnessed and heard about :The claim that media interest in e-cigarette debates served to unhelpfully simplify debates by positioning actors as ‘for’ or ‘against’ e-cigarettes is unsurprising since it is in line with existing analyses of mass media coverage of other scientific debates (e.g. ). Howeve“It becomes difficult. So suppose… British American Tobacco were to buy Johnson & Johnson nicotine replacement therapy range… and became one of the biggest suppliers of medicinal nicotine in the UK, how would we react then? And that, so there would need to be some sort of change in the way that we dealt with tobacco industry in those circumstances. At the moment, the tobacco industry, or until now, has made it very easy for us to portray them as evil and […] organisations that we shouldn’t have anything to do with because they haven’t produced any products that we actually want. But whether they’re in that situation now is far from clear...”Interviewee 1 (academic researcher): Our interviewees did not generally proffer their own explanations as to why they felt some actors were working to attract others to particular ends of the spectrum Fig.  or why, Several interviewees, including the one quoted above, questioned whether tobacco control may need to alter its approach to excluding commercial interests, given the tobacco industry’s new role in e-cigarettes. In contrast, others emphasised some of the successes of tobacco control to date, noted previous efforts by tobacco industry actors to prevent and undermine tobacco control efforts, and stated they felt there remains a clear need to protect public health from tobacco industry influence .‘Like a The concept of a ‘policy paradigm’ comes from Hall  who, emp“If you look at the smoking rates… they are going down which is good, but it’s a really big health inequalities issue, and the people in the kind of poorer, more deprived areas have much higher smoking rates than the people in the more affluent areas. So whilst the overall trend is down, I think that there’s a lot still that needs to be done to support those people that are still smoking to become free of that addiction.”Interviewee 25 (industry manufacturer association): Viewed from this perspective, policy responses to the growth of e-cigarettes in the UK could involve one of three options: (i) the current policy paradigm continues and e-cigarettes become part of the existing ‘smoking cessation toolbox’ (interviewee 14 – pharmacy); (ii) e-cigarettes are tightly regulated, possibly banned, as policy shifts to a ‘neo-prohibitionist’ paradigm; or (iii) e-cigarette growth continues and informs a shift to a new ‘harm reduction’ policy paradigm . In our data, there was very little evidence of actors working to actively promote a shift towards a ‘neo-prohibitionist’ policy paradigm . Howeve“I’ve been promoting harm reduction as an alternative strand or a complementary strand to tobacco control ‘cause we aren’t doing enough for people who are addicted to tobacco. We’re doing quite well at preventing smoking uptake but we’re not doing very well at helping established smokers to quit. And harm reduction is an opportunity to do that.”Interviewee 1 (academic researcher): Some interviewees, such as the following, explicitly used claims about the failure of current approaches as a rationale for considering new approaches:may be on the cusp of significant changes in the way we think about tobacco policy.To explore this further, Fig. scientific authority, even though these debates are often entangled with more normative divisions (notably around the appropriate role of tobacco industry actors in policy discussions).From the perspective of the ‘policy paradigms’ literature it is unsurprising that e-cigarettes debates feel divisive since, at least for those clearly favouring one ‘paradigm’ over the other, the stakes are very high. Where potential policy paradigm shifts emerge, Hall  suggests“Recent research suggests that the second-hand smoke from some e-cigarettes may contain higher levels of certain harmful metals than in traditional tobacco cigarettes […]. Coupled with this, the World Health Organisation has already proposed banning the use of e-cigarettes indoors as they increase the background level of toxicants and nicotine. Children in Scotland believes that in light of this evidence that there should be further regulation of e-cigarettes.” (Children in Scotland).“E-cigarette vapour does contain toxicants however this is usually at levels which are far lower than those found in tobacco cigarettes. […] The consensus is that any harm from e-cigarettes is likely to be far less than that from tobacco. Therefore, we believe that there is currently insufficient evidence to justify a ban on the use of e-cigarettes indoors and in enclosed public spaces on the basis of harm from second-hand vapour.” (Cancer Research UK).Reflecting the importance of scientific authority for public health, we found scientific evidence regularly being enlisted by actors to support their positions. Indeed, for each e-cigarette regulatory issue, we can find submissions expressing directly contrasting preferences, each citing supporting evidence. The following two extracts - responses to consultation questions about restricting vaping in enclosed public places - are illustrative:For the most part, where this occurred, participants either appeared to be drawing on different sources of evidence or framing the implications of the same evidence in contrasting ways due to the normative lens through which they were interpreting it. However, we also found occasional examples of evidence being actively misrepresented. For example, a consultation submission from Totally Wicked Ltd. claimed that research undertaken by John Moores University demonstrated that there is no correlation between electronic cigarette advertising and use amongst children when, on checking the original qualitative research , we founInterestingly, where actors acknowledged that evidence was lacking, our data provide examples of this absence being employed to reach directly opposing conclusions, as Fig. ‘when the goal is monopolization of professional authority and resources, boundary-work excludes rivals […] by defining them as outsiders with labels such as “pseudo,” “deviant,” or “amateur”’.Such accounts are examples of what Gieryn terms scientific ‘boundary work’, in which participants construct a ‘boundary’ between ‘science’ and ‘non-science’ to reinforce the authority of their preferred perspective . AlthougIn fact, there are so many examples of ‘boundary work’ in action within our data that it is impossible to include them all so, for brevity, Table Overall, while there are occasional references to markers of scientific credibility relating to methods and processes in our data (e.g. one reference to the peer-review process and another highlighting the credibility of systematic reviews compared to single studies), we can also see in Table  that perRevisiting our earlier conversations with a sub-set of follow-up interviewees in 2019–2020 suggests that, while e-cigarette products and markets have continued to evolve, the intensity of policy discussions has temporarily abated. The primary explanation for this seemed to be the fact that initial UK regulatory responses have been agreed and enacted. Additionally, interviewees suggested other issues had distracted attention from e-cigarettes, including the emergence of cannabis related products and the need for broader organisational preparations for the UK’s exit from the European Union .No follow-up interviewee suggested there had been any significant new primary studies, though several noted there had been some major reviews to synthesise available evidence . InterviOverall, despite some feeling that media outlets continued to present e-cigarettes as divisive, our follow-up interviews suggested research and policy debates were less fraught in 2019–20 than they had been in 2016–17 but that this was due to reduced policy activity, rather than to substantial shifts in the positions of key players or the emergence of any research consensus. For now, e-cigarettes appear to have been accommodated within the UK’s dominant tobacco control policy paradigm, as a means of enabling smokers to reduce or quit smoking. However, it was also clear that post-Brexit regulatory discussions are being positioned as an opportunity for changing the UK’s regulatory approach to e-cigarettes. Indeed, it was notable that interviewees involved in the e-cigarettes industry were already expressing some confidence that post-Brexit changes will enable the further promotion and growth of e-cigarettes in the UK. One interviewee also noted ongoing discussions about the possibility of prescribing e-cigarettes via the National Health Service. All of this suggests that any lull in policy debates about e-cigarettes is likely to be temporary.In an overview of sociology of science, Pinch and Leuenberger argue that: ‘By studying a scientific controversy one learns something about the underlying dynamics of science and technology and their relations with wider society’, including ‘the normally hidden social dimensions of science’ . In thisThe ensuing debates have clearly been fraught and, like Hawkins and Ettelt , we founIn theoretical terms, our analysis suggests that Sabatier and Jenkins-Smith’s ACF, which had previously been widely applied to analyses of tobacco policy debates , 19, 27,Looking to the future, our findings suggest that e-cigarette debates are likely to settle only when a majority of participants in the uncertain ‘middle ground’ agree a more concrete position along the spectrum depicted in Fig. In the meantime, our follow-up interviews suggest that policy debates are experiencing a temporary lull, during which time there is a reduced spotlight on the ‘scientific showdowns’ we identified. However, these ‘scientific showdowns’ seem likely to re-intensify as new (e.g. post-Brexit) policy debates develop. This means evidence will continue to be constructed and interpreted by actors against a tense backdrop of contrasting claims to scientific authority. The importance attached to scientific authority in public health policy means academics are key actors in these debates and future research should explore not only how actors are deploying evidence in policy debates but also how and why the e-cigarettes evidence-base is being constructed as it is, by whom and to what ends.Our data support previous research in identifying policy debates around e-cigarette as highly contentious . Indeed,Additional file 1. Generic version of the main interview schedule, which lists the questions we put to interviewees .Additional file 2. Generic version of the follow-up interview schedule, used in the follow-up interviews conducted August 2019–March 2020.


Knowledge translation (KT) platforms are organisations, initiatives and networks that focus on supporting evidence-informed policy-making at least in part about the health-system arrangements that determine whether the right programmes, services and products get to those who need them. Many descriptions and evaluations of KT platforms in low- and middle-income countries have been produced but, to date, they have not been systematically reviewed.We identified potentially relevant studies through a search of five electronic databases and a variety of approaches to identify grey literature. We used four criteria to select eligible empirical studies. We extracted data about seven characteristics of included studies and about key findings. We used explicit criteria to assess study quality. In synthesising the findings, we gave greater attention to themes that emerged from multiple studies, higher-quality studies and different contexts.. Evidence briefs and deliberative dialogues were the activities/outputs that were the most extensively studied and viewed as helpful, while rapid evidence services were the next most studied but only in a single jurisdiction. None of the summative evaluations used a pre–post design or a control group and, with the exception of the evaluations of the influence of briefs and dialogues on intentions to act, none of the evaluations achieved a high quality score.Country was the most common jurisdictional focus of KT platforms, EVIPNet the most common name and high turnover among staff a common infrastructural featureA large and growing volume of research evidence suggests that KT platforms offer promise in supporting evidence-informed policy-making in low- and middle-income countries. KT platforms should consider as next steps expanding their current, relatively limited portfolio of activities and outputs, building bridges to complementary groups, and planning for evaluations that examine ‘what works’ for ‘what types of issues’ in ‘what types of contexts’. Our definition of evidence-informed policy-making has two parts. The first part involves using the best available data and research evidence – systematically and transparently – in the time available in each of the four phases of the policy-making process , namely The organisations, initiatives and networks supporting evidence-informed health policy-making can be differentiated, based on previous works by the authors, according to which phase(s) of the policy-making process they focus on and whether that focus is about: (1) clinical programmes, services and products (e.g. prescription drugs) that target individuals; (2) public health programmes and services that target groups and populations; and/or (3) health system  arrangements that determine whether the right programmes, services and products get to those who need them, in ways that improve population health and the patient or citizen experience, while keeping per capita costs manageable , 5.most data-analytics organisations focus on (1) understanding policy problems and possibly monitoring implementation and (2) clinical or public health topics;most clinical practice guideline initiatives focus on (1) informing which programmes, services and products clinicians should provide (i.e. policy or programme development) and (2) clinical practices;most health technology assessment (HTA) networks focus on (1) informing which programmes, services and products a health system should provide and (2) clinical and, less commonly, public health ‘technologies’;most modelling organisations focus on (1) estimating the expected reach and impact of selected practices/technologies and related financial, human, and other resource needs and (2) practices/technologies and, less commonly, health system arrangements;most implementation research/behavioural insights initiatives focus on (1) informing which implementation approach a health system should use to improve the reach and impact of selected practices/technologies and (2) health system arrangements; andmost evaluation networks focus on (1) understanding the impact of selected policy choices (e.g. practices/technologies) and (2) clinical or public health topics.Consider the following six examples, which complement the type of organisation that is the focus of this study, described in greater detail in the next section :most daOur focus here is what we call KT platforms, which we define as organisations, initiatives and networks that focus on supporting evidence-informed policy-making at least in part about the governance, financial and delivery arrangements that determine whether the right programmes, services and products get to those who need them  –9. WhileOur focus is specifically KT platforms in low- and middle-income countries (LMICs), where policy-making in over-burdened and under-resourced health systems may present particular challenges, including , 11 (1) While our interest in KT platforms arose from our involvement in the WHO-sponsored Evidence-Informed Policy Networks (EVIPNet) or in similar entities seen as peers to EVIPNet , 12, 13,Describing and evaluating KT platforms becomes increasingly difficult as one moves from (1) description through formative evaluation and on to summative evaluation and (2) activities and outputs (i.e. the approaches used) through outcomes and impacts (i.e. whether the approaches are making a difference) and on to context and infrastructure .When it comes to summative evaluation, for example, the KT field continues to search for the ‘holy grail’ of outcome and impact measures that are light touch and can be applied across both ‘intervention’ and ‘control’ groups . The golMany descriptions and evaluations of KT platforms in LMICs have been produced but, to date, they have not been systematically reviewed. The objective of this systematic review is to describe the findings of empirical studies that 1) describe the activities and outputs of KT platforms; (2) formatively evaluate these activities and outputs; (3) summatively evaluate whether activities and outputs achieve outcomes and impacts; (4) describe the KT platforms’ context and infrastructure; and (5) examine other types of linkages among variables , such as (a) context and infrastructure influencing whether activities and outputs achieve outcomes and impacts; (b) context influencing decisions about infrastructure design, activities and outputs selected, or the baseline measure of key outcomes; and (c) infrastructure influencing decisions about activities and outputs selected. We provide an illustration of these variables and their potential inter-relationships in Fig.  describeWe used the Preferred Items for the Reporting of Systematic Review and Meta-Analysis (PRISMA) statement to guide the design of the systematic review . We did We began developing our search parameters by identifying synonyms for three conceptual domains relevant to our review question (and combining synonyms within a domain with ‘OR’ and then across domains with ‘AND’), namely knowledge translation, policy-making and LMICs (which we addressed using a filter developed by the Norwegian satellite of Cochrane’s Effective Practice and Organisation of Care review group in 2012) [We conducted searches in five electronic databases, namely Cumulative Index to Nursing and Allied Health (CINAHL), Embase, Global Health, MEDLINE and Web of Science. Searches were conducted on two separate occasions, first in January 2015 (week 2) and again in September 2016 (week 4) to capture studies published in the intervening time period. We complemented the electronic database searches with a variety of approaches to identify additional literature (including grey literature), namely requests for studies sent to experts in the field; manual review of the reference lists of included studies; ‘related articles’ search in PubMed for all included studies (in July 2017); and manual review of a list of studies published by EVIPNet affiliates, whether or not EVIPNet was the focus of the evaluation, that was maintained by the study authors as part of their evaluation work .We used four criteria to assess eligibility for inclusion in the review: (1) does the article discuss (a) organisations, initiatives or networks located in or targeted at (b) LMICs whose goal is at least in part to (c) support the use of research evidence in health systems policy-making?; (2) does the article have a methods section?; (3) does the article report empirical data based on the application of these methods?; and (4) do the empirical data pertain to context, infrastructure, activities/outputs, outcomes and/or impacts of these organisations, initiatives and networks?All criteria needed to be met in order to be included in the systematic review. We did not exclude articles using specific exclusion criteria such as studies in non-peer-reviewed publications .We assessed eligibility in three phases: (1) title and abstract review, which was completed by one reviewer (ACRP) on the full sample and by a second reviewer (HR) on a 20% sample; (2) first full-text article review, which was again completed by one reviewer (ACRP) on the full sample and by a second reviewer (HR) on a 20% sample; and (3) a second full-text article review, which was completed by three reviewers  on all articles that were short-listed for inclusion (i.e. that made it through the first full-text article review), to ensure that criteria 1 and 4, which were the most difficult to judge, were appropriately assessed. Any disagreements that arose among the reviewers were resolved by consensus. We calculated the agreement between reviewers on the two 20% samples using Fleiss’ Kappa coefficient. The Fleiss’ Kappa coefficient on the two 20% samples were 0.66 and 0.71, respectively. We retained a list of ‘near miss’ papers that were excluded in case a reader wanted to double-check the application of our inclusion criteria. These are available in Additional file We extracted data about the following characteristics of included studies: (1) lead author, year and citation; (2) jurisdictional focus of KT platform(s); (3) name(s) of KT platform(s); (4) categories of variables and/or relationships addressed ; (5) time period studied; (6) data collection method(s) used; and (7) objective(s). We also extracted the key findings from included studies, with a focus on the variables and relationships described above (e.g. findings about activities and outputs were extracted and organised using the Cochrane KT framework subheadings). When more than one article described the same empirical study, we treated them as a single study for data-extraction purposes only if the same data were re-presented across articles. After extensive pilot testing, data were extracted by one reviewer who was not involved in any of the studied KT platforms (ACRP). A second reviewer (CM) checked all extracted data and resolved any concerns through discussion with the first reviewer.For all included studies, we used the following explicit criteria to assess quality, with a focus on those including a formative and/or summative evaluation: (1) two or more data collection methods were used  , 22; (2)We used the first three criteria to assess descriptive studies and formative evaluations (yielding a score out of three) and all five criteria to assess summative evaluations that measured impact (yielding a score out of five). We did not use more traditional risk-of-bias criteria  to assess the summative evaluations because none of the included studies used formal effectiveness designs such as a randomised controlled trial. We specified cases where scores for individual formative, summative and/or linkage evaluations were different from overall study scores. Articles were deemed to be high quality if they received a score of at least two points if they were assessed using the first three criteria or at least three points if they were assessed using all five criteria. After extensive pilot testing, quality was assessed by one reviewer who is involved with a KT platform but not one that has yet been studied (CM) and was checked by a second reviewer (ACRP). Discrepancies were resolved through discussion between the reviewers and any conflicts were reviewed and resolved with a third reviewer (JNL).In synthesising the findings, we gave greater attention to themes that emerged from (1) multiple studies, (2) higher-quality studies and (3) different contexts. The first draft of the synthesis was completed by one reviewer who was not involved in any of the studied KT platforms (ACRP) and the second and final drafts were completed in conjunction with a second reviewer (JNL).As illustrated in Fig. the years of publication ranged from 2008 to 2017, with the median year of publication being 2014;the most common jurisdictional focus of the KT platforms was country rather than sub-national, regional  or global, while the most common country focus was Uganda (appearing in 13 studies) , 52, 53,the most common name used by the KT platforms was EVIPNet (appearing in 17 studies) , 52, 53,n = 33 of 38 studies) [n = 18) [n = 15) [n = 4) [◦ descriptions (studies) , 58–60 w[n = 18) , 57–60, [n = 15) , 58, 59, [n = 4) , 45;n = 24 of 33 studies) [n = 17) [n = 14) [◦ of the descriptive studies, context (studies) –54, 56 w[n = 17) –60 and i[n = 14) ;n = 8 of 18 studies) [n = 6) [◦ of the formative evaluations, evidence briefs (studies) , 57, 59  [n = 6) , 57, 59 n = 4 of 15 studies) [◦ of the summative evaluations, deliberative dialogues were the most common activities examined (studies) , 55, 59;n = 3 of 4 studies) [n = 3) [n = 1) [◦ of the studies of other linkages among variables, the relationships between context and activities and outputs (studies) , 36, 45  [n = 3) , 33, 45  [n = 1) ;the most common variables and relationships addressed, were as follows:the time period studied ranged from 1996 to 2015, with the median year of time period studied  being 2010; andn = 20 of 38 studies) [n = 17) [n = 4) [the most common data collection methods used were interviews (studies) , 58–60, [n = 17) , 57, 59  [n = 4) , 59, 60  [n = 4) , 56–60 aBased on our analysis of the characteristics of included studies provided in Additional file for formative evaluations, the mean and median quality scores were 1.0 and 1.0 out of 3, respectively, and the most common criterion responsible for a lower score was a response rate lower than 60%; andfor summative evaluations, the mean and median quality scores were 1.3 and 1.0, respectively, but this time out of 5, and the most common criterion responsible for a lower score was two or more types of competing variables examined.Turning now to the quality assessments for studies including a formative and/or summative evaluation, which we present in Additional file evidence briefs and deliberative dialogues were the activities/outputs that have been the most extensively studied and the most widely undertaken ; andrapid evidence services and capacity-building workshops  were arguably the next most extensively studied and widely undertaken .Finally, we turn to the summary of findings from included studies  described in the included studies are a jurisdiction-specific summary of what is known from local data and studies and from systematic reviews about (1) a problem and its causes, (2) options for addressing it and (3) key implementation considerations. The deliberative dialogues  involve a diverse group of policy-makers, stakeholders and researchers – informed by a pre-circulated evidence brief – deliberating about the same three topics as well as next steps for different constituencies, with the key themes captured in a dialogue summary. The combination of the evidence brief and dialogue summary are intended to provide policy-makers with the best available research evidence and systematically elicited stakeholder insights. The rapid evidence services described in the included studies provide a summary of what is known, typically from systematic reviews and sometimes from local data and studies, and from jurisdictional scans. These are typically in time frames of days to weeks and about one of a problem, policy options, or implementation considerations. The capacity-building workshops are intended to help policy-makers and stakeholders to find and use research evidence on their own as part of a systematic approach to examining priority issues, whether they have minutes, hours or days to inform a policy-making process. The systematic approach maps policy questions about problems, options and implementation considerations onto types of research evidence and then those types of research evidence onto appropriate sources of pre-appraised, synthesised research evidence .evidence briefs and deliberative dialogues were the activities/outputs that have been the most extensively studied and the most widely viewed as helpful both in general and in terms of the specific design features commonly used by EVIPNet; andrapid evidence services were the next most extensively studied but only in a single jurisdiction (Uganda).Moving on to formative evaluations of activities and outputs Table , we founThese formative evaluations took the form of surveys administered to participants in deliberative dialogues, with one survey about the pre-circulated evidence brief being completed before the dialogue began and a second survey about the dialogue itself after the dialogue was completed.KT platforms as a whole have been the most extensively studied in terms of both (1) impacts on policy-making processes and (2) influence on outcomes such as stronger relationships between policy-makers and researchers and raising awareness about, and building demand for, using research evidence; andevidence briefs and deliberative dialogues were the next most extensively studied, both in terms of their impact on policy-makers’ and stakeholders’ intentions to act on what was learned and on select policy-making processes.Continuing on to the summative evaluations of outcomes and impacts Table , we founHowever, none of these summative evaluations used a pre–post design or a control group and, with the exception of the evaluations of the influence of briefs and dialogues on intentions to act , none ofmany descriptions of the political system context in which the KT platforms are operating, some descriptions of their research system context and no descriptions of their health-system context;some descriptions of the KT platforms’ infrastructure; andno formative evaluations of the KT platforms’ context or infrastructure per se, just statements made in single studies about helpful aspects of the context or infrastructure (which their designs did not permit them to examine rigorously).With respect to context and infrastructure Table , we founThe key features of the political and research system context in which the KT platforms are operating  include (1) policy-makers have limited capacity for finding and using research evidence in policy-making; (2) policy-making processes have many veto points where key interests can block evidence-informed policy proposals or support competing alternatives; and (3) there are low levels of researcher engagement in supporting evidence-informed policy-making. The key features of the KT platform’s infrastructure (again those identified in four or more studies) include (1) variability in whether decision-making authority for the KT platform was located in government or elsewhere; (2) short-term, unpredictable or scarce ongoing funding; (3) a lack of skilled human resources to draw upon; and (4) high turnover among KT platform staff. Examples of the statements made about helpful aspects of KT platforms’ context and infrastructure include (1) high-level political support is key; (2) ‘home-grown’ models can have a greater likelihood of success; (3) strong, independent advisory or governance structures are helpful; and (4) staff who are well trained and proactive are essential.Concluding with other types of linkages among variables, we found only four studies , 36, 45 country was the most common jurisdictional focus of the KT platforms and EVIPNet the most common name used by them;descriptions (33 of 38 studies) were more common than formative evaluations , summative evaluations  or studies of other linkages among variables (4);interviews and surveys were the most common data collection methods used ;quality scores were generally low for both formative and summative evaluations;evidence briefs and deliberative dialogues were the activities/outputs that have been the most widely undertaken , with rapid evidence services and capacity-building workshops the next most common;evidence briefs and deliberative dialogues were also the activities/outputs that have been the most extensively subjected to formative evaluation and the most widely viewed as helpful both in general and in terms of the specific design features commonly used by EVIPNet, while rapid evidence services were the next most extensively studied but only in a single jurisdiction (Uganda);KT platforms as a whole, followed by evidence briefs and deliberative dialogues, have been the most extensively subjected to summative evaluation; however, none of these summative evaluations used a pre–post design or a control group and, with the exception of the evaluations of the influence of briefs and dialogues on intentions to act, none of the evaluations achieved a high quality score;the key features of the political and research system context in which the KT platforms are operating  include (1) policy-makers have limited capacity for finding and using research evidence in policy-making; (2) policy-making processes have many veto points where key interests can block evidence-informed policy proposals or support competing alternatives; and (3) there are low levels of researcher engagement in supporting evidence-informed policy-making;the key features of the KT platform’s infrastructure include (1) variability in whether decision-making authority for the KT platform was located in government or elsewhere; (2) short-term, unpredictable or scarce ongoing funding; (3) a lack of skilled human resources to draw upon; and (4) high turnover among KT platform staff; andthe four studies examining linkages among other variables found that political support (context) and networks/multi-institutional arrangements (infrastructure) can influence the demand for, and supply of, activities and outputs.Although it remains premature to make definitive statements about whether KT platforms in general or their particular approaches are effective, let alone whether and how context and infrastructure influence whether the approaches translate into outcomes and impact, we have a large and growing volume of research evidence (38 studies) suggesting that KT platforms offer promise in supporting evidence-informed policy-making in LMICs. Our principal findings include the following:Our systematic review has five main strengths, as follows: (1) we conducted an exhaustive, broad-based search for both published and grey literature; (2) we used explicit criteria to set a relatively low threshold for inclusion  and provided reassurance about reliability when only one reviewer was involved in their application (e.g. ensuring we had achieved an acceptable Kappa statistic); (3) we used explicit quality criteria that reflected best practices in studies of policy-making processes ; (4) we gave greater attention to themes that emerged from multiple studies, higher-quality studies, and different contexts; and (5) we took steps to ensure that team members who are involved in operating, supporting and evaluating KT platforms were not involved in several key steps in the review .There are three main weaknesses in our review, two of which have more to do with the difficulties of describing and evaluating KT platforms than they do with the design and execution of the review, namely (1) study searches were undertaken in 2015 and 2016 and the related articles search in 2017; (2) none of the summative evaluations used formal effectiveness designs, such as a randomised controlled trial , likely in no small part because of the lack of outcome and impact measures that can reliably perform as well as of case studies focusing on untangling actual influences on policy-making that often take place over long periods of time, behind ‘closed doors’, and in other ways that make simple metrics difficult to apply; and (3) the near absence of studies of linkages among variables highlights the sample-size challenge of finding enough contexts and infrastructures supporting a common approach that can be evaluated using standardised methods.To the best of our knowledge, this is the first systematic review of descriptions and evaluations of KT platforms in LMICs. The ever-growing number of systematic reviews undertaken to inform efforts to support evidence-informed policy-making –64 diffeWe offer three implications of our review for those creating, leading or supporting KT platforms: (1) consider expanding the current, relatively limited portfolio of activities and outputs ; (2) consider aligning with evaluation initiatives using a common approach to outcomes and impact measurement to facilitate cross-platform learning (as EVIPNet did in its first decade); and (3) consider giving greater attention to context  and to infrastructure , particularly given that this review suggests that the prospects for robust empirical answers to these questions are limited.While not implications arising from this review per se, this work did prompt us to reflect on where KT platforms may need to move in future, as follows (1) consider building bridges to organisations, initiatives and networks working in complementary areas such as guideline and HTA units  but also data-analytics, modelling, implementation research and evaluation units (that are building data and evidence de novo to support a particular phase of the policy-making process); (2) consider partnering with allies both within government  and outside government ; (3) consider what can be done to institutionalise the use of research evidence in policy-making processes ; and (4) consider collaborating with organisations, networks and initiatives that are operating in the broader Sustainable Development Goals space but that have not had the long-term focus on supporting evidence-informed policy-making that has been a hallmark of the health sector  across all Sustainable Development Goals except the three about the environment .While we are in urgent need of a monitoring and evaluation plan 2.0 for KT platforms, our experience with the KT platform monitoring and evaluation plan 1.0 used for EVIPNet and now with this review reinforce for us how very difficult it will be to design one that is sensitive both to best practices in studies of policy-making processes and to best practices in summative evaluations (i.e. effectiveness studies). The paradox is that KT platforms attempt to hold others to account to use research evidence in their decision-making, yet they themselves work in a space that is so difficult to evaluate in robust ways.A large and growing volume of research evidence suggests that KT platforms offer promise in supporting evidence-informed policy-making in LMICs. While our review had many strengths, many of which reflect best practices in studies of policy-making processes, unfortunately, none of the summative evaluations used formal effectiveness designs, which limits our ability to make statements about ‘what works’. KT platforms should consider as next steps expanding their current, relatively limited portfolio of activities and outputs (e.g. to include citizen panels), building bridges to complementary groups , and planning for evaluations that examine ‘what works’ for ‘what types of issues’ in ‘what types of contexts’.Additional file 1. MEDLINE search string.Additional file 2. Citations for ‘near miss’ papers that were excluded.Additional file 3. Characteristics of included studies with reference list.Additional file 4. Quality assessments for included studies.Additional file 5. Summary of findings from included empirical studies.


We herein summarise the evidence concerning the impact of sperm DNA fragmentation in various clinical infertility scenarios and the advances on sperm DNA fragmentation tests. The collected evidence was used to formulate 41 recommendations. Of these, 13 recommendations concern technical aspects of sperm DNA fragmentation testing, including pre‐analytical information, clinical thresholds and interpretation of results. The remaining 28 recommendations relate to indications for sperm DNA fragmentation testing and clinical management. Clinical scenarios like varicocele, unexplained infertility, idiopathic infertility, recurrent pregnancy loss, intrauterine insemination, in vitro fertilisation/intracytoplasmic sperm injection, fertility counselling for men with infertility risk factors and sperm cryopreservation have been contemplated. The bulk evidence supporting the recommendations has increased in recent years, but it is still of moderate to low quality. This guideline provides clinicians with advice on best practices in sperm DNA fragmentation testing. Also, recommendations are provided on possible management strategies to overcome infertility related to sperm DNA fragmentation, based on the best available evidence. Lastly, we identified gaps in knowledge and opportunities for research and elaborated a list of recommendations to stimulate further investigation. The male factor is found in nearly 10% of all couples and is responsible for about 50% of infertility cases . Many conditions, including varicocele, chronic illness, accessory gland infections, advanced paternal age, lifestyle, obesity, occupational and environmental factors, medications, ionising and nonionising radiation and heat exposure, have been associated with elevated SDF levels  and infertile men. While time to pregnancy (TTP) of less than 12 months from stopping contraception and ability to conceive should be ideally used as the criterion to classify fertile men  is influenced by the SDF level. Moreover, a growing body of evidence referenced in this manuscript supports the hypothesis that SDF is associated with various pre‐conception developmental impairments and also post‐conception issues such as miscarriages and increased susceptibility to progeny diseases.Despite the robust association between SDF and infertility, the limited knowledge of SDF tests' characteristics and a common opinion that SDF is untreatable have prevented the broad application of testing in routine practice  role and utility. Currently, about eight guidelines on male infertility have been developed by expert panels from many societies  and the American Society for Reproductive Medicine (ASRM) have not recommended the use of SDF testing during the routine infertility evaluation mainly due to insufficient data and lack of effective treatment options to overcome infertility in such cases  is a vast document that contains a subsection specifically addressing SDF testing  guidelines on male infertility dedicated a few sections to SDF testing and the impact of SDF in varicocele and unexplained infertility . The coordinator (SCE) drafted the key questions and invited experts in the field, including reproductive urologists , scientists with well‐known expertise in the technical aspects of SDF tests  and one reproductive endocrinologist (PH). Based on defined keywords , the literature search was performed in PubMed/MEDLINE from inception up to 31 May 2020.2.2The coordinator prepared a summary of findings based on existing systematic reviews and meta‐analyses and controlled trials or relevant cohort studies and case reports when the former were not available. The guideline development group (GDG) discussed the summary evidence and provided additional supporting evidence if applicable, which served the basis for the draft recommendations.2.3The coordinator prepared the draft recommendations and discussed them with the GDG to reach an agreement on the final recommendations. For each recommendation, a strength rating based on GDG expert judgment and the grade of recommendation, according to the Oxford Centre for Evidence‐Based Medicine Levels of Evidence , was inc3In healthy spermatozoa, the chromatin is characterised by a linear disposition of the nucleotides along each DNA strand and the lack of both single and double DNA strand breaks, nucleotide modifications or base loss , lifestyle , infection, fever, radiotherapy, chemotherapy and ageing have been related to SDF , The existing tests can be group in methods that use (a) enzymatic reactions to label the DNA breaks, (b) controlled DNA denaturation combined with protein depletion as intermediates to reveal the DNA breaks and (c) dyes that bind to relaxed GC‐rich motifs. The first category comprises tests that utilise a terminal transferase  SDF tests, namely TUNEL, ISNT, SCSA, SCD, and Comet, and (2) Sperm chromatin compaction tests, namely, chromomycin A3 staining, acridine orange staining, toluidine blue staining and aniline blue staining.Tests that measure SDF  may be preferable to those that measure chromatin compaction due to the role of oxidative stress in male infertility  for SDF measured using the SCSA , SCD and TUNEL , and SCSA and TUNEL , whereas moderate correlations were reported for alkaline Comet and SCD , alkaline Comet and SCSA , and alkaline Comet and TUNEL  is not supported by current evidence, as results cannot predict the likelihood of pregnancy  provide reliable information about sperm DNA integrity in subfertility. However, it is vital to understand how each test reports results.In SCD and conventional TUNEL, the assessments are carried out manually on one to several hundred spermatozoa, under bright‐field (SCD) or fluorescence microscopy . However, thresholds may vary slightly according to individual studies and methods. For instance, a clinical threshold of 17% has been reported for male infertility diagnosis with the flow cytometry TUNEL assay  best discriminate confirmed and presumed fertile men from infertile men, with an area under the curve (AUC) of 0.844  discriminate fertile from infertile men with an AUC of 0.925  are clinically useful for classifying infertile couples into a statistical probability of prolonged time to achieve natural pregnancy, decreased likelihood of pregnancy by IUI, IVF or ICSI and increased risk of miscarriage  low‐grade (e.g. grade 1) varicocele and borderline to normal semen parameters , and (b) moderate (grade 2) or large (grade 3) varicocele and semen parameters within normal ranges  and SDF remains equivocal. Although a controlled study involving 337 men reported a remarkable improvement in semen parameters and increased clinical pregnancy rates after repair of subclinical varicoceles , as pregnancy rates by ICSI are less affected by elevated SDF than with the use of IUI and conventional IVF (discussed in next sections).5.3Recurrent pregnancy loss is defined as two or more pregnancy losses from conception to 24 weeks of gestation  involved in RPL in couples with SDF is not known. However, it has been speculated that DNA fragmentation not repaired by the oocyte may contribute to poor blastocyst development, implantation failure and miscarriage  seems to modulate the final effect. Recent evidence suggests that the presence of double DNA strand breaks (DS‐DBs) is more lethal than single‐DNA strand breaks and potentially associated with RPL, implantation failure and spontaneous miscarriage after IVF or ICSI  exceed 20%  and delivery rates  was associated with reduced pregnancy rates , possibly causing SDF. After treatment, the patient might be retested to check if SDF was reduced to allow the continuation of IUI treatment. If no causative factor is identified, or elevated SDF persists after treatment, ICSI may be considered.5.5p < 0.003) or ICSI .Most IVF/ICSI meta‐analyses concur that sperm DNA integrity impacts reproductive success. The studies of Li et al., Zini et al. and Zhao et al. showed that elevated SDF was associated with reduced pregnancy rates with conventional IVF but not ICSI  and miscarriage rates  were negatively affected by the presence of elevated SDF; however, live birth rates were not apparently impacted . Although the adverse impact of SDF on IVF and ICSI cycles has not been reported unequivocally , varicocele, genital infections, advanced age and exposure to toxicants .A positive association between exposure to air pollutants toxicants  and SDF has been documented  compared with the Western diet  was shown to help reduce SDF rates . However, no data were provided concerning the effect of SDF reduction on fecundity and offspring health. Lastly, an uncontrolled cohort study of six obese men with unexplained infertility showed that a nutritionist‐led dietary program associated with exercise over a 3‐ to 8‐month period was able to help reduce SDF values; in this study, the couples achieved full‐term deliveries after the intervention , indicated that SDF rates may be decreased by adopting meditation and yoga‐based lifestyle  are used to determine the ideal number of specimens to bank and prognosticate the assisted conception modality required for future conception. Moreover, a frozen aliquot is thawed for sperm cryosurvival assessment, which also helps estimate the total motile sperm number available and advise about the prospects of assisted conception.p < 0.0001) and progressive motility  was significantly higher following vitrification than conventional freezing. However, SDF (reported in four studies) was not significantly affected.The cryopreservation process can harm semen quality as it can increase ROS production, leading to excessive OS  select the optimal method for sperm freezing and (b) guide the later decision to use IUI or IVF/ICSI if both options are available.6In this section, we provide recommendations concerning the technical aspects, indications and interpretation of SDF testing based on the evidence that has been identified, collated and analysed  with vast clinical experience but also a group of eminent scientists/andrologists  with seminal contributions to the development of the four major SDF tests . For the first time ever, the latter group worked together in a project of its kind, bringing readers the most detailed and practical information regarding SDF testing. Also, our guidelines included a distinguished reproductive endocrinologist (PH) with vast clinical experience, who added unique insights concerning the application of SDF testing in couples undergoing ART.Second, we included several relevant studies in each subsection, strengthening the clinical utility of SDF testing and the recommendations made Tables  and 3. TWe also expanded the indication of SDF testing to all couples considering IUI or IVF/ICSI provided the minimum requirements for running an SDF test are met .Clarify the role of antioxidant therapy for men with SDF.Compare the cost‐effectiveness of early ICSI versus expectant management in couples with unexplained infertility, RPL, family planners with infertility risk factors (preferably in prospective studies).Compare laboratory techniques to select spermatozoa with low DNA fragmentation for ICSI, in prospective trials involving couples with elevated SDF, controlled for age and other confounders.Compare the clinical efficacy of testicular spermatozoa for ICSI, in prospective randomised trials involving couples with elevated SDF, controlled for age and other confounders.Further research is needed on the clinical SDF thresholds to be used with each SDF test on IUI, IVF, and ICSI, using different endpoints .Clarify the role of oocyte quality on SDF repair.Establish the value of preimplantation genetic testing in couples with elevated SDF undergoing IVF/ICSI.Study the role of cryoprotectants and cryopreservation techniques to protect spermatozoa from DNA damage.Based on the published data and discussion of the available evidence, we identified various topics for which evidence is inconclusive or inexistent. The GDG recommends that future research focuses on the gaps in knowledge listed below.9Male infertility is a common medical condition and a public health concern as it is associated with adverse effects on reproduction, overall health, reduced life expectancy and impaired quality of life. A comprehensive evaluation of male infertility can reveal severe and potentially life‐threatening underlying medical conditions. The prevention and management of male and female infertility are integral components of comprehensive sexual and reproductive health services needed to attain a sustainable development goal. Our CPG translates the best existing evidence into recommendations to provide the foundation for standardising care while maintaining clinicians′ autonomy. We herein reviewed the data supporting the indications of SDF testing in different infertility scenarios and elaborated recommendations based on best evidence and expert judgment.10Infertility is a couple′s problem; thus, a single test of gamete dysfunction from just one partner is limited to predict the treatment outcome. However, SDF thresholds may reflect the probability of a successful reproductive outcome influenced by the SDF level and modulated primarily by females age.While SDF testing is not a replacement for the current tools for infertility diagnosis, it may add independent information about sperm quality, and its integration into fertility clinics may provide better counselling, diagnosis and treatment planning.Sperm DNA fragmentation testing in the clinic can help to:Identify patients with potentially correctable underlying factors causing SDF, including the optimal selection of patients for varicocele repair.Provide laboratory evidence of defective sperm chromatin to couples seeking fertility counselling and family planning, particularly when the male partner has infertility risk factors, as a way to counsel about fecundity prospects and reinforce the importance of lifestyle changes and avoid exposure to risk factors.Better assess the semen quality of subfertile men of advanced paternal age for counselling and guiding clinical management.Monitor the effects of interventions ;Identify and guide management in couples where elevated SDF might contribute to unexplained/idiopathic infertility.Identify and guide management in couples in whom elevated SDF might contribute to recurrent pregnancy loss and could cause IUI or IVF/ICSI failure.Better assess the sperm quality of cancer patients who wish to bank sperm for fertility preservation to guide the choice of assisted conception modality optimally.The male partner of any infertile couple who is found to have elevated SDF should be evaluated by a reproductive urologist/andrologist to rule out varicocele and other occult male factors. The reduction in SDF rates may help the couple achieve natural conception, downgrade the assisted reproduction method and increase the likelihood of successful reproductive outcomes with IUI, IVF and ICSI.SCE coordinated the GDG and had a leading role in collecting the evidence, drafting the manuscript and handling the GDG’s comments. All participants contributed to the guideline development, discussed the key questions, synthetised the evidence, drafted recommendations and writing sections of the manuscript. All authors read and approved the submitted version.SCE declares the receipt of unrestricted research grants and lecture fees from Merck outside the submitted work. PH reports receipt of unrestricted research grants from Merck, IBSA, Gedeon Richter, and MSD, and lecture fees from Merck, Gedeon Richter, MSD, and IBSA outside the submitted work. AZ declares shares in YAD‐Tech neutraceuticals. DPE is president director of SCSA Diagnostics, a company with a commercial interest in sperm DNA damage. SEML is an employee of Examenlab Ltd., a university spin‐out company with a commercial interest in sperm DNA damage. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.Fig S1‐S5Click here for additional data file.


The field of digital health, and its meaning, has evolved rapidly over the last 20 years. For this article we followed the most recent definition provided by FDA in 2020. Emerging solutions offers tremendous potential to positively transform the healthcare sector. Despite the growing number of applications, however, the evolution of methodologies to perform timely, cost-effective and robust evaluations have not kept pace. It remains an industry-wide challenge to provide credible evidence, therefore, hindering wider adoption. Conventional methodologies, such as clinical trials, have seldom been applied and more pragmatic approaches are needed. In response, several academic centers such as researchers from the Institute of Global Health Innovation at Imperial College London have initiated a digital health clinical simulation test bed to explore new approaches for evidence gathering relevant to solution type and maturity. The aim of this article is to: (1) Review current research approaches and discuss their limitations; (2) Discuss challenges faced by different stakeholders in undertaking evaluations; and (3) Call for new approaches to facilitate the safe and responsible growth of the digital health sector. The FDA considers digital health as a broad scope that includes categories such as mobile health, health information technology, wearable devices, telehealth and telemedicine, and personalized medicine3, a definition we follow in this article. Indeed, the numbers of digital health solutions are booming, for example, more than 300,000 health applications exist with more than 200 added daily4. Digital solutions can be grouped as follows, based on potential risk to patients5: (1) Solutions that improve system efficiency but with no measurable patient outcome benefit; (2) Mobile digital health, that inform or deliver basic monitoring, and encourage behavior change and self-management; (3) Clinical decision support (CDS), and prediction models, that guide treatment, deliver active monitoring, calculate and/or diagnose.Digital health has evolved rapidly since the concept was first introduced in 2000 by Seth Frank6. In addition, aging adults, considered amongst the most digitally divided demographic group7, present unique challenges and dedicated efforts exist to develop strategies for implementation10. Conversely, challenges exist for healthcare innovators to best demonstrate solution impacts and to ensure compliance with standards, these include: unclear end-user expectations; uncertainty of evidence generation approaches; and, keeping up to date with the evolving compliance landscapes.The evidence requirements of regulators are determined by a product’s intended use claims, as such, a large proportion of digital health solutions (e.g. administrative tools and wellness apps) fall outside of their jurisdiction. Therefore, a huge challenge for end users, such as patients and providers , is how to determine a new solution’s credibility and compliance with standards. Furthermore, end users have different thresholds for acceptance of innovation and can be grouped into five archetypes: innovators, early adopters, early majority, late majority, and laggardsThis article discusses the challenges for providing timely and robust evidence, to meet end-user expectations, in the context of digital health solutions. Specifically, we consider how the cadence of traditional research approaches are misaligned with the “fail fast, fail often” mantra espoused by technology start-ups. In addition, we introduce clinical simulation-based research as a potential opportunity to bridge the evidence gap.Over the last 10 years a plethora of guidance has been developed for digital health innovators. In Table 14 , and proposes to assess both development organization and product capabilities. Notwithstanding, current guidance does not go far enough to enable innovators and end-users to know what evidence generation approaches are appropriate, and practical, for all classes of digital health solutions throughout the product lifecycle.Until recently regulators relied upon modifications to existing medical device (software) regulations and innovators were encouraged to conform to development standards, as shown in Table 4 Tables . For exa16, yet, only a handful of products have been tested in this way as shown by recent systematic review17 and our searching results in Table 17. In our opinion, this lack of studies indicates that these methods are no longer practicable, likely due to the speed of digital product development and iterative upgrading. In Fig. The most commonly recognized evidence for healthcare interventions is the randomized controlled clinical trial (RCT)18. Surveys and/or interviews are often employed, which are low-cost, efficient, scalable tools to collect attitudes, user experience, and suitability insights. Commonly used methods include usability testing, user-center design, net promoter score survey (e.g. to rate likelihood to recommend a product), online surveys, and log-file data analyses 19. Such approaches have been used to explore user views on the usefulness of digital storytelling20, to assess a web-based network for MS patients21, and to collect attitudes towards digital treatment for depression22. Despite being common, few efforts are turned into peer-reviewed publications19, likely because the main purpose was to generate insights for internal use (e.g. product development) or external customer communication , and can be challenging to pass the peer-review for such work due to its relatively lower evidence strength23.In the early stages of development innovators seek to establish product usability, feasibility, and efficacy26. Typically, an intended user completes tasks and is observed for where they encounter problems. This can be exploratory, to identify new features or functionalities, or comparative testing A vs. B28. Studies are conducted by UX researchers, who synthesize results and translate to actions (e.g. product improvements). Data collected can be qualitative (e.g. observations of problems) and/or quantitative . Evidence strength depends upon study design, for example, task-based and controlled studies that collect quantitative data and can be replicated in other settings/sites, generate stronger evidence, whilst surveys and self-reported behaviors provide weaker evidence, as suggested by UX practitioners29. Controversy exists regarding the appropriate number of participants. Whilst there is no “single correct number”, for formative testing 5 participants is common (“the magic number 5”), compared with 20 participants for summative tests, which offer a tighter confidence interval30.A key approach for digital solution development is usability testing which has been widely utilized to examine whether specified users can achieve intended use effectively and efficiently31. For end-users, not considered “early adopters”, such studies are critical to justify adoption decisions. The randomization unit can be individuals, groups (“clusters”), or even specific solution components32. Choice of the study designs heavily depends on the digital solution and objectives of the evaluation.Prospective RCTs are the most accepted method for evaluating healthcare interventions33) or clinician-level randomization . This is traditionally the most commonly used experimental design in healthcare research 35, however for digital health solutions, we found few studies employed strict individual randomized designs (Table 38). One reason is that individual randomization is not always possible or appropriate as in the examples provided below.Individual-randomization trials (IRTs) are well-suited for digital solutions targeting an individual user, such as patient-level randomization (e.g. symptom self-monitoring39), and this approach has been increasingly adopted by public health researchers42. CRTs are often used in situations when contamination may occur; for example, where individuals in the same cluster have been randomized to different intervention groups, or for logistic, feasibility or ethical reasons43. Attractive features include: increased administrative efficiency; decreased risk of experimental contamination 43; and, enhancement of subject compliance44. In addition, CRTs allow both direct and indirect effects of an intervention to be evaluated—a particular advantage when both effects are hypothesized to be important, e.g., in vaccine field trials45. Disadvantages include: reduced statistical efficiency relative to IRTs46; overmatching; and, subsampling bias48. Analysis commonly employs multi-level modeling50.Cluster-randomization trials (CRTs), by contrast, are better suited for digital solutions supporting group efforts 52, and can be particularly powerful in the early stages of product development51. MRTs generate longitudinal data with repeated measures of participants’ behaviors, context, and psychosocial factors, and can be analyzed by methods, such as multilevel models and generalized estimating equation54.Micro-randomization trials (MRTs) are helpful when researchers want to determine empirically the efficacy of a specific component 17 and supported by our own searches , to allow familiarization and to limit bias related to implementation56; post-phase to collect data on solution effectiveness. Existing studies are often undertaken at a single site (vs. multi-site), which is typically more practical and affordable. Typically, this design requires a longer duration, making it difficult to evaluate continuous solution upgrades (i.e. new features and/or bug fixes), which are often observed in digital health products. In addition, it is not optimal for testing medium-term or longer-term clinical outcomes, because it is difficult to determine independent effects when patients may appear in both pre-phase and post-phase. Data analysis generally employs methods, such as analysis of variance (ANOVA) and analysis of covariance (ANCOVA) or non-parametric tests (depending on the underlying distributions)57.The most commonly used method for evaluating digital health solutions, however, is the pre–post design, as demonstrated by a previous systematic reviewes Table . A stand17 , which can be difficult to achieve. The main advantage is reduction in study duration. Disadvantages include: higher set-up efforts; increased cost; and, challenges to identify matched sites. Various tests are employed such as t-test, non-parametric tests, or other advanced techniques (depending on the underlying distributions)62.Relatively few multi-site studies have been conducted63 than prospective studies because data are already collected, and are commonly used to generate hypotheses for further investigation by prospective studies. The disadvantages are, that they are subject to biases and confounding factors, such as patient information loss or distortion during data collection64, risk factors present but not captured, normal growth or maturation influence, attrition bias (e.g. patients with unfavorable outcome(s) less likely to attend follow-up)65, and selection bias due to non-random assignment of participants66. Such biases threaten internal validity, therefore, retrospective studies are considered (particularly by the academic groups) inferior as compared to RCTs66. It remains as an open question whether this is still the case for digital health solutions, particularly for the ones of lower-risk class.Retrospective studies can be employed to analyze pre-existing data, such as patient charts or electronic medical records. Types of retrospective studies include case series, cohort, or case-control studies. They are typically quicker, cheaper, and easier67 . Nevertheless, one such study from India investigated concordance between the treatment recommendations of an artificial intelligence (AI) algorithm compared with actual tumor board recommendations67 Table . Strictl70. Reviews on a specific solution can provide stronger evidence for its impacts, but require a sufficient number of individual evaluation studies. A possible limitation for such work in digital health is that included studies would need to be matched to the same mechanism of intervention, disease area, and measurable outcome.Systematic reviews have a key role in evidence-based medicine and the development of clinical guidelines73. Frameworks and tools exist to facilitate this including: prediction model risk of bias assessment tool (PROBAST), quality in prognosis studies (QUIPS), revised Cochrane randomized comparative design (ROB), risk of bias in nonrandomized studies of interventions (ROBINS-I). Details provided in Table Systematic reviews of prediction models are a new and evolving area and are increasingly undertaken to systematically identify, appraise, and summarize evidence on the performance of prediction models18) including: cost-effectiveness analysis, cost–benefit analysis, cost-consequence analysis, cost-minimization analysis, etc. However, for all of the aforementioned methods, tracking usage and performance data of users compared to non-users, is required.Demonstration of positive economic benefits are critical for the majority of end-users to justify solution adoption. In addition, such data is important for other critical actors  to endorse the need for change. The World Health Organization (WHO) guidelines provide a good overview of options for economic evaluation . As such, traditional approaches present fundamental limitations for researchers to create evidence for digital health solutions. In fact, evaluation of digital health solutions has been identified as requiring improvement, and has been cited as a major obstacle for wider adoptionDigital solution evaluation requires collective efforts from multiple parties, such as health authorities, healthcare providers , and manufacturers such as small and medium-sized enterprises (SMEs), multinational corporation (MNCs). Whilst they face shared difficulties with the current approaches for evidence generation (e.g. significant time and cost), they also have circumstance-specific challenges.19. As demonstrated in a recent report19, the biggest obstacle for providing evidence of effectiveness reported by companies, is the cost and timeframe for evaluation.SMEs typically prioritize and allocate their research and development budget to product development. Anecdotal evidence suggests that close relationships between innovator and adopter are a critical driver of initial adoption decisions. Wider implementation requires robust evidence of benefit, yet this is difficult to prioritize given the many challenges for establishing new ventures. In addition, well designed and executed studies require skilled researchers, often via collaboration with academia, adding further complexity. Moreover, it has been estimated that the timescale for submitting a research proposal and receiving ethical approval for a pilot or trial study can take as long as 3 yearsLarger corporations have more resources to develop evidence but are equally limited by time. For internal budget allocation, it can be difficult to provide rationale for investments into expensive and time-consuming clinical studies for early-stage solutions when such products are constantly evolving. Given it typically takes 2–3 years to conduct a study, evidence published today may reflect a product that has been updated and refined multiple times. Furthermore, for many companies’ investments in sales and manufacturing, for example, are more tangible with more predictable return on investment than those in clinical studies.The same challenges (as SMEs) exist around navigating the complex infrastructure of the healthcare system, dealing with the cultural resistance to digital solutions, and identifying appropriate principle investigators for the evaluation studies. Despite the long-existing collaborations between large health abd life science companies and principal investigators in, for example clinical trials for drug development, this group of researchers may not necessarily be willing to conduct studies to evaluate digital solutions, as they require different settings, capabilities and also deliver different scientific output—benefits on the operational level impacting cost and indirectly patient outcome versus a drug that can improve patient outcome directly.A growing number of academic centers have created digital health research programs to develop and evaluate digital health solutions. However, such research units generally favor traditional research methodologies because of the increased likelihood of high-impact publication. As such, the timeliness of studies is largely immaterial, therefore, potentially valuable solutions may be delayed and/or are never implemented at scale. Obtaining sufficient research funding can also be a challenge.79). The “gold standard” RCT remains but for later-stage final assessment.In our opinion, large differences exist between the evidence required for initial adopters , and that required for the majority (prospective RCT studies). Other research areas, such as drug development, have demonstrated that pragmatic approaches can be adopted to control cost at early stages . The assumption is that people behave similar to real-life if key components of the scenarios are extracted and fidelity maintained. Various approaches for simulation could be applied to evaluate digital solutions, such as, computational, system, and clinical simulation.80. Verification checks if a system was built according to specification, and validation checks that a system meets user expectations. The most common application of computational has been for verification. Typically, this involves simulated outcomes based on synthesized or real cases, before involving users/clinicians. Recent efforts have extended its use to non-regulated and on-market products , however, to date we are not aware of the use of system simulation to evaluate a digital health solution, perhaps because of the significant complexity to establish models that represent a healthcare system.System simulation adopts a system engineering view and methodology to model the effect of an intervention on a healthcare system  without disrupting the real health care setting84. In our opinion, can be complementary to many of the traditional approaches reviewed above that require the use of a digital solution in real clinical practice, and could bridge the evidence needs between those of “early adopters” and the “majority”. Clinical simulation provides a good balance between the strength of evidence , whilst remaining cost-effective and timely for fast version updates , feasibility in being implemented as remote and/or distributed87, and ability to collect behavioral and/or cognitive metrics. Sophisticated approaches and equipment can be employed, such as eye-tracker analysis or measurement of EEG, which would not be possible in real clinical practice. Furthermore, clinical simulation may also be helpful in facilitating patient engagement and/or Patient and Public Involvement and Engagement (PPIE), an initiative aiming to involving patients and/or representatives from relevant public bodies in the research88.Clinical simulation was traditionally developed and used in training medical residents, and it was further developed as an approach to test systems and digital solutions with representative users doing representative tasks, in representative settings/environmentstes Fig. . Previou89. For example, in one study90 primary care physicians interacted with videoclips of professional patient actors providing standardized responses to clinical scenarios and utilized a CDS tool of clinical prediction rules via an EMR system. In another recently published study91, cognitive load and performance of physicians was evaluated for different conditions by randomly assigning participants to baseline EHR (control) or enhanced EHR (simulated environment with features such as automatic sorting and decision support instructions). Moreover, a recent interview study of 10+ companies reported that they found this approach feasible for evidence generation for their own digital solution19.Clinical simulation has been increasingly used in evaluating digital health solutions, including five studies in Table 84; and, The Institute of Global Health Innovation (IGHI) 92. Indeed, researchers from IGHI have established a simulation test bed specifically to explore application to test digital health solutions. Initial work evaluated the impact of a digital solution on the conduction of cancer multidisciplinary team (MDT) meetings. 56 healthcare professionals , who were regular participants at lung cancer tumor boards, were recruited to take 10 simulated MDT sessions. High-fidelity mock patient cases were developed by the study team and clinical experts93. Participants discussed up to 10 patient cases, using a standard UK approach to conduct MDTs (paper handout and PACS system) in the control condition, compared with the NAVIFY Tumor Board solution. A manuscript detailing the learnings and results from this pioneer work is under development.Several academic centers have established clinical simulation test environments, including: The School of Health Information Science (University of Victoria); The Department of Development and Planning ; The IT Experimentarium (ITX) lab 96; and can demonstrate solution utility .Whilst clinical simulation offers opportunities to prospectively test a digital solution quickly, safely and cost-effectively prior to implementation, there are a few limitations in its use. First, high-fidelity is a prerequisite for generating valid and effective evidence. Therefore, researchers should take efforts to create scenarios representing real clinical practice, recruit the most representative end-users as participants, and provide comprehensive trainings of the digital solutions to the participants before their simulation sessions. Second, while the regulatory space evolves fast, we think clinical simulation results itself alone probably are not adequate for approval application from Health authorities, particularly for higher-risk group of digital solutions that would need to be approved as SaMD. Nevertheless, in these cases, clinical simulations can help to provide initial insights for product development, reduce safety risk for patients, and guide the design of large-scale real clinical studies. Third, for digital solutions that are already adopted in clinical practice, leveraging real-word data (RWD) is probably more suitable. RWD studies could be systematically employed to undertake near real-time evaluation during pilot implementation and post-market monitoring. Indeed, studies utilizing real-world data (RWD) have been encouraged to support regulatory decision making (e.g. The 21st Century Cures Act; Table Finally, we believe clinical simulation can be employed in combination with traditional study designs, e.g., individual-randomization, cluster-level randomization, and micro-randomization to examine different types of digital solutions. For example, clinical simulation-based study with micro-randomization design can be a powerful and pragmatic approach to evaluate the digital solutions with multiple components at early stage of the product development.Innovators face significant challenges to overcome the “no evidence, no implementation—no implementation, no evidence” paradox in digital health. We believe that innovative approaches, such as simulation-based research, can enable the generation of higher-quality, lower-cost, and more timely evidence. By considering such methods, end-users will encourage developers to undertake research activities, rather than be intimidated by the complexity, cost, and duration of traditional approaches.