According to the Joint Monitoring Programme (JMP), an estimated 844 million people do not use improved water sources and 2.3 billion lack access to even a basic sanitation service . Worldwide, 892 million people still practice open defecation. Rural, poor and, vulnerable households have particularly limited access to adequate facilities and inequities are often regionally focused. Populations in sub‐Saharan Africa and Oceania are lagging behind in access to improved drinking water sources, whilst South Asia and sub‐Saharan Africa have the highest concentrations of open defecation.Limited, or no, access to safe facilities for eliminating human waste, gathering clean drinking water, or practicing hygienic washing and food preparation practices exposes individuals to higher‐levels of contagious pathogens. There is evidence to suggest that poor water, sanitation, and hygiene (WASH) conditions are associated with high levels of diarrhoeal disease , parasitInadequate access affects disadvantaged groups disproportionately, but women and girls are particularly badly affected by the costs of having limited access to WASH facilities. They often carry the majority of the burden associated with collecting water , and can be placed in high‐risk situations when using unsafe places to defecate . Women aIn 2015, more than 150 world leaders adopted the new 2030 Agenda for Sustainable Development, which sets new goals for 2030 that build upon, and go even further, than the Millennium Development Goals (MDGs). Sustainable Development Goal (SDG) 6 aims to ‘ensure the availability and sustainable management of water and sanitation for all’ by 2030 . In order to help achieve these universal targets, which includes reaching the most disadvantaged populations, decision makers need access to high quality evidence on what works in WASH promotion in different contexts, and for different groups of people. Both impact evaluations and evidence syntheses can be useful to decision makers. Single impact studies are useful for providing information on how a programme functions in a specific context; for example, the recent WASH‐Benefits trials were unable to detect effects of combined or single water, sanitation, or hygiene interventions on child linear growth in Bangladesh and Kenya . HoweverFor policymakers, practitioners and commissioners of research to make informed decisions, they need to be able to identify where high quality evidence exists in usable formats, and where more evidence is needed. There are also concerns about approaches used to measure outcomes in WASH sector primary research, such as self‐ and carer‐reporting of diarrhoeal disease . These concerns necessitate, firstly, examining the critically appraised evidence (from systematic reviews) and, secondly, evidence on a wide range of behavioural, health and socio‐economic outcomes.What remains an issue, therefore, is the extent of evidence on the effectiveness of interventions to improve access to WASH services for households, communities, schools and health facilities on outcomes in the round, and an assessment of what primary and synthesised evidence is still needed across different low‐ and middle‐income countries and regions.Water, sanitation and hygiene (WASH) interventions have two important components to them – the ‘what’ and the ‘how’. The ‘what’ describes the technology that the participants end up with  and the ‘how’ describes the mechanism of the intervention . Prior to the early‐2000s, the focus of the conversation was principally on ‘what’ works; research was centred on understanding and demonstrating the short and long term consequences of providing a technology. However, over the last 15 years the conversation has increasingly switched from not just what technology to provide but what is the best way to both get it into the community and have it be regularly used. This has seen the rise of behaviour change and systems‐based approaches. Due to this changing focus, the principal interventions will be defined by the mechanisms (the ‘how’); this means that the evidence gap map will present intervention mechanisms against outcomes. There will then be a filter for the technology provided by the intervention; this will allow for easy comparison of the evidence for different mechanisms of providing, for example, latrines.Mechanisms for providing WASH technologies can be classified into four main groups; direct provision, health messaging, psychosocial ‘triggering’, and systems‐based interventions. The below definitions have been adapted from relevant literature in the field :Direct provision mechanisms cover all interventions where hardware (such as a latrine or water purifier) is provided for free and has been chosen by an external authority .Health messaging, most often focused on sanitation or hygiene, is typically a directive educational approach designed to help individuals, or communities, improve their health through increasing their knowledge and/or skills.Psychosocial ‘triggering’ falls into two subcategories of directive and participative approaches. Both subcategories use behavioural factors which have been derived from psychosocial theories  to motivate behaviour change, rather than reason. An example of this approach is community‐led total sanitation (CLTS) where the community is encouraged to discuss how they would like sanitation practices to change, identify problem areas , and use social cohesion and pressure to motivate people to construct latrines and stop practicing open defecation .Systems‐based mechanisms try to change people's behaviour by changing the wider system around them. These approaches include pricing reform, improving operator performance, private sector (PS) and small‐scale independent provider (SSIP) participation, and community driven development (CDD).The behavioural change communication (BCC) approaches – health messaging and psychosocial ‘triggering’ – are often combined with both direct provision and systems‐based approaches in an attempt to simultaneously overcome the social and financial barriers to accessing appropriate WASH services.WASH technologies for household and personal consumption can be classified into four main, related, groups: water quantity, water quality, sanitation hardware and sanitation software (hygiene) :Water quantity technologies provide a water supply or distribution system. Water may be supplied to communities at source, such as through a public standpipe, or at point‐of‐use (POU), such as being piped directly to households.Water quality technologies provide the means to protect water from, or treat water to remove, microbial contaminants. Examples of water treatment technologies include filtration, chlorination, flocculation, solar disinfection, boiling, and pasteurising. Water quality improvements are most commonly undertaken in the household, rather than at the source, though this class of interventions also includes treatment at source and provision of containers for safe transportation and storage of water.Sanitation technologies provide means to dispose of excreta (such as faeces), through new or improved latrines or connection of existing latrines to the public sewer.Hygiene technologies consist of hygienic practices, and facilitators of these such as soap, hand sanitisers, and washing stations. Hygiene practices are most often focused on handwashing but can also include food hygiene, such as proper food storage and washing dishes appropriately, as well as wearing appropriate footwear, or menstrual hygiene management.A third important dimension to any intervention is how, or where, participants interact with it in terms of both their social and physical environments. Interventions that seem similar can be very different in nature, and their outcomes not necessarily comparable, due to the space they inhabit. An ecological model can be integrated into the types of technology to reflect where a technology is used. The place of use is important in the WASH sector as it affects the convenience to users, and therefore adoption rates, as well as how the intervention disrupts the causal chain of disease transmission. The four main spaces in which WASH technologies are provided are in the home , in the community (to be shared), at a school, and at a health facility.Multiple mechanisms can be used in one programme; for example, soap could be directly provided with a social marketing campaign on handwashing. Multiple WASH technologies are also often be provided together in programmes where they are combined. A common example is combined water supply and sanitation (WSS) programmes.The quality of water supply, sanitation and hygiene facilities – that is, the extent to which they are likely to provide potable drinking water or safe removal of excrement from the human environment, or enable hygienic hand‐washing – is dependent on the type of water or sanitation facility. The conceptual framework links WASH interventions with impacts along the causal chain . Sector The links in the chain are not automatic. For example, in the particular case of water quality, faecal contamination of drinking water between source and point‐of‐use (POU) means that hygienic approaches may be needed to store clean water collected at source, or treat water for contaminants in the household (POU). Better access to water supply (quantity) may improve health by reducing contamination in the environment by enabling better personal hygiene (e.g. handwashing) and environmental hygiene . Factors such as environmental faecal contamination may prevent impacts from clean drinking water provision being realised. Sustainability of impacts requires continued (permanent) adoption and acceptance by beneficiaries as well as appropriate solutions to reduce ‘slippage’ in improved behaviour and financial barriers to uptake and technical solutions to ensure service delivery reliability. Scalability requires that impacts which are demonstrated under ‘ideal settings’ of trials are achievable in the context of ‘real world’ programme implementation, where beneficiaries may not constantly be reminded to use technologies appropriately.Progress towards the Millennium Development Goals (MDGs) was uneven in the sector. The MDG drinking water target to “halve the number without access to safe drinking water (defined as access to water from an improved source within one kilometre of the household)” was declared met in 2012, but of those who did gain improved access to drinking water since 1990, supplies are mainly provided at the community level and are often unreliable . The MDG sanitation target to “halve the number without access to sanitation by 2015” was missed .The Sustainable Development Goals (SDGs) are aspirational, aiming for universal coverage by 2030, and adding targets for hygiene.To provide safe and affordable drinking water for all, measured by population using safely managed drinking water that is an improved drinking water source, located on premises, available when needed and free from contamination (SDG 6.1).To end open defaecation and provide adequate and equitable sanitation for all, measured by population using safely managed sanitation services and a basic handwashing facility with soap and water (SDG 6.2). Safely managed sanitation is defined as an improved facility where excreta is treated and disposed of in situ or off‐site.To ensure all men and women have access to basic services, including basic drinking water, sanitation and hygiene (SDG 1.4).In order to move towards these ambitious targets, it is likely that substantial improvements in resource allocation will be needed to promote interventions which are effective in improving behaviours and outcomes in particular contexts. The purpose of this evidence gap map is to assist policy‐makers and practitioners in gaining access to evidence on the effectiveness of WASH interventions.In 2014, 3ie produced an evidence gap map (EGM) on the effectiveness of WASH interventions in improving quality of life outcomes. That map includes evidence until February 2014 and only considered quality of life outcomes  as primary outcomes. Behaviour change outcomes were included as secondary outcomes, provided the study also included primary outcomes. In addition, the map excluded interventions in health facilities. This update aims to capture studies conducted since, as well as broadening the included interventions and outcomes to better reflect the state of evidence on WASH in 2018.In 2014, 3ie produced an evidence gap map for household and community interventions for promoting water, sanitation, and hygiene consumption in LMICs.The overarching aim of the evidence map is to gather and present the rigorous empirical research on the effectiveness of interventions to improve consumption of water, sanitation and hygiene in the household, communities, schools and health facilities. This protocol provides the project plan for an update to the 2014 WASH evidence gap map (EGM) to take stock of the existing evidence, and capture newly published work, on the effects of interventions in these areas.The aim of the EGM is to identify, map, and describe existing evidence on the effects of interventions to improve access to, and quality of, WASH infrastructure, services, and practices in low‐ and middle‐income countries. This update of the original map aims to capture additional studies conducted in the last three years and extend the scope of the EGM, in particular to cover behavioural outcomes and WASH interventions at healthcare facilities. The primary outcomes for this gap map include morbidities (e.g. diarrhoea), mortality, psychosocial health, nutritional status, education, income, and time use. In addition, behavioural outcomes will also be included as primary outcomes, such as water treatment practices, hygiene behaviour, and latrine construction in CLTS.The update of the EGM addresses three objectives:(1) To identify existing evidence from high quality impact evaluations and systematic reviews (SRs), particularly those published since 2014, which can be used to inform policy.(2) To expand the scope of the EGM to better capture WASH behaviour change and programmes implemented at healthcare facilities, with the aim of improving the map's policy relevance.(3) To identify existing gaps in evidence where new primary studies and systematic reviews could add value.The results from this EGM aim to inform the direction of future research surrounding WASH, and discussions based on systematic evidence about which approaches and interventions are most effective in the WASH sector, whether they are used in small scale projects or large scaled‐up programmes.Evidence gap maps aim to establish what we know, and do not know, about the effectiveness of interventions in a thematic area .The framework for this evidence map  is basedThe finalised updated evidence map will be structured around a framework of policy relevant WASH mechanisms and outcomes, with a filter for technologies, and will be available online at 3ie's evidence gap map portal.The evidence map will highlight the best available evidence from systematic reviews and provide access to user‐friendly summaries and appraisals of those studies.The evidence map will also show where completed and, through the inclusion of trial registries, on‐going primary studies  have been conducted.The evidence map will highlight absolute gaps in evidence (lack of studies for particular interventions and/or outcomes).The evidence map will highlight synthesis gaps where there are sufficient studies for a new systematic review or an update of an existing systematic review.The evidence map will have filters to highlight the evidence in particular countries and regions, targeted at particular populations, using specific methodologies, and specific intervention approaches.We will include any study populations regardless of age, sex or socio‐economic status. However, populations are restricted to those in low‐ and middle‐income countries (LMICs), as defined by the World Bank, at the time the research was carried out.Water, sanitation, and hygiene can be categorised into groups and sub‐groups of related intervention mechanisms as shown in We then propose that the interventions should be further classified into groups and sub‐groups of related WASH technologies as shown in As mentioned before, we will also be integrating an ecological model focused on where a technology is physically used. Ecological models stress the importance of the dynamic relationships between the personal and environmental factors that shape an individual's behaviours and lifelong human development. They, and their socio‐ecological counterparts, are often applied in crime prevention  Health impacts including, but not necessarily limited to:a. diarrhoeal diseaseb. acute respiratory infections (ARIs)c. other water related infections such as helminthsd. pain and musculoskeletal disorderse. psychosocial health and safetyf. reproductive health outcomesg. mortality.(2) Nutritional impacts including, but may not be limited to:a. measures of stunting b. wasting c. underweight .(3) Social and economic impacts, for example:a. educational outcomes (e.g. absenteeism)b. time usec. labour market outcomes (e.g. employment and wage)d. measures of income, consumption, and income poverty.We expect most studies will focus on outcomes among children but would not exclude studies that only report outcomes for adults.(4) We will also include studies even if they only report on the following types of behavioural and attitudinal outcomes:a. water quantity used/consumedb. water treatment practicesc. latrine use or defaecation practices (including construction of facilities for ‘triggering’ interventions)d. hygienic behaviour e. willingness to pay.We will exclude studies that only report measures of knowledge and attitudes; for example, a hygiene education programme that reports the proportion that know that bacteria can cause infections would be excluded.Any adverse or unintended outcomes found to be reported, but not captured in the above list, will be included.This evidence gap map will include impact evaluations and systematic reviews of the effectiveness of technologies and intervention mechanisms. Impact evaluations are defined as programme evaluations or field experiments that use quantitative approaches applied to experimental or observational data to measure the effect of a programme relative to a counterfactual representing what would have happened to the same group in absence of the programme. Impact evaluations may also test different programme designs. We will include both completed and on‐going impact evaluations and systematic reviews; to capture the latter, we will include prospective study records in trial registries or protocols when available. We will include a broad range of intervention study designs in order for the map to provide a comprehensive look at the evidence provided by researchers working in the sector in different disciplines . We include randomised and non‐randomised controlled studies. We also include methods such as natural experiments which may provide ‘as‐if randomised’ evidence when well conducted . We allow broader inclusion criteria for particular outcomes, such as case‐control studies in the case of mortality, which may not be ethically collected in trials, and uncontrolled before versus after for time‐use outcomes, which are crucially important for water collectors and which arguably do not require controls .Study design criteria for includable intervention studies are as follows:a) Prospective studies allocating the participants to the intervention using randomised or quasi‐randomised mechanisms at individual or cluster levels.a. Randomised control trial (RCT) with assignment at individual or cluster level b. Quasi‐RCT using a quasi‐random method of prospective assignment b) Non‐randomised designs with selection on unobservables:a. Natural experiments using methods such as regression discontinuity (RD)b. Panel data or pseudo‐panels with analysis to account for time‐invariant unobservables c. Cross‐sectional studies using multi‐stage or multivariate approaches to account for unobservables c) Non‐randomised designs with selection on observables:a. Cross‐sectional or panel (controlled before and after) studies with an intervention and comparison group using methods to match individuals and groups statistically (e.g. PSM) or control for observable confounding in adjusted regression.d) The following impact evaluation study designs will only be included in the specific circumstances described.a. Reflexive controls (pre‐test/post‐test with no control/comparison group) will be included for studies reporting time use outcomes.b. Case‐control and cross‐sectional exposure designs will be included for studies conducted at healthcare facilities measuring mortality.e) Studies explicitly described as systematic reviews and that describe methods used for search, data collection, and synthesis.We will include impact evaluations where the comparison/control group receive no intervention (standard WASH access), a different WASH intervention, a placebo (e.g. school books) or the study employs a pipeline (wait‐list) approach.We do not plan to include qualitative research.All included impact evaluations must have been conducted in low‐ and middle‐income countries (LMICs) as defined by the World Bank at the time of the intervention. We will also exclude systematic reviews only containing evidence from high‐income countries. We will include studies in challenging circumstances such as refugee camps, but exclude studies which are conducted under outbreak conditions, such as epidemics of cholera (‘extremely watery diarrhoea’) as this map aims to describe the evidence on what works under normal conditions.As we are focusing on personal and household WASH interventions, we will exclude studies that look at WASH interventions in agriculture, commercial food preparation, and ones that focus on animal excreta. We will, however, include WASH interventions at medical facilities if they meet the above intervention definitions. Studies on medicalised hygiene (such as sterilising wounds) will be excluded.We will search for and include completed and on‐going studies. We will not exclude any studies based on language or publication status or publication date.We will automatically include all studies that were included in the 2014 evidence gap map for which thorough searches were conducted for both published and ‘grey’ literature. Therefore, the search strategy will cover two main components: updating the searches already conducted from February 2014 onwards, and conducting new electronic searches for the expanded scope from 2000 onwards. We will use the following strategies to identify completed and on‐going new potential studies:1) Database and trial registry searches: We will search MEDLINE(R) (Ovid), Embase (Ovid), CAB Global Health (Ovid), CAB Abstracts (Ovid), Cochrane Library, ERIC (Proquest), Social Sciences Citation Index (Web of Science), Social Sciences Premium Collection (Proquest), Popline, WHO Global Health Library, Econlit (Ovid), Ebsco Discovery, and Campbell Library.2) Organisation searches: We will search for literature using 3ie's impact evaluation database and through the online repositories of organisations who are known to produce impact evaluations and systematic reviews of WASH interventions. These include the Asian Development Bank, African Development Bank, Inter‐American Development Bank, Department for International Development, IMPROVE International, International Water and Sanitation Centre (IRC), Oxfam, UNICEF, US Agency for International Development, WaterAid, the World Bank  .3) Bibliographic searches: Several recent systematic reviews  are relevant to topics in our expanded scope and we will screen these systematic reviews to locate additional primary studies.4) We will also conduct bibliographic back‐referencing of reference lists of all included systematic reviews to identify additional primary studies and systematic reviews.We will use EPPI reviewer to assess studies for inclusion at both the title / abstract and full‐text screening stages. Due to time and resource constraints, at the title / abstract stage, we will use EPPI reviewer's machine learning capabilities to prioritise studies in order of likelihood of inclusion. We will screen until we are no longer finding any studies to include (at least 50 studies with 0 includes). Two researchers will screen each title / abstract and each full‐text. Any disagreements on inclusion will be resolved through discussion.For impact evaluations, we will use a standardised data extraction form to extract descriptive data from all studies meeting our inclusion criteria. Data extracted from each study will include bibliographic details, intervention types and descriptions, outcome types and descriptions, study design, context / geographical information, details on the comparison group, and on the quality of the implementation. We will also extract data on the sex disaggregation of outcomes.A full list of data to be extracted is described in the coding tool in For systematic reviews, a modified version of the tool will be developed for the data extraction. All systematic reviews for inclusion will undergo a critical appraisal following the 3ie systematic review database protocol for appraisal of systematic reviews . Critical appraisals will be completed by one experienced researcher.We will critically appraise included systematic reviews according to the 3ie tool  which draws on We will not be critically appraising the quality of the included impact evaluations, but will collect data on study design. For the purpose of the present map it is not necessary to critically appraise the impact evaluations, beyond indicating whether the evidence is from randomised, natural experiment, or non‐randomised studies, as the systematic reviews provide overviews of the body of evidence, including their quality, where they exist. A major purpose the map is to provide access to the body of work on particular outcomes and interventions to encourage further syntheses of those studies by WASH sector researchers.Where multiple papers exist on the same study (e.g. a working paper and a published version), the most recent open access version will be included in the evidence map. If the versions report on different outcomes, an older version will be included for the outcomes not covered in later versions.The matrix and filters are described above and in The matrix and filters are described above and in We have engaged stakeholders on the evidence matrix at various organisations who provide WaSH sector policy and programmes. These include Aga Khan Foundation, the Independent Evaluation Group of the World Bank, Japan International Cooperation Agency, Sanitation and Hygiene Applied Research for Equity (SHARE) consortium, the Water Supply and Sanitation Collaborative Council, and WaterAid.Lead EGM author: The lead author is the person who develops and co‐ordinates the EGM team, discusses and assigns roles for individual members of the team, liaises with the editorial base and takes responsibility for the on‐going updates of the EGM.Content: Sandy Cairncross has substantial expertise in water, sanitation and hygiene interventions for the control of disease. Hugh Waddington led a previous review of WASH impacts.Systematic review methods: Sandy Cairncross has led several major systematic reviews of WASH interventions. Hugh Waddington led a previous systematic review and has supported a large number of systematic reviews and meta‐analyses for 3ie and Campbell.Information retrieval: John Eyers has substantial expertise in devising and running systematic searches for published and unpublished literature, including for reviews of WASH interventions.We thank JICA and the Water Supply and Sanitation Collaborative Council (WSSCC) for financial support. Ritsuko Yamagata, Jeff Tanner, Midori Makino, Ryotaro Hayashi and Marie Gaarder provided helpful comments.Sandy Cairncross has been involved in the development of sanitation and hygiene interventions and he and Hugh Waddington have authored primary studies and systematic reviews that may be eligible for inclusion in the evidence map. Inclusion decisions and critical appraisal of any studies these two authors have been involved in will be undertaken by other members of the team. We are not aware of any other conflicts that might affect decisions taken in the review and results presented.Approximate date for submission of the EGM: July 2018.This map is itself an update of a EGM published online in 2014. We plan to update the map (or support others in doing so) when sufficient further studies and resources become available.


Public health policies and recommendations aim to be informed by the best available evidence. Evidence underpinning e-cigarettes policy recommendations has been necessarily limited due to the novelty of the technology and the lack of long-term epidemiological studies and trials. Some public health bodies have actively encouraged e-cigarette use whilst others have raised concerns over introducing new health risks and renormalising tobacco smoking. Using citation network analysis we investigated the author conflicts of interest and study funding statements within sources of evidence used by public health bodies when making recommendations about e-cigarette policy.We conducted citation network analysis of public health recommendation documents across four purposively selected diverse jurisdictions: WHO, UK, Australia, and USA. We extracted all citations from 15 public health recommendation documents, with more detailed data collected for influential citations (used in 3+ recommendation documents). We analysed the relationships between the sources of evidence used across jurisdictions using block modelling to determine if similar groups of documents were used across different jurisdictions. We assessed the frequency and nature of conflicts of interest.1700 unique citations were included across the 15 public health recommendation documents, with zero to 923 citations per document . The evidence base underpinning public health recommendations did not systematically differ across jurisdictions. Of the 1700 citations included, the majority were journal articles (n = 1179). Across 1081 journal articles published between 1998–2018, 200 declared a conflict of interest, 288 contained no mention of conflicts of interest, and 593 declared none. Conflicts of interest were reported with tobacco , e-cigarette , and pharmaceutical companies , with such conflicts present even in the most recent years. There were 53 influential citations, the most common study type was basic science research without human subjects (e.g. examination of aerosols and e-liquids) (n = 18) followed by systematic review (n = 10); with randomised control trial being least common (n = 4). Network analysis identified clusters of highly-cited articles with a higher prevalence of conflicts of interest.Public health bodies across different jurisdictions drew upon similar sources of evidence, despite articulating different policy approaches to e-cigarettes. The evidence drawn upon, including the most influential evidence, contained substantial conflicts of interest (including relationships with e-cigarette and tobacco industries). Processes to explicitly manage conflicts of interest arising from the underlying evidence base may be required when developing public health recommendations. Public health policies and recommendations aim to be informed by the best available evidence , 2. The Electronic cigarettes  are an important case study to understand how evidence informs public health recommendation documents, given the rapid development of the evidence base. Numerous public health bodies across the world have released reports, recommendation documents, and statements on their position on e-cigarettes. A range of policy/regulatory and public health approaches towards e-cigarettes products has been pursued across the globe from being completely prohibited to being regulated as consumer products, tobacco products, or medicinal devices , 7. DiffThe potential for conflicts of interest (COI) is present in all areas of public health. It occurs when “a set of circumstances that creates a risk that an individual’s professional judgement or actions regarding a primary interest  is, or could be, impaired or influenced by a secondary interest ” . COI mayThis study investigates the sources of evidence in relation to the types of evidence used by public health bodies across four diverse jurisdictions when making e-cigarette policy recommendations.Further, it examines the author conflicts of interest and study funding statements within these sources, to deepen our understanding of the diffusion of industry funded and industry-supported evidence in public health recommendation documents.We purposively selected four different influential jurisdictions, considered important for setting the agenda on policy recommendation for e-cigarettes policy: WHO, UK , Australia, and USA. These jurisdictions were selected to reflect the different approaches towards e-cigarette regulation . The UK Sub-national level bodies within the UK were included in the sample to investigate the diversity within a jurisdiction. The UK has four public health systems and they correspond to its four different political systems. Scotland, Wales, and Northern Ireland each have an autonomous legislature that makes health policy while the UK Government directly runs England’s National Health Service (NHS) , 20. ThiWithin each of the chosen contexts, we identified public health bodies that had produced public health recommendation documents, position papers, or policy statements on e-cigarettes that included recommendations for health practice, public health and/or health policy. A ‘public health body’ was defined as an organisation whose aims stated, or whose role within local/national/international policy is to protect and improve the health of a population. Several public health bodies had been identified during the literature review stage of the research and through correspondence with experts in the field. Additional public health bodies were identified using online searching. The online search for the public health bodies and recommendation documents was conducted between July and August 2019. As the literature surrounding e-cigarettes is continuously evolving, another online search was conducted in December 2019 to ensure no documents had been missed from the sample. Websites of public health bodies were searched for any publicly available documents using the key terms “e-cigarettes”, “electronic cigarettes”, “e-liquids”, and “tobacco”. Citation lists within the identified documents were examined for additional relevant recommendation documents, position, or policy statements. The criteria for sample inclusion is shown in Through snowballing from websites, policy documents, and personal contacts, a list of relevant experts within each jurisdiction was compiled. These experts were emailed with a list of the documents making up the sample and asked to provide details of any recommendation documents, positions, or policy statement documents they believed to be influential that were not included in the original sample. The search strategy is presented in A total of 15 documents across 10 public health bodies  met the inclusion criteria for further analysis.Citation analysis measures the importance or impact of an author, an article, or a publication by counting times cited in other works, and network analysis can be used to study patterns of connections between documents, where a citation is considered a link between documents in the network . We extrWe assessed COI statements in the publications and checked supplementary material (such as ICMJE forms) when referenced and available. If an author did not provide a COI statement in the manuscript and did not refer to supplementary material elsewhere, this was classified as ‘no mention’ of COI. We constructed three further graphs of high-impact citations, colouring by the number of times cited, type of publication, and type of COI. Publication types were classed as classed basic science research without human subjects (e.g. examination of aerosols and e-liquids), systematic review (SR), non-systematic review, longitudinal observational study, cross-sectional and randomised control trial (RCT).We conducted in-depth analysis of the 53 influential citations to determine if the interpretation of the citations varied across recommendation documents. This was done by examining the surrounding text of when the reference was cited within the recommendation document. Where COI were declared in a cited document, we recorded whether this was assessed in the recommendation document. The primary coder was MS and a random sample of 20% was double coded by KS.To retrieve information and categorise each of the citations, we built a Shiny interactive web app . Shiny iTo analyse whether recommendation documents drew upon similar evidence, we used bipartite stochastic block modelling to detect clustering within the citation network. The method aims to detect if there are groups of documents that are similar based on their connections to other documents in the citation network. We examined which recommendation documents drew upon similar sources of evidence and created groups of recommendation documents by strength of connection. The clustering of evidence sources was determined by their co-occuring citations in recommendation documents. We fitted a series of block models, with between 1 and 10 blocks (referred to as ‘groups’ for the remainder of the article) of recommendation documents, and between 1 and 15 blocks (referred to as ‘clusters’) of evidence sources . We usedA total of 1700 unique citations were included across the 15 public health recommendation documents, with zero to 923 citations per recommendation document  . The NHSThe year of publication of the 1700 citations ranged from 1947–2019 . Citations were categorised by type; the most common was journal article (n = 1189), followed by Government/official report (n = 267). E-cigarette company press releases (n = 3) and commentaries on papers (n = 1) and were the least common. Please see The network graph in As shown in The most common study type was non-epidemiology followed by systematic review (SR); with randomised control trial (RCT) being the least common.et al. [Out of the 53 influential documents, 13 citations declared a COI, 14 made no explicit mention of COI, and 26 declared none . See S4 et al. , PHE 201et al. , p. 183 et al. , p.9.In addition, we analysed the 13 citations that declared a COI and found that the presence of COI was not explicitly taken into account when it was presented as evidence in any of the recommendation documents. The NHMRC document highlighted the importance of considering COI of authors when reviewing the evidence base.Blockmodelling was fitted to the network of citations included in more than one recommendation document, comprising 192 evidence sources (11% of 1700 citations). The best-fitting model contained four groups of recommendation documents (Groups 1–4) and five clusters of cited references (Clusters 5–9) . The logEach black rectangle in Investigating the distribution of COI statements across recommendation groups, we saw clear differences in proportions of evidence sources declaring or not declaring the presence or absence of COI . Groups Of the 1179 cited journal articles 44 texts were unavailable and therefore were excluded from the analysis. Between 1965–1997 the majority of articles cited had no mention of COI, therefore, more detailed analyses of distributions of COI were restricted to publications post-1998. This demonstrates the reporting of COI has improved over time, as shown in Out of the 1081 journal articles published between 1998–2018, 288 contained no mention of COI, 593 declared no COI, and 261 declared COI. The reporting of COI has substantially improved over time, as shown in Public health bodies across four jurisdictions vary in their approach to citing evidence to justify their recommendations, with some citing numerous sources whereas others not indicating the evidence used to develop their recommendations. There was some overlap in the sources of evidence drawn upon by public health bodies when making e-cigarette recommendations. However, this evidence was used to articulate different policy approaches; the UK adopting a ‘harm reduction’ approach and WHO, Australia, and USA adopting a ‘precautionary’ approach. The majority of the evidence cited was not shared across the recommendation documents, with relatively few influential citations.Our analysis demonstrated that evidence influencing public health recommendation documents stems from research where COI are not declared or where important conflicts exist. A substantial proportion of cited evidence contained pharmaceutical (12%), e-cigarette (7%) or even tobacco (3%) COI, including amongst the most influential research featuring across multiple recommendation documents. While reporting of COI has substantially improved over time, there is still a substantial proportion of articles that do not explicitly report potential COI. The presence of COI associated with e-cigarette companies coincides with the introduction of e-cigarettes into the European and US market in 2006 and 2007, respectively. Several journals will no longer accept submissions of articles that have ties with the tobacco industry , 33. Theet al. and Fabbri et al. argue that disclosure of COI and funding should be mandatory [Several studies have investigated the relationship between industry COI and/or funding and research outcomes , 34–36. andatory . Our stuet al. and Rasmussen et al. showed that COI are often not declared [et al. found that almost half of all authors had undisclosed COI in clinical trials [Our study has several strengths. We systematically identified e-cigarette recommendation documents from four purposefully selected jurisdictions. We carried out a detailed investigation of the citations included in the 15 recommendation documents . The use of citation network analysis to investigate and illustrate the sources of evidence drawn upon by public health bodies when making recommendations is a relatively novel method that highlights the inter-relationships between the evidence used by different public health bodies. However, some limitations should be noted. First, we examined only citations in relation to the types of research and COI rather than the quality of evidence and broader forms of evidence use. Second, it is highly likely that we are underestimating the presence of COI as we are reliant on what has been declared within each article and how COI are interpreted. Bindslev declared , 38. ResOur study highlights several areas of research that contribute to understanding the sources of evidence used in public health recommendations. There is a need to better understand the process used by different public health bodies when creating recommendations and how recommendation committees handle evidence where vested interests exist. To address this, an investigation of the views of those involved in the development of public health recommendations to explore the development process, the role of evidence, how COI are managed during the development process, and how contextual factors influence the development of recommendations. This would help to deepen our understanding of the development process and the role of evidence in public health recommendations. Our study was not able to determine and understand why different public health organisations have pursued different policy approaches based on the evidence. Further, more detailed analysis involving policy stakeholders may be required to understand this. The variation in the number of citations per recommendation document  cited 11 or fewer citations from the total 1700 citations) impacted the visualisation of the citation network. The aim of this study was to investigate the sources of evidence used by public health bodies when making e-cigarette policy recommendations and COI within these sources rather than to investigate the quantity of citations included within a recommendation document. Future research could usefully explore how citations reflect the development of e-cigarette recommendations.Public health recommendations aspire to be evidence-informed. Our study shows that the evidence relied upon when developing policy recommendations is subject to COI. The presence of COI could threaten the validity of the evidence base, therefore, shaping subsequent policy recommendations resulting in inappropriate public health actions. Using e-cigarettes as a case study we have demonstrated the need for robust methods to manage evidence derived from industry funding or incorporating industry COI within public health recommendations. These COI extend to even the most concerning industries, such as tobacco, and an urgent debate is needed about whether such evidence should inform public health policy.S1 Appendix(DOCX)Click here for additional data file.S2 Appendix(DOCX)Click here for additional data file.S3 AppendixDistribution of the number of journal articles published between 1998–2018.(DOCX)Click here for additional data file.S1 Table(DOCX)Click here for additional data file.S2 Table(DOCX)Click here for additional data file.S3 Table(DOCX)Click here for additional data file.S4 Table(DOCX)Click here for additional data file.S5 Table(DOCX)Click here for additional data file.S6 Table(DOCX)Click here for additional data file.


In our manuscript we acknowledged that the MR analysis suffers from relatively low power influenced both by the power of exposure in the outcome GWAS, and “winners’ curse” in the exposure GWAS. We concluded that we find limited statistical evidence for vitamin K1’s role in COVID-19 (but do not claim that this lack of evidence is evidence of no effect): the point estimates in the main method (IVW) point to slightly reduced very severe COVID and hospitalizations  being associated with higher genetically predicted circulating vitamin K1 levels.In their correspondence arising from our recent manuscript , JanssenHowever, previous MR investigations using the same circulating phylloquinone GWAS for genetic instrument selection and effect-size estimates did uncover strong evidence for a modest causal effect on type 2 diabetes  and largFurthermore, in our manuscript we concluded that effect estimates obtained in our MR analysis relate to the preventative effect of the naturally occurring range of circulating phylloquinone concentration prior to infection; moreover, we confirm that our analysis was not intended to mirror the potential therapeutic effect of high-dose clinical interventions which may be needed for seriously ill, hospitalized individuals.Contrary to the implicit suggestion in Janssen et al. , there iIn their closing paragraph, Janssen et al.  state th2 supplementation in COVID-19, which is led by Janssen’s co-authors and industry collaborators , and includes pulmonary damage and coagulopathy as secondary outcomes. As with other medical interventions, vitamin supplementation can lead to side effects, and therefore, multiple strands of evidence  may be needed to establish whether vitamin K has a role in prophylaxis and/or the therapy of COVID-19 before any clinical recommendations can be made; we certainly made no such recommendations in our own manuscript.The previous track record of vitamins (such as D) and micronutrients (such as zinc) with plausible mechanistic pathways and much more accumulated observational evidence than vitamin K in randomized controlled trials (RCT) has not been encouraging . This is


This is a non-experimental study which aims to obtain the academic procrastination data of students at Makassar, South Sulawesi, Indonesia. It used the non-random sampling method to obtain data from 586 students, consisting of 310 men and 276 women through questionnaires. The data were then descriptively analyzed using an academic procrastination scale, which was compiled and developed based on the Solomon & Rothblum (1984) theory. This scale consist of two sections which aim to determine the areas or fields of academic function, and the reasons for academic procrastination. Structural equation modelling and factor analysis were used to validate the construct. Furthermore, it was descriptively analyzed using an academic procrastination scale, which was compiled and developed based on the Solomon & Rothblum (1984) theory [2Data was obtained from participants through a non-experimental method, using questionnaires, after which it was then descriptively analyzed The participants of this study were students in the XIII semester at several universities in Makassar, South Sulawesi, Indonesia, during the 2020 academic year. It made use of the non-random sampling method to select participants from three universities, namely Makassar State University, Hasanuddin University, and the Department of Psychology, Makassar State Islamic University. Subsequently, questionnaires were distributed directly to 586 students at XIII semesters in Makassar, comprising of 310 men and 276 women. The data collection process took place from January 5 to 28, 2020.An academic procrastination scale contains a total of 44 items, divided into two sections. The first section was used to determine the areas of academic procrastination. Scales for these areas have different labels, but in each case they were coded from 1 (strongly disagree) to 5 (strongly agree). The second section examined the reasons for procrastinating academic assignments, and it used a Likert scale coded from 5 (strongly agree) to 1 (strongly disagree), This research was reviewed and agreed upon by the institutional review board of the university in Makassar based on decree number 5.11.1/UN32.8.4.1/DT2019. Explanation and approval were obtained from each participant before data collection.This research was not sponsored by any institution or organization.


In 2018/2019 there were a number of initiatives for collaboration between Member States in the European Economic Area (EEA) and the European Commission published a Proposal for a Regulation on Health Technology Assessment. In view of the perceived benefits from collaboration, the experiences and challenges of these collaborative initiatives and the possible implications of the proposed legislation, a study of the evidence on attitudes, perceived impacts and the motivational factors towards European Member State collaboration regarding the pricing and reimbursement of medicines was conducted. This study adopted an evidence–based management approach by Barends and Rousseau. The main findings showed that Member States differed in their motivation for collaboration for different pharmaceutical activities. Member States favoured voluntary co-operation for all activities of pricing and reimbursement except for relative effectiveness assessments where Member State authorities had divergent attitudes and prioritised activities related to the sustainability of their healthcare systems and access to medicines. Contrastingly pharmaceutical companies strongly favoured mandatory cooperation for evaluation. Member States motivation for collaboration was highly dependent on the purpose, political will, implementation climate and cultural factors. Currently, with the experiences of ongoing collaborations, following the progress of the discussion at Council, and with a number of inititatives for new pharmaceutical strategy and policy, it is proposed that Member States use their trust, expertise and knowledge of application of evidence-based decision making for pricing and reimbursement of medicines and apply it to decide the future model for Member State collaboration. The applicability of principles of evidence-based management to pharmaceutical policy can be used as a starting point. The area of new medicines is very significant for European Member States (MSs). Continued high prices for new medicines particularly for orphan diseases and cancer affect the sustainability of European healthcare systems . Access A number of initiatives have been instigated by governments and by public authorities for pricing, reimbursement and funding of new premium priced medicines. These initiatives have included innovative models to improve the managed entry of new medicines, horizon scanning, new ways of financing innovative medicines, strategies to improve prescribing and medicines use, as well as post-launch activities and initiatives for regional collaboration between MSs .MS collaboration in the area of pricing and reimbursement is a paradigm shift in the pharmaceutical framework. A number of MSs consider that by collaborating together they can benefit from increasing access to new medicines. This is achieved by developing a stronger stakeholder position within the pharmaceutical policy framework and through augmented negotiation power, particularly with pharmaceutical companies increasingly entering into managed entry agreements including confidential discounts .In the last 25 years, there have been a number of initiatives among MSs for collaboration. These were mainly driven by the MSs, were voluntary and were sporadic and lacked coordination between them . CouncilThe European Commission tried to achieve coordination, and there were several initiatives to this effect. Pricing and reimbursement authorities of most MSs collaborated on a voluntary basis within the “European Network for Health Technology Assessment” (EUnetHTA), a network of governmental organisations, regional agencies and non-profit organisations which collaborate on HTA. EUnetHTA achieved a number of joint outputs particularly tools, guidelines and methodologies for HTA and formed a number of Joint Actions (JA) to support research, communication and work between the collaborators . EUnetHTIn 2016 the European Commission started studying alternative long-term solutions for MS collaboration for HTA. The Final Report , hencefoA “Proposal for a Regulation of the European Parliament and of the Council on health technology assessment and amending Directive 2011/24/EU” , hencefoIn parallel, initiatives for the development of “regional co-operations” within groups of MSs were implemented including the Valletta Declaration, BeNeLuxAIr, FINOSE and Fair Pricing Initiatives. These were voluntary and collaborated on different activities according to priorities. . These cDifferent bodies such as the European Commission and the World Health Organisation office for Europe are drafting proposals for pharmaceutical strategies. The European Commission has just launched a document for a “Pharmaceutical Strategy for Europe”  while thDuring the COVID-19 pandemic, MS collaborated in joint procurement for different medical supplies, for vaccines and for remdesivir .This study aims to collect and present evidence about the specific attitudes, perceived impacts and motivational factors for MS collaboration for pricing and reimbursement of medicines. It focuses on Member States within the European Economic Area (EEA), operating within the framework of EU pharmaceutical policy and legislation.Presented in Theoretical Framework for Collaboration).The first objective was to formulate the theoretical framework to identify the themes for the key topics studied: attitudes, perceived impacts and motivational factors for MS collaboration (Method and Results).The second objective was to collect and present the evidence about MS collaboration using these themes . Search terms were identified using the PICOC and from the literature described in the The search terms were run through different databases (MEDLINE COMPLETE (EBSCO); Pro Quest ABI/INFORM Global; Cochrane database of systematic reviews; EBSCO host; SCOPUS; PLOS ONE; Psychology and Behavioural Science Collection). A number of search strategies were conducted. The PRISMA methodology for reporting of reviews  was used as a guide to present the literature review. The Critical Appraisal Skills Programme (CASP) checklist for qualitative research was used to evaluate the quality of the papers . The greArticles published from 1st January 2000 to end May 2019 were considered. The inclusion criteria included articles directly relevant to the subject area (or related terms); articles related to the different activities for pricing and reimbursement and articles making recommendations for MS (inter) collaboration in their recommendations. Exclusion criteria included articles which covered cooperative partnership and networking between different stakeholders within the supply chain; articles focusing on techniques for evaluation, methodology, and procedures for pharmaceutical activities; papers describing cases for a specific medicine or medicinal product; activities such as manufacturing and the supply chain; articles which covered technologies other than human medicines; papers covering activities outside the scope of this dissertation such as pharmacy practice, prescribing, guidelines, supply chain, integrated healthcare networks, community care; and articles before 1st January 2000.The number of articles for each search strategy was identified. The articles were screened through title and abstract. Those abstracts which possibly fitted the criteria and were directly relevant to the subject of collaboration and the abstracts which were possibly relevant for the building of the general logic model and for the description of the problem in the introduction were identified. Further screening was undertaken of the abstracts and further stratification was performed and duplicates were removed. Generally the articles up to 2014 gave more of a historical background and considered international collaboration as a wishful recommendation, with the exception of the EUnetHTA exercise which was considered primarily for the generation of guidelines and tools to be used by the MSs.Organisational internal data to supplement the formal literature review was mainly obtained through hand searches, searching relevant websites and from reports of the MS entities for pricing and reimbursement activities and of the different regional collaborations. The cut-off date for collection of documents was the end of May 2019.The main researcher (PVB) performed an analysis of the report of the stakeholder consultation conducted by the European Commission as part of the Better Regulation Exercise in preparation for the Proposal for a Regulation for Health Technology Assessment using the ‘Framework’ for this study. The evidence presented in the study was collected from different stakeholders, the specific stakeholder who submitted the information was also specified. While the scope of “the Study” covered pharmaceuticals, medical technologies and other technologies; for this study only pharmaceuticals were considered.A focus group discussion was conducted to supplement the other methods. A professional European organisation was approached. The participants consisted of representatives from authorities and entities involved in pricing and reimbursement activities from different Member States. The President and the Secretary of the organisation accepted to include this focus group discussion as part of the agenda of a scheduled meeting which took place in Brussels in April 2019. There were thirteen participants for the focus group discussion. It was agreed that the researcher (PVB) was to record the discussion, prepare a transcript, circulate it to participants, obtain any feedback and their consent for future activities. The participants had diverse opinions on a number of aspects, and there were topics where there were clear opposing attitudes and critical views of certain activities. It was agreed that the transcript of the focus group discussion would not to be published. However, general inferences and observations could be made and included in future publications.The evidence for each method used was collated and analysed by thematic analysis . Care waThe evidence from each of the methods was thematically placed in the “Framework” according to the themes. The classification of concepts was subjective and care was taken to keep it as standardised as possible. At times, the same concept could be fitted under different alternative themes depending on the perspective, the specific aspect and point of view considered e.g., IT infrastructure can be a challenge and can be a benefit, depending of the perspective. The themes used in the “Framework” and the indicators for the themes, supported the stratification of the evidence. The “Framework” was found to be comprehensive to represent the evidence collected from the different methods.As detailed in The principal researcher (PVB) prioritised the information which was directly related to MS collaboration between pricing and reimbursement authorities. Care was taken to minimise subjective interpretation and selection of evidence. As this aggregation was based on the raw data collected, there was a second round of classification of the evidence under the relevant themes by the principal researcher (PVB), and some repositioning of the evidence was made. The method from which the evidence was obtained was noted to support cross referencing of the relevant information sources. Consequently, these were broken down as follows: SL – scientific literature; GL – grey literature; SIA – Study on Impact Analysis; FG – focus group.In the discussion, the authors present their perspective of application of evidence for decision making, and present some recommendations for ongoing pharmaceutical strategy initiatives and challenges.The papers identified from this review to cover specific activities of pricing and reimbursement are presented in There were different types and quality of papers. No systematic reviews were identified in the literature review. In some published papers, collaboration was the main, or one of the main, objectives of the paper. Some papers were specific to the measurement of opinions about collaboration, used clear and specific methodologies for measurement of opinions, and reported the results through structured titles/themes.Different methodologies were used though for the purpose of obtaining opinions. Some papers gave results from questionnaires or other methods within the paper either as the raw data or else in a collated manner. The presentation of raw data within a paper enabled direct evaluation of the results by the reader rather than just having access to the reporting and the interpretations given by the authors of papers. The interpretation between the authors of the paper and the reader could be different. In other papers, the opinion or recommendation for collaboration was made in an indirect manner.Five papers clearly explained the methodology which they used usually involving both quantitative and qualitative information and the results were presented in a comprehensive manner. These papers used questionnaires for project participants and for external stakeholders to evaluate the experience of EUnetHTA first Joint Action and gave the results in a global manner. Three papers were “perspectives” papers: Eight papers reported actual experiences of collaboration and gave a description of personal or third-party experiences of practitioners who worked in organisations. Some papers went to the level of reporting the achievement or progress of the collaboration. These papers were of high quality in relation to the CASP tool and had clear objectives related to the collaboration through existing projects, mainly the EUnetHTA project. These papers were presented in a structured way with clear methodology and five of these papers were published in the same journal. The greatest experience in collaboration to date was regarding activities of the EUnetHTA JA projects. A number of papers were published between 2010 and 2014 coinciding with the first EUnetHTA Joint Action (JA 1), which dealt with the development of tools and the EUnetHTA Core Model. These included A number of papers focused on specific activities related to pricing and reimbursement. These papers often made no specific recommendations for collaboration as a means for the improvement of the activity. In these papers, it was difficult to apply the criteria of the CASP tool directly. The objectives of these papers were the activities and not collaboration. There was a pattern of activities over time. There were activities with good build up of experience , while some activities such as real-world evidence gathering and disinvestment were still being developed, defined and described.Four papers described initiatives for national HTA in different countries, mainly at the time when the specific country was undertaking initiatives for improvement/development of their national system e.g., Belgium: Four papers compared differences and similarities in considerations and recommendations by HTA agencies in different countries during HTA evaluations. These included Some papers included in this review were used to update the Logic Model, for example: horizon scanning systems in Some authors were involved as main authors and/or co-authors in more than one paper. This led to a level of standardisation in the presentation of the papers e.g., the papers on the outcome of EUnetHTA Joint Actions.Reference price systems were considered as initiatives for MS collaboration because there were organised systems for exchange of information on prices and on policies.The highest reason for exclusion of papers was in cases where papers dealt with or recommended collaboration between different stakeholders (interdisciplinary collaboration) e.g., payers, life-sciences companies, industry, and stakeholders within the supply chain; and did not cover international cooperation between MSs. One paper was excluded because it considered collaboration across healthcare services in councils within the same country. Another reason for exclusion of papers were papers which only considered countries outside the European Union. The EuroScan network was not considered as a MS collaboration for pricing and reimbursement because it consists of an international network of publicly funded agencies undertaking horizon scanning which are not necessarily national agencies for pricing and reimbursement but include different types of agencies such as public health agencies. Members of EuroScan were often service providers for Pricing and Reimbursement agencies . EuroScaA number of papers considered technical aspects of pricing and reimbursement but did not consider collaboration between MSs. Examples of aspects considered included medicines adaptive pathways, pricing frameworks, aspects of decision making for HTA, e.g., multiple criteria decision analysis, sequence of activities for reimbursement decisions, personalised medicines, medicines availability and affordability and big data.A number of papers covered comparison of the requirements, similarities and differences between reimbursement evaluations in different countries. These papers were included if they linked these aspects to recommendations regarding collaboration. There was also increased literature about experiences of disinvestment in different countries.A summary of the different types of documents used as grey literature and the sources of evidence served is predented in Overall, the sourced grey literature was very useful to study different perspectives and in particular in-depth insights from different stakeholders; although, in some cases the evidence could not be directly linked to a specific stakeholder. A number of documents from the grey literature represented the perspectives and priorities of organisations: individual member states or collective opinions. The media gave information and insights of happenings which would otherwise have remained hidden. Often journals used the tactic that they reported information from someone  who was not allowed to divulge information.The Policy Briefs typically gave a comprehensive and balanced overview of the relevant topic. Policy briefs mainly adopted the MS’ perspective/s. Council Conclusions gave joint positions from the MS. Conferences and conference proceedings were particularly relevant because they gave an overview of a topic and also provided an insight about evidence from sources of information, such as organisational information or stakeholder perspectives, which was typically not published in scientific literature unless the conference proceedings were published as a Supplement to a Journal. A few of the documents included in the grey literature were prepared and organised according to systematic methodology; these included studies contracted out through bodies such as WHO, for example the “Research study on impact and benefits of cross border collaboration in WHO Europe region” by From the evidence, it was difficult to clearly qualify and quantify the achievements and outcomes from the activities of the MS collaboration. In reality, there were few significant achievements beyond sharing of information.The “Study on impact analysis for policy options for strengthened EU cooperation on HTA, Final report” , referreDetailed analysis of “The Study” revealed some limitations. The Commission pre-conditioned the options with its criteria and did not present a blank drawing sheet. The authors of “The Study” adopted the Policy Options and conditions set by the Commission. “The Study” did not consider an implementation mechanism without EC funding because the Commission considered that intergovernmental collaboration without input from the EU was strictly the responsibility of the Member States. The authors of “The Study” expressed their collective opinion about specific issues in quite a definitive and specific manner; and this could bias the output of “The Study.”The presentation of the results was based on the Better Regulation framework of impacts and the impact analysis focused on impacts. Greater emphasis was placed on the economic impact rather than the social impact. This reflects the priorities and perspectives of stakeholders, with companies typically more concerned with economic rather than social impacts.The response on impacts was presented according to different stakeholder groups; however, not all stakeholder groups responded. The pharmaceutical industry and public administrations provided most of the feedback. The point of view of these stakeholders (particularly the industry with most respondents) overpowered the results and also the recommendations. The recommendations of these two major stakeholders differed, with the recommendations of the industry dominant and most reflected in the final legislative proposal. Not all countries responded to this study and consequently the opinion of the MSs as reported in “The Study” did not reflect certain opposing views which were later expressed during the discussion of the Proposal for a Regulation on HTA at the European Council.The participation in the survey by patient organisations was far too low and the authors decided that they could not analyse the feedback from the patient response. Consequently, nothing was included on the patient response in the summary of the results. However, the detailed report had some information on the perspectives of patient associations, where concern was expressed about uncertainty due to conditional approval. This response was very significant and very contrasting to the feedback by the industry.The participants at the focus group considered the topic of collaboration as sensitive and political, particularly in view of the discussions on the Proposal on a Regulation on HTA which at that time was being discussed at Council. There was a general positive consideration on collaboration for certain activities for pricing and reimbursement including horizon scanning and collection of real-world data but not for HTA. There was a divergence of opinion on the experience and success of regional collaborations. Some participants came from countries which were involved in a regional collaboration while some participants were from countries that were sceptical of such collaborations. The discussion about the regional collaborations was quite challenging. There were in-depth insights regarding the politics and the challenges being faced in the interaction between the industry and the Member States within the regional co-operations, and this is likely to remain.The evidence on the different processes for pricing and reimbursement from the different methods was used to update the Logic Model for the system of pricing and reimbursement. The final and updated Logic Model for the system of Pricing and Reimbursement as in September 2019 is presented in The updates to the Model which were derived from scientific literature were highlighted in yellow while the updates derived from grey literature were highlighted in grey.There were a number of changes in the process for pricing and reimbursement, including collaboration between pricing and reimbursement authorities, which was a relatively new concept. More recently there was a high level of initiative for “coordinated collaboration alongside the life-cycle”  and thisAs discussed in the Method, during aggregation of the evidence, the evidence was referenced according to the source of the evidence: SL: scientific literature; GL: grey literature; SIA: Study on Impact Analysis, “The Study”; FG – focus group.A main theme concerning attitudes on collaboration was whether collaboration should be voluntary or mandatory. The policy options for the Proposal for a Regulation on HTA presented different options for the type of participation and for the uptake of joint outputs . In 2016, The Council of the European Union recognised that a number of MSs expressed interest in pursuing voluntary cooperation and stressed that these activities should remain voluntary and be focused on added values (GL). The European Commission proposed that collaboration would be voluntary for all activities except for HTA (GL). During the focus group discussion, a number of practitioners were vociferously supportive of voluntary collaboration and against mandatory collaboration. The opinion towards mandatory collaboration was overshadowed (FG). “Strong” countries which have well established systems for pricing and reimbursement considered that they did not gain from collaboration; in fact they stood to lose .The stakeholders had differing opinions on collaboration on HTA. The authors of “The Study” considered that economic impacts were particularly relevant to the industry; sustainability of healthcare systems was mainly relevant to public administration authorities and social impacts were relevant to citizens, patients and healthcare professionals (SIA). The industry considered that collaboration on HTA would be positive for mandatory participation and uptake of REA but would be negative for full HTA, i.e., the economic part of the assessment was not to be considered because of the high level of agreement that would be needed. Industry graded legislation for full HTA (including REA) negatively for all themes (SIA). In “The Study,” public administrations were reported to favour full HTA including the economic part with a legislative framework and mandatory participation and uptake (SIA). Patients also favoured policy options with mandatory participation and uptake as these were considered to increase the availability of medicines and to ensure standardised monitoring of health technologies prior to market access (SIA).Collaboration could take place at different levels, ranging from loose collaboration such as exchange of information and development of common methodologies, to joint collaboration on cross-border assessments (SIA). Traditionally (up to 2015) the MSs collaborated by setting joint tools, guidelines and methodologies which were subsequently to be used at a national level, bringing a level of standardisation but allowing for differences between MSs. The experience of the project-based cooperation of EUnetHTA resulted in a number of challenges. These included the fact that joint work was used to a limited extent by the MSs, there was low uptake of joint work by national HTA authorities mainly due to legal and administrative hurdles, there were concerns on quality assurance, the joint timelines needed to be aligned with national timelines, and there was lack of sustainability of the work produced at the project setting (GL). More recently, it is preferred to undertake working together as one output . While practitioners seemed to generally agree on the technical aspects related to tools, guidelines and methodologies for HTA, at the political level there was divergence of positions on joint work, which were more at the level of national politics than the perspectives of individual practitioners (FG).As mentioned, “The Study” showed that the European Commission and the authors of “The Study” considered that intergovernmental cooperation without input from the European Commission was not to receive EU funding and was strictly the responsibility of the MSs (SIA). Practitioners considered that the Proposal for a Regulation on HTA, as published, was much more supportive of the economic rather than the public health perspective. They felt that with the introduction of a Regulation for HTA, the position of the MSs within the power balance will decrease as compared to industry. Practitioners had mixed opinions on the power position. Those who felt less influential and powerful in the MS hierarchy (the “weaker” MSs) preferred to have legislation which brought all MSs at par rather than them being bullied by “stronger” MSs. MSs which had well-established HTA systems and had negotiating power (the “stronger” MSs) strongly resisted mandatory collaboration. Practitioners felt that there were advantages from collaboration including production of evaluations of high quality. They considered that the European Commission proposed collaboration which was “coercive.” Practitioners believed that power within HTA collaborations would be restricted. If the methodology for HTA including the criteria for REA were to be set through legislation, as set in the original Proposal, this was considered to restrict the collaborators from evolving and adapting the methods. MSs were concerned with transferring the authority for REA outside national jurisdiction and the Proposal was considered to take over national legislation (SL).As stated earlier, during the focus group discussions, there were divergent opinions with respect to regional collaborations, particularly with regards to their outcomes and success. There was consensus though that the industry was not typically willing to participate in joint negotiations (FG). The regional collaborations were considered as the reaction of the MSs to the industry’s game of divide and rule (GL).In 2019, the main innovative industry association (EFPIA) considered that some initiatives for collaboration could support increased access to medicines and sustainability of healthcare systems, while in some cases national procedures were preferred. EFPIA considered that collaboration was in its infancy and there was little successful experience of enhanced access to medicines through collaborative initiatives. EFPIA suggested that until there was evidence of benefit from collaboration, national access processes were to remain the preferred way for timely access (GL). In 2018 the media criticised the BENELUXA cooperation because of lack of concrete negotiation deals (GL). The BENELUXA stated that at times industry did not want it publicly known that negotiations were underway (GL).MSs had a general positive attitude for voluntary collaboration for different activities, with the exception of a clear divergent position on voluntary/mandatory REA. The scientific literature was the main source of information to inform of or promote new activities such as the generation of real-world data. MSs tended to keep the information on their actions and initiatives for collaboration internal e.g. the building of the horizon scanning initiative within BENELUXA and the regional collaborations kept the information within their organisations. There was consensus from the different sources that sharing of information on HTA was the first step towards collaboration, and the one which is most likely to be successful. The attitudes on other activities are not so clearly stated, probably because there was uncertainty on the possible level of achievement of outcomes. The MSs participating in regional collaborations were more interested in activities which they considered to increase access to medicines, particularly joint negotiation. The industry was supportive of mandatory REA and early dialogues but not of the other activities. Patients were mainly focused on the attitude that there should be patient involvement in activities and in decision making.The themes of the “Framework” and the indicators for each theme for the perceived impacts supported the strategic aggregation of the evidence from the different sources and standardisation with the relevant theme. The themes were divided into social health impacts and economic impacts in line with the Guidelines for Better Regulation of the European Commission. At times, the same information could fit into different themes.The scientific literature mainly reported the positive experiences and perspectives (benefits) in relation to improved quality of output and the benefits from joint methodology and tools. “The Study” specifically studied the perspectives of the impact for different stakeholder groups and was the most informative method in this respect. “The Study” showed that there were conflicting perspectives between and within stakeholder groups. The industry  considered economical benefits as key, while the MSs emphasised the social impacts such as access, sustainability of health care systems and public health. “The Study” considered that there would be improved governance, improved quality of assessment, standardisation of tools and methodology and reduced administrative burden for MSs (SIA). Contrastingly the scientific literature , the focThe themes in the “Framework” were considered adequate for the presentation and aggregation of the available evidence on motivational factors. There was some overlap between the concept of motivational factors and perceived benefits and risks. The grey literature was the main source of evidence on motivational factors and gave an insight mainly from the point of view of MS organisations. There seemed to be more evidence on challenges than facilitators for collaboration. The main driver for collaboration for MSs was increased access to medicines. The main challenges were not of a technical nature but concerned mainly cultural, national and political factors such as safeguarding national jurisdiction, autonomy over activities for pricing and reimbursement, problems with harmonisation across MSs, the national level of engagement, building of trust, national legislations, the need for specific resources for collaboration and political will and commitment.The aggregation of the evidence from the different methods and its the corroboration enabled the building of a holistic and realistic picture.There was an imbalance in the publication of evidence. A great deal has been published about the Proposal for a Regulation on HTA; however, less has been published regarding certain new activities related to pricing and reimbursement. The generation of evidence came in waves, for example the concept of real-world data in response to conditional marketing authorisation was still evolving and the evidence followed the same patterns. There was also the limitation that confidential data, such as minutes of meetings, which could not be used.The aggregation of the evidence from different sources showed that some individuals and/or organisations participated in more than one initiative, and there were “opinion leaders.” Certain activities such as funded projects and high-level meetings between different stakeholders could be used to change the political balance in relationships and possibly reorganised the power between stakeholders.Although grey literature is generally considered as lower quality of evidence in terms of the hierarchy of evidence for HTA, for the purpose of evidence-based management it proved to cover a more holistic picture and was particularly useful to show organisational insights and professional opinions. It is recommended that a different “pyramid for the hierarchy of evidence” is drawn, specific for the study of attitudes and of perspectives, with methods that provide insight being ranked high in this pyramid. The media was specifically useful to uncover the controversial points and to bring them to light. The media was like a balance check for the evidence, although it was still biased. The focus group discussion was probably “too good” a method to bring out the issues, so much so that the participants decided to block it. This gave an indication of what may actually happen in reality, and with the exception of the media , the evidence which gives too much insight of reality may tend to get blocked.‘The Study’, which followed the methodology of EU Better Regulation, focused on the perceived impacts of the intervention. It did not consider the attitudes of the main stakeholders and the challenges and motivators related to the intervention. This is considered as a deficiency of the EU Better Regulation methodology of the European Commission.One major lesson learnt from this research was the risk of bias when reading literature and interpreting the various studies. When “The Study” was published, the first author (PVB) had given it a good viewing. After a thorough evaluation of “The Study” as part of this research, it was clear that although the methodology of “The Study” was clear and according to the rules, the final outcome of “The Study” gave a strong prominence to the position of industry and made less emphasis to certain points made by other stakeholders such as the patient organisations, which had much less representation.Generation and processing of the evidence was very time consuming. It would be difficult to generate such a lot of evidence in the future to address decision making in routine practice. This methodology would probably be limited to very important decisions, as were the decisions on initiatives for Member State collaboration for pricing and reimbursement.The main aspects of attitude for collaboration were voluntary/mandatory participation in collaboration, the uptake of the joint output from collaboration, and the attitude towards the formation of regional collaborations and different perspectives of stakeholders.This research showed that the power and attitude of the MSs involved will determine whether countries will be willing to collaborate together and which model of organisational structure will be chosen for the collaboration The differences in attitude on voluntary/mandatory collaboration between the “stronger” MSs and the “weaker” MSs could be explained in terms of motivational factors. The “stronger” MSs considered that their economic strength gave them enough power for negotiation with the industry, and thus they were not motivated to collaborate with other MSs, while “weaker” MSs needed to build power for negotiation through grouping. This is aligned with The power and attitude of the MSs involved will determine whether countries will be willing to collaborate together and which model of organisational structure will be chosen. Forming smaller groups, such as the regional cooperations, may make it easier for the collaborating MSs to find an aligned scope and benefit. The regional co-operations are being formed by MSs voluntarily and therefore these only form if the MSs feel that they are mutually benefitting from the collaboration in one way or another. It is clear from the evidence that inititiatives for collaboration must be introduced voluntarily and that MSs are to be left to participate out of their own will.A high emphasis was placed by certain stakeholders particularly the European Commission and the industry on benefits for “governance and administration” in terms of improved quality of assessment, with standardisation and harmonisation. This is in line with the results of the systematic review by The results of this study showed that “factors related to purpose,” cultural factors, and the implementation climate were key motivators for MS collaboration. The factors identified in the results of this study were aligned with the factors mentioned by As shown by The principle of bounded rationality  considerOne bold consideration for the regional collaborations is for the MS involved to introduce transparency of prices of medicines between themselves. It is not possible for all the Member States to be gaining from the current confidentiality on prices. The Member States know the published (list) prices of each other, which in reality may not be the true price when considering different forms of discounts involved. Transparency of information on prices should be between the countries that agree to the terms of the collaboration and should not extend beyond the countries participating within the regional collaboration because of the system of reference pricing. It may also be problematic for patients if co-payments are based on list rather than actual prices. In addition, concerns whether confidential discounts will be seen as undemocratic as debates within parliaments are not possible when ministers have already negotiated confidential discounts .In spite of progress achieved, MSs still have much to learn about the possible benefits and synergies from collaborating together. Strong political will is needed as there can be a lack of trust and of motivation among MSs to collaborate. Consequently, the ongoing collaborations need to take bold steps to surmount the main hurdles and fears. They need to be assertive, they need to trust each other and they must address fears of the unknown. It takes time to build trust and political will. Ideally the main ongoing regional collaborations will take actions at the same time, if not jointly. The positive experience and the willingness to collaborate exhibited by MS during the current COVID pandemic shows that where there is strong motivation hurdles can be overcome.The method of evidence generation described by The evidence from this research can be used to inform future decisions about initiatives for MS collaboration. The project shows that different MSs have different levels of motivation to collaborate on different activities and the main determining motivational factor for realisation of collaboration is political. This information corroborates well with the level of difficulty encountered during the discussions on the Proposal on HTA and the way that joint procurement for the COVID pandemic was handled.Member States already have extensive experience and knowledge in applying evidence for decision making with respect to pricing and reimbursement of medicinal products within their national health care systems. They should apply the principles of evidence-based practice to their management and to policy decisions too. The Logic Model for the system for pricing and reimbursement  shows thStakeholders within the pharmaceutical framework have contrasting interest and whenever a strategic decision is to be taken, understanding the different political positions and powers of the stakeholders concerned is key and influence/interest mapping should be undertaken formally using the best evidence and incorporating the involvement of the different stakeholders concerned. This will be the basic principle for MS to consider when they need to establish collaborations with other stakeholders such as the pharmaceutical industry.Currently, there are a number of ongoing initiatives for setting of a pharmaceutical strategy by the different authorities including the European Commission  and the World Health Organisation. Now is the time to look at the legislative and policy aspects which could have hindered initiatives for collaboration in the past and to address them. The multitude of ongoing concurrent initiatives may lead to overlap in policy or conflicting directions. Ideally there is collaboration at the policy and strategy level as well.Patricia Vella Bonanno works with the Ministry for Health in Malta and is a member of the Valletta Technical Committee. All opinions expressed in this publication express her personal positions as an academic and are not representative of her post.Patricia Vella Bonanno and Brian Godman are members of the Piperska Group and have participated in joint publications as part of this Group. The opinions expressed in this publication are personal and not in the name of the Piperska Group.https://www.um.edu.mt/library/oar/handle/123456789/49528.This study was conducted by Patricia Vella Bonanno for a dissertation “Attitudes, perceived impacts and motivational factors for European Member State Collaboration for pricing and reimbursement of medicines: a review of the evidence,” submitted in partial fulfilment of the requirements of the Degree of MA in Management at the University of Malta. Vincent Cassar was the main academic tutor for this dissertation. Brian Godman contributed as pharmaceutical expert for the dissertation as well as for this publication. The full dissertation including the raw data (presented in its Appendices) is published by the Library of the University of Malta on Ethical Clearance for the study was carried out as part of the requirements of the Dissertation on the 29th of May 2019 by the Faculty Research and Ethics Committee of the Faculty of Economics, Management and Accountancy at the University of Malta.