In the last decade, electronic health  literacy has attracted the attention of the scientific community, as it is associated with the self-management of patients with chronic diseases and the quality and cost of care. It is estimated that 80% of people with chronic diseases are cared for at home by a family member, friend, or relative. Informal carers are susceptible to physical and mental health problems, as well as social and financial hardships. Nevertheless, there seems to be a research gap in terms of carers’ needs, skills, and available resources in the age of new technologies, with the vital role of eHealth literacy of the carers remaining unexplored.The aim of this study was to investigate the level of eHealth literacy and health literacy of primary and secondary carers of people with dementia, to explore the association between health and eHealth literacy, as well as their association with the caregiving variables: self-efficacy, coping, and caring perceptions.A sample of 200 primary carers (the carer who supports the people with dementia in everyday living) and 200 secondary carers  will be recruited from dementia day care centers and Alzheimer’s associations in Greece and Cyprus. The study will be a cross-sectional correlational descriptive study. Tools to be used include the eHealth Literacy Scale adapted for carers to measure eHealth literacy, European Health Literacy Survey Questionnaire 16 (HLS-EU-Q16), Single Item Literacy Screener, Revised Scale for Caregiving Self-Efficacy, Carers of Older People in Europe (COPE) index for caregiving perceptions, and COPE brief to measure selected coping strategies. Descriptive statistics will be reported, and correlations between different variables will be explored with parametric and nonparametric measures.As a preliminary study, the HLS-EU-Q16 has been validated in 107 older people. The internal consistency of the scale as estimated using Cronbach alpha coefficient was .77, somewhat lower than other validation studies. Recruitment of pilot study participants started in May 2017.Carers’ eHealth literacy is a new field. Whereas previous studies have focused on the role and impact of low eHealth literacy and health literacy among older adults, the eHealth literacy of carers, and in fact carers of people with dementia, has not been explored. We hypothesize an association between eHealth literacy and health literacy level with carers’ perceptions about caregiving role, self-efficacy, and coping strategies. A possible moderator in these associations is the secondary carers’ eHealth and health literacy level, which will also be explored. By confirming the above hypotheses, tailored eHealth literacy interventions for carers of people with dementia and their families will be developed as a direct outcome of this research. In the new digital era, new technologies are developed to support carers in their everyday role. However, most of the time, this is done without taking into consideration carers’ health literacy, digital skills, and electronic health  literacy level. According to Eurocarers Association, “a carer is the person who provides unpaid care to someone with chronic illness, disability, or other long lasting health and care needs, outside a professional or formal framework.” People who provide care at least once or twice per week are those in the age range of 50 to 64 years, followed by the 35 to 49 age group according to the third European Quality of Life Survey . In the , the level and the role of eHealth and health literacy among carers and, in particular, carers of people with dementia is very limited. There is, nevertheless, abundant information, mainly of descriptive nature, with regard to the type of Internet use among carers of people with different chronic diseases, without any further exploration or recommendation.Older adults are considered to be the population group with the most difficulty in using new technology. In recent years, many studies have investigated the eHealth literacy of older adults, providing evidence that increased age and lower educational level are good predictors of lower eHealth literacy level and low Internet use -7. AlthoIn a recent study in the United States, Kanthawala et al  investigIn another study in the United Kingdom, Blackburn et al  exploredAccording to Lam and Lam , the mosIn a qualitative study, carers visiting a caregiving website mostly looked for health information and practical, legal, and financial issues. These preferences were directed by the type of caregiving. Kernisan et al  categoriThere is also a large number of studies looking at the effectiveness and the usability of Web-based support programs such as online communities, fora, and psychoeducational programs that aim to improve education and communication of carers . A recenAs a term, eHealth literacy has gained considerable attention in recent years with the increased use of new technologies in health. Nevertheless, there is accumulating evidence that available technologies provided to people with chronic diseases or their carers are not properly used, or people are not using them because of lack of digital skills. In 2006, Norman and Skinner  presenteTraditional literacy, which includes basic skills to read, understand, write, and speak language.Information literacy, which describes the skills needed by a person to find, select, and use information available of any type.Media literacy, which is defined as a process of metacognitive reflective strategies to place the information from several media sources in a social and political context.Health literacy, for which several definitions have been used in the literature. One of the most frequently cited definitions is the one proposed by Ratzan and Parker , which rIn the context-specific type of skills, Norman and Skinner include (5) computer literacy, which is the ability to use computers, and (6) scientific literacy, which is the skill to understand the aims, methods, implementation, limitations, and politics of creating knowledge. As part of this theory, Norman and Skinner  developeChan and Kaufman  have proNorman  discusseAfter the revision of the Lily model, which actually included the cognitive factors of users, additional attempts to expand the model have taken place ,23. GilsMore recent suggestions are the ones proposed by Norgaard et al  and BautConsidering the important role that carers play for the national health systems, both the scientific community and policy makers alike have become more interested in maintaining carers’ health in recent years. Carers experience more stress than the general population, and they report higher use of antidepressants, are more susceptible to infections and cognitive decline, and have high mortality rates -28. FurtThe stress process model  includesThe coping strategies and the social support of the carer in combination with the different types of the stressors, according to Pearlin’s model, act as mediators of the mental and physical health of the carer .Social support is included in Pearlin’s stress process model as part of the personal resources that are important to cope with life stressors . The modIn the initial model by Pearlin , as part, discussing the primary and secondary appraisal. In primary appraisal, the person focuses on the importance of the event, if it is irrelevant to their own well-being, benign, positive, or stressful. In the secondary appraisal, we encounter the contextual factor and the ability of the person to cope with the stressor.Additionally, Lazarus and Folkman  distinguThere is limited research on the associations between the abovementioned concepts, with research especially limited in terms of the role of eHealth literacy. In Concerning the selected caregiving variables, self-efficacy is related to cognitive appraisal and acts as a motivator of action and selection of coping strategies . PerceptWe presume the effect of health and eHealth literacy of the secondary carer and primary carer’s perceived social support on the health and eHealth literacy of the primary carer and the selected caregiving variables.Social support is also a concept connected with health literacy, acting as a possible moderator in the relationship between low health literacy and poor health and is defined as “the degree to which individuals have access to social resources, in the form of relationships, on which they can rely” ,42. The The second dimension of social support, which may possibly interact with the level of health literacy, includes the emotional, informational, health reminder support, and tangible aspect of support and is referred to as functional support ,43. AccoAccording to a recent review , we findAs part of the study, health and eHealth literacy tools, as well as the Revised Scale for Caregiving Self-Efficacy will be validated in the Greek language for use among this population group.The main research questions areRQ1a: What is the level of health literacy and eHealth literacy of dementia patients’ primary carers?RQ1b: What is the level of health literacy and eHealth literacy of dementia patients’ secondary carers?RQ2: Is there a difference between health literacy and eHealth literacy level of dementia patients’ primary and secondary carers, given the generation gap?RQ3: What is the association between health literacy and eHealth literacy of dementia patients’ primary and secondary carers?RQ4: What is the association (if any) between health literacy and eHealth literacy of dementia patients’ carers and caregiving self-efficacy?RQ5: What is the association (if any) between health literacy and eHealth literacy of dementia patients’ carers and their ability to cope with the stressors of caring?RQ6a: What is the association (if any) between health and eHealth literacy of dementia carers and their perceptions toward the caregiving role?RQ6b: What is the association (if any) of the health literacy and eHealth literacy of the dementia patients’ secondary carer and the primary carers’ self-efficacy, coping, and caregiving perceptions, and to what extent does the observed association between health literacy or eHealth literacy and caregiving variables in the primary carer differ according to the health and eHealth literacy of the secondary carer?RQ7: What is the association (if any) between social support and caregiving variables and to what extent the observed association between health literacy or eHealth literacy and caregiving variables in the primary carer differ according to the levels of social support?The study will be a cross-sectional correlational descriptive study design to explore the level of health literacy, eHealth literacy, and their association with caregiving self-efficacy, coping strategies, social support, quality of support, positive value, and negative impact of caregiving in Greece and Cyprus.Before the full scale research study, a pilot phase will be conducted to assess the appropriateness of selected questionnaires, the mode of data collection and length of interview, the acceptance of the research material by the primary and secondary carer, and expected challenges in sample recruitment. According to Connelly , the adeCarers of people with dementia will be recruited from dementia centers and Alzheimer’s associations in Greece,  and Cyprus. They will be invited to participate in the study following informed, signed consent. The sample will include primary carers (the carers who support the people with dementia in activities of daily living) and secondary carers . For each primary carer, a secondary carer who provides support to the primary carer will be identified. The secondary carer will be named by the primary carer as the closest person who supports the primary carer in his or her caring role. Selected questions will assist the primary carer to identify the supporter carers.As there are many social cultural similarities related to caregiving between Greece and Cyprus, given the common language and historical and sociocultural background of both countries, it was decided to recruit one sample from both countries. Carers in both countries have the most important role in the care of people with dementia substituting for gaps in the national health care systems. The non-for-profit associations have undertaken the role of supporting and providing services to carers in Greece and Cyprus. In Greece, a number of services provided to carers by the not-for-profit associations are funded by the Ministry of Health through the mental health reform program .Furthermore, the inclusion of two metropolitan cities from Greece, Athens and Thessaloniki, offers the opportunity to involve very active Alzheimer’s associations in Greece with both a high as well as more heterogeneous number of users, in an effort to achieve the inclusion of as wide as possible set of members from the target population in terms of their sociodemographic characteristics, as well as the variables of interest. As this is a correlational study, the multicenter convenience sampling aims to increase the observed variability in the variables of interest.r=.25 (type I error 5%) is 168 primary carers and 168 secondary carers. To account for issues with possible inconsistencies in data, incomplete questionnaires, and missing values, it was decided to increase the recruitment to a sample of 200. Moreover, in this way, we ensure that the number of the secondary carers (and thus primary-secondary carers dyads) will not fall under the minimum required sample size, as it is likely that not all secondary carers may agree to participate. Estimated duration of the recruitment period will be 12 months.The sample size was calculated considering carers in Greece and Cyprus as one sample according to the above requirement The minimum required sample size with 95% power to detect a statistically significant correlation of the aforementioned variables of the magnitude of In Cyprus, prospective participants will be recruited from the Pancyprian Association of Alzheimer’s Disease and from the Alzheimer’s day centers, Ithaki, which are located in the city of Limassol and Pafos. We have selected these two day centers as they are currently the only services for carers. In Athens and Thessaloniki, recruitment will be done through the Alzheimer’s association. In Athens, there are currently six dementia day care centers: in the municipalities of Marousi (1), in Halandri (1), in the city of Athens (3), and in Ilioupoli (1). In Thessaloniki, there are two dementia centers. Furthermore, a sample will also be selected during the events on Carers’ day, which is usually organized by the associations annually.Inclusion criteria for the primary carer include being a self-appointed carer of a person with dementia; supporting the person in activities of daily living, irrespective of the relationship with the person ; being over 18 years of age; and able to read and write in Greek.The carers will be first approached by the manager of the centers and/or associations who will explain to them the aims of the study. If a carer fulfills the inclusion criteria and is willing to participate, she or he will be referred to the researcher for data collection.Secondary carers will be nominated by the primary carer and will also be invited to participate in the study. The primary carer will initially contact the secondary carer asking if they are interested in participating, and the researchers will follow this communication to arrange the face-to-face or telephone survey interview.The face-to-face surveys will be conducted at a place and time convenient for the primary carer. In the case of the secondary carer, an effort will be made to collect the data in face-to-face survey interviews, but the option for a telephone survey interview will be provided to reduce the likelihood of nonparticipation by the secondary carers. The primary carers will respond to the full questionnaire pack, whereas the secondary carers will be asked to respond to the health literacy and eHealth literacy scales (using the same tools as in the case of the primary carers), as well as providing information with regard to sociodemographic characteristics.Information on sociodemographic characteristics will be collected from both the primary and secondary carers, as well as for the people with dementia they are caring for. Primary and secondary carers information will include age, gender, education, employment status, living situation, hours of care per week (primary carer), years of care (primary carer), number of care recipients (primary carer), relationship with the person with dementia (primary carer), care professional help (primary carer), relationship with primary carers (secondary carer), and type of support provided to primary carers (secondary carer). Information of the person with dementia will include age, gender, diagnosis, stage of the disease, and functional level.eHeals, a self-report tool measuring eHealth literacy based on the Lily model, will be used . The scaFor this study, the eHeals will be translated into the Greek language using backward-forward translation of the original English version. The questionnaire items will be adapted accordingly where necessary to address carers based on a review by an expert panel. The metric properties of the Greek version will be assessed using the Content Validity Index (CVI) based on the responses of an expert panel in the field of eHealth and health care to assess its content validity. Furthermore, the construct validity of the scale will be assessed in exploratory and confirmatory factor analyses as necessary. The internal consistency of the scale will be assessed using Cronbach alpha coefficient. The validation will be part of the analysis of data derived from the final sample.The Internet use carers profile will be measured using a series of 10 questions that assess the frequency and type of use, for example, use of websites, emails, e-learning, social media, interactive services, forums, blogs, mobile, and the Internet. It was deemed important to supplement the eHeals scale with these profile questions, as there has been much criticism with regard to the lack of relevant questions in the eHeals scale, given the Web evolution during the last decade ,48.(HLS-EU-Q16) [In addition to eHealth literacy, the health literacy of the primary and secondary carers will be assessed using the European Health Literacy Survey Questionnaire 16 -EU-Q16) ,53. The -EU-Q16) ,55, Bulg-EU-Q16) , and Swe-EU-Q16) . As far Single Item Literacy Screener (SILS) assesses inadequate health literacy and together with the HLS-EU-Q16 provides the information on the health literacy level of the study participants. SILS has been part of 16 questions developed by Chew et al . InitialThe scale assesses the self-efficacy of carers . It consCOPE index measures carers’ perceptions toward positive and negative values of caring . It consBrief COPE assesses the coping strategies adopted by carers . It consMultidimensional scale of perceived social support consists of 12 items measuring social network support, including three factors: significant other, family, and friends. The items are scored on a Likert scale from 1 (very strongly disagree) to 6 (very strongly agree) ,65. The t test, analysis of variance) will be used as needed, for example, to investigate differences in eHealth and health literacy according to sociodemographic characteristics of the participants. The association between dependent variables  and independent variables  will be assessed in multiple regression models before and after adjusting for sociodemographic variables. The extent to which the observed association between health literacy and coping and caregiving perceptions among primary carers differs according to self-efficacy, social support, and the secondary carer’s eHealth and health literacy (moderators) will also be explored.Descriptive statistics will be reported, and bivariate correlations between all variables of interest will be explored with parametric and nonparametric measures. Sociodemographic correlates associated with health literacy will also be assessed. Additional data analysis  will be assessed using exploratory and confirmatory factor analyses and internal consistency reliability analysis. Analysis will be performed by Statistical package for the Social Sciences (SPSS) version 22 (IBM Corp) and exploratory factor analyses with SPSS AMOS.Permission to conduct the study was granted by the National Committee of Bioethics in Cyprus on January 10, 2017, according to the National Law (EEBK ΕΠ 2016.01.151). The commissioner of personal data protection in Cyprus has been notified accordingly and confirmed notification on December 19, 2016 (study number 3.28.460). In Greece, the scientific committee of the Athens Association of Alzheimer’s Disease and Related Disorders have also been notified and approved the study on March 17, 2017, with a decision by the Executive Board. This process will be repeated for the Alzheimer’s association in Thessaloniki.All participants will be fully informed about the purpose and the requirements of participation in the study. Consent forms will be signed, and participants will have the right to withdraw at any time. Confidentiality of the participants will be respected. Researchers will safeguard the well-being of the participants during the data collection.Participants who are interested in receiving feedback will be contacted by email or telephone as soon as the results are analyzed and drafted. Researchers will try to make the participants feel comfortable and resolve any kind of conflict concerning the time, the place of the meetings, and the way that the secondary carers will be contacted.To safeguard personal sensitive data, a database protected by a password will be developed and will be stored by the research team university computers. Only members of the research team will have access to the database. Hard copies of all measurements will be stored and locked in the Office of the Scientific Supervisor.The pilot phase of the study is in progress. In the following section, we report some preliminary results of the validation of HLS-EU-Q16 in Greek for the purposes of this protocol.A convenience sample of 107 older people from an outpatients’ eye clinic in Cyprus and open clubs for leisure activities for older people in Athens, Greece, participated in the validation of the scale .The internal consistency of the scale as estimated using Cronbach alpha coefficient was .77 and was adequate, even though it was somewhat lower that the respective figure observed in validation studies elsewhere. CVI for each item, as well as the overall scale was also calculated with a panel of experts (N=6) and a panel of health professionals (N=20), providing high scores for item-level CVI and scale-level CVI/average (S-CVI/Ave) in both groups. S-CVI/universal agreement (S-CVI/UA) was lower among health professionals compared with the group of experts .In-depth analysis of the results derived by the validation of HLS-EU-Q16 will be presented in a subsequent paper. The data collection of the pilot study started in May 2017, and the data collection for the main study is projected to start in October or November 2017.In this study protocol, we have presented the preliminary results of the HLS-EU-Q16 validation. The validation was carried out among 107 older people in Greece and Cyprus, providing information for the comprehensive health literacy level of older people in these two countries. The main study will investigate the relationship of eHealth literacy and health literacy with caregiving self efficacy, coping strategies, and care management perceptions of carers of people with dementia. Previous studies have explored the associations between health literacy and coping strategies, health literacy and self-efficacy, coping strategies and care management, caregiving and self-efficacy, social support and self-efficacy, and social support and health literacy in different target groups. However, no previous study has adopted a unified approach or explored these issues in carers of people with dementia ,66-69. FeHealth literacy is a rather underresearched concept among this population, taking into consideration the age of the majority of carers above 50 years). The idea of connecting eHealth literacy with caregiving becomes more challenging. New technologies are a core part of everyday life for a large percentage of the population worldwide, but still there are specific groups with low access to technological advances. Low income, low socioeconomic status, and racial or ethnic minorities are considered a predictor of Internet nonuse [0 years. Carers and especially spouses could be considered to be a minority in the use of technology. On the other hand, several projects are funded to develop technological innovations to support carers in their role, including Web-based psychoeducational programs and support groups -73, inteThis is also confirmed by the systematic review by Chi et al . Six typEducation using mainly telephone-based, Web-based, and video interventionsConsultation using videoconferencingPsychosocial or cognitive behavioral therapy intervention using telephone and videoconferencing toolsSocial support using videoconferencing toolsData collection or monitoring, including response center, sensors, and fall detectorsClinical care delivery using videoconferencesTaking into consideration the large amount of research available on the usability and feasibility of this type of research, it is interesting that there is little focus on the skills required by this target population to use the aforementioned services.The challenges of this study concern the recruitment of carers, both in terms of access (hence a convenient sample of people in contact with services), as well as the time requirements and other elements of the recruitment procedure, mainly the survey completion time (estimated at 60 min) and potential difficulties in contacting and recruiting secondary carers. We expect that the majority of secondary carers will be the children or friends of the primary carer, making the arrangement of the survey interview challenging both in terms of time and location but also in terms of motivation to participate.This study presents numerous strengths. Even though a convenience sample will be recruited, the recruitment will be from a variety of settings to increase the heterogeneity of the sample in terms of their sociodemographic characteristics, as well as the variables of interest. Furthermore, the eHeals questionnaire will be adapted to the needs of carers, and the HLS-EU-Q16 will be used and validated for the first time in this specific population. More importantly, the study will assess the level of health and eHealth literacy of Greek and Cypriot carers of people with dementia for the first time, as well as explore the role of these constructs in the caregiving process. This has important implications about the services provided.Moreover, screening tools will be available to measure health and eHealth literacy levels for this specific population, and future research on eHealth literacy training of carers in Greece and Cyprus will follow.ConclusionsTaking into consideration the fast technological progress, the demand for Web-based training and eHealth literacy training is only a matter of time. More and more resources are being developed to support carers on the Web, and the use and assessment of this type of technologies by carers are becoming essential skills that in future years will become obligatory. Focusing on training and developing, training classes and e-learning courses could facilitate the development of these specific skills among this population. Furthermore, the usage of new technologies and the Internet could act as a facilitator in the caregiving demands of carers.


Some outcomes for children with mental health problems remain suboptimal because of poor access to care and the failure of systems and providers to adopt established quality improvement strategies and interventions with proven effectiveness. This review had three goals: (1) assess the effectiveness of quality improvement, implementation, and dissemination strategies intended to improve the mental health care of children and adolescents; (2) examine harms associated with these strategies; and (3) determine whether effectiveness or harms differ for subgroups based on system, organizational, practitioner, or patient characteristics.Sources included MEDLINE®, the Cochrane Library, PsycINFO, and CINAHL, from database inception through February 17, 2017. Additional sources included gray literature, additional studies from reference lists, and technical experts. Two reviewers selected relevant randomized controlled trials (RCTs) and observational studies, extracted data, and assessed risk of bias. Dual analysis, synthesis, and grading of the strength of evidence for each outcome followed for studies meeting inclusion criteria. We also used qualitative comparative analysis to examine relationships between combinations of strategy components and improvements in outcomes.We identified 18 strategies described in 19 studies. Eleven strategies significantly improved at least one measure of intermediate outcomes, final health outcomes, or resource use. Moderate strength of evidence (from one RCT) supported using provider financial incentives such as pay for performance to improve the competence with which practitioners can implement evidence-based practices (EBPs). We found inconsistent evidence involving strategies with educational meetings, materials, and outreach; programs appeared to be successful in combination with reminders or providing practitioners with newly collected clinical information. We also found low strength of evidence for no benefit for initiatives that included only educational materials or meetings (or both), or only educational materials and outreach components. Evidence was insufficient to draw conclusions on harms and moderators of interventions.Several strategies can improve both intermediate and final health outcomes and resource use. This complex and heterogeneous body of evidence does not permit us to have a high degree of confidence about the efficacy of any one strategy because we generally found only a single study testing each strategy.CRD42015024759.PROSPERO,  Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV) criteria in any given year [Approximately one in five children and adolescents living in the USA has one or more mental, emotional, or behavioral health disorders according to the ven year . These dven year .Several key publications in the mid- to late-1990s suggested that usual care in children’s mental health had, at best, no  and someDespite advances in the evidence base , 7, someGiven the gap between observed and achievable processes and outcomes, one way to improve the mental health care of children and adolescents is to adopt QI practices; another is to develop strategies to implement or disseminate interventions with known effectiveness. Such strategies target changes in the organization and delivery of mental health services , 14. TheSome investigators consider implementation and dissemination strategies as a particular subset of initiatives to improve the quality of care. However, the field of implementation and dissemination is so new that the conceptual framework and terminology in relationship to QI efforts have not been fully standardized yet . We do nWe developed the topic, key questions (KQs), outcome list, and analytic framework for this systematic review through a comprehensive, evidence-based, and public process. We refined review criteria used by two recent reviews , 25 and http://effectivehealthcare.ahrq.gov/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productid=2030 on December 30, 2014, and registered on PROSPERO (Registration number: CRD42015024759). Following release of a draft of the systematic review and peer review, we amended our protocol to include additional review and analysis methods suitable for complex interventions , implementation, and dissemination strategies employed in outpatient settings by health care practitioners, organizations, or systems that care for children and adolescents with mental health problems to improve:KQ 2: What are the harms of these mental health strategies?KQ 3: Do characteristics of the child or adolescent or contextual factors  modify the effectiveness or harms of strategies to improve mental health care and, if so, how?Our key questions included the following:Figure KQ 3 was intended to evaluate the effect of moderators on any outcomes reported in KQ 1  or KQ 2 . Because the effect of the moderator may vary based on the nature of the outcome, our planned analyses did not combine categories of outcomes.Methods Guide for Effectiveness and Comparative Effectiveness Reviews from AHRQ . The review uses the PRISMA  checklist to facilitate the preparation and reporting of the systematic review [We followed procedures specified in the c review .We searched MEDLINE® for eligible interventions using a combination of Medical Subject Headings (MeSH), abstract keywords, and words or phrases found in the title of the paper, limiting the search to human-only studies, from database inception through February 17, 2017. We also searched the Cochrane Library, PsycINFO, and CINAHL  using analogous search terms. The full report  providesClinicalTrials.gov, the World Health Organization’s International Clinical Trials Registry Platform, the National Institutes of Health Research Portfolio Online Reporting Tools, the Database of Promoting Health Effectiveness Reviews, and CMS.gov. To avoid retrieval bias, we manually searched the reference lists of landmark studies and background articles on this topic to look for any relevant citations that our electronic searches might have missed.In addition, we searched the gray literature for eligible studies. Sources of gray literature include www.nrepp.samhsa.gov). These criteria specify that the intervention needs to have produced one or more positive behavioral outcomes in at least one study using an experimental or quasi-experimental design with results published in a peer-reviewed journal or similar publication. In addition, implementation materials, training and support resources, and quality assurance procedures for these interventions need to be ready for use.We specified our inclusion and exclusion criteria based on the patient populations, interventions, comparators, outcomes, timing of outcomes assessment, and setting (PICOTS) we identified as relevant during our scope development through the topic refinement process Table . We inclWe use the term “strategy” to mean the approach used to target health care systems, practitioners, or both to improve the quality of care for children and adolescents with mental health problems. We use the term “intervention” to denote a specific EBP used as part of a strategy.Because strategies tended to be complex in nature and the number and types of components that varied between the treatment arm and comparison group arm differed by study, we also recorded components of each strategy. We relied on the Cochrane Review Group’s Effective Practice and Organization of Care (EPOC) Group taxonomy, which categorizes strategies by whether they include one or more professional, financial, organizational, and regulatory components .Because many comparison group strategies also had several components, we noted and compared the components in each study arm. This allowed us to describe fully the numerous components that were being combined and tested in each strategy; it also enabled us to determine whether the study arms differed by a single or multiple components.We required each included study to report at least one intermediate outcome in a minimum of one of three major categories: (1) practitioner intermediate outcomes , (2) system intermediate outcomes , and (3) patient intermediate outcomes . Harms of interest included those at the patient, provider, and/or system that are potentially associated with the strategies themselves.As noted earlier, the choice of outcomes in the review was based on existing evidence and theory , feedbacFor all KQs, we excluded study designs without comparison groups to ensure that our pool of included studies provided strong evidence on the causal link between the strategy and outcomes. We excluded studies in which the strategy  and the intervention being tested both differed between groups, because the effectiveness of the QI, implementation, or dissemination strategy could not be isolated from the baseline intervention effects.Our exclusion of non-English-language studies is based on limitations of time and resources. However, we examined English language abstracts of non-English-language studies to assess the potential size of the literature that we might otherwise miss through this approach.To begin to understand salient contextual factors and guide our analyses of effect modifiers, we applied the Consolidated Framework for Implementation Research (CFIR)  to reseaMethods Guide [To assess the risk of bias  of studies, two independent reviewers used predefined, design-specific criteria based on guidance in the ds Guide . We resods Guide . We asseds Guide  and A Cods Guide . MinimumIn general terms, a study with no identifiable flaws has a low risk of bias. A study with medium risk of bias is susceptible to some bias but probably not sufficient to invalidate its results. A study with high risk of bias has significant methodological flaws  that may invalidate its results. We considered the risk of bias for each relevant outcome of a study. When studies did not report sufficient detail to assess the validity of the design or study conduct, we judged the risk of bias to be unclear.Trained reviewers abstracted important information from included studies into evidence tables, which were uploaded to AHRQ’s Systematic Review Data Repository. A second senior member of the team reviewed all data abstractions for completeness and accuracy. Reviewers resolved conflicts by discussion and consensus or by consulting a third member of the review team.To determine whether quantitative analyses were appropriate, we assessed the clinical and methodological heterogeneity of the studies under consideration following established guidance . For allWe employed several other methods to provide additional information about the nature of the strategies tested and what components of the strategies had the most impact on outcomes. First, we performed additional search approaches of related publications (known as “cluster searching”) to identify sibling (multiple publications on the same study) or kinship studies  . We hopeMethods Guide [We graded the strength of a body of evidence based on the updated guidance in the ds Guide , 43. The. Senior members of the review team graded the strength of evidence.Two reviewers assessed each domain for each key outcome and resolved any differences by consensus discussionGrades reflect the confidence that the reviewers have that various estimates of effect are close to true effects with respect to the KQs in a systematic review. A high grade indicates that we are very confident that the estimate of effect lies close to the true effect for this outcome. A moderate grade indicates moderate confidence that estimate of effect lies close to the true effect for this outcome and the body of evidence has some deficiencies. A low grade indicates limited confidence that the estimate of effect lies close to the true effect for this outcome. The body of evidence has major or numerous deficiencies. A grade of “insufficient” applies when we have no evidence, we are unable to estimate an effect, or we have no confidence in the estimate of effect for this outcome .Risk of bias assessments for individual studies feed into the rating for the first strength of evidence domain, study limitations. Specifically, we rated bodies of evidence comprising trials with a high risk of bias as having high study limitations. Medium or unclear risk of bias studies resulted in medium study limitations. Low risk of bias studies resulted in low study limitations. In keeping with GRADE and strength of evidence guidance, we rated observational studies as having high study limitations , 44.As described above, study design and study limitations together set the baseline strength of evidence grade. Other domains  then could either reduce or increase the grade. A body of evidence with high study limitations, with no other reasons to increase confidence  or decrease it , would generally have a low strength of evidence grade. A body of evidence with low study limitations, with no reasons to decrease confidence , would generally have a high strength of evidence grade.For each source of uncertainty, we consistently used the following rubric to evaluate its effect on the overall strength of evidence across outcomes. Specifically, for indirectness, we rated intermediate outcomes as direct, rather than indirect, evidence. Typically, strength of evidence grading systems will downgrade all evidence from intermediate outcomes as a matter of course. For this review, such an approach would have penalized strategies from studies whose intent was to examine implementation outcomes.For this systematic review, these outcomes can be interpreted as direct measures of process change. Regarding inconsistency, we rated it as unknown for bodies of evidence with single studies; the rating of unknown consistency did not lower the overall grade. We relied on established guidance to judge imprecision . RegardiWe summarize results by KQ below for searches through February 17, 2017. The full report  providesFigure http://srdr.ahrq.gov/projects/530.This evidence base for KQ 1 consisted of 19 studies , 46–62. Table Improved practitioner adherence to EBPs or guidelines: by training practitioners to monitor metabolic markers , providiImproved practitioner morale, engagement, and stress: by implementing a program to enhance organizational climate and culture Improved patient access to care, parent satisfaction, treatment engagement, and therapeutic alliance: by training nurses to educate parents about EBPs Improved patient functional status: by giving practitioners weekly feedback on patient symptoms and functioning Improved service utilization: by training practitioners about monitoring medications  and apprThe strongest evidence in the review (for KQ 1) comes from a study of pay for performance. Therapists in the pay for performance group were more than twice as likely to demonstrate implementation competence  . In thisA strategy to test augmented active learning versus computerized routine learning versus routine practitioner workshop to implement an EBP A collaborative consultation treatment service to promote the use of titration trials and periodic monitoring during medication management versus control An Intensive Quality Assurance system versus workshop to implement an EBP intervention Use of additional computerized assisted training or computerized training plus supervisory support to implement an EBP versus using a workshop and resources only A Contextualized Feedback Systems trial to test the provision of session-by-session clinician feedback versus clinician feedback provided after 6 months of visitsA strategy to test the implementation of a school-based EBP by school counselors versus psychologistsSix strategies (one study each) consistently provided insufficient evidence or evidence of no benefit across all reported outcomes. These included:The absence of evidence on several factors of interest further limited our conclusions. We found no evidence of studies examining several intermediate outcomes, particularly system-level intermediate outcomes. In addition, we identified no studies that measured final patient health outcomes such as co-occurring conditions. We also found no evidence of strategies testing several components of the EPOC taxonomy, including any regulatory components, and little evidence on strategies with financial components.Of the 19 studies in our review, we rated risk of bias as follows: low, one study; medium, five studies; high, six studies; and unclear, seven studies. Various issues with study design, attrition, and incomplete information reported by study authors explain the risk of bias issues for high and unclear ratings.The uncertain or high risk of bias of most of these studies affected the overall strength of evidence grades, as did the fact that generally we had only single studies for each strategy examined.Only one study evaluated any harms (KQ 2), in this case associated with professional training to identify and refer cases to early-intervention services for untreated first-episode cases of psychosis . The stuOverall, we found evidence on four strategies that examined moderators of the effectiveness of strategies to improve mental health care for children and adolescents (KQ 3). Three examined whether training intensity influenced the degree of effectiveness; of these, two strategies were graded as having insufficient strength of evidence. The third strategy had low strength of evidence for benefit for patient intermediate outcomes (access to care) and patient health outcomes .A fourth study examined the moderating effects of fidelity to the EBP (A-CRA) in therapists meeting individual patient targets specified in the A-CRA program. We graded this evidence about fidelity as low strength for no benefit on patient health outcomes and patient remission status.We did not find studies that examined most of our previously specified list of moderators—namely, as patient characteristics, intervention characteristics other than training intensity, outer context, inner context, characteristics of involved individuals, process characteristics other than fidelity to the training, or other moderators such as length of follow-up.To understand better what combinations of components (“condition sets”) might serve as solutions or “recipes for success”, we turned to QCA. We examined several different models that had different combinations of intervention components; we tested two different outcomes. We chose the model that best fit our data with the highest level of consistency (proportion of solutions resulting in success or outcome) and coverage (proportion of observations explained by the solutions). Our model included the presence or absence of several professional components , any financial component, organizational structural-oriented components , and organizational provider-oriented component . We defined success as having a statistically significant improvement in either a majority of practitioner-, system-, and patient-level intermediate outcomes or at least one patient health or service utilization outcome with at least low strength of evidence for benefit Fig. .Fig. 3QuOur analysis included 19 studies; 9 of 18 studies showed at least low strength of evidence of benefit and significant improvements in the majority of practitioner, system, or patient intermediate outcomes tested (one study did not test any intermediate outcomes). An additional 3 of the 10 remaining studies showed at least low strength of evidence of benefit for at least one patient health or service utilization outcome. Seven studies did not meet either of these benefit criteria.Having educational materials or meetings, educational outreach, and patient-mediated intervention componentsHaving educational materials or meetings, educational outreach, and reminder componentsHaving educational materials or meetings, patient-mediated intervention components, reminders, and organizational structural components not educational materials or meetings, not audit and feedback, not organizational structural components, and not organizational provider components  , 49, 56A patient-mediated component  , 55, 56Enhanced referrals and patient choice of treatment Overall, 12 of the 19 studies demonstrated effectiveness as measured by moderate or mainly low strength of evidence for benefit for at least one outcome of interest. Our confidence in these results is limited by the paucity of studies on any strategy. We graded benefits of pay for performance as moderate strength of evidence . We gradEducational materials and/or educational meeting components only  , 59Educational materials and outreach components only  , 57We found low strength of evidence of no benefit for intermediate outcomes for strategies that involved the following combinations of professional components:We were unable to judge the potential for harms associated with these strategies that might mitigate benefits. We had only a single study on early intervention for first-episode psychosis, and it reported no adverse events and no differences in false-positive referral rates. In addition, the available evidence from four studies on two moderators does not permit us to make general conclusions about the conditions under which these strategies might work optimally.The studies varied with respect to the numbers and types of active components, i.e., we observed considerable differences in components in treatment group strategies and comparison group strategies. In some studies, the treatment group contained several components and the comparison group contained none of those components. In other studies, both the treatment and comparison groups tested strategies with multiple components, with different numbers of variations in components across arms. Because both arms often received active interventions, the Hawthorne effect may explain lack of effectiveness. We did not find any consistent patterns of effectiveness involving the number of active components. That is, we did not find that studies that employed strategies with a single active component had any better or any worse effect on outcomes than those that employed multiple active components.Additional heterogeneity arose from several other sources and precluded any quantitative synthesis of our findings. Except for two studies reported in one publication , two triChallenges in this systematic review arose with defining the intervention of interest, constructing the search strategy, and applying prespecified inclusion/exclusion criteria. The lack of consistency in the terminology used in the published literature meant that the use of self-selected descriptors such as “QI,” “implementation,” or “dissemination” by study authors did not conform either to our a priori definitions of these types of studies or to the other similarly labeled studies in the field. This lack of consistency led to our reliance on the EPOC taxonomy as our primary analytic framework.Strategies differed considerably in the number of components; the reporting on these components was not always clear enough either to describe the strategy adequately or to let us understand fully the relative importance of component parts. New taxonomies are continually emerging, such as the revised EPOC taxonomy  and the existing mental health issues  meant we could not examine issues of prevention. In addition, although we included a broad range of eligible comparators in our protocol , we did encounter otherwise eligible studies in which the intervention combined both a patient-level intervention and a system-level strategy to implement or disseminate that intervention. Because the use of a “usual-care” arm did not permit the original investigators to draw conclusions about the effect of their implementation or dissemination strategy apart from the underlying intervention, we excluded these studies for having a wrong comparator [Trying to specify the population and comparison criteria to ensure greater homogeneity of included interventions posed additional challenges. For example, our focus on children and adolescents with mparator –72. In aReporting issues in the literature also hindered our ability to derive firm conclusions on the effectiveness of included strategies. Authors reported complex analyses but often did not report other aspects of their methods well enough to permit an independent evaluation of the effect size , precisiWe did not find evidence on the majority of the outcomes that we had specified a priori. Of particular note, seven strategies (two from a single publication) relied on EBPs and did not report patient health outcomes , 58, 59.When researchers maintain fidelity to the original intervention, assuming that the same level of effectiveness will occur in a new trial is reasonable; adopting this assumption can, therefore, produce efficient use of research funds. Unfortunately, not all studies measured fidelity adequately. New strategies relying on EBPs must, at a minimum, report on fidelity so that practitioners and policymakers can judge whether the strategy is, in fact, a new intervention rather than implementation or dissemination of an existing intervention. Information on pragmatic issues related to implementation  may not necessarily require new studies on strategies with existing information; support of analyses done with data from existing studies may fill some of the gap.The sparse evidence base of a set of diverse strategies and outcomes focusing on intermediate and health outcomes and resource use highlights the fact that clinicians and health plan administrators still do not have adequate knowledge of best methods to introduce EBPs successfully into clinical settings for children and adolescent populations. Third-party payers are paying increasing attention to quality metrics, as health care systems move to accountable care models. We found no studies on regulatory components and just one study testing the effectiveness of a financial component, specifically for pay for performance . Future The majority of included studies appropriately used cluster designs. Cluster RCTs, like pragmatic trials, typically need more resources than conventional RCTs. Moreover, they can be harder to analyze than conventional studies. An additional consideration  related to inadequate reporting, a problem often noted in the literature , 74. TheAlthough the failure to use EBPs can lead to gaps between potential and achieved outcomes, closing such holes in the knowledge base requires more than just using an array of EBPs. What continues to be unknown is how to bridge the gap in the context of the finite resource of time allocated for a patient encounter and what implications changes will have on current work flow processes. As expectations increase for documenting or checking off quality metrics for each action within a patient encounter, the risks of errors of both omission and commission rise. For new information to be actionable, more persuasive evidence is needed on the relative merits of each action or strategy.More research is needed on strategies for the QI, implementation, and dissemination of EBPs relating to both psychological and behavioral treatments and medication interventions for treating youth suffering from mental illness. Other important research targets include developing and testing dissemination strategies for introducing mental health care into geographic areas lacking in mental health care, such as very rural areas with fewer mental health providers than more urban locations have. In these areas especially, focusing more attention on primary care providers may be essential.no benefit for initiatives that included only educational materials or meetings (or both) or only organizational provider components.The evidence does not give us a high degree of confidence about the efficacy of any one strategy to improve the implementation, dissemination, or quality of mental health care for children and adolescents, generally because we had only a single study testing a given strategy. Although we had insufficient or low strength of evidence of no benefit for more than half of the outcomes that we evaluated, our findings suggest that several approaches can improve both intermediate and final health outcomes and resource use. Of the 19 included studies, 12 significantly improved at least one such outcome or measure. Moderate strength of evidence (from one RCT) supported using provider financial incentives such as pay for performance to improve the competence with which practitioners can implement EBPs. We found inconsistent evidence involving strategies with educational meetings or materials; programs appeared to be successful in combination with outreach, reminders, providing practitioners with newly collected clinical information, or organizational structural components including quality monitoring and changing the scope or nature of benefits. We also found low strength of evidence for


An objective measure of nutrition literacy is unavailable for use in the primary care population. The Nutrition Literacy Assessment instrument (NLit) is a tool designed to measure nutrition literacy across six domains and has been previously piloted in breast cancer and parent populations. The purpose of this research was to engage nutrition experts and patients to guide revisions of the NLit for use in adult primary care.n = 5) reviewed each item in the NLit using a survey to assign rankings of their agreement according to relevance, clarity, and reading difficulty. Relevance rankings were used to calculate Scale Content Validity Index. After suggested revisions were made, patients (n = 12) were recruited from urban primary care clinics of a University Medical Center located in the Midwestern United States and were interviewed by trained researchers using the cognitive interview approach to generate thoughts, feelings, and ideas regarding NLit items. Data analysis involved qualitative and quantitative methods.Experts  contains supplementary material, which is available to authorized users. Health Literacy, or “the degree to which individuals have the capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions”  is a skiNutrition education research has produced little evidence base to inform best practices for improving health literacy with regard to nutrition information, despite the critical importance of nutrition for preventing and treating chronic disease. Nutrition is a major underlying factor in both the development and treatment of diabetes , hyperteThe ability of individuals to navigate nutrition-related information is a critical component of nutrition literacy. However, health literacy assessments lack a nutrition focus and generally only identify print literacy and/or numeracy in the context of health care . AlthougWe took several steps in an effort to address this gap in instrumentation. In previously reported work, we developed an instrument specific to nutrition literacy, based upon input from experienced nutrition educators and registered dietitians , pilot-tThe purpose of this study was to build upon our previous work by revising the Nutrition Literacy Assessment Instrument (NLit) for use in the primary care setting with patients who have nutrition-related chronic disease. To this end, we tested the hypothesis that the revised instrument (NLit) is content valid and clearly understood by the general healthcare population.We engaged two audiences to guide revisions of the NLit: 1) nutrition experts, and 2) primary care patients. The Human Subjects Committee of the Institutional Review Board approved all methods for this study with expedited review (HSC# 13805). All data were collected between August 2014 and December 2014.The NLit includes measures of print literacy and numeracy, similar to health literacy assessments, while also including measures of nutrition knowledge and skills that nutrition educators identified were needed for following a healthy diet. Prior to content review, the NLit was expanded from five domains totaling 40 items  to six dn = 1) and current or recent experience with nutrition education (n = 4). The four content experts, who are published academics with clinical nutrition experience, evaluated each item in the expanded pool for relevance to the content domain, clarity, and reading difficulty using a survey to assign rankings of their agreement .” Similar statements were made by other participants. While many of the participants rated this section as “very easy” or “easy,” the frequency of incorrect answers for this domain was highest compared to the other domains.No major concerns were uncovered for the Food Label & Numeracy domain. Participants identified the difficulty of performing some of the calculations as the main challenge. The last question  was the most difficult for participants (or most often answered incorrectly), and this involved calculating a percentage.The fewest interviews were completed on the Food Groups domain because previous testing of these items during our breast cancer patient pilot indicated familiarity with all food items. Most interesting were requests for more specificity with item descriptions, such as “is the rice white or brown?” and “is the tortilla a corn tortilla or flour tortilla?” Also, because the 'Added Sugars' category was added as a result of expert suggestion, researchers explored the lemonade and fruit punch items more than others. Lemonade was confused with the fruits group by 2/3 participants, both noting that it depends on the type of lemonade. One classified “fresh lemonade” as a fruit and “bottled lemonade” as added sugars, while frozen lemonade “would just depend on the brand.”No major concerns were identified with the electronic version of the Consumer Skills domain, though one participant referencing the paper version found the formatting problematic. He noted that he automatically disregarded option C  because it appeared to him as a statement, and it did not have a food photo underneath this answer option similar to other potential answers.As a result of content review and cognitive interviews, the NLit was reduced to 66 items. In total, 17 items were modified, five items were deleted, instructions for two domains were modified, and the formatting of question presentation was modified in one domain. There were some themes that arose in which no revision was made because the research team concluded from the interviews and review of transcripts that items missed were due to inaccuracies in nutrition knowledge, and not due to misunderstanding the intent of the questions and/or answer options.The NLit is the first instrument designed to assess the nutrition literacy of adults in the primary care setting. Individual item review by nutrition and psychometric experts provided confirmation of content validity while also aiding in revisions to increase clarity. Involvement of patients from the target population using the cognitive interviewing approach largely assured researchers that the instrument was understood as intended, while also identifying potential confusing language that those interviewed were able to help rewrite for clarity.In some cases, those who struggled with items often demonstrated lesser understanding of nutrition information. For example, the inabilities of some to acknowledge that homemade lemonade contains sugar and yet classify it as a fruit rather than an added sugar may represent a nutrition literacy problem. This could be important considering the potential ramifications of excessive intake of sugar sweetened beverages in the US population .Assessing skills with portion sizing, represented by the Household Food Measurement domain in the NLit, proved difficult throughout the revision process. Research demonstrates that extensive training is required for accurate estimation of portion size , and eveFurther complicating matters, sources providing portion size recommendations are inconsistent. For example, a portion of rice (1/3 cup) recommended by the American Diabetes Association is different than a portion (1/2 cup) according to the USDA . The FDAAn important limitation of this study is the time intensive nature of cognitive interviewing, which limits the sample size. As noted by Willis, the goal of sampling in cognitive interviewing, is not a large sample size, but to include a variety of individuals who are believed to represent the target population of the survey to be tested . AlthougAs a result of revisions made to the NLit, courtesy of expert and patient review, the NLit is both content valid and understood by a sample of the primary care patient population. These steps are necessary and helpful in producing an instrument that achieves its target constructs and is understood as intended by the target population. Although further testing is required to establish validity and reliability, review by these important audiences increases the likelihood that the final instrument will accurately identify nutrition literacy issues rather than difficulties with navigating the instrument itself.Additional file 1:Transcribed Cognitive Interviews. (DOCX 161 kb)


This is the sixth in a series of papers reporting Sustainability in Health care by Allocating Resources Effectively (SHARE) in a local healthcare setting. The SHARE program was established to investigate a systematic, integrated, evidence-based approach to disinvestment within a large Australian health service. This paper describes the methods employed in undertaking pilot disinvestment projects. It draws a number of lessons regarding the strengths and weaknesses of these methods; particularly regarding the crucial first step of identifying targets for disinvestment.Literature reviews, survey, interviews, consultation and workshops were used to capture and process the relevant information. A theoretical framework was adapted for evaluation and explication of disinvestment projects, including a taxonomy for the determinants of effectiveness, process of change and outcome measures. Implementation, evaluation and costing plans were developed.Four literature reviews were completed, surveys were received from 15 external experts, 65 interviews were conducted, 18 senior decision-makers attended a data gathering workshop, 22 experts and local informants were consulted, and four decision-making workshops were undertaken. Mechanisms to identify disinvestment targets and criteria for prioritisation and decision-making were investigated. A catalogue containing 184 evidence-based opportunities for disinvestment and an algorithm to identify disinvestment projects were developed. An Expression of Interest process identified two potential disinvestment projects. Seventeen additional projects were proposed through a non-systematic nomination process. Four of the 19 proposals were selected as pilot projects but only one reached the implementation stage. Factors with potential influence on the outcomes of disinvestment projects are discussed and barriers and enablers in the pilot projects are summarised.This study provides an in-depth insight into the experience of disinvestment in one local healthcare service. To our knowledge, this is the first paper to report the process of disinvestment from identification, through prioritisation and decision-making, to implementation and evaluation, and finally explication of the processes and outcomes.The online version of this article (doi:10.1186/s12913-017-2269-1) contains supplementary material, which is available to authorized users. This is the sixth in a series of papers reporting Sustainability in Health care by Allocating Resources Effectively (SHARE). The SHARE Program is an investigation of concepts, opportunities, methods and implications for evidence-based investment and disinvestment in health technologies and clinical practices in a local healthcare setting. The papers in this series are targeted at clinicians, managers, policy makers, health service researchers and implementation scientists working in this context. This paper reports the exploration of methods to identify health technologies and clinical practices suitable for disinvestment; establish prioritisation and decision-making processes; and develop, implement and evaluate evidence-based disinvestment projects.The need for disinvestment has emerged in response to increasing costs and a growing awareness of ineffective practices and systemic waste in healthcare services. Although there is no clear single definition, disinvestment is generally understood to be removal, reduction or restriction of health technologies and clinical practices (TCPs) that are unsafe or of little benefit, seeking to improve patient outcomes and use available resources more efficiently .Following successful implementation of a rigorous evidence-based program for introduction of new TCPs , leadersThe ‘Sustainability in Health care by Allocating Resources Effectively’ (SHARE) Program was established to investigate an organisation-wide, systematic, transparent, integrated, evidence-based approach to disinvestment. The SHARE Program was funded as a 3-year demonstration project by the Victorian Department of Human Services (DHS) and Monash Health, and was undertaken by the Centre for Clinical Effectiveness (CCE), an in-house resource to facilitate Evidence Based Practice (EBP). An overview of the SHARE Program, a guide to the SHARE publications and further details about Monash Health and CCE are provided in the first paper in this series .Following preliminary investigations to understand the concepts related to disinvestment, identify current decision-making practices at Monash Health, learn from local experiences of disinvestment and consider the implications of the proposed changes, a plan for the SHARE Program was created . This inThe first aim of the SHARE Program was to explore systems and processes for decision-making that could proactively and systematically identify opportunities for disinvestment. The second aim was to investigate pilot disinvestment projects to gain detailed insight into the change processes involved, assess the resources required to deliver effective projects, provide practical guidance for future projects and, if successful, be used as positive examples to promote subsequent disinvestment activities.The preliminary work also identified that if the first two aims were to be achieved, services to support the proposed activities and build staff capacity would be required . Four suThe need to investigate methods to identify and prioritise potential target TCPs and undertake projects to disinvest them is noted in the literature , 17–24. The aim of this aspect of the SHARE Program was to undertake disinvestment pilot projects. This would be achieved via three objectives: identifying potential disinvestment opportunities; establishing prioritisation and decision-making processes; and developing, implementing and evaluating disinvestment projects.The aim of this paper is to describe, explore and explain the processes and outcomes of undertaking these objectives and the factors that influenced them.What methods are available to identify potential disinvestment opportunities in a local health service?What methods are available for prioritisation and decision-making to initiate disinvestment projects in a local health service?What methods are available to develop, implement and evaluate disinvestment projects in a local health service?What were the processes and outcomes of application of these methods at Monash Health?What factors influenced the decisions, processes and outcomes?The SHARE papers use a case study approach to address the limited understanding of resource allocation processes in health services, particularly regarding disinvestment , 21, andDescriptive: findings are reported in detail to describe events, processes and outcomes to enable replication when successful and avoidance or adaptation when unsuccessfulExploratory: literature reviews, surveys, interviews, workshops and consultation with experts are used to explore what is known and identify actual, preferred and ideal practicesExplanatory: theoretical frameworks are used to understand and explain the events, processes and outcomesAll three case study approaches are used .DescriptCase studies are characterised by multiple sources of quantitative and qualitative evidence . An overThe SHARE Program was undertaken using the SEAchange model for Sustainable, Effective and Appropriate change in health services . The modAction research was undertaken based on the ‘researcher as facilitator for change’ model defined by Meyer: researchers working explicitly with and for people rather than undertaking research on them , 33. In a priori. Methods developed during the course of the investigation are reported in the Several of the activities reported in this paper were to develop methods that would be undertaken in subsequent activities. The methods reported in this section are those determined Mixed methods were used to capture frameworks, methods and tools, and stakeholder perspectives and experiences. These included literature reviews, a survey, interviews, workshops, consultations, and document analysis. Participant validation for factual accuracy was undertaken following interviews and workshops. An overview is provided in Fig. a priori. Details of individual project protocols are provided in Additional file Outcomes of consultations and findings from initial interviews with small numbers of participants were documented and collated using MS Word or Excel. Workshop and subsequent interview findings were collated in MS Word, Excel and/or Nvivo  and analUsing the principles of evidence-based change, the SHARE team worked with stakeholders and external experts to synthesise the findings from the literature and local research into discussion papers and workshop presentations.Decisions were made by the SHARE Steering Committee composed of executive directors, committee chairs, clinical program directors, legal counsel, support service managers and consumer representatives , impact (Did this achieve a change in practice?) and outcome . These were not all undertaken due to reduced funding in the final year of the program.Training in evaluation methods was provided to the pilot project teams through the Capacity Building Service and assistance in data collection and analysis was provided through the Project Support Service.Factors that influenced outcomes of the piloting process were identified using a framework for evaluation and explication of evidence-based innovations . Based oSome of the planned implementation and evaluation activities were not completed due to reduction of funding in the final year by the program funder and changes in requirements for the pilot project by the project funder; details and impact are discussed below.Results of the literature reviews and the response rates and representativeness of participants in the survey, interviews and workshops are included in Additional file Data collected from these activities informed a range of research questions. Findings related to the research questions in this paper are presented and discussed below; findings related to topics not addressed here are reported in other SHARE publications , 38–40.Although Monash Health staff were not aware of the term ‘disinvestment’, they were familiar with the concept of removal, reduction or restriction of current practices. Surveys and interviews with a range of decision-makers and project staff who had undertaken these and other resource allocation activities provided details of strengths, weaknesses, barriers and enablers in these processes. These have been combined into positive and negative influences to remove duplication; they are collated in Table The investigation of potential methods for identification, prioritisation and decision-making, and implementation and evaluation of disinvestment projects are summarised in Fig. Seven methods to identify disinvestment opportunities in a local health service were investigated. The focus of Aim 1 was to explore methods that could be integrated into organisational infrastructure for systematic consideration of disinvestment in routine health service decisions. Six potential mechanisms were identified  meetings regarding introduction of new TCPs, clarification of authorisation processes for new equipment or consumables prior to purchase, and inclusion of a CCE staff member on the Clinical Purchasing Committee to facilitate evidence-based decision-making. However no changes regarding identification of opportunities for disinvestment were implemented. The Purchasing Policy Guidelines were due for routine review and those responsible welcomed participation of the SHARE team to address these issues; however the review was not undertaken during the life of the SHARE Program.DiscussionThere are discussions in the current literature about smart, innovative and evidence-based purchasing , 42 and In addition to processes that allocate funding, systematic mechanisms for allocating non-monetary resources were also recognised by repondents as potential methods to identify disinvestment opportunities. Local guidelines and protocols determine allocation of resources for specific conditions, patient groups or clinical procedures by stipulating use of drugs or equipment, recommending diagnostic tests, selecting health professional groups, prioritising staff time, specifying referral mechanisms and allocating capacity in clinics, operating rooms and other facilities. There are potential opportunities for disinvestment in all of these activities. Prompts, triggers and mandatory requirements to consider disinvestment could be introduced into document development and authorisation processes. Requirements for local guidance to be based on the best available evidence would ensure that harmful, ineffective or inefficient TCPs would be identified in the systematic review process and steps to discontinue these practices could be included in the resulting guidance document. Evaluation, audit and review of guidelines and protocols may also identify opportunities for disinvestment. Mechanisms involving local guidelines and protocols could be implemented quickly and, once established, delivered with no additional costs.The CCE staff members involved in SHARE were simultaneously developing a new Policy and Procedure Framework for Monash Health. No examples of using local guideline and protocol development to identify disinvestment opportunities were identified from the literature or local consultations in this process.“If the procedure involves introduction of new practices, identify the current practices that are being replaced. Cessation or restriction of specific activities in current practice must be addressed with active interventions in the same way as introduction of new practices.” [A prompt to consider whether any current practices could be discontinued was included in the instructions to developers of guidance documents. ctices.” . A requiAfter developing the new framework, CCE staff handed it over to the department that had responsibility for organisational documents for implementation and ongoing governance. The disinvestment prompts and requirement for systematic reviews, along with other instructions, were removed by the implementers with the intention of making the process less onerous for document developers.DiscussionSeveral authors refer to the potential to use guidelines for implementation of disinvestment recommendations –49 but wScoping searches of the health databases in preparation for the literature review revealed a growing body of evidence about practices that are harmful, of little or no clinical benefit, or where a more effective or cost-effective alternative is available. Searches for evidence-based disinvestment opportunities could be undertaken and the findings delivered directly to decision-makers. Workshops with the Steering Committee determined that to avoid wasting time and resources considering information that does not represent the best available evidence, only high quality synthesised information such as systematic reviews, health technology assessments and evidence-based guidelines should be used proactively to drive decisions.It was clear from interviews with decision-makers that Monash Health had no mechanisms to use research evidence proactively. The SHARE team developed a catalogue of disinvestment opportunities to enable this  process where health service staff nominated their own projects could potentially provide quick results.Monash Health staff were invited to submit applications to receive training and support from the SHARE Program for a disinvestment project. An EOI form was drafted to include criteria agreed by the SHARE Steering Committee. To facilitate completion of the new document, the content and format of existing Monash Health templates familiar to the applicants were adapted to address the EOI requirements. A disinvestment project was defined as one that removes a TCP that is unsafe or ineffective, restricts a TCP to more appropriate patient groups, or replaces a TCP with an equally safe and effective but more cost-effective option. Training in implementation and evaluation methods was provided by the Capacity Building Service. Support available from the Project Support Service included administration; project planning and implementation advice including analysis of barriers and enablers; evaluation advice including establishing systems to monitor and evaluate change and identify sources of data; and economic evaluation or cost comparison study . Clinical trials and projects already underway were excluded.Invitations to submit an EOI were distributed via the Clinical Program Directors. Two applications were received.DiscussionThree more-recently published frameworks for disinvestment also propose applications from stakeholders in the identification process –97; howead hoc proposals for potential disinvestment projects were received. At each meeting, members of the Steering Committee nominated TCPs which the SHARE team were asked to investigate. This process was given priority over development of criteria to ascertain suitable TCPs from the catalogue of evidence-based project opportunities. Each proposed TCP had one or more attributes that made it seem promising, but no assessment using explicit criteria was undertaken. Seventeen TCPs were nominated in this way.Many Including the two EOIs, 19 TCPs were investigated as potential pilot disinvestment projects. The nature of the change and reason for nomination are summarised in Table Discussionsoft intelligence” [Proposals based on individual’s observations or local knowledge have been referred to as “ligence” ; this haligence” , 23, 99.The priority setting exercises described above clearly include a prioritisation process, however initiatives that identify disinvestment targets by other means may need a specific prioritisation process to choose between the available options.A literature review to identify frameworks and tools for prioritisation found a Spanish guideline and assessment tool specifically for disinvestment , a frameThe Australian priority setting framework , 102 wasDiscussionSubsequently, lists of criteria for consideration in prioritisation and decision-making have been published for disinvestment , 107–110Prioritisation tools primarily focus on characteristics intrinsic to the TCP. However additional criteria may influence whether a TCP is selected to be the focus of a practice change initiative. These might be factors that affect the outcome of a project such as likelihood of success or sustainability and potential usefulness of the evaluation, or pragmatic features that enhance initiatives chosen specifically as pilot or demonstration projects such as opportunities for ‘quick wins’.Criteria for the EOI process were developed based on information from the literature and stakeholder consultations, and refined in consultation with the SHARE Steering Committee. The EOI criteria stipulated that the project must be based on high-quality evidence, be endorsed by Program and Department Heads, have appropriate resources allocated to undertake the project, have a documented clinical pathway and clear measurable outcomes. These and additional criteria that emerged in general discussion during SHARE meetings are outlined in Table The decisions made were pragmatic, based on likelihood of ‘quick wins’ and unspecified factors related to the proposed TCP. Prioritisation did occur, but the reasoning was not transparent. The final outcomes and reasons for the decisions are summarised in Table DiscussionDeciding between several alternatives can be a complex process requiring consideration of multiple factors. This has been addressed in more recently developed tools. Multi-criteria decision analysis (MCDA) allows consideration of all factors simultaneously , 116 andThe initial literature review and survey of external experts did not identify any information to guide development, implementation or evaluation of disinvestment projects in the local health service context. Interviews and workshops with Monash Health staff found that, although they did not use the specific term, they had experience of ‘disinvestment’ processes and other resource allocation activities. Most of the issues they identified Table  were conOnly one of the proposed pilot disinvestment projects reached the implementation stage Table . NursingThe clinical project team attended workshops on evidence-based change, implementation and evaluation and worked with SHARE staff to develop project, implementation, evaluation, reporting and cost-comparison plans. The funding agency required Monash Health to include four other health services in this project but no additional time or resources were provided. Many of the activities in the planning and development phase of the project were not undertaken as this time was spent liaising with the other health services. Analysis of barriers and enablers was delayed until midway through the implementation process which precluded development of strategies to avoid or minimise problems before they arose; however identifying actual, rather than anticipated, influencing factors provides more accurate information for future use Table . The ProAs the SHARE Program concluded earlier than expected, the implementation phase had not been completed and the planned evaluation was not undertaken. While we understand that the new technology was implemented and the transition from the old procedure to the new procedure was generally successful, final outcomes were not measured. The clinical project team agreed to complete the same template used by the SHARE project team to capture their experiences: ‘what worked, what didn’t work, how could it be improved?’ There is considerable overlap between these findings and the barriers and enablers. They have been combined and collated under the headings of the determinants of effectiveness in Table Discussionresource centres” to provide expertise, access to relevant methods and tools, and education, training and capacity-building [picking low hanging fruit” before tackling more difficult projects [The current literature acknowledges generic needs for implementation strategies and methods for monitoring and evaluation of disinvestment outcomes. In concert with the responses from Monash Health staff, several authors call for dedicated resources and in-house “building , 125. A building , some aubuilding . A rangebuilding , 109; fibuilding , 127; rebuilding , 128; clbuilding , 77; clibuilding ; qualitybuilding , 127; prbuilding ; promptsbuilding , 127; rebuilding ; public building , 127; trbuilding ; and “piprojects . The Schprojects . Others projects  and listprojects . A systeprojects . The defprojects , 78, 133The factors identified in relation to the determinants of effectiveness are summarised in Table “invest to save” as an alternative to disinvestment [The challenges in identifying suitable disinvestment projects are well documented. Decision-makers find it difficult to identify appropriate disinvestment opportunities , even whvestment . Decisiovestment .ad hoc process to select targets for disinvestment, local barriers beyond the scope of the SHARE Program, and lack of clarity and substance in proposals for change. These are discussed below.The experiences at Monash Health are consistent with these. Only one suitable project emerged from 19 nominations. Three factors played a significant role in this lack of success: dominance of an The absence of standardised methods for disinvestment decision-making is well-recognised , 23, 99.Ad hoc approaches to disinvestment decisions have been reported as “non-sustainable, reliant on chance or not conducive to independently identifying local opportunities for disinvestment” [“a lot of decisions are taken on gut feeling” and the problematic “tendency to adopt a short term perspective whilst searching for a ‘quick fix’ instead of taking a whole systems perspective based on consideration of long-term sustainability” [estment” , compromestment” . The gapestment” . The expability”  reflectsad hoc manner.Although the SHARE Program was underpinned by a commitment to systematic, transparent, accountable and evidence-based systems and processes, this was not achieved in the process of delivering pilot disinvestment projects. Potential target TCPs in the evidence-based catalogue were not considered and nominations were accepted and pursued in an SHARE had all the recognised enablers to systematic use of synthesised evidence in decision-making –140. TheThe non-systematic approach also led to a lack of transparency. All discussions were documented in minutes of the meetings and there were no attempts to be covert, however in the absence of a specified process and explicit criteria, it was not always clear how decisions had been made. The decisions themselves were transparent but the methods to reach them were not.ad hoc approach: two projects were based on a systematic, explicit EOI process and two had been through a rigorous application process for VPACT funding. These were the four projects finally accepted.There were four exceptions to the “Understanding how the technology got on the agenda, where it came from and who was pushing for it” have been reported as important factors for senior health decision-makers [n-makers . When inn-makers , 70.This is also true of the SHARE process. Eight proposals were made by people who had no connection with the TCP pathway. In addition, two were proposed because they were proceeding anyway  and two were proposed by the state health department unit (VPACT) providing funding to implement new TCPs Table . In totaNoted barriers to EBP include lack of authority to make the change , 139–142Most of the SHARE activities were either within the remit of CCE or the portfolios of the executives and senior managers on the Steering Committee. However the SHARE team did not have ownership of the data services, purchasing and procurement processes, and guideline and protocol documentation, or authority to make decisions in these departments. Although managers in these areas were generally supportive, their heavy workloads and competing priorities unrelated to SHARE activities prevented successful implementation of change in these areas.Disinvestment has been associated with a perceived focus on ‘cost cutting’ and ‘taking away’ in preference to ‘evidence-based care’ , 62, 146financial drivers were stronger than clinical drivers” in previous decision-making processes , an y others , 70, 99.There are also positive messages from this work. Although the objective to deliver disinvestment pilot projects was largely unsuccessful, there is much to learn from these experiences and the findings contribute in part to addressing the paucity of information about the disinvestment process. The single project undertaken was underpinned by a rich list of enabling factors, also contributing to the knowledge base in this area.It has been argued, within the SHARE Program and by others, that disinvestment would be more successful when considered in conjunction with investment decisions , 85, 161While it may not be productive to specifically seek ‘to disinvest’, it is appropriate and worthwhile to remove practices that are harmful, ineffective and inefficient. There are many potential sources of information and decision-making mechanisms to identify these practices. The opportunities for research lie in development of proactive methods and systematic prompts and triggers to utilise these resources.Seven potential methods of identifying disinvestment opportunities were investigated. While system redesign and PBMA were not feasible as methods of identifying disinvestment targets at Monash Health, both approaches are now well-researched, including their role in disinvestment –85, 94. The framework and taxonomy for evaluation and explication of disinvestment projects, and the algorithm for identifying disinvestment projects from a catalogue of potential TCPs, can be tested and refined for use in this context or extended into other decision-making settings.Local barriers were responsible for the limited success in applying the novel methods in this project. Further exploration of proactive methods to identify suitable disinvestment targets, systematic prompts and triggers to initiate disinvestment decisions, and strategies for project development, implementation and evaluation is warranted. Detailed documentation of the processes undertaken and the factors influencing them provide insight into elements to build upon and others to be avoided in future investigation of disinvestment in the local healthcare setting.