General practice websites are an increasingly important point of interaction, but their readability is largely unexplored. One in four adults struggle with basic literacy, and there is a socioeconomic gradient. Readable content is a prerequisite to promoting health literacy.To assess general practice website readability by analysing text and design factors, and to assess whether practices adapted their website text to the likely literacy levels of their populations.Websites for all general practices across Scotland were analysed from March to December 2019, using a cross-sectional design.Text was extracted from five webpages per website and eight text readability factors were measured, including the Flesch Reading Ease and the Flesch-Kincaid Grade Level. The relationship between readability and a practice population’s level of deprivation, measured using the Scottish Index of Multiple Deprivation (SIMD), was assessed. Overall, 10 design factors contributing to readability and accessibility were scored.n = 813/941) of Scottish practices had a website; 22.9% (n = 874/3823) of webpages were written at, or below, the government-recommended reading level for online content (9–14 years old), and the content of the remaining websites, 77.1% (n = 2949/3823), was suitable for a higher reading age. Of all webpages, 80.5% (n = 3077/3823) were above the recommended level for easy-to-understand ‘plain English’. There was no statistically significant association between webpage reading age and SIMD. Only 6.7% (n = 51/764) of websites achieved all design and accessibility recommendations.In total, 86.4% (Changes to practice websites could improve readability and promote health literacy, but practices will need financial resources and ongoing technical support if this is to be achieved and maintained. Failure to provide readable and accessible websites may widen health inequalities; the topic will become increasingly important as online service use accelerates. Most practices in the UK have a website and there is impetus from the UK and Scottish government to increase the provision of services that GPs offer online, such as appointment booking and repeat prescription requests.2,5,6 A small number of website providers operate in the primary care market, with one company supplying nearly half of the general practice websites in the UK; consequently, many websites are similar in basic design and structure. As patients are sometimes required — and, increasingly, expect — to interact with health services via the internet, poorly produced websites can create a barrier to accessing health care.The basis for general practice websites is commonly the practice leaflet, a contractually required document that provides information about the surgery’s services, opening times, appointments, prescriptions, data protection policy, and staff.‘the state or quality of being readable.’7 Text factors, such as word length or the number of syllables in a word, and design factors, such as line spacing and typeface, influence readability.8–10 Context is also important: familiar formatting — for example, having opening times written in table format — may help people understand complex information.9,11The comprehensibility of text is often termed ‘readability’, which is defined as 12 similar results were found in Scotland.13 Healthcare jargon and context adds complexity, even for those with otherwise good literacy levels.14 It has been found that 43% of written health information is too complex for UK adults to fully understand, a figure that rises to 61% when numerical information is included.15 Health literacy has been defined as the skills of individuals to ‘gain access to, understand, and use information to promote and maintain good health’;16 however, in order to promote health literacy, text must be readable.17 Low basic literacy and low health literacy are associated with higher levels of socioeconomic deprivation.13,18 In Scotland, those living in the 15% of areas with the greatest levels of deprivation, according to the Scottish Index of Multiple Deprivation (SIMD), were twice as likely to only reach a basic level of literacy compared with those in all other areas.13 Computing literacy also varies with socioeconomic status: individuals in areas of greatest deprivation are least likely to be able to use, have access to, or know about online services.19–21Readability matters: in the latest major review of adult literacy 16.4%, or around 5.8 million people, in England and Northern Ireland score at the lowest level of proficiency in literacy (at or below Level 1) that is, they are only able to comprehend short sentences and identify single pieces of information if they were identical or synonymous with the information in a question;‘into consideration the health literacy and/or accessibility needs of the population it is aimed at.’22 Ensuring information is understandable is vital to enabling equitable access to health services.The NHS Information Standard states that, when creating information, providers should take This study explores general practice website readability by analysing text and design factors, and assessing whether readability varied according to the SIMD measure of a practice population’s level of deprivation.23 The SIMD divides Scotland into 6976 neighbourhoods; each area is scored against 38 indicators of deprivation.24The authors used publicly available information from the Scottish Government — namely, the Information Services Division’s (ISD’s) list of all GP practices in Scotland, ranked by the percentage of each practice population’s level of multiple deprivation, as measured by the SIMD.appointments: how to make an appointment with a doctor;clinics and services: description of the clinics or extra services offered;repeat prescriptions: how to order repeat medicines; andnew patient information: how to register.Between January and July 2019, one author searched the internet to identify which of the 941 practices on the ISD/SIMD list had their own website. Directory-style entries on websites such as NHS Inform were not counted as independent general practice websites. Practice websites hosted by their local health board were included. If practices had merged with others and had a single group-practice website, the data were collected under the code of the practice whose physical site they shared; no data were collected for the relocated practice. Data were extracted from webpages that on discussion as a team were determined would be visited most often, such as the homepage (introductory page when clicking from a search result) and those with information on the following:25 These are well-established tools, and proxies for gold-standard comprehension tests.18 They consider average sentence length and syllables per word,19 and both are widely used and freely available in word-processing software.19 Both formulae have correlation coefficients of >0.9 with comprehension tests.20The primary measures of text readability were the Flesch-Kincaid Grade Level (FKGL) and Flesch Reading Ease (FRE) scores.8,25 Text should follow the principles of ‘plain English’ and:use short, everyday words;avoid jargon;be written in the first person; and22,23use an active, rather than a passive, voice.UK government website designers and literacy campaigners suggest that websites should be comprehensible by a 9–14 year old.22The readability statistic target for ‘plain English’ is an FRE of ≥60/100.character count;characters per word;word count;words per sentence;sentences per paragraph; and1paragraph count.Six secondary measures were recorded, in line with recommendations from a previous review of readability:One author checked each of the five webpages for all of the general practice websites. If practices did not have a separate webpage for the different areas of information , but had the relevant text on another part of the website, that text was extracted. Where possible, only the main area of webpage text (‘body text’) was extracted; navigation information, headers, and footers were not assessed.Body text was manually selected, copied, and pasted into Microsoft Word (2016) and formatting elements such as bulleted or numbered lists, headings, titles, tables, figures, and paragraph breaks were discarded.Where there had been a bulleted or numbered list, full stops were added to the end of each line; without full stops, the software calculated the whole list as a single sentence, which was misleading as the purpose of a list was, generally, to improve readability.26A different researcher checked the readability statistics for 10% of the websites, selected at random. This revealed 100% agreement, so the first researcher extracted the remaining data. With R software (version 3.6.2), linear regression was used to model the SIMD ranking for each practice against the FKGL score.A significant change in readability score was considered to be one grade level — that is, one UK school year.10,11,22 Typeface size was not investigated because it adjusts automatically based on individual settings, making it difficult to reliably record. The appointments page  was assessed, as it was thought that there would be appointments content on most websites and it has relevance to both new and existing patients. The body text  was scored. A score out of eight was given if there were no images; a score out of 10 was given if images had been used. To allow for a true comparison of webpage scores, a scaled design score was calculated — each score was divided by the maximum possible score for that webpage, giving a final score of 0.0–1.0.Design factors that contributed to readability and accessibility were also assessed. Informed by NHS England’s Information Standard, the UK government website’s design system, and recommendations from the Plain English Campaign, a 10 factor design score of desirable features was created .10,11,22The first researcher scored each website’s appointments page or section, and the second scored a random 10% of pages. Discrepancies were discussed and a third researcher reviewed those that could not be resolved.27 Using linear regression, the authors also assessed whether there was a correlation between the design score and the FRE readability statistic.Whether design scores varied within, and between, website providers was assessed. R software (version 3.6.2) was used to calculate the scaled design scores, mean scaled score, and standard deviation for each provider, and the number of webpages that scored full marks.Sensitivity analyses were conducted for both the readability and scaled design-score investigations by excluding webpages with <150 words.A random 10% of the ISD list of 941 practices (95 websites) were independently scored. There were no discrepancies in readability scoring. Of the 95 webpage design scores, seven had discrepancies of a maximum of 1 point (Cohen’s k coefficient: 0.98); these discrepancies were resolved by discussion.n = 3823/4065) of possible webpages (see Supplementary Table S1). Of all 3823 webpages, 77.1% (n = 2949) featured information that exceeded the recommended 9–14-year-old reading age for online content; 22.9% (n = 874/3823) featured information at, or below, the recommended age range. If all 813 functioning websites had five webpages with extractable data, there would have been 4065 potential webpages to analyse. Readability statistics were calculable for 94.0% (n = 3077/3823) of webpages scored below the recommended FRE cut-off of ≥60 for ‘plain English’ (data not shown).The FRE results were similar . In totaThere was no statistically significant association between the webpage reading ages (FKGL converted to UK age ranges) and the practice population’s level of multiple deprivation (SIMD quintile) (data not shown); Supplementary Figure S1 presents the reading-age level for the five webpage types by SIMD quintile. Secondary readability statistics are reported in Supplementary Table S2.n = 764/813) of practice websites had an appointments section (n = 51/764) scored full marks for design and accessibility (data not shown).In total, 94.0% .There was a spread of scaled design scores for each provider , but a sP = 0.23). Supplementary Table S3 and Supplementary Table S4 detail further descriptions of the remaining outliers.Webpages with <150 words were excluded and the analyses were repeated; results are shown in Supplementary Figures S2–S5. This removed the most extreme outlying readability scores  of webpages include content that exceeds the recommended target reading age of 9–14 years old,8,27 and 80.5% (n = 3077/3823) include content that exceeds the target for ‘plain English’.26 There was no evidence that practice websites were adapted to meet the likely literacy levels of the populations they serve. Only 6.7% of websites met accessibility and design recommendations28 . Both studies found a similar proportion fell within the respective readability targets (23.0% for webpages versus 24.3% for PILs).17Patient information leaflets (PILs) and online condition-specific information has been more widely studied; poor readability has been universally reported.The data presented here suggest most general practice websites across Scotland do not meet the standards recommended by the NHS, government, and literacy campaigners; in addition, there is no consistent evidence that practices in more socioeconomically deprived areas adjust the readability of their websites to meet the likely lower literacy levels of the populations that they serve, and vice versa.39It is possible that the hastened uptake of digital health due to the COVID-19 pandemic could exacerbate health inequalities, especially if literacy is not considered.25In Scotland, a national template for practices to adapt is being considered. Although website design may improve, practices will need support to create accessible content; pre-population with user-tested accessible text could help, as could the development of an NHS style guide, like that developed by Government Digital Service.9While awaiting national efforts, practices can take steps to improve the situation: the authors’ simple 10-point design score  could be14,31,40The authors’ assumption is that readability and design improvements promote comprehension and health literacy, but this can only be assessed by testing websites with their target users.Practices will require financial resources and technical support on an ongoing basis to enact and maintain such recommendations, but failure to do so may inadvertently widen health inequalities.In a time of scarce resources, partnerships between patient participation groups, literacy charities/campaign bodies, government, and practices may be necessary to ensure digital changes are inclusive.


This study aimed to develop a scale for evaluating and grading the evidence of prevention and treatment in ancient books of traditional Chinese medicine (TCM), in view of providing a reference for TCM clinicians, supporting the compilation or revision of evidence-based practice guidelines for TCM, improving the level of evidence-based research on ancient TCM books, and supplementing the development of evidence-based ancient TCM books.  The Delphi method was used for consultation among 40 experts in relevant fields. Excel 2016 and SPSS 21.0 were used to analyze the positive coefficient, authority coefficient, degree of concentration, degree of coordination, and degree of expert consensus. p < 0.05) and 0.081 (p < 0.05), respectively.  In the first round of the questionnaire, a total of 17 evaluation indexes were formulated in three aspects: 5 were deleted, 1 was modified according to the expert opinions, and no additional index was added. In addition, quantitative standards, weight assignment, and grading standards were developed according to the findings of the expert consultation. The positive coefficients of experts in the first and second rounds of questionnaires were 87.5% and 82.9%, respectively. The authority coefficient was 0.835 (>0.7). The coefficient of variation of the first and second rounds were 0.14∼0.29 and 0.09∼0.27, respectively. Kendall's coefficient of concordance of the first and second rounds were 0.135 ( The evaluation indexes and quantitative reference values of the developed scale conform to the characteristics of prevention and treatment evidence in ancient TCM books. It can provide a useful exploration tool for the evaluation and grading of evidences in TCM ancient books. Ancient books of traditional Chinese medicine (TCM) compiled and published in 1911 and before are the only books to carry the text on hand-made paper and are important and direct carriers of TCM knowledge. Among them, the contents of ancient books on prevention and treatment, which are known to have very important practical value, have been repeatedly verified by physicians in the past dynasties in clinical practice. With the rapid development of evidence-based medicine, evidence plays an increasingly important role in medical practice. The ancient books of TCM are the main source of evidence for the treatments in TCM clinical medicine. Currently, there are no relevant standards for evidence-based research in evidence quality evaluation and classification and the current research status in this field is still being explored , 3. Ren Based on the abovementioned problems and the present situation, to develop effective evaluation system for Chinese medical evidence has a wide range of applications and meets the demand for “best evidence” of evidence-based medicine. This study draws lessons from the current research of evidence-based practice in classifying the evidence and characteristics of classification, considering the particularity of ancient literature of TCM in writing style and content of the report, to develop an evaluating and grading scale for the prevention and treatment evidence of ancient Chinese medical books, in order to provide a reference for clinical physicians and medical researchers to participate in the formulation of TCM clinical practice guidelines .Synopsis of the Golden Chamber (Jin Gui Yao Lve), which is a classic in internal medicine with miscellaneous diseases and theories. Formularies of the Bureau of People's Welfare Pharmacy (Taiping Huimin Hejiju Fang), which is the world's first large-scale formula book compiled by the official auspices. Treatise on the Spleen and Stomach (Piwei Lun), which is a representative work of one of the four famous medical scientists in the Jin and Yuan dynasties, and Classified Case Records of Celebrated Physicians (Mingyi Leian), which is the foundation work of TCM medical cases. Meanwhile, we browsed extensively the record forms and characteristics of various works in the Encyclopedia of Traditional Chinese Medicine , which is the major TCM work of all dynasties with 1,156 ancient Chinese medical books, totaling over 10,000 volumes and more than 400 million words before 1949 [What elements should be included in the establishment of an evidence evaluation index system of ancient books of TCM? This study first investigated the related research by retrieving information from China National Knowledge Infrastructure (CNKI), Wanfang Data, China Science and Technology Journal Database (VIP), and Chinese biomedical database (CBM) using the search terms “Ancient books of TCM” OR “Classical medical literature for medicines” OR “Ancient literature of TCM” AND (“Evidence-based medicine” OR “Evidence”). It aimed to analyze the method of evaluating the quality of TCM evidence in ancient books in modern literature. But on account of the particularity of content and style in TCM ancient books and in order to get the comprehensive evaluation index system in this study, we selected the The questionnaire was designed mainly based on the pool of evaluation entries established. The content of the questionnaire includes a letter to the experts, which explains the purpose, content, and method of expert consultation in brief. Subject of the questionnaire, which is an expert evaluation score table set the importance score for each index of the evaluation index of evidence of ancient books, experts are invited to assign the importance of the indexes according to the Likert 5-level scoring method , and a column of suggestions for modification is attached for experts to put forward suggestions and reasons for modification, addition, or reduction. Basic information of experts, including the general information of experts, the evaluation of experts' judgment on the consulting content, and the evaluation of experts' familiarity with the consulting content are included. The judgment basis includes four dimensions: practical experience, theoretical analysis, peer understanding, and intuitive selection. The influence degree of expert judgment of each dimension is divided into three grades: large, medium, and small. Those are practical experience , theoretical analysis , peer understanding , and intuitive selection . The familiarity degree is divided into five grades, including very familiar, relatively familiar, general familiar, not very familiar, and not familiar, and the values are 1.0, 0.8, 0.5, 0.2, and 0 in sequence.40 experts were invited from across China for consultation. The selection criteria of experts were as follows: experts had a deputy senior professional title or above, engaged in TCM clinical practice, TCM ancient literature research and TCM research, had a rigorous academic attitude, showed some enthusiasm for our research, and were willing to answer the expert consultation questionnaire. We sent questionnaires by expressage, and prior to this, the experts were informed about the main purpose, method, and time schedule of this study via phone calls or SMS, so that the experts could fully understand the intended purpose of the questionnaire and offer their support.The results of the questionnaire were checked and entered in Excel 2016 and SPSS 21.0 by two researchers. According to the Delphi method, the expert positive coefficient, that is, the ratio between the number of returned questionnaires and the total number of sent questionnaires . The autApply the expert consultation method of the subjective weighting method to determine the weight of each index; that is, experts judge the importance of the index and normalize it according to the average importance degree of each index. The basic principle of the method is that more important indicators should be given a larger weight. The weight of each indicator is judged by experts according to their own experience and subjective judgment of the actual situation.The results of literature survey are shown in the In the first round of expert consultation, a total of 40 questionnaires were sent out to experts, and 35 were recovered, with a recovery rate of 87.5%. In the second round, a total of 35 questionnaires were issued, and 29 were recovered, with a recovery rate of 82.9%. The recovery rate of each round of questionnaires was higher than 70%, indicating that the experts participating in this study were highly motivated and cooperated well with the study team.In our study, the value of Cs is 0.83, the value of Ca is 0.84, and the value of Cr is 0.835 by formula, which is bigger than 0.7, indicating that the experts consulted have high authority and can improve the reliability of the research.Results of the full mark ratio and the arithmetic mean of each evaluation index in the two rounds of questionnaires are shown in The results of the CV of each evaluation index in the two rounds of questionnaires are shown in p < 0.05). In the second round, Wi was 0.081(p < 0.05) and the difference was statistically significant, indicating that the experts participating in the two rounds had consistent opinions on the evaluation indicators.Results of Kendall's coefficient of concordance are shown in The results of the calculation of the importance score and consensus degree of each evaluation index are shown in The second round of expert questionnaire survey is mainly about consultation on the quantitative methods of each evaluation index. According to the different contents of the evaluation index, qualitative and quantitative methods were adopted. The qualitative method is a common method to score the relevant evaluation items according to the qualitative description. For example, the AMSTAR scale is a scale to evaluate the quality of systematic evaluation research. The evaluation result of each evaluation index of this scale is a qualitative result of “yes” or “no”. Jin et al.  applied The quantitative method is to score the relevant evaluation items according to the quantitative description. In this study, the method of pre-evaluation was adopted, and the treatment formulae selected from the “TCM Clinical Practice Guide for Chronic stable Angina pectoris” in the TCM Evidence-based Clinical Practice Guide: Internal Medicine of TCM  were useGeneral Catalogue of Ancient Chinese Medical Books (Zhongguo Zhongyi Guji Zongmu), most ancient books have two to three versions and only a few have dozens of editions. Therefore, it is unreasonable to quantitatively measure the number of editions (suggestions are shown in Appendix 1). Appendix 1 also shows the second round's questionnaire body content and the experts' consensus degrees, opinions on the treatment, and modification conditions of the evaluation indicators.According to the results of the second round, the consensus degree was calculated, which indicated that the experts' consensus degree on all evaluation indexes was above 60%. In the experts' opinion and suggestion session, some experts believe that in the According to the results of the second round of expert questionnaire, the weight ratio between the ancient books and the evidence content was 3 : 7. The subjective weighting method was applied to calculate the weight value of each evaluation index, and normalized meanwhile. The sum of each weight value was 1, and the results are shown in Appendix 1. In addition, for the convenience of calculation, all results were multiplied by 10 and approximated values were used. For example, 1.9 is adjusted to 2, while 3.4 is adjusted to 3.5. The specific results are shown in Appendix 1.According to the preliminary formation of the scale, different weights of the evaluation indexes, score arrangements of evidence's content (evidence of knowledge or evidence of case), and the total score were evaluated. The grading standards are as follows with advice from the consulted experts in the questionnaires: high quality evidence: 35 points or above; intermediate quality evidence: 20 points or above; and low quality evidence: less than 20 points .History of Chinese Medicine[The final evaluation rating scale consists of three main parts: the body of the scale, the rating standard for each index, and the grading standard for evidence quality. Specific quantitative measures and operations are illustrated in the rating standard in detail (Appendix 2). When evaluating the popularity of ancient books, this study sorted out an evaluation reference material according to the  Medicine in order“bi zheng,” “bi,” “bi disease,” “gu bi,” “he xi feng,” “li jie,” “wang bi,” “wan bi,” “knee bi,” “li jie feng,” “li jie disease,” “jin bi,” “white tiger disease,” “knee pain,” and “tong feng.” We use the abovementioned search terms to obtain evidence from the Encyclopedia of Traditional Chinese Medicine .With reference to Zhongyi Neike Xue  , ZhongyiThis study sorted out the evidence contents of ancient Chinese medicine books for the treatment of KOA referring to the studies of May et al. –27, who Since ancient times, the ultimate goal of all medical research has been to save lives and improve the clinical efficacy of diseases. The origin and development of evidence-based medicine to clinical diagnosis and treatment with a focus on the process of using evidence to clinical curative effect, more specifically, the application of the evidence-based medicine method to systematically find evidence, screening, and evaluation of evidence and combining it with the experience of the patients and clinicians using evidence to make up for the lack of medical experience, is considered to be an irreplaceable scientific method in clinical practice and clinical decision. As an important part and source of TCM evidence, the idea and method of evidence-based medicine are applied to establish a perfect evaluation system of TCM evidence, so as to improve the reliability and scientificity of TCM clinical research and promote the application of TCM to the world.We first applied the Delphi method to construct an evaluating and grading scale for evidence of disease prevention and treatment in ancient books of TCM based on sufficient literature investigation under the guidance of theory and practice of evidence-based medicine. The evidence from the ancient books of TCM is divided into knowledge evidence and case evidence, and every kind of evidence was evaluated from both the source of evidence and the content of evidence separately. In terms of the evaluation of evidence sources, this study evaluates ancient Chinese medicine books from three different perspectives; they are the number of citations, the number of versions, and the popularity of ancient Chinese medicine books, which play an important role in ensuring the source reliability of evidence in ancient Chinese medicine books. Li et al. also believed that compared with modern evidence-based evaluation, the evaluation of evidence in ancient books should focus more on which ancient book the evidence is from . In termFinally, an evaluation grading scale containing 12 evaluation indexes is formed, which includes the specific quantitative score, weight assignment, and grading standard of each evaluation index. At the same time, a detailed description of each scoring standard of the scale is attached at the end of the whole scale. The scale made a specific quantitative evaluation method and the classification standard for ancient books' evidence. It has a supporting role both in the clinical application of TCM and in developing or revising the TCM clinical evidence-based practice guidelines. The scale makes an appropriate supplement for evaluation and recommendation of evidence of ancient TCM books.The following specific problems should be paid attention to in the application of the scale: First is to determine the search term. In modern clinical medicine, a disease may correspond to a variety of diseases of TCM. We should correspond to the names of diseases in western medicine, Chinese medicine, and the names of diseases in TCM ancient books, search for information comprehensively, and determine the search terms as comprehensive as possible, so as to make the retrieved evidence more comprehensive. Another problem is to set out the inclusion and exclusion criteria for evidence. The inclusion and exclusion criteria for evidence in TCM ancient books should be specific and clear in their formulation. The two criteria should be complementary rather than contradictory, and the criteria should be adjusted at any time according to the target disease.The evaluation indexes and quantitative reference values of the developed scale conform to the characteristics of prevention and treatment evidence in ancient TCM books. It can provide a useful exploration tool for the evaluation and grading method of TCM ancient book evidence.


How is health insurance literacy associated with coverage choices between traditional Medicare and Medicare Advantage, as well as within Medicare Advantage?In this cross-sectional study of 6627 Medicare beneficiaries, higher health insurance literacy—particularly, annual review and comparison of coverage choices—was associated with higher Medicare Advantage enrollment and choice of low-cost or high-rated Medicare Advantage plans. However, health insurance literacy was low among Medicare beneficiaries with low socioeconomic status.These findings suggest that policy makers should develop programs to make health insurance information accessible and understandable and to encourage annual review and comparison of coverage options, especially for vulnerable populations. This cross-sectional study of Medicare beneficiaries uses data from the 2015-2016 Medicare Current Beneficiary Survey to examine the association between health insurance literacy and coverage choices between traditional Medicare and Medicare Advantage, as well as within Medicare Advantage. Health insurance literacy helps individuals make informed choices. However, evidence suggests that Medicare beneficiaries experience low health insurance literacy, leading to high-cost or poor-quality coverage choices.To examine how health insurance literacy was associated with coverage choices between traditional Medicare (TM) and Medicare Advantage (MA), as well as within MA.This cross-sectional study included 6627 TM and MA enrollees, using data from the 2015-2016 Medicare Current Beneficiary Survey. Data analyses were conducted between May 1 and June 30, 2021.Three self-reported measures of health insurance literacy .Enrollment in TM vs MA and enrollment in an MA plan with different characteristics .We included 6627 Medicare beneficiaries . A total of 77 individuals were Asian (1.2%), 696 were Black (10.5%), 488 were Hispanic (7.4%), 5277 were non-Hispanic White (79.6%), and 225 (3.4%) were single races not of Hispanic origin (including American Indian or Alaska Native and Native Hawaiian) or were 2 or more races. Medicare Advantage enrollment was higher among individuals with higher health insurance literacy than those with lower health insurance literacy, especially for those who reviewed or compared coverage options annually than among those who did not . Among MA beneficiaries, those who reviewed or compared coverage options annually were more likely to enroll in plans with 4 to 4.5 stars and plans with monthly premiums of $1 to $50 by 4.6 percentage points  and 4.8 percentage points , respectively. However, enrollment in plans with 5 stars was 3.8 percentage points lower  among individuals who reviewed or compared coverage options annually than among those who did not. Among individuals with low socioeconomic status, the likelihood of reviewing or comparing coverage options annually was lower for those with Medicare and Medicaid dual eligibility than for those without it .Results of this study suggest that higher health insurance literacy—particularly, annual review and comparison of coverage choices—is associated with higher MA enrollment and choice of a particular MA plan. Policy makers should develop programs to encourage frequent review and comparison of coverage options for informed decision making. If individuals do not have the information or ability to make an adequate choice, they may make suboptimal choices that do not meet their preference or care needs.Choosing health insurance is among the most important financial and health-related decisions a person can make, but this can be challenging owing to the complexity and difficulty of terms and concepts. Thus, navigating health insurance requires considerable health insurance literacy, defined as a person’s ability to seek, obtain, and understand insurance coverage.3 and lower prices4 than TM. Once beneficiaries have decided to enroll in MA, they are confronted with a choice between multiple plans. Medicare beneficiaries had access to an average of 33 MA plans in 2021.5 Medicare Advantage plans vary significantly in terms of costs, coverage, and quality ratings.Health insurance literacy is particularly important to Medicare beneficiaries because they can choose among several options to access Medicare coverage. First, they can obtain their Medicare coverage either through the federally administered traditional Medicare (TM) program or the privately run Medicare Advantage (MA) program. Medicare Advantage plans must provide coverage for at least the same services covered by TM, but MA plans can offer additional supplemental benefits9 which may be attributable to limited health insurance literacy.10 Indeed, most Medicare beneficiaries do not use Medicare’s official information sources.11 Furthermore, 30% of Medicare beneficiaries reported difficulty understanding the Medicare program,11 and 57% did not review or compare coverage options annually.11 This low health insurance literacy may lead to impaired decision making, resulting in high-cost or poor-quality coverage choices. Consequently, many Medicare beneficiaries tend to stay in TM or remain in the same MA plan.12 However, little is known about the association between health insurance literacy and Medicare beneficiaries’ choice between TM and MA or their choice between different plans offered within MA.Evidence suggests that Medicare beneficiaries choose suboptimal coverage in terms of cost and quality,In this study, we examined the association between health insurance literacy and coverage choices between TM and MA as well as within MA. We conducted 4 analyses. In the first analysis, we examined whether enrollment in TM vs MA differed by health insurance literacy. In the second and third analyses, we investigated whether enrollment in an MA plan with a specific type of characteristic varied by health insurance literacy. We also studied what plan-level characteristics were associated with MA plan choice and how this differed by health insurance literacy. Finally, we examined which individual-level factors were associated with high health insurance literacy.13 The data combine information from Medicare claims and administrative data, with data collected by interview. The survey was conducted via 3 rounds per year. This study was approved by the University of Pennsylvania’s institutional review board and received a waiver of informed consent and HIPAA  authorization because the data were deidentified. This study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline.In this cross-sectional study, we used data from the 2015-2016 Medicare Current Beneficiary Survey, which provides a nationally representative sample of the Medicare population.We identified Medicare beneficiaries with 12-month continuous enrollment in TM or MA. We excluded individuals who did not have Parts A and B benefits, those who died within the year, and those whose original Medicare eligibility was attributable to end-stage renal disease. In most analyses, the unit of analysis was at the person level. To examine MA plan choice in each MA enrollee’s actual choice set, however, we restructured the data so that the unit of analysis was the plan-person.We included 3 types of coverage choice outcomes. The first outcome was a dichotomous indicator of whether the participant enrolled in TM vs any MA plan. For the second and third coverage choice outcomes, we limited analyses to individuals enrolled in an MA plan. The second type of outcome was a series of categorical variables describing the characteristics of the specific plan in which each participant enrolled. We included the following plan characteristics: star rating, monthly plan premium, in-network maximum out-of-pocket limit, plan type, and provision of supplemental benefits. The Centers for Medicare & Medicaid Services (CMS) has implemented a 5-star quality rating system to improve care quality while helping enrollees compare and choose high-quality MA plans. Specifically, star ratings are appraised according to 47 performance measures. The third type of outcome was a dichotomous indicator of whether the participants enrolled in each of the plans in their choice sets. We reshaped the data such that there was 1 observation for every possible MA plan available to each MA enrollee in their county. Thus, the unit of analysis was the plan-person. We included all possible pairings of each enrollee and each plan available in their choice set, from which each enrollee enrolled in exactly 1 plan.Our primary independent variables were 3 self-reported measures of health insurance literacy, which were selected because of their high relevance. Before the enrollment period for TM and MA, participants were asked to answer the following about these variables: “I have the information I need to make an informed comparison among different health insurance choices,” “How easy would you say it is for you to review and compare your Medicare coverage options?” and “How often do you review or compare your Medicare coverage options?” For each measure, respectively, we categorized responses into 2 levels: disagree (completely or somewhat disagree) vs agree (completely or somewhat agree), difficult (very or somewhat difficult) vs easy (very or somewhat easy), and less than annually  vs annually (at least once every year).To control for differences in patient characteristics, we included the following individual-level covariates: age, sex, race and ethnicity (self-reported), education, income, Medicare and Medicaid dual eligibility, marital status, comorbidities, general health status, number of limitations in activities of daily living, and baseline year. Because differential access to MA plans may lead to differential enrollment, we included county-level MA plan offerings as follows: total number of MA plans overall and the number of MA plans by star rating, plan type, monthly plan premium, and maximum out-of-pocket limit.14 As a sensitivity analysis, we conducted the analysis described earlier after applying inverse probability of treatment weights.15 Specifically, we computed the inverse probability of treatment weights as a propensity for enrolling in MA according to the variables described.For our analysis using enrollment in TM vs MA as an outcome, we included both TM and MA enrollees. We estimated adjusted rates of MA enrollment across levels of the 3 health insurance literacy measures. For each measure of health insurance literacy, we conducted a linear probability model of MA enrollment while controlling for individual-level characteristics and county-level MA plan offerings described earlier, as well as health insurance literacy. We also included county-fixed associations to additionally adjust for remaining variations in county-level factors. Then, we estimated the adjusted mean values of MA enrollment by health insurance literacy. Prior research has found that MA enrollees tend to be healthier than TM enrollees, suggesting that a direct comparison between TM and MA enrollees is potentially biased.To examine coverage choice within MA, we limited an analysis to individuals who enrolled in an MA plan. For our analyses using MA enrollment by plan characteristics as an outcome, we estimated adjusted rates of MA enrollment by plan characteristics and health insurance literacy. For each measure of health insurance literacy, we conducted multinomial logit regressions of MA enrollment by each plan characteristic while controlling for individual-level characteristics and county-level MA plan offerings described earlier, as well as health insurance literacy. We also included other 4 types of plan characteristics  to adjust for differences in other plan characteristics. Then, we estimated the adjusted mean values of benefit-specific enrollment in MA by health insurance literacy. We also estimated differences in the adjusted mean values by health insurance literacy.16 For each MA enrollee, we first constructed a choice set that included all plans available in his or her county. Then, using a conditional logit model with a dichotomous outcome of the chosen plan, we estimated the differential associations between an MA plan’s characteristic and the probability that an enrollee would select that MA plan.18 Our goal was to assess how the influence of MA plan characteristics on plan choice differed by health insurance literacy. Thus, we included plan characteristics described earlier, as well as their interaction terms with health insurance literacy. Then, we estimated the marginal association between a (1-unit or 1-level) change in a plan-level characteristic and the probability of selecting an MA plan across different levels of health insurance literacy.Because MA plan decisions may be made with several characteristics in mind, we used a conditional logit choice model to account for the simultaneous association of different plan characteristics with plan choice.To examine which individual-level factors were associated with low or high health insurance literacy, we included both TM and MA enrollees. Then, we conducted logistic regression of each measure of health insurance literacy while controlling for individual-level characteristics and county-level MA plan availability described earlier.P values were from 2-sided tests, and results were deemed significant at P < .05. All analyses were conducted in Stata version 16.1 (StataCorp LLC).Data analyses were conducted between May 1 and June 30, 2021. All We included 6627 Medicare beneficiaries  . A totalMedicare beneficiaries with high health insurance literacy were more likely to choose MA than those with low health insurance literacy . ParticuIndividuals who reviewed or compared coverage options annually were more likely to enroll in plans with 4 to 4.5 stars and in unrated plans  than those who did not, but were less likely to enroll in plans with 5 stars  . Also, eOverall, MA enrollees were marginally sensitive to monthly premiums and maximum out-of-pocket limit regardless of health insurance literacy . For allHigh health insurance literacy tended to be particularly low among individuals with poor health or low socioeconomic status . The likOur study presents 3 key findings. First, individuals with high health insurance literacy were more likely to choose MA than those with low health insurance literacy, consistent across all 3 measures of health insurance literacy. Second, annual review or comparison of coverage options had a stronger association with plan characteristics than the associations between either access to or understanding of plan information and plan characteristics. Specifically, individuals who reviewed or compared coverage options annually were more likely to enroll in low-cost or high-rated MA plans. Third, overall health insurance literacy was particularly low among individuals with low socioeconomic status.19 This finding also indicates that TM enrollment decisions may be attributable to lack of health insurance literacy, because approximately one-half of Medicare beneficiaries did not know about MA during an open enrollment period.20 This may be amplified by a status quo bias.12 However, some beneficiaries may choose TM owing to preferences such as freedom of provider choice or eligibility for Medigap purchase.Our findings suggest that health insurance literacy is an important factor in enrollment choice by enabling beneficiaries to more closely align their health insurance needs and preferences. This association may be more relevant to MA because of its low premiums, expanded benefits, and more coordinated care.7 which may align with our finding that those who more frequently reviewed their coverage options more often selected lower-premium plans. We also found that individuals who reviewed or compared coverage options annually had higher enrollment in 4- to 4.5-star plans than those who did not, but lower enrollment in 5-star plans. This finding suggests that MA enrollees prefer highly rated plans. Because they must balance preferences across multiple plan characteristics, however, the best-rated plan may not always represent the preferred choice.21 One explanation for this finding may be that MA enrollees recognized that a higher star rating was associated with improved patient outcomes, but the magnitude of the improvement was marginal.22Individuals who reviewed or compared coverage options annually were more likely to enroll in lower-premium plans or high-rated plans than those who did not. First, prior research showed that individuals with sufficient information about plan options enrolled in less costly plans,23 There may be a mediating role of socioeconomic status on health insurance literacy. Because individuals who might have potentially improved coverage choices were the least likely to review or compare coverage options annually, programs targeted to them may help to enhance decision-making quality for disadvantaged populations, possibly reducing disparities in enrollment decisions. Because we did not specifically examine each of the associations, however, further research is warranted to explore the underlying mechanisms for each socioeconomic factor. We also found that health insurance literacy varied by disease, which may come from disease-specific variations in insurance coverage, possibly leading to variations in health insurance literacy.Our findings suggest disparities in health insurance literacy by socioeconomic status, consistent with prior research.24 probably because of a combination of low technological skills and health insurance literacy. To improve health insurance literacy and help beneficiaries make an informed choice, policy makers need to consider developing more decision support tools. Because beneficiaries have various preferences and needs, the tools need to account for individual-specific characteristics. Specifically, tools can be designed to personalize choice framing and recommendations based on an individual’s specific characteristics. Prior research found that consumers with personalized information were more likely to switch plans and to choose a low-cost plan than those without it.25Our findings provide policy implications for enrollment decisions. Many resources are offered by CMS for beneficiaries to better understand and navigate the Medicare program. Overall, seniors tend to prefer person-to-person interaction over existing information resources from CMS,18 Second, estimates of the association between health insurance literacy and TM vs MA enrollment may be biased because of differences in individual-level characteristics that influence both literacy and enrollment. Third, there may be selective enrollment within MA by plan characteristics. Healthy enrollees may enroll more in high-rated plans than sick enrollees. Fourth, we could not account for all possible factors that may affect insurance coverage. Because our data did not allow us to determine the extent to which beneficiaries were making optimal choices, our findings should be interpreted with caution. Fifth, our analysis is associational, and our results do not necessarily have a causal interpretation.Our study has several limitations. First, an individual’s plan choice may be affected by additional aspects of plan benefits. However, evidence suggests that the role of other benefits is limited in enrollment decisions.In this cross-sectional study, higher health insurance literacy—particularly, annual review and comparison of coverage choices—was associated with higher MA enrollment and choice of a particular MA plan. Policy makers should develop programs to make health insurance information accessible and understandable across diverse and older adult populations and to encourage annual review and comparison of coverage options.


Digital innovation has scaled exponentially in many sectors including tourism, banking, and retail. It is well cited that the health sector is slower to embrace digital health innovations (DHI) beyond the pilot stage and consequently, many successful DHI pilot projects have failed to scale up. Such failure arises in part from a knowledge gap around what type and level of evidence are needed to convince implementers and decision makers to fund, endorse, or adopt new innovations into care delivery systems and sustainable practice. Much is known about the range of DHI evaluation methods used; however, less is published on the evidence that decision makers need to move innovations to scale. This paper draws on interviews (N = 18) with decision makers/project leads engaged in DHI in Scotland to identify what evidence matters when making DHI adoption/scale decisions. The results are used to present a heuristic service readiness level (SRL) framework that captures the changing nature of the evidence base required over a project lifecycle for progression to scale. We utilise this framework to discuss ‘what evidence’ is required and ‘how data accumulate’ over time to assist project teams to build a ‘DHI case for scale’. In the last 20 years or so, various industry sectors have transformed themselves by capitalising on new and innovative digital technologies, creating unprecedented benefits, efficiencies, and convenience advances to common services such as banking, tourism and retail . AdoptinIn this paper, we explore one dimension of the process of scaling up DHI, focusing on the nature of the evidence base that is required by key decision makers to translate pilot schemes into wider adoption and cases for scale. Through interviews with decision makers and project leads, we mapped this evidence to DHI examples and associated ‘levels of readiness’ for digital health innovation. In doing so, we have produced a heuristic service readiness level (SRL) framework that can be used by project teams and commissioners to ensure all parties gather the right assurances and evidence throughout the process to scale DHI if and when appropriate. Following a review of previous research on scaling digital health innovations and an outline of the methods used, this paper focuses on the different themes and evidence bases which are required to support scaling. Together, this allows us to present a new framework, based around service-level readiness, which offers insights to the varying and desirable evidence sought to assist in building a case for scaling DHI. This paper concludes with a brief discussion of the implications and potential use of the framework.Scaling DHI is not a new issue and has been discussed in many publications over the last ten years. The related work is therefore described in the following section as part of the literature review relative to the key benefits, challenges, and approaches to scaling up DHI.The benefits of adopting DHI have been widely acknowledged, offering opportunities to be ‘transformative’  and evenIt is accepted that DHI can play a significant role in modernising health care and that digital health technologies (at least in theory) lend themselves to scaling to the benefit of the populace . For at The adoption of technologies for health and care in general has been far from a smooth process, with innovation and transformation using digital for this sector often viewed as a ‘huge challenge’ . To dateIn explaining this absence of scaling in the UK, Lennon et al.  suggest In the context of DHI, complexity is not only centred on organisational and operational contexts, but also on the nature and form of evaluation required to reflect the wide array of health-related settings covered by DHI. A European Commission report  noted, ‘This inherent complexity can be problematic for the project team to collect the correct evidence that data decision makers require. In an attempt to address the issue of complexity in the UK, the Medical Research Council (‘MRC’) established a framework for complex evaluations in 2008, which is due to be updated later in 2021. Although presented as an iterative approach, this framework emphasizes four phases of a project lifecycle: intervention development; feasibility and piloting; evaluation; and implementation. Notwithstanding the merits of this framework and the insight that assisted this study, it is unclear to project teams what evaluation methods should be adopted at a particular stage, and what is the expected level of evidence and assurance required in relation to how ready the service is to adopt the DHI; what we term service readiness levels SRL, see . FurtherDecision makers require to understand whether a DHI is viable and capable of being adopted and scaled. To focus some of the uncertainties and interdependencies which accompany innovation adoption, researchers have developed the Non-Adoption, Abandonment, Scale-up, Spread, Sustainability (‘NASSS’) Framework . This frIn pursuing the efficacy of scaling a DHI, there remain issues around how research and empirical evidence is generated through project lifecycles. The lack of understanding on how the evidence is aggregated, iterated and built upon to create a final case for scale can be daunting to inexperienced DHI project staff. Projects should plan from the early stages to not fall into the “no evidence, no implementation—no implementation, no evidence” paradox often seen in digital health and highlighted recently by Guo  as a keyIn this paper, the questions we seek to address include what is the nature of the evidence base required across the entire project lifecycle to enable decision makers to be confident when considering scaling up interventions? What are the stages of development at which different decision makers are involved in evaluation and engaging with evidence bases? How can key decision makers become the ‘champions’ that Desveaux et al.  suggest The aim of the research was to explore, identify, and understand existing and appropriate future evaluation methods and evidence for DHI, in relation to building cases for scale. The end goal was to establish a usable framework which corresponds with the readiness levels identified through the common DHI project lifecycle stages and evidence required to be gathered at each stage.In order to achieve our objective of establishing a reliable and usable framework, the authors adopted a qualitative approach in the form of (i) a scoping review of the literature using the methodology set out by Arksey and O’Malley , (ii) seOverview/briefing paper detailing the purpose of the research, emailed in advance to interviewees (N = 18). The interviewees were drawn from senior management involved in DHI in Scotland, covering finance (2), clinical care (9), service management (3), and technical (4) . Their selection covered the main institutions involved in DHI decision making, including Scottish government, NHS, Innovation group project teams. In relation to the length of DHI experience of those interviewed, 15 interviewees had over 10 years’ experience in this field, 2 had over 5 years and 1 had under 5 years, with a gender split of 50% male and 50% females. The interviews were semi-structured, encompassing their experience with DHI projects, examples of good practice, barriers to scaling and their views of how scaling might be advanced   frameworService readiness levels (SRL) described using specific headline titles that were summarised from the illustrations offered and leading examples of DHI projects that had or were moving towards national scale. These SRL titles were then arranged in chronological order and titled to describe the types of activities and journey observed, as per the initial interview content. The evolution of this framework took place with title headings changing and reordered where necessary as part of the latter consultations and feedback when the SRL framework see  was beinThis SRL framework was then tested and validated by N = 14 interviews with key DHI leaders to gain further feedback and detail to optimise usefulness of the framework. These interviewees included 5 who were involved in the original research as well as an additional 9 who had comparable senior management roles in relation to DHI in Scotland, and these interviews took place over 2020/21.The research highlights two key elements to DHI scaling. First, it was evident that there were a number of thematic areas where a strong evidence base needed to be developed to provide the confidence and robustness required to support upscaling. However, second, it was also clear that this of itself was not sufficient to enable scaling, as the evidence had to be aligned to different stages of project development. This was strongly related to the notion of service readiness. In this section, we explore further these two strands.Analysis of the interviews identified six main evidence-related priority themes that were regarded as important by decision makers in reviewing a DHI case for scale . TogetheIn terms of organisational capacity to move to scale, the interviews revealed that this stage was often unclear, firstly on which overarching organisation would take ownership for building the case for the national scale and secondly who would be ultimately responsible for making the decision to scale nationally. It was noted that NHS Boards in Scotland did not think they had the mandate, capacity, or capability for building a national case for scale. This was said to cause delays and diminish pace, momentum, and morale, especially when project teams and partners believed that they had ‘good enough’ evidence for scale at that point. As one key stakeholder noted:“there are very few people who could really change business aspects of the service…. And then there’s even fewer people who are given the authority and the capacity to actually take action. So even the people who are interested in the change and can express what the future should be like, don’t really have the means to move forward to scale” (T&X2).C&Y2 expressed it, the evidence needed to answerIn making a case for scaling therefore in terms of organisation and service dimensions, evidence at the initial project stage in understanding demand and conveying a strong vision was particularly significant. This helped address questions as to whether a DHI would make a difference to the service, and create enough interest nationally against a major strategic priority, if scale was the end goal. As “does it solve or contribute to our real challenge in the system…so people see its value?”“addresses demand and capacity challenges”.Or as C&Y4 noted, it provided assurances that the DHI Evidence on the benefits of introducing a new transformed service using DHI were deemed essential. Each interview raised the need to articulate benefits from different perspectives, including benefits to (1) them, at a personal level, (2) a service level and (3) for their customers, ‘patients’ and ‘carers’. It was accepted that a range of common benefits would be expected, with hospital and health demand metrics rating most highly in persuading decision makers of the value to scale.Alongside this was a desire for evidence to reassure service delivery would not be compromised by introducing DHI, demonstrating that service quality should be at least maintained, if not improved, using DHI. To this end, there was broad agreement that clear baseline evidence of the current service provision and performance metrics from multiple viewpoints was desirable in any case for scale, allowing opportunities for transformation options to surface early in the process, and to enable comparisons to be made between traditional approaches, pilot studies and future states when delivered at scale.Beyond this, concern was raised about service continuity for projects aiming to reach scale fast, and as to how the project would be sustained while it awaited review and subsequent approval for national case for scale. Interviewees noted a frequent lag between project pilot conclusion and moving to a business-as-usual (BAU) service; the latter requiring specific skills, agreements, and a level of support for continuity to be confirmed to ensure the organisation responsible was confident to offer this as a service. As C&X1 expressed it“you have to draw on quite a lot of know-how, I think you’ve got to have quite a range of skills to actually bring that together and to put it into something as business as usual”,Creating an inherent reluctance by organisations to decide on scaling up.The complex inter-organisational working that typifies decision making in the health sector means that additional evidence is needed to assist multi-stakeholders in the co-designing process, including agreement over the desired scaled up form of the DHI. Evidence mapping that demonstrated a review of best practise—referred to as ‘landscape/literature reviews’ and ‘horizon scanning’—from other stakeholders, regions, and countries was noted as imperative. Such evidence at different stages helped to ensure a strong reference base, allowed shared learning, and offered reassurance practices were ‘not reinventing the wheel’.Interviewees stated it was easier to convince those that have the authority to ‘take action’ if there was real-world evidence (RWE) on the benefits/impacts and also if there was guidance on ‘how to do it’. Strong proof that it was achievable, and the service would accept a ‘new way of working’, was also important to build confidence with the decision makers. There was a general sense that evidence of this kind, often a process evaluation and softer related evidence (views and endorsement), was often either missed or lacked depth and clear sponsorship, resulting in decision delays. As projects moved towards implementation at scale, there was a need for practical evidence to allow for clear guidance on ‘how to implement’ the DHI and ‘create the conditions’ for the service change to be understood, realised and endorsed internally.Further, everyone interviewed recognised that clinical efficiency was a top priority for reviewing a case for scale, and that fundamental pieces of evidence on clinical efficacy and patient safety were key, including an evidence base that enabled opportunities to:“re-evaluate its position around what conditions can we make to make sure that it is safe, and that it is complying” (S&X1).Predominantly, the interviews revealed that there was a strong preference that as the DHI project matured to a higher state of readiness that this evidence was in the form of clinical trials; seen as generating a strong and robust clinical evidence base that would include efficiency and effectiveness and would withstand clinical scrutiny. The clinical trials discussion featured in many of the interviews with much debate on the best method and whether ‘Randomised control trials’ (RCTs) referenced as the ‘gold standard’ was appropriate for DHI. It was noted by most that a hybrid of a pragmatic RCT was best used, if the service criticality required it as evidence, noting not all DHI require this level of evidence if the risk of harm to the patient is considered to be low.“…without this clinical backing it unlikely that even if an innovation is proved to be effective, efficient and convenient for end users it will be too difficult to mandate without clinical leadership and champions being in place to promote to their peers and drive this forward as an acceptable option” (T&X1).There was also consensus that evidence of other clinical benefits was both desirable and often necessary to achieve support and reduce the barriers for scaling, this often manifested itself in benefit plans and cost benefit analysis formats. The demonstration of clinical effectiveness and the delivery of better use of resources were essential. Such evidence was important if the necessary clinical leadership were to be achieved:Such leadership not only had to endorse the adoption and scaling DHI, but to feel a degree of ‘investment’ and ‘ownership’ in the opportunity.“...you can’t look at these projects just as health projects in isolation, they have an economic dimension, education and industry dimensions, various facets to them that need to be taken into account”. (F&Y1).Unsurprisingly, financial issues along with legal and standards evidence are viewed as key indicators when introducing a DHI. Interviewees stated that clear evidence of the cost of the new service was a vital piece of data required for the case for scale, closely tied with the return the DHI would provide in respect to the investment. Interviewees highlighted a range of cost benefits that could be evidenced were discussed, including ‘cost savings’, ‘cost avoidance’, and ‘cost neutral’; all of which would potentially lead over time to a cost benefit. In assessing such benefits, evidence that demonstrated value through a more holistic approach and ‘whole system’ view was seen as advantageous:A major sub-theme raised under finance evidence was in connection to affordability to launch the new service and the sustainability of scaling, this related to the service and costs associated in moving it into a business-as-usual (BAU) state, which in most cases was noted to require a recurring budget. The noted BAU challenge and often barrier identified were linked to references of short-term funding structures used for projects of this nature. The realisation being that immediate savings would not be achieved in the short term with systemic political and organisation constraints. Common barriers included, budgets unable to be redirected easily or quickly and due to the lack of dedicated parallel innovation budgets for DHI sustainability being in place within most governments and service organisations. This was exacerbated by the need to evidence compliance, relating to legal, procurement, regulatory  and general sector specific standards.Whilst robust evidence of cost benefits and cost effectiveness would be a strong driver for DHI scaling, the interviews revealed that opportunity cost pressures and other challenges around risk and liability meant that there was a reluctance to commit to scaling without clearer evidence of future implications, with a gap identified within the DHI case for scale being the ‘consequence of not scaling’. To date, little research exists about the future implications of not approving a case for scale with regard to DHI.Although clinical and medical evidence formed the basis of much of the evidence base sought for scaling, there was an appreciation that evidence of support from other stakeholders, citizens and political decision makers should form part of an evidence framework.“I’m looking at evidence of workability, acceptability”.In making the case for citizen support, it was evident that this label was commonly applied to describe not only patients, but also health care customer (including carers), and service users, although for some interviewees this was broadened to the wider population. There was general agreement that the DHI should be evidenced to be ‘patient centric’, regarded as a crucial principle that strongly aligns with government strategies for this sector. Clear evidence that patients had been consulted and were part of the co-design for the potential future service was made clear by those interviewed, along with evidence on usability, acceptability, along with uptake/adoption and demonstration of equitable access to prove citizen approval. In short, as C&X3 expressed it: That said, across the interviews, there was also a desire for evidence that assessed demand and quality of a new DHI enabled service from a citizen’s perspective. For most, this indicated not only likely uptake of the service but also linked with the health services’ agenda of empowerment. The type of evidence of citizen benefits sought included real stories of the difference this had made to patients and citizens, or the benefits to others such as ‘carers’ lives, which also highlighted how their quality of life was or would be improved by this new service innovation. Importantly, interviewees also noted that if the DHI concept was found to be unpopular with the public, then organisations and governments would be unlikely to support the DHI for scale.“political enthusiasm” (F&Y1)—to provide endorsement and backing to evidence how this aligns with strategic priorities and major challenges in the case for scale was seen as essential.The political dimensions of introducing a DHI for the national scale mattered. With most DHI cases for the national scale in Scotland having to be reviewed and funded, at least at the initial scaling stage, by Scottish government. There was a compelling case for evidence which showed alignment with government strategy and policy priorities, with clear critical success factors and vision required to be set out early as part of the case for scale. Further, it was desirable that project teams should ensure that there was clear and easy to digest evidence to communicate alignment with policy and preferably a confirmed senior sponsor within government, especially as the state of readiness was maturing—what one respondent termed The final theme—and one that underpins all DHI initiatives—relates to the technology that enables the introduction of a DHI service. Surprisingly, technology evidence was mentioned least by the interviewees; suggesting that evidence of the existence of technology and support was being ‘taken for granted’.Those interviewed made it clear that the level of technology evidence was dependent on where the innovation was in relation to the technology readiness level (TRL). Existing technologies that had a proven track record and a strong evidence base required less specific technological evidence than a technology that was emerging and was regarded a disrupter, the latter often lacked a robust evidence base within this sector, regarded as ‘not trusted’, leading to heightened risk when considering a case for scale. The technology had to be proven with evidence that it was safe, accessible, usable, and reliable. Further, it was desirable for technical endorsement to be demonstrated for all users: citizens (patient and carers), professional staff  and the NHS board health IT departments .“in terms of evidence … you’re buying a thing that isn’t connected to other things. There’s the whole interoperability.... You’re having to plug something into our existing infrastructure” (T&Y1).Interviewees flagged a necessity that for to be both interoperable and able to be integrated easily into existing NHS legacy systems if required:However, it was equally important that there was clear evidence that the technology could flex with local needs, proving adaptability and transferability to other regions and conditions in building a case for scale would be beneficial. This is turn offered a degree of what was termed as ‘future proofing’ and contributed to the ‘economies of scale’ argument within a ‘case for scale’.“there’s lots of different people involved at different stages” and that “the business case was only the end point of quite a long process” that “targeted multiple different structures, each with a different purpose … continual throughout. So, it’s not a position where you’re delivering a business case, which then is a surprise to people. Actually, the reality of the business case was almost decided” (C&Y5) due to decisions being made along the way with “a lot of people to convince” (C&Y2) to accumulate a sense of “confidence and assurance” (S&Y1).What emerged from the interviews and is currently absent from previous research is an appreciation that the nature of evidence varies dependent on the stage in the lifecycle of the project, criticality of the service, and state of readiness to accept and adopt and scale the innovation. Cross-cutting the thematic evidence bases noted above in In our analysis, we identified that there was an emerging view that the form and nature of evidence required could be conceived of as being linked to different degrees of service readiness levels. DHI service readiness levels (SRLs) is a framework that supports the assessment of the maturity of an organisation in adopting a digital health innovation into a health service. The SRL approach and format was constructed from that of the NASA TRL  structurUsing this general SRL model and insights from the interviews, we have created a heuristic framework which details the varying nature of evidence that should be gathered at each stage of the DHI project lifecycle, corresponding to the different levels of SRL. This provides a simple representation of how best to collect the evidence, demonstrating that the evidence accumulates as the project matures to ultimately build a case for scaling the DHI.In constructing this framework , we haveThe titles and contents of these columns reflect the thematic evidence findings with key stakeholders as part of the interviews conducted and were synthesized and organized to allow the researchers to present the findings in an easily to understand format. This provided a clear ‘staged approach’ to build an evidence base that would enable progression if the criteria was met, to exit points (as per column 4), allowing diligent movement towards the next stage of readiness and eventually to scaling the DHI (if appropriate), please note it was found certain stages can be iterative (SRL 6 and 7) and some may run in parallel (SRL 3 and 4) and other stages required to be refreshed (SRL3) at the final case for scale (SRL 8), dependent on the time lag between (SRL3 and SRL8).The development of the above framework offers a novel and easy way to detail the evidence required at each stage of the decision-making process, how evidence accumulates as DHI projects mature, that ultimately assists in building a robust evidence base that supports a case for scaling DHI. It also provides insights into how those involved in DHI, as Desveaux et al.  argue, aIt is acknowledged that further research and application of the framework are clearly needed to test and validate further such assertions but given the current difficulties in scaling up small DHI pilot studies, we suggest that this framework provides a significant contribution to enable an appropriate robust evidence base to be generated to assist scaling. However, one implication of bringing together the different thematic forms of evidence required to make robust cases for scaling with different stages of project development and service readiness is that it is feasible to envisage an overarching framework of evidence for DHI scaling. In contrast to the successful transformation of different industry sectors through the adoption of new and innovative digital technologies, the arena of DHI has been marked by small-scale pilot projects unable to be scaled up to create health and well-being benefits and service efficiencies. The existence of a mature technical solution does not guarantee that such a product will be adopted by a health care system and certainly does not guarantee widespread adoption into business-as-usual service delivery and or scaling across a whole institution or national care delivery system. This paper has sought to fill a major knowledge gap around what type and level of evidence are needed to convince implementers and decision makers to reduce the barriers and advance large-scale DHI.Technical readiness levels (TRLs) have become accepted internationally as way to classify the evolution of a technology since proposed by NASA in 1973. We have highlighted that of equal, if not greater importance is a systems willingness and readiness to change and adopt a DHI. To secure a digitally supported and enabled future, we must support organisations to better understand where they are starting from on their journey, how ready they are and what evidence they need to gather to secure successful adoption and scaling of DHI.Through interviews with key stakeholders, we have created and tested the potential utility of a heuristic service readiness level framework to detail the type and range of evidence sought to allow DHI scaling to be more easily assessed. This offers for the first-time clear pathways in how evidence can be accumulated to enable scaling up of digital health innovation and building a robust ‘case for scale’ for decision makers. Further research on the application of the framework is needed to test this SRL framework and understand in more detail if there are gaps that could be further progressed to evolve this study and aid the progression of scaling DHI.