A phase 1 trial of a clade B HIV vaccine in HIV-uninfected adults evaluated the safety and immunogenicity of a DNA prime co-expressing GM-CSF (Dg) followed by different numbers and intervals of modified vaccinia Ankara Boosts (M). Both vaccines produce virus-like particles presenting membrane-bound Env.th the DNA dose as DgDgMMM given at 0, 2, 4, 6 and 8 months, or full dose DgDgM_M or DgDgMM_M regimens, given at 0, 2, 4, and 8 months, and 0, 2, 4, 6, and 10 months, respectively. Peak immunogenicity was measured 2 weeks post-last vaccination.Four US sites randomized 48 participants to receiving 1/10rd dose of MVA increased the avidity of elicited antibody (7.5% to 39%), the ADCC response to Bal gp120 (14% to 64%), and the one-year durability of the IgG3 responses to gp41 by 4-fold (13% vs. 3.5% retention of peak response). The co-expressed GM-CSF did not enhance responses over those in trials testing this vaccine without GM-CSF.All regimens were well tolerated and safe. Full dose DgDgM_M and DgDgMM_M regimens generated Env-specific IgG to HIV-1 Env in >90%, IgG3 in >80%, and IgA in <20% of participants. Responses to gp140 and gp41 targets were more common and of higher magnitude than to gp120 and V1V2. The gp41 antibody included reactivity to the conserved immunodominant region with specificities known to mediate virus capture and phagocytosis and did not cross-react with a panel of intestinal flora antigens. The 3This DNA/MVA prime-boost regimen induced durable, functional humoral responses that included ADCC, high antibody avidity, and Env IgG1 and IgG3 binding responses to the immunodominant region of gp41. The third, spaced MVA boost improved the overall quality of the antibody response. These products without co-expressed GM-CSF but combined with protein boosts will be considered for efficacy evaluation.NCT01571960ClinicalTrials.gov  The RV144 vaccine reigmen of canarypox (ALVAC) and rgp120 subunit protein (AIDSVAX) demonstrated 31% efficacy against HIV acquisition among 16,000 Thai women and men when the clinical trial concluded at 42 months , 8. ImmuThe Clade B HIV-1 vaccines furthest along in development are the GeoVax  DNA (pGA2/JS7) vaccine and recombinant modified vaccinia Ankara poxvirus (MVA/HIV62B) vaccine. A unique property of these vaccines is that both produce non-infectious virus-like particles (VLPs) displaying trimeric membrane-bound forms of the HIV-1 envelope glycoprotein. A previous Phase 2 study of these vaccines demonstrated a predominant gp41 IgG response when two DNA priming doses were followed by two MVA boosting doses; giving three MVA doses without the DNA prime improved the gp120 IgG response . HoweverTherefore, the HVTN launched a novel Phase 1 trial of the GeoVax DNA prime, MVA boost regimen, to advance understanding and enhance immunogenicity of these two vaccines used in combination. The trial compared 2 vs. 3 MVA boosts after 2 DNA primes, and a 2- vs. 4-month pause prior to the final MVA dose based on improved immunogenicity with this pause in influenza vaccination . BecauseForty-eight HIV-uninfected adult male and female vaccinia-naïve participants were enrolled at HVTN Clinical Research Sites in 4 US cities  from May through November 2012. Subjects were required to meet the following criteria for enrollment: age 18–50 years; good general health on the basis of medical history, physical examination, laboratory tests, and electrocardiogram (ECG) findings; and assessed as low risk for HIV infection. Pregnant women were excluded, and volunteers who could become pregnant agreed to consistently use effective contraception. All participants provided written informed consent.6 HEK293T cells. Modified Vaccinia Ankara MVA/HIV62B (MVA62B) encodes HIV-1 VLP from the same HIV sequences mutated for expression by a poxvirus as GEO-D03 [The GEO-D03 co-expresses GM-CSF and non-infectious VLPs from clade B HIV-HXB2/BH10 (Gag-Pol) and HIV-ADA sequences (gp120/gp41 cleavage site intact), from a single transcript by subgenomic splicing . Approxi GEO-D03 . In MVA6 GEO-D03 . Vaccineth dose group and for each full-dose group was 6 months and 12 months, respectively after their last vaccination: month 14 for Group 1, month 22 for new Group 2 and month 20 for Group 3 during the main study. Participants were also contacted annually by telephone for completion of health questionnaires for 3 years following enrollment.HVTN 094 was a randomized, double-blind, placebo-controlled trial among participants considered to be at low risk for HIV infection . Immune responses were measured at baseline, 2 weeks after each MVA/placebo dose, and 6-  and 9- or 12-months (Groups 2 and 3) after the last MVA/placebo dose.Serum for measuring GM-CSF levels was collected pre- and 3 and 7 days post- each DNA vaccination. Serum for measuring anti-GM-CSF binding antibodies was collected at baseline before the first DNA vaccination and at 2 weeks after each DNA and first MVA vaccinations. Levels of GM-CSF were measured using the Quantikine HS GM-CSF Immunoassay (R&D Systems).Serum HIV-1-specific IgA and IgG responses (1:50 dilution) and IgG1- IgG4 subclass responses (1:40 dilution) were performed by binding antibody multiplex assays (BAMA) under cGLP-compliant conditions using 21CFR Part 11 compliant software. The readout is expressed as mean fluorescence intensity (MFI) , 24–26 aThe affinity maturation of the vaccine-elicited antibody responses to the immunodominant region of gp41 (Bcon03 ID epitope tetramer) was measured by an avidity index assay using the BAMA described above with the following modifications. Avidity index is measured by inclusion of a denaturation step  and comparing the MFI in the treated well vs. MFI in the untreated well. Untreated wells contain Phosphate Buffered Saline (PBS) during the avidity incubation step. Avidity index (AI) is defined as 100*MFI in the treated divided by MFI in the untreated well , 33.Antibody dependent cellular cytotoxic activity mediated by the participants’ plasma/sera was detected according to our modification of the previously described GranToxiLux (GTL) cell-mediated cytotoxicity procedure . The CEMA validated intracellular cytokine staining (ICS) assay with detection by flow cytometry was used to examine HIV-1-specific CD4+ and CD8+ T-cell responses as previously described . PreviouEnd of study HIV diagnostics were performed on samples drawn 6 months after the last vaccination to evaluate ongoing seroreactivity to licensed HIV-1 antibody tests. Four enzyme-linked immunosorbent assays (ELISAs) were used: Abbott Architect HIV Ag/Ab Combo, Abbott Prism, BioRad GS HIV Combo Ag/Ab EIA, and Multi-spot HIV-1/HIV-2 Rapid Test. Western blot testing was also performed, but this test is no longer CDC-recommended and therefore these results are not included in the summary VISP results. Verification of HIV-1 uninfected status on reactive samples was confirmed with negative HIV nucleic acid testing.http://www.hiv.lanl.gov/content/nab-reference-strains/html/home.htm.Neutralizing antibodies were measured as a function of reductions in luciferase (Luc) reporter gene expression after a single round of infection in TZM-bl cells , 38. TZM−5, and with a Bonferroni multiplicity adjustment for the number of peptide pools [The sample size of the study was determined to provide a reasonable precision in the assessement of the primary safety and immuenogenicity endpoints (See details in Section 6.1 of the Protocol). The randomization allocation sequence was obtained by computer-generated random numbers and provided to each clinical site through the HVTN statistics and data monitoring center’s web-based randomization system. At each site, the pharmacist with primary responsibility for dispensing study products was charged with maintaining security of the treatment assignments. Participants and site staff (except for site pharmacists) were blinded as to participant treatment arm assignments. All randomized participants were included in the safety analyses. The number and percentage of participants experiencing each adverse event or reactogenicity symptom were tabulated by severity and vaccination regimens. All randomized participants with reliable assay data based on blood draw dates within the allowable visit window were included in the immunogenicity analysis. Fisher’s exact and Wilcoxon rank sum tests were used to compare the response rates and magnitudes, respectively, between two groups. For the comparison of two visits within a group, McNemar’s and Wilcoxon signed rank tests were used to compare the response rates and magnitudes among responders, respectively. Two-sided 95% confidence intervals (CI) for binomial proportions were calculated using the Wilson score method . All tesde pools . For theTo summarize multiple assay readouts simultaneously, radar plots were used to display the response rate and median response magnitude of each assay readout by vaccine regimen. To assess the relationship between different assay readouts and the groups of observations in a reduced dimensionality, principal component analysis (PCA) biplot was used. PCA was performed on scaled, centered data. In PCA, the original assay readouts were algebraically converted and combined into a new set of uncorrelated composite variables, ranked in order of their contribution to explaining the variation in the data. These new variables were termed “principal components” (PCs). In the biplot, each point represented one participant and each arrow represented one assay readout in the plane defined by the first two PCs, along with the percent of variation explained by each PC.th dose) DNA and 1x108 TCID50 (full dose) MVA in a DgDgMMM regimen given at 0, 2, 4, 6 and 8 months, or full dose  in a DgDgM_M or DgDgMM_M regimens, given at 0, 2, 4, and 8 months, and 0, 2, 4, 6, and 10 months, respectively  and predominantly female (63%). The participants self-identified as white (63%), Hispanic (13%), Black or African-American (10%), Asian (8%), and multi-racial (6%). Demographic characteristics did not differ significantly by randomization arm . All parnd dose of DNA and 3rd dose of MVA respectively, and 1 reported a severe headache after the first MVA vaccination. All recovered spontaneously within 3 days. Participants receiving full dose vaccine regimens did not have significantly more local or systemic reactogenicity than those in the 1/10 dose arm. Adverse events (AEs) were reported by 94% of participants, with no significant difference in AE reporting between active and placebo groups. Eight participants experienced AEs attributed to the study product by the investigator; all were mild and transient. There were no serious adverse events nor myocarditis or pericarditis events reported by any of the participants.The vaccinations in all arms were well-tolerated, with mild to moderate local reactogenicity . Only 3 As GEO-D03 contains GM-CSF DNA, serum levels of GM-CSF and anti-GM-CSF antibodies were measured. Anti-GM-CSF antibodies are known to be present in healthy people at levels of approximately 1 mcg/ml, but are elevated above 10 mcg/ml in patients with pulmonary alveolar proteinosis . The medrd MVA boost was assessed by comparing responses after the 2nd vs. 3rd dose within the DgDgMM_M arm. To evaluate the effect of a 4- vs. 2-month pause before the last MVA dose, the two study arms were compared after 2 doses of MVA (DgDgM_M vs. after 2 doses in the DgDgMM_M arm). Peak responses were measured 2 weeks after the final MVA dose; durability was compared between the two arms at peak, 6, 9, and 12 months post the last MVA immunization.Because the full-dose regimens were safe and tolerable and will likely be considered or modified in future trials, all immunogenicity results presented here compare data within or between the 2 full-dose arms. The effect of the 3Overall rates of humoral and cellular immune responses at the peak time-points are depicted in To further define the specificity of antibody binding, nd and 3rd MVA doses, respectively  in both arms of the study against gp140 and gp41 . The proTotal IgG and IgG3 magnitudes are shown for both arms through 12 months post last vaccination in None of the DgDgM_M participants had subsequent measurable IgA against ConS gp140 at 12 months , panel ATo determine if cross-reactive antibodies to the microbiome were elicited by this vaccine regimen, as previously described for gp41 antibodies , we measAt the peak time-point, the IgG avidity index to the IDR was significantly higher after the third vs. second MVA in the DgDgMM_M arm  among IgG responders, but was similar between the 2- vs. 4-month pause . At 6 moAt the peak time-point, the proportion of ADCC responders was significantly higher after the 3- vs. 2- MVA in the DgDgMM_M arm for BaL gp120 , but not for the two other gp120 antigens although with a similar trend . No signOverall response rates from ICS assays for IL-2 and/or IFN-y were higher for CD4+ than CD8+ T cells . CD4+ anCollectively, the 3-dose MVA arm generated comparable or better antibody-mediated and cellular immune responses compared to the 2-dose MVA arm, whereas the overall immune response profile were generally comparable between the 2- vs. 4-month pause arms . In addiAt the final study visit (6 months post-last MVA vaccination for the 1/10th dose arm and 12 months post- for each full-dose arm), each participant was assessed for the detection of HIV-1 antibodies by commercial assays, with confirmatory HIV nucleic acid testing performed for reactive samples. No participants were HIV-1 infected. The Vaccine-Induced Seropositivity/Reactivity (VISP/R) rates were 90%, 93% and 94% for the 1/10th dose arm, DgDgM_M, and DgDgMM_M arms, respectively.These GEO-D03/MVA HIV62B regimens induced an HIV-1 humoral immune profile that included Env IgG1 and IgG3 binding responses, high antibody avidity, and low level ADCC and neutralizing antibody responses, without substantial IgA; the cellular immune response was biased toward CD4+ T cells. The RV144 trial  had a siAs noted above, the targeted epitopes were completely different between RV144 and HVTN 094. HVTN 094 vaccinees predominantly targeted gp41 while the protective responses in RV144 were directed at gp120. A preclinical trial using a SHIV prototype of the vaccine used here found that avidity to the IDR of gp41 correlated inversely with peak viremia when the animals were challenged with a single high dose of SHIV-162P3 . SimilarA recent paper hypothesized that the predominance of the gp41 humoral response in many HIV-1 vaccine trials originates from cross-reactive priming by intestinal microbial antigens . B cell Virtually all of the IgG produced by participants in the HVTN 094 trial were of the IgG1 and IgG3 subtypes, with almost no IgG2 or IgG4 responses. In RV144, HIV-1 Env IgG3 correlated with decreased HIV-1 risk , and medOn the other hand, IgA blocked ADCC-mediating antibodies to envelope in the RV144 trial, and ADCC was only associated decreased risk of HIV-1 infection in the absence of IgA . Becauserd MVA dose. A previous study demonstrated improved antibody responses to an influenza DNA/monovalent inactivated vaccine prime-boost regimen if the pause between immunizations was at least 12 weeks [The third MVA dose appeared to improve both the quality and longevity of the antibody response. ADCC responses were substantially improved after the third MVA dose, and the magnitude of IgG to gp140 remained stable from 6 to 12 months post-vaccination, in contrast to waning values in the two-dose MVA arm. Response rates and titers of neutralizing antibody, also increased for MN3, the virus with the most substantial neutralizing antibody responses, with the 312 weeks . In thisA non-human primate study using an earlier HIV prototype vaccine demonstrated greater protection when the GM-CSF co-expressing DNA was used as the prime . This inThis study is limited by its small size, calling for confirmation of the findings of this study using a DDMM_M regimen moving forward. In addition, in RV144, certain class I and class II HLA genotypes have been found to be associated with immune responses and protection , 56, as The Global HIV Vaccine Enterprise has called for a rational approach to moving forward vaccine candidates, particularly those that build upon previous evidence of protection while exploring a novel immune space . The DNAS1 Protocol(PDF)Click here for additional data file.S1 CONSORT checklist(PDF)Click here for additional data file.S1 Table(DOCX)Click here for additional data file.S1 Fig(A) Local reactogenicity. (B) Systemic reactogenicity. Frequency of maximum severity of reactogenicity symptoms by vaccination for placebo recipients (combined for arms 1–3) and vaccine recipients in each treatment arm. DgDgM_M refers to immune responses after the last MVA in the DgDgM_M group; DgDgMM and DgDgMM_M refer to immune responses after the 2nd and 3rd MVA in DgDgMM_M group, respectively.(EPS)Click here for additional data file.S2 Fig(A) Peak IgG binding antibody response rates. (B) Peak IgG binding antibody response magnitudes. Binding antibody response to individual antigens at two weeks after the 2nd and 3rd MVA or placebo in the Placebo, DgDgM_M and DgDgMM_M groups. DgDgM_M refers to immune responses after the last MVA in the DgDgM_M group; DgDgMM and DgDgMM_M refer to immune responses after the 2nd and 3rd MVA in DgDgMM_M group, respectively. Shown are total IgG binding antibody to Clade B Lab Adapted Env, Clade C Transmitted/Founder Env, and Non-Env antigen. Positive responses are shown as filled circles and negative responses are shown as open circles. Box-plots represent the distribution for the positive responders only.(EPS)Click here for additional data file.S3 Fig(A) IgA response rates. (B) IgA response magnitude. Binding antibody responses to individual antigens at two weeks after the 2nd and 3rd MVA or placebo in the Placebo, DgDgM_M and DgDgMM_M groups. DgDgM_M refers to immune responses after the last MVA in the DgDgM_M group; DgDgMM and DgDgMM_M refer to immune responses after the 2nd and 3rd MVA in DgDgMM_M group, respectively. Shown are total IgA binding antibody responses to HIV-1 Env gp140 (ConS gp140), HIV-1 Env gp120 (Con 6 gp120 B), the V1V2 loop, and gp41. Positive responses are shown as filled circles and negative responses are shown as open circles. Box-plots represent the distribution for the positive responders only.(EPS)Click here for additional data file.S4 FigPeak response rates of neutralization antibody responses at two weeks after the 2nd and 3rd MVA or placebo in the Placebo, DgDgM_M and DgDgMM_M groups. DgDgM_M refers to immune responses after the last MVA in the DgDgM_M group; DgDgMM and DgDgMM_M refer to immune responses after the 2nd and 3rd MVA in DgDgMM_M group, respectively. Neutralization IC50 antibody titers were measured in TZM-bl cells against a panel of heterologous Env-pseudotyped viruses .(EPS)Click here for additional data file.S5 FigDgDgM_M refers to immune responses after the last MVA in the DgDgM_M group; DgDgMM and DgDgMM_M refer to immune responses after the 2nd and 3rd MVA in DgDgMM_M group, respectively. The x- and y-axes are the values from the 1st and 2nd PC, respectively, that explain the most variation in the data. Points on the plot represent the values of the PCs of each observation. Points that are close together correspond to observations that have similar values in the two PCs. The top axis is the 1st PC loadings and the right axis is the 2nd PC loadings, where loadings are the weights by which each original assay readout is multiplied to get the value of the corresponding PCs. An arrow (vector) is drawn for each assay readout from the origin to the point defined by its first two PC loadings. Vectors that point in the same direction correspond to readouts that have similar response profiles on the basis of the first two PCs. The observations whose points project furthest in (opposite of) the direction in which the vector points are the observations that have the most (least) weight of the corresponding readout. The angle between two arrows conveys information about the correlation of the assay readouts, with a zero degree angle denoting perfect correlation and a 90 degree angle denoting no correlation.(EPS)Click here for additional data file.


From Galileo to Bacon, the science has always tried to bring the scientific discipline on the road of “evidence” in face of many possible and arbitrary “opinions.” For instance, the Cochrane Library exists so that healthcare decisions could get better. This organization tried to certify this “evidence” using a methodologic approach. Nevertheless, in oral and maxillofacial surgery, most of systematic reviews founded weak or no evidence that any particular type of intervention is better than any other.Evidence medicine (EBM) has been defined as “the judicious use of the best current evidence in making decisions about the care of the individual patient” . It was The rationale behind the EBM is to analyze only the effect of the drug, removing the patients' and operators' variables. To cancel patient variability, sample size should be in the order of thousands. At the same time, the only skill required to the operator is to display a pill. EBM has been applied to surgery over the past decades with increasing numbers of randomized controlled trials (RCTs). Nevertheless, interpretation of its clinical meaning might be problematic in the dental surgical field. In fact, despite a test technique can be compared to a control one in a randomized modality, to ensure that the techniques can be precisely defined and then performed in a standardized manner for each operator remains challenging. The skill of the operator, which remains the most important variable, must not be underestimated. Finally, due to the ethical conflicts intrinsic to the surgical field, sample size is very often small, and despite statistical tricky analyses and strict inclusion/exclusion medical criteria, patient variability cannot be excluded. Very often, in fact, to statistically increase numbers and apparently generalize the meaning of the message, the biology behind the cases is underestimated and the outcomes are so generalized to lose a specific clinical effectiveness.Although the start of this methodologic approach was really promising, its, so far, extensive and maybe misinterpreted use in the surgical field might represent only a tricky way to publish.Controversially, in the modern implant dentistry, bio-logic (a composite neologism to express the need to combine knowledge of biology and the use of the logic) could be the key factor to interpret and choose the correct surgical (and not only) techniques to treat a clinical case. At the light of the third millennium clinical demands, interpretation of the evidence in surgical dentistry through Cochrane's guidelines might appear problematic due to the nature of surgery itself. Outcomes of systematic reviews  should be used as approximate indications.Randomized controlled trial and systematic review will continue to be a safety margin not to go beyond, but the message to spread to the scientific community should be to encourage the production of long-term prospective and also retrospective study conducted in clinical practice. These might provide less high levels but without forgetting the importance of the experience and the operator skill, just focusing on the potential limits and complications of the technique itself.Luigi CanulloEitan MijiritskySilvio Mario Meloni


Simulation plays a vital role in health professions assessment. This review provides a primer on assessment validation for educators and education researchers. We focus on simulation-based assessment of health professionals, but the principles apply broadly to other assessment approaches and topics.Validation refers to the process of collecting validity evidence to evaluate the appropriateness of the interpretations, uses, and decisions based on assessment results. Contemporary frameworks view validity as a hypothesis, and validity evidence is collected to support or refute the validity hypothesis . In validation, the educator or researcher defines the proposed interpretations and decisions, identifies and prioritizes the most questionable assumptions in making these interpretations and decisions (the “interpretation-use argument”), empirically tests those assumptions using existing or newly-collected evidence, and then summarizes the evidence as a coherent “validity argument.” A framework proposed by Messick identifies potential evidence sources: content, response process, internal structure, relationships with other variables, and consequences. Another framework proposed by Kane identifies key inferences in generating useful interpretations: scoring, generalization, extrapolation, and implications/decision. We propose an eight-step approach to validation that applies to either framework: Define the construct and proposed interpretation, make explicit the intended decision(s), define the interpretation-use argument and prioritize needed validity evidence, identify candidate instruments and/or create/adapt a new instrument, appraise existing evidence and collect new evidence as needed, keep track of practical issues, formulate the validity argument, and make a judgment: does the evidence support the intended use?Rigorous validation first prioritizes and then empirically evaluates key assumptions in the interpretation and use of assessment scores. Validation science would be improved by more explicit articulation and prioritization of the interpretation-use argument, greater use of formal validation frameworks, and more evidence informing the consequences and implications of assessment. Educators, administrators, researchers, policymakers, and even the lay public recognize the importance of assessing health professionals. Trending topics such as competency-based education, milestones, and mastery learning hinge on accurate, timely, and meaningful assessment to provide essential information about performance. Assessment of professional competence increasingly extends beyond training into clinical practice, with ongoing debates regarding the requirements for initial and ongoing professional licensure and certification. Front-line educators and education researchers require defensible assessments of health professionals in clinical and nonclinical settings. Indeed, the need for good assessments has never been greater and will most likely continue to grow.Although workplace-based assessment is essential –3, simulThe purpose of this article is to provide a primer on assessment validation for educators and education researchers. We focus on the context of simulation-based assessment of health professionals but believe the principles apply broadly to other assessment approaches and topics.Validation refers to the process of collecting validity evidence to evaluate the appropriateness of the interpretations, uses, and decisions based on assessment results . This deSecond, validation involves the collection of validity evidence, as we discuss in a following section.Third, validation and validity ultimately refer to a specific interpretation or use of assessment data, be these numeric scores or narrative comments , and to Rigorous validation of educational assessments is critically important for at least two reasons. First, those using an assessment must be able to trust the results. Validation does not give a simple yes/no answer regarding trustworthiness ; rather, a judgment of trustworthiness or validity depends on the intended application and context and is typically a matter of degree. Validation provides the evidence to make such judgments and a critical appraisal of remaining gaps.Second, the number of assessment instruments, tools, and activities is essentially infinite, since each new multiple-choice question, scale item, or exam station creates a de facto new instrument. Yet, for a given educator, the relevant tasks and constructs in need of assessment are finite. Each educator thus needs information to sort and sift among the myriad possibilities to identify the assessment solution that best meets his or her immediate needs. Potential solutions include selecting an existing instrument, adapting an existing instrument, combining elements of several instruments, or creating a novel instrument from scratch . EducatoIn addition, simulation-based assessments are almost by definition used as surrogates for a more “meaningful” clinical or educational outcome . Rarely content, construct, and criterion; see Table Classical validation frameworks identified at least three different “types” of validity: Content evidence, which is essentially the same as the old concept of content validity, refers to the steps taken to ensure that assessment items  reflect the construct they are intended to measure. Internal structure evidence evaluates the relationships of individual assessment items with each other and with the overarching construct(s), e.g., reliability, domain or factor structure, and item difficulty. Relationships with other variables evidence evaluates the associations, positive or negative and strong or weak, between assessment results and other measures or learner characteristics. This corresponds closely with classical notions of criterion validity and construct validity. Response process evidence evaluates how well the documented record  reflects the observed performance. Issues that might interfere with the quality of responses include poorly trained raters, low-quality video recordings, and cheating. Consequences evidence looks at the impact, beneficial or harmful, of the assessment itself and the decisions and actions that result [The first contemporary validity framework was proposed by Messick in 1989  and adopt result –29. EducThe most recent validity framework, from Kane , 12, 30,Educators and researchers often ask how much validity evidence is needed and how the evidence from a previous validation applies when an instrument is used in a new context. Unfortunately, the answers to these questions depend on several factors including the risk of making a wrong decision , the intended use, and the magnitude and salience of contextual differences. While all assessments should be important, some assessment decisions have more impact on a learner’s life than others. Assessments with higher impact or higher risk, including those used for research purposes, merit higher standards for the quantity, quality, and breadth of evidence. Strictly speaking, validity evidence applies only to the purpose, context, and learner group in which it was collected; existing evidence might guide our choice of assessment approach but does not support our future interpretations and use. Of course, in practice, we routinely consider existing evidence in constructing a validity argument. Whether old evidence applies to a new situation requires a critical appraisal of how situational differences might influence the relevance of the evidence. For example, some items on a checklist might be relevant across different tasks while others might be task-specific; reliability can vary substantially from one group to another, with typically lower values among more homogeneous learners; and differences in context (inpatient vs outpatient), learner level , and purpose might affect our interpretation of evidence of content, relations with other variables, or consequences. Evidence collected in contexts similar to ours and consistent findings across a variety of contexts will support our choice to include existing evidence in constructing our validity argument.In addition to clarifying the four key inferences, Kane has advanced our understanding of “argument” in the validation process by emphasizing two distinct stages of argument: an up-front “interpretation-use argument” or “IUA,” and a final “validity argument.”As noted above, all interpretations and uses—i.e., decisions—incur a number of assumptions. For example, in interpreting the scores from a virtual reality assessment, we might assume that the simulation task—including the visual representation, the simulator controls, and the task itself—has relevance to tasks of clinical significance; that the scoring algorithm accounts for important elements of that task; that there are enough tasks, and enough variety among tasks, to reliably gauge trainee performance; and that it is beneficial to require trainees to continue practicing until they achieve a target score. These and other assumptions can and must be tested! Many assumptions are implicit, and recognizing and explicitly stating them before collecting or examining the evidence is an essential step. Once we have specified the intended use, we need to (a) identify as many assumptions as possible, (b) prioritize the most worrisome or questionable assumptions, and (c) come up with a plan to collect evidence that will confirm or refute the correctness of each assumption. The resulting prioritized list of assumptions and desired evidence constitute the interpretation-use argument. Specifying the interpretation-use argument is analogous both conceptually and in importance to stating a research hypothesis and articulating the evidence required to empirically test that hypothesis.Once the evaluation plan has been implemented and evidence has been collected, we synthesize the evidence, contrast these findings with what we anticipated in the original interpretation-use argument, identify strengths and weaknesses, and distill this into a final validity argument. Although the validity argument attempts to persuade others that the interpretations and uses are indeed defensible—or that important gaps remain—potential users should be able to arrive at their own conclusions regarding the sufficiency of the evidence and the accuracy of the bottom-line appraisal. Our work is similar to that of an attorney arguing a case before a jury: we strategically seek, organize, and interpret the evidence and present an honest, complete, and compelling argument, yet it is the “jury” of potential users that ultimately passes judgment on validity for their intended use and context. It is unlikely that any single study will gather all the validity evidence required to support a specific decision. Rather, different studies will usually address different aspects of the argument, and educators need to consider the totality of the evidence when choosing an assessment instrument for their context and needs.Of course, it is not enough for researchers to simply collect any evidence. It is not just the quantity of evidence that matters, but also the relevance, quality, and breadth. Collecting abundant evidence of score reliability does not obviate the need for evidence about content, relationships, or consequences. Conversely, if existing evidence is robust and logically applicable to our context, such as a rigorous item development process, then replicating such efforts may not be top priority. Unfortunately, researchers often inadvertently fail to deliberately prioritize the importance of the assumptions or skip the interpretation-use argument altogether, which can result in reporting evidence for assumptions that are easy to test rather than those that are most critical.Imagine that we are teaching first year internal medicine residents lumbar puncture (LP) using a part-task trainer. At the end of the training session, we wish to assess whether the learners are ready to safely attempt an LP with a real patient under supervision.Define the construct and proposed interpretationIn our example, the construct is “LP skill” and the interpretation is that “learners have fundamental LP skills sufficient to attempt a supervised LP on a real patient.”Validation begins by considering the construct of interest. For example, are we interested in the learners’ knowledge of LP indications and risks, their ability to perform LP, or their non-technical skills when attempting an LP? Each of these is a different construct requiring selection of a different assessment tool: we might choose multiple-choice questions (MCQs) to assess knowledge, a series of skill stations using a part-task trainer to asses procedural skill with an Objective Structured Assessment of Technical Skills (OSATS) , or a reMake explicit the intended decision(s)In our example, our foremost decision is whether the learner has sufficient procedural competence to attempt a supervised LP on a real patient. Other decisions we might alternatively consider include identifying performance points on which to offer feedback to the learner, deciding if the learner can be promoted to the next stage of training, or certifying the learner for licensure.Without a clear idea of the decisions we anticipate making based on those interpretations, we will be unable to craft a coherent validity argument.Define the interpretation-use argument, and prioritize needed validity evidenceIn our scenario, we are looking for an assessment instrument in which a “pass” indicates competence to attempt a supervised LP on a real patient. We anticipate that this will involve a physician rating student performance on a skills station. Assumptions in this context include that the station is set up to test techniques essential for LP performance (vs generic skills in sterile technique or instrument handling), that the rater is properly trained, that a different rater would give similar scores, and that learners who score higher on the test will perform more safely on their first patient attempt. Considering the evidence we might need to support or refute these assumptions, and using Kane’s framework as a guide, we propose an interpretation-use argument as follows. We do not know at this stage whether evidence has already been collected or if we will need to collect it ourselves, but we have at least identified what to look for.Scoring: the observation of performance is correctly transformed into a consistent numeric score. Evidence will ideally show that the items within the instrument are relevant to LP performance, that raters understood how to use the instrument, and that video-recording performance yields similar scores as direct observation.Generalization: scores on a single performance align with overall scores in the test setting. Evidence will ideally show that we have adequately sampled performance  and that scores are reproducible between performances and between raters (inter-station and inter-rater reliability).Extrapolation: assessment scores relate to real-world performance. Evidence will ideally show that scores from the instrument correlate with other LP performance measures in real practice, such as procedural logs, patient adverse events, or supervisor ratings.Implications: the assessment has important and favorable effects on learners, training programs, or patients, and negative effects are minimal. Evidence will ideally show that students feel more prepared following the assessment, that those requiring remediation feel this time was well spent, and that LP complications in real patients decline in the year following implementation.In making our interpretations and decisions, we will invoke a number of assumptions, and these must be tested. Identifying and prioritizing key assumptions and anticipating the evidence we hope to find allows us to outline an interpretation-use argument .In our sWe cannot over-emphasize the importance of these first three steps in validation. Clearly articulating the proposed interpretations, intended decision(s), and assumptions and corresponding evidence collectively set the stage for everything that follows.Identify candidate instruments and/or create/adapt a new instrumentWe find a description of a checklist for assessing PGY-1’s procedural competence in LP . The cheWe should identify a measurement format that aligns conceptually with our target construct and then search for existing instruments that meet or could be adapted to our needs. A rigorous search provides content evidence to support our final assessment. Only if we cannot find an appropriate existing instrument would we develop an instrument de novo.Appraise existing evidence and collect new evidence as neededscoring evidence by describing the development of checklist items through formal LP task analysis and expert consensus. It provides generalization evidence by showing good inter-rater reliability, and adds limited extrapolation evidence by confirming that residents with more experience had higher checklist scores. Other studies using the same or a slightly modified checklist provide further evidence for generalization with good inter-rater reliabilities [implications evidence by counting the number of practice attempts required to attain competence in the simulation setting [We begin our appraisal of the validity argument by searching for existing evidence. The original description  offers sbilities , 36, andbilities , 37 and bilities . One stuAlthough existing evidence does not, strictly speaking apply to our situation, for practical purposes we will rely heavily on existing evidence as we decide whether to use this instrument. Of course, we will want to collect our own evidence as well, but we must base our initial adoption on what is now available.Keep track of practical issues including costFor the LP instrument, one study  tracked An important yet often poorly appreciated and under-studied aspect of validation concerns the practical issues surrounding development, implementation, and interpretation of scores. An assessment procedure might yield outstanding data, but if it is prohibitively expensive or if logistical or expertise requirements exceed local resources, it may be impossible to implement.Formulate/synthesize the validity argument in relation to the interpretation-use argumentWe find reasonable scoring and generalization evidence, a gap in the extrapolation evidence , and limited implications evidence. As is nearly always the case, the match between the interpretation-use argument and the available evidence is not perfect; some gaps remain, and some of the evidence is not as favorable as we might wish.We now compare the evidence available  against the evidence we identified up-front as necessary to support the desired interpretations and decisions (the interpretation-use argument).Make a judgment: does the evidence support the intended use?Based on the evidence summarized above, we judge that the validity argument supports those interpretations and uses reasonably well, and the checklist appears suitable for our purposes. Moreover, the costs seem reasonable for the effort expended, and we have access to an assistant in the simulation laboratory who is keen to be trained as a rater.We also plan to help resolve the evidence gaps noted above by conducting a research study as we implement the instrument at our institution. To buttress the extrapolation inference we plan to correlate scores from the simulation assessment with ongoing workplace-based LP assessments. We will also address the implications inference by tracking the effects of additional training for poor performing residents, i.e., the downstream consequences of assessment. Finally, we will measure the inter-rater, inter-case, and internal consistency reliability in our learner population, and will monitor costs and practical issues as noted above.The final step in validation is to judge the sufficiency and suitability of evidence, i.e., whether the validity argument and the associated evidence meet the demands of the proposed interpretation-use argument.Although the above concepts are essential to understanding the process of validation, it is also important to be able to apply this process in practical ways. Table As a thought exercise, let us consider how the above would unfold if we wanted to use the same instrument for a different purpose and decision, for example as part of a high-stakes exam to certify postgraduate neurologist trainees as they finish residency. As our decision changes, so does our interpretation-use argument; we would now be searching for evidence that a “pass” score on the checklist indicates competence to independently perform LPs on a variety of real patients. We would require different or additional validity evidence, with increased emphasis on generalization , extrapolation , and implications evidence . We would have to conclude that the current body of evidence does not support this argument and would need to either (a) find a new instrument with evidence that meets our demands, (b) create a new instrument and start collecting evidence from scratch, or (c) collect additional validity evidence to fill in the gaps.This thought exercise highlights two important points. First, the interpretation-use argument might change when the decision changes. Second, an instrument is not “valid” in and of itself; rather, it is the interpretations or decisions that are validated. A final judgment of validity based on the same evidence may differ for different proposed decisions.In our own validation efforts –41 and iOur review  found thAs noted above, validation frameworks add rigor to the selection and collection of evidence and help identify gaps that might otherwise be missed. More important than the framework chosen is the timing  and manner (rigorously and completely) in which the framework is applied in the validation effort.Comparing the scores from a less experienced group against those from a more experienced group  is a common approach to collecting evidence of relationships with other variables—reported in 73% of studies of simulation-based assessment . Yet thiCook provided an extended discussion and illustration of this problem, concluding that “It is not wrong to perform such analyses, … provided researchers understand the limitations. … These analyses will be most interesting if they fail to discriminate groups that should be different, or find differences where none should exist. Confirmation of hypothesized differences or similarities adds little to the validity argument.” Validation researchers often focus on data they have readily available or can easily collect. While this approach is understandable, it often results in abundant validity evidence being reported for one source while large evidence gaps remain for other sources that might be equally or more important. Examples include emphasizing content evidence while neglecting internal structure, reporting inter-item reliability when inter-rater reliability is more important, or reporting expert-novice comparisons rather than correlations with an independent measure to support relationships with other variables. In our review, we found that 306/417 (73%) of studies reported expert-novice comparisons, and 138 of these (45%) reported no additional evidence. By contrast, only 128 (31%) reported relationships with a separate measure, 142 (34%) reported content evidence, and 163 (39%) reported score reliability. While we do not know all the reasons for these reporting patterns, we suspect they are due at least in part to the ease with which some elements  can be obtained.This underscores the importance of clearly and completely stating the interpretation-use argument, identifying existing evidence and gaps, and tailoring the collection of evidence to address the most important gaps.As noted above, validity is a property of scores, interpretations, and uses, not of instruments. The same instrument can be applied to different uses , and much validity evidence is context-dependent. For example, score reliability can change substantially across different populations , an asseThe interpretation-use argument cannot, strictly speaking, be appropriately made without articulating the context of intended application. Since the researcher’s context and the end-user’s context almost always differ, the interpretation-use argument necessarily differs as well. Researchers can facilitate subsequent uptake of their work by clearly specifying the context of data collection—for example, the learner group, task, and intended use/decision—and also by proposing the scope to which they believe their findings might plausibly apply.It is acceptable to talk about the validity of scores, but for reasons articulated above, it is better to specify the intended interpretation and use of those scores, i.e., the intended decision. We strongly encourage both researchers and end-users (educators) to articulate the interpretations and uses at every stage of validation.We have often observed researchers merely report the evidence without any attempt at synthesis and appraisal. Both educators and future investigators greatly benefit when researchers interpret their findings in light of the proposed interpretation-use argument, integrate it with prior work to create a current and comprehensive validity argument, and identify shortcomings and persistent gaps or inconsistencies. Educators and other end-users must become familiar with the evidence as well, to confirm the claims of researchers and to formulate their own judgments of validity for their specific context.Volumes have been written on the development, refinement, and implementation of assessment tasks, instruments, and procedures , 45–48. First, the sample of tasks or topics should represent the desired performance domain. A recurrent finding in health professions assessment is that there are few, if any, generalizable skills; performance on one task does not predict performance on another task , 50. ThuSecond, the assessment response format should balance objectification and judgment or subjectivity . The advIt is frustrating to identify an assessment with relevance to local needs and validity evidence supporting intended uses, only to find that the assessment is not specified with sufficient detail to permit application. Important omissions include the precise wording of instrument items, the scoring rubric, instructions provided to either learners or raters, and a description of station arrangements  and the sequence of events. Most researchers want others to use their creations and cite their publications; this is far more likely to occur if needed details are reported. Online appendices provide an alternative to print publication if article length is a problem.Too often as educators, we allow the availability of an assessment tool to drive the assessment process, such as taking an off-the-shelf MCQ exam for an end-of-clerkship assessment when a performance-based assessment might better align with clerkship objectives. This issue is further complicated with simulation-based assessments, where the availability of a simulator may drive the educational program as opposed to designing the educational program and then choosing the best simulation to fit the educational needs . We shouThere are three problems with labeling an instrument as validated. First, validity is a property of scores, interpretations, and decisions, not instruments. Second, validity is a matter of degree—not a yes or no decision. Third, validation is a process, not an endpoint. The word validated means only that a process has been applied; it does not provide any details about that process nor indicate the magnitude or direction (supportive or opposing) of the empiric findings.We hope to see greater use of simulation-based assessment as part of a suite of learner assessments. Simulation-based assessment should not be a goal in and of itself, but we anticipate more frequent assessment in general and believe that simulation will play a vital role. The choice of modality should first consider what is the best assessment approach in a given situation, i.e., learning objective, learner level, or educational context. Simulation in its various forms will often be the answer, especially in skill assessments requiring standardization of conditions and content.We hope that simulation-based assessment will focus more clearly on educational needs and less on technology. Expensive manikins and virtual reality task trainers may play a role, but pigs feet, Penrose drains, wooden pegs, and cardboard manikins may actually offer more practical utility because they can be used with greater frequency and with fewer constraints. For example, such low-cost models can be used at home or on the wards rather than in a dedicated simulation center. As we consider the need for high-value, cost-conscious education , we encoBuilding off the first two points, we hope to see less expensive, less sophisticated, less intrusive, lower-stakes assessments take place more often in a greater variety of contexts, both simulated and in the workplace. As Schuwirth and van der Vleuten have proposed , this moWe hope to see fewer new assessment instruments created and more evidence collected to support and adapt existing instruments. While we appreciate the forces that might incentivize the creation of novel instruments, we believe that the field will advance farther and faster if researchers pool their efforts to extend the validity evidence for a smaller subset of promising instruments, evaluating such instruments in different contexts, and successively filling in evidence gaps.We hope to see more evidence informing the consequences and implications of assessment. This is probably the most important evidence source, yet it is among the least often studied. Suggestions for the study of the consequences of assessment have recently been published .Finally, we hope to see more frequent and more explicit use of the interpretation-use argument. As noted above, this initial step is difficult but vitally important to meaningful validation.Although we do not pretend to know the future of simulation-based assessment, we conclude with six aspirational developments we hope come to pass.


Both financial literacy  and health literacy  become increasingly important for older adults, potentially impacting their quality of life. Resources in these constructs of literacy tend to be distinct, although the skills and decision-making involved overlap as financial issues impact healthcare choices. Thus the primary purpose of this commentary is to propose a new area of research focus that defines the intersection of financial and health literacy .We conducted a limited literature review related to financial, health, and health insurance literacy to demonstrate gaps in the literature and support our position. Online search engines were utilized to identify research in our primary areas of interest.We define the intersection of financial and health literacy as an area of need labeled financial health literacy, with a focus on four domains. These include: 1) the ability to manage healthcare expenses; 2) pay medical bills; 3) determine health needs and understand treatment options; and 4) make sound healthcare decisions with financial resources available. Despite some overlap with health insurance literacy, financial health literacy would define an area of need encompassing health management choices and health plan selections integrated with other financial management issues including living arrangements, financial planning, and retirement planning.Potential initiatives should be considered to help at-risk older adults find resources to improve their financial health literacy, which in turn will enhance their abilities to manage medical choices in the environment of an increasingly complex healthcare system. In the US, over 46 million Americans are age 65 or older, a number that is increasing rapidly. This population is projected to nearly double by 2050 –3 makingFinancial literacy encompasses the ability to understand financial concepts and make sound financial decisions, primarily regarding housing, investments, and long-term savings , 13–16. A gap in the literature exists regarding the significant overlap of financial literacy and health literacy, as well as its critical importance to older adults. Thus the primary purpose of this commentary is to propose a new area of focus to better define this intersection as financial health literacy Figure : the capfinancial health literacy” and reviewed the results, which provided no existing research studies specifically dedicated to the domains of FHL as we propose to define it. Therefore we gathered and summarized the most relevant literature on the related but separate areas of literacy discussed here .To inform this commentary and support our position, we conducted a thorough yet limited review of the scientific literature relevant to our purpose in the areas of financial, health, and health insurance literacy. We first searched for “financial literacy, financial literacy measures/scales, financial literacy resources older adults/seniors, financial planning, health literacy, health literacy measures/scales, health literacy resources older adults/seniors, health insurance literacy, health insurance literacy measures/scales, health insurance literacy resources older adults/seniors, older adults/seniors, Medicare, retirement savings, and retirement planning. Each search term returned a number of results too large to review individually. Thus we narrowed our results to limit the search for these exact phrases to better identify research pertaining directly to older adults. Additionally, we focused on publications dated 2006 or later as these would be most relevant to the current healthcare system and Medicare Part D, although earlier research papers with important background information or definitions were considered. Finally, publications specifically related to older adults’ health or medical decisions based on their treatment options (with no consideration for costs) or other health factors alone were excluded, as they were not relevant to our objective of examining the overlapping roles of financial and health literacy. Research studies describing the reasoning for, or the process of, making medical decisions due to a lack of money could have been relevant to this commentary; however, we did not identify specific studies in this area to include. Thus our commentary is based on the publications deemed most relevant to our purpose.Online search engines, primarily PubMed, Medline, Google Scholar, and a mainstream Google search, as well as references from relevant articles, were utilized to identify research in our areas of interest. Across these sources, we used several primary search terms, including: Financial literacy returned approximately 1700 results; health literacy returned over 10,000 results; and health insurance literacy returned 390 results, most of which overlapped with general health literacy and were not specific to health insurance literacy. Furthermore, many results for both financial literacy and health literacy overlapped with general literacy topics or areas not useful for this commentary. Thus the following numbers of publications were selected to review in detail: financial literacy: 59; health literacy: 11; health insurance literacy: 10.For brevity, results are summarized for only the primary search terms used to gather research. After further review of the most relevant publications, the final number ultimately cited in this commentary totals 43, with 42 articles published in 2006 or later, and the remaining published earlier. The resulting commentary is based on this limited literature review, which was sufficient to support our position on FHL as a unique area of need.Financial literacy has been defined with validated measures in the scientific literature, with several variations: the ability to understand financial information to make informed decisions ; to perfHealth literacy has been defined with validated measures as the capacity to obtain, communicate, process, and understand basic health information and services to make appropriate health decisions –19. HealCompared to financial and health literacy, health insurance literacy is an emerging area of focus, with fewer measurement scales in use or research studies published , 22. In Research indicates that financial, health, and health insurance literacy are closely intertwined and especially impactful in terms of later-life decision-making , 25. Amo“Suppose you had $100 in a savings account and the interest rate was 2% per year. After 5 years, how much do you think you would have in the account if you left the money to grow?” The second item, also with five responses, reads: “Imagine that the interest rate on your savings account was 1% per year and inflation was 2% per year. After 1 year, how much would you be able to buy with the money in this account?” Finally, the third item is as follows: “Please tell me whether this statement is true or false. Buying a single company’s stock usually provides a safer return than a stock mutual fund.” Among Americans over age 50, only half answered the first two items correctly; only one-third answered all three items correctly [Currently, there is no single screening question to measure basic financial literacy, but longer validated measures exist. Several commonly used surveys, primarily the University of Michigan Health and Retirement Study (HRS)  and the orrectly , 16. In Other measures of financial literacy include the Financial Industry Regulatory Authority (FINRA) National Financial Capability Survey , which lwww.usaa.com) that allows members to assess their preparedness for retirement and offers access to financial advisors by phone [www.aarp.org) features several financial offerings including free tax preparation, fraud and scam alerts, and other tools [The common chosen resource for financial literacy would most likely be a financial planner specializing in later-life financial planning and retirement needs. Existing money management resources designed for older adults tend to focus on basic financial concepts, such as investment planning to cover living arrangements and expenses in retirement as well as Social Security options. Websites designed for this purpose often feature financial tools related to economic security, debt management, scams, and tips for aging workers . Larger by phone . As anoter tools . These rfinancial literacy returned approximately 1700 results while health literacy returned over 10,000 results, indicating a greater research base. Furthermore, measurement of health literacy across various settings and populations is commonly based on one widely used and accepted validated screening item: “How confident are you filling out medical forms by yourself?” [Compared with financial literacy, health literacy research has been developed more extensively. Notably, in our review of the literature on these topics, a search for urself?” , 37. Usi“How many years is it recommended that the average person who has had a normal colonoscopy should wait before having another colonoscopy?” Although an objective question, this item may not reflect common knowledge, as the frequency of recommended colonoscopies varies depending on test results. Furthermore, another item in this section of the HRS appears to be more opinion-based, and thus responses may be an inaccurate representation of health literacy: “Do you think regular colon cancer screening for people over age 50 does or does not reduce the risk of dying from colon cancer?”Research on literacy assessment suggests the importance of measuring the impact of both objective  and subjective (perceived) knowledge in an individual’s decision-making process . The sinVarious health-related resources provide general information about common concerns and condition management. For older adults, a general practitioner or other physician is likely to be a common chosen source of information. Older adults may trust their physicians to provide the best advice on their health needs, although physicians may not address medical costs or insurance billing with their patients. Multiple websites, such as those of the Centers for Disease Control and Prevention (CDC), Mayo Clinic, and WebMD, do provide guidance on medical conditions; some offer an online symptom checker or search feature to match symptoms with certain conditions. For older adults seeking in-person assistance, local health fairs or senior centers may provide access to nurses or other providers to discuss health concerns. Free healthcare services such as weight measurement and blood pressure, cholesterol, and diabetes testing are often available as well. Other personalized resources include pharmacists at retail pharmacy or drugstore outlets. Many older adults who take prescription medications likely know where to go for this advice, such as a neighborhood pharmacy, and may even have a pharmacist they personally know and trust. Finally, mainstream media publications targeting older audiences often feature articles on health care, common conditions, healthy lifestyle strategies, and treatment/medication options. Compared to some of the financial resources, these health-related options may be more commonly known or easier to access. However, they tend to focus strictly on health issues rather than the expenses associated with managing health later in life.While health insurance literacy has been defined, this area of literacy has not been measured extensively, and no widely used assessment scale exists , 22. HeaFor health insurance literacy, a commonly used resource would likely be a representative with a specific insurance plan or an independent broker to assist with health plan choices. These individuals can help consumers to select healthcare plans or medical care options and provide information on healthcare expenses. Other existing resources geared toward evaluating insurance options and planning for healthcare expenses could serve as examples to inform further development of specific tools to support the additional domains of FHL. For instance, online resources have emerged as a common service offered by various entities. AARP’s Benefits Quick Link, for one, helps members determine whether they qualify to save money on health care, and an online long-term care calculator provides guidance in planning for long-term care expenses, which would be an important component of FHL . ElsewheAs an alternative to online resources, older adults may seek assistance on their own from well-known brands, such as AARP or BlueCross BlueShield (BCBS), as they have highly visible products and readily available contact information. These resources could potentially assist those who need help using their limited finances most effectively. However, additional, easily accessible options in this area are still needed to improve and support FHL among older adults.“How confident are you with managing healthcare expenses independently, including paying bills?” Further development of a longer measure would also be instrumental in establishing FHL and identifying those at risk for low FHL. The key overarching domains should first be established , or through other organizations such as local United Way offices. Resources would need to be user friendly, accommodate all reading levels, and include enough detail to be useful in planning. In addition, those without regular computer access or computer literacy would likely need alternative options, such as services accessible over the phone or in person, brochures, handouts, or other written materials, or individual sessions and classes covering aspects of FHL. For instance, local senior centers could potentially serve an important role in this area by providing accessible, useful, and cost-free offerings, and these centers would likely be well known to seniors and their families. Senior centers could also offer free or low-cost access to professionals specializing in these areas who would provide individualized advice or assistance in managing overlapping late-life expenses. In addition, organizations such as USAA (http://www.usaa.com) offer retirement planning and financial advice, often along with healthcare planning resources.Another option with potential promise may be services and advisors providing unbiased assistance with managing healthcare expenses and/or making medical plan and treatment choices. This type of offering could be comparable to resources available through the National Association of Area Agencies on Aging, which provides educational programs in local areas throughout the US as well as online  plan selections, and important medical care decisions, resources to support and assist with decision-making become increasingly necessary. Recent estimates suggest that a 65-year-old couple should plan to have about $250,000 saved strictly to cover medical expenses post retirement , 43. In “How confident are you with managing healthcare expenses independently, including paying bills?” could also be useful in developing a longer comprehensive measure. Based on our proposed definition, the key domains of FHL would encompass the conceptual skills listed in Table Establishing FHL would require development of a measurement scale to assess the conceptual skills involved, perhaps initially based on one screening question. As we have suggested, a screener for FHL, perhaps Defining FHL as a unique area of need may be the first step in simplifying the decision-making process for older adults as they address the burdens of managing healthcare expenses, other living expenses, and life-changing decisions. In this commentary, we have proposed a definition of FHL, an area of need encompassing four main domains: the ability to manage healthcare expenses, pay medical bills, determine needs and understand treatment options, and make sound healthcare decisions with the financial resources available. Although FHL would overlap with certain areas of health insurance literacy, it would involve other unique factors, such as the ability to plan for and pay for all late-life expenses and make affordable treatment choices with limited resources. In addition, we have suggested the conceptual skills within each domain that would be important to assess with a validated measurement scale to identify those most at risk for low FHL. By establishing the need for FHL and subsequently identifying those with low FHL, resources can be better targeted to help older adults and their caregivers more successfully manage their healthcare needs, options, and other living expenses while making affordable and sound decisions.


Up-to-date, high quality, evidence-based clinical practice guidelines (CPGs) that are applicable for primary healthcare are vital to optimize services for the population with chronic musculoskeletal pain (CMSP). The study aimed to systematically identify and appraise the available evidence-based CPGs for the management of CMSP in adults presenting in primary healthcare settings.A systematic review was conducted. Twelve guideline clearinghouses and six electronic databases were searched for eligible CPGs published between the years 2000 and May 2015. CPGs meeting the inclusion criteria were appraised by three reviewers using the Appraisal of Guidelines Research and Evaluation (AGREE) II.Of the 1082 records identified, 34 were eligible, and 12 CPGs were included based on the inclusion and exclusion criteria. The methodological rigor of CPG development was highly variable, and the median domain score was 66%. The median score for stakeholder involvement was 64%. The lowest median score was obtained for the domain applicability (48%). There was inconsistent use of frameworks to aggregate the level of evidence and the strength of the recommendation in the included CPGs. The scope and content of the included CPGs focussed on opioid prescription.Numerous CPGs that are applicable for the primary healthcare of CMSP exists, varying in their scope and methodological quality. This study highlights specific elements to enhance the development and reporting of CPGs, which may play a role in the uptake of guidelines into clinical practice. These elements include enhanced reporting of methodological aspects, the use of frameworks to enhance decision making processes, the inclusion of patient preferences and values, and the consideration of factors influencing applicability of recommendations.CRD42015022098.PROSPERO The online version of this article (doi:10.1186/s13012-016-0533-0) contains supplementary material, which is available to authorized users. Chronic musculoskeletal pain (CMSP) is a global healthcare concern. The condition is classified as a part of chronic non-malignant pain, which encompasses musculoskeletal, neuropathic and visceral pain, and pain from sickle cell disease . CMSP neThe prevalence of chronic pain is high and increasing , 6, 7. AAccess to high quality, evidence-based, and up-to-date information resources can assist clinicians in making decisions about the care of patients with CMSP. Clinical practice guidelines (CPG) are one way of providing information on evidence-based and person-centred options for CMSP care. CPGs provide systematically developed clinical recommendations derived from best available evidence to guide clinicians and patients in making decisions about healthcare for specific clinical circumstances , 13. CPGThe validity of CPGs depends on how well they were designed and conducted . CliniciHigh quality, evidence-based, and up-to-date CPGs that are applicable for primary healthcare are crucial to optimize care for CMSP . The mulidentify profession-specific or inter-professional CPGs on the management of CMSP in primary healthcare settings;critically appraise the quality of the included CPGs using the AGREE II instrument;determine the currency of the CPGs; and todetermine the grading systems used to demarcate the level of evidence and the strength of the recommendation in the CPGs.The primary objectives were to:The study criteria were formulated using the PIPOH format  for guideline reviews . The typThe types of interventions could include evaluation, diagnosis, and management of CMSP. Examples of such interventions include inter-professional interventions, pharmacological, and non-pharmacological management, rehabilitative options, and self-management. CPGs that targeted any healthcare professionals involved in the management of CMSP were eligible for inclusion. The expected outcomes of CPGs could include patient outcomes, system outcomes, or public health outcomes. For the purpose of this review, only CPGs focused on primary healthcare settings were included. CPGs that focussed solely on secondary, tertiary, or specialist healthcare settings were excluded due to different management options offered.The primary investigator searched the electronic databases of the following guideline clearinghouses: the US National Guideline Clearinghouse (US NGC); Agency for Healthcare Research and Quality (AHRQ); Guidelines International Network (G-I-N); Scottish Intercollegiate Guidelines (SIGN); United Kingdom’s National Institute for Health and Clinical Excellence (NICE); New Zealand Guidelines Group (NZGG); WHO guidelines; TRIP database; National Institutes of Health (NIH); Monash University Centre for Clinical Effectiveness; Australia’s National Health and Medical Research Council (NHMRC); Canadian Medical Association Clinical Practice Guidelines Infobase, and the Institute for Clinical Systems Improvement (ICSI). In addition, the following online databases were searched to include those guidelines that were peer reviewed and published in journals: CINAHL, PEDro; PubMed, EBSCO host, and Medline. The broad search terms included: clinical practice guidelines; OR clinical guidelines; OR care pathway; OR clinical pathway; OR care protocol; AND chronic pain; OR chronic musculoskeletal pain; OR chronic non-malignant pain; OR chronic non-cancer pain; AND adults; AND primary care; OR primary healthcare.The initial search was conducted from July to October 2014, and the search was updated during May 2015. The inclusion and exclusion criteria were applied by screening the identified CPG titles and objectives to select the eligible CPGs. The data was extracted in the PIPOH format to enable this analysis. The process was verified by the co-authors on a random audit basis. A record of search yields and decision-making was kept.The clinical question formulation using the PIPOH format.Guideline currency: the publication date of the CPG and periods covered by the literature search in the guideline, date of revision.General information: the developing organization/authors; country of publication; language of publication.The primary investigator extracted the following data into custom built data extraction sheets:The methodological quality of eligible CPGs was independently assessed by three reviewers using the AGREE II instrument . The AGRThe reviewers’ AGREE scores were entered into a custom built Excel spreadsheet by the primary investigator. Any difference in score higher than two points was discussed amongst the project team to reach consensus. A quality score was calculated for each of the six AGREE II domains using the guiding principles and the following formula provided in the user manual:Domain score = (obtained score–minimum possible score)/(maximum possible score–minimum possible score) × 100 = %. All item scores in a domain were summed, and the total was standardized as a percentage of the maximum possible score for that domain. The median domain score and interquartile range (IQR) for each domain was then calculated .The results of the systematic search are summarized in Fig. Table The AGREE II domain scores (median and range) are provided in Table CPGs consistently inadequately adhered to topic 5 in domain 2 (views and preferences of target population). Only 3 of the 12 included CPGs reported that they sought patient perspectives as part the CPG development. The methods used in these CPGs included focus groups with patients (CPGs 3 and 2); patients as part of the guideline development group (CPG 5); a literature search of patients’ preferences (CPG 5); an environmental scan through surveys, key informant interviews and focus groups (CPG 2). The strengths and limitations of evidence (topic 9), was also inconsistently adhered to. Within domain 3 (rigor of development), topic 14 had the lowest scores, indicating that few CPGs included a procedure for updating the CPG. All four topics in domain 5 (applicability) were challenging for the CPG developers to report on, as indicated by the low scores.The analysis of domain 5 (applicability) indicated that algorithms and outcome measure tools for putting recommendations into practice was the most common strategies provided to facilitate implementation of CPG recommendations  was adequate; however, the range of these scores was wide. The criteria that presented challenges were the aggregation of the strengths and limitations of the body of evidence and updating the CPG. More than half of the included CPGs were current and were published 5 years prior to our systematic search. Guideline developers put little emphasis on providing procedures and dates for updating the CPGs, which was also a key deficiency found in Vernooij’s et al.  systematThis systematic review highlights the inconsistent use of frameworks to aggregate the level or body of evidence and the strength of the recommendation by the included CPGs. The criterion of using a framework to rate the strengths and limitations of the body of evidence is evaluated in the AGREE II domain of methodological rigor (topic 9). The AGREE II considers formal or informal methods for this domain. A framework that guides the decision-making process for recommendations ensures transparency and objectivity  and provThe inclusion of stakeholders in the development process of CPGs is assessed as part of AGREE II (domain 2), and adherence to this quality criterion is thought to enhance a sense of ownership, thereby facilitating the uptake of the guideline . While mThe AGREE domain of applicability (relevance) considers context factors that may impact CPG implementation, namely, the facilitators, barriers, and resource implications for implementation; tools for putting recommendations into practice, as well as monitoring and auditing criteria that influence the uptake and implementation of a CPG. These criteria therefore represent the features of the healthcare system, resources, cultural, and organizational factors that should be considered in framing guideline recommendations , 19, 36.Transparency and specificity regarding context factors that may impact recommendations is becoming increasingly important due to the option of adopting, adapting, or contextualizing CPGs for different settings, when de novo development of CPGs is not a feasible option. Knowledge about the context sensitive recommendations, may inform the above processes of CPG development. It is therefore an option that future CPGs on CMSP highlights preference or context sensitive recommendations ; and linCPGs can be an important information resource for the evidence-based and holistic management of CMSP and has the potential to influence policy and practice about CMSP care. Given the multidimensional nature of CMSP, it is important that the scope of future CPGs include a multidisciplinary approach, which implies that healthcare systems in which CMSP is managed should ideally allow for a multidisciplinary management approach. While some included CPGs elaborated on the cost and cost-effectiveness of interventions, the findings of this review indicate that there is great opportunity for CPGs to elaborate on cost and policy implications that guideline recommendations may have. The RNAO  includedWe identified several strengths in the CPGs, which included that guideline developers clearly communicated the overall objectives, scope, and focus of the CPG. The domain on clarity of presentation obtained a high score, indicating that CPGs presented their recommendations in a clear, easily identifiable, and specific format. Editorial independence had also been adhered to in most included CPGs.Include decision-making frameworks for the level of evidence, strength of recommendation, and framing recommendations in the CPG development process.Include the relevant stakeholders and end-users in the process of CPG development.Consider and document barriers and facilitators that may influence the uptake of clinical guideline recommendations in the intended setting.Develop and document strategies to facilitate the successful uptake and implementation of CPGs into practice.Provide dates and procedures for updating the CPG.This systematic review highlights the strengths and limitations of current CPGs on the management of CMSP in primary healthcare settings. Guideline developers who are considering adopting, adapting, contextualizing, updating existing CPGs, or developing new CPGs for CMSP should consider the findings of this systematic review to optimize the methodological quality and uptake of CPGs. Based on the findings of this review, the authors recommend that future CPGs should:We further recommend that, due to the many CPGs on the topic, existing high quality CPGs be contextualized to local circumstances, particularly in resource-constrained environments. In this way, resources may be used to further the uptake of CPG recommendations with a rigorous implementation and sustainability strategy, instead of resources being spent on de novo CPG development . Where dThe similarities between the findings of this systematic review and that of Shaneyfelt et al.  and MissOur review had some limitations of its own. We applied the AGREE II instrument, which evaluates the reported rigor of development and not the content of the CPG. The next step in our process is the content analysis of high quality CPGs for CMSP in primary healthcare settings. Our focus was specific to primary healthcare settings, and this may have excluded some guidelines that are applicable in this setting, but not stated overtly to be so. The CPGs we identified were predominantly from developed countries, so the appropriateness of these guidelines in other countries is uncertain. Furthermore, we acknowledge that although we did an extensive literature search, we may have missed eligible CPGs.The systematic review found several CPGs that are applicable for the primary healthcare of CMSP. The included CPGs vary in their scope and methodological rigor. There was little emphasis on non-pharmacological strategies in the composition of the guideline development team, the stated scope, and the content of the CPGs.This study highlights specific elements for enhancement in the development and reporting of future CPGs, which may play a role in the uptake of CPGs into clinical practice. These elements include enhanced reporting of rigor of development, the inclusion of stakeholder preferences and values and the consideration of context specific factors . Further research on methods to include patient preferences and how this impacts CPG content and implementation is needed.


Electronic health  literacy is a growing area of research parallel to the ongoing development of eHealth interventions. There is, however, little and conflicting information regarding the factors that influence eHealth literacy, notably in chronic disease. We are similarly ill-informed about the relationship between eHealth and health literacy, 2 related yet distinct health-related literacies.The aim of our study was to investigate the demographic, socioeconomic, technology use, and health literacy predictors of eHealth literacy in a population with moderate-to-high cardiovascular risk.Demographic and socioeconomic data were collected from 453 participants of the CONNECT (Consumer Navigation of Electronic Cardiovascular Tools) study, which included age, gender, education, income, cardiovascular-related polypharmacy, private health care, main electronic device use, and time spent on the Internet. Participants also completed an eHealth Literacy Scale (eHEALS) and a Health Literacy Questionnaire (HLQ). Univariate analyses were performed to compare patient demographic and socioeconomic characteristics between the low (eHEALS<26) and high (eHEALS≥26) eHealth literacy groups. To then determine the predictors of low eHealth literacy, multiple-adjusted generalized estimating equation logistic regression model was used. This technique was also used to examine the correlation between eHealth literacy and health literacy for 4 predefined literacy themes: navigating resources, skills to use resources, usefulness for oneself, and critical evaluation.P=.01), had lower level of education (P=.007), and spent less time on the Internet (P<.001). However, multiple-adjusted generalized estimating equation logistic regression model demonstrated that only the time spent on the Internet (P=.01) was associated with the level of eHealth literacy. Regarding the comparison between the eHEALS items and HLQ scales, a positive linear relationship was found for the themes “usefulness for oneself” (P=.049) and “critical evaluation” (P=.01).The univariate analysis showed that patients with lower eHealth literacy were older (68 years vs 66 years, This study shows the importance of evaluating patients’ familiarity with the Internet as reflected, in part, by the time spent on the Internet. It also shows the importance of specifically assessing eHealth literacy in conjunction with a health literacy assessment in order to assess patients’ navigational knowledge and skills using the Internet, specific to the use of eHealth applications. People are increasingly managing their health with the aid of electronic tools ,2. This Despite its increasing importance, to date there has been limited investigation into the demographic, socioeconomic, and technology use determinants of eHealth literacy. In healthy adults, lower age and higher education correlate to higher eHealth Literacy ,7-9, as People with cardiovascular disease (CVD) are required to self-manage many aspects of their condition, and this requires a minimal level of health literacy. Health literacy is defined as the “knowledge, motivation, and competences to access, understand, appraise, and apply health information in order to make judgments and take decisions in everyday life concerning health care” . In a poPeople diagnosed with or at risk for CVD were assessed to explore the relationship between demographic characteristics, socioeconomic factors, use of technology, health literacy, and eHealth literacy. The sample consisted of 453 participants in the Consumer Navigation of Electronic Cardiovascular Tools (CONNECT) Study . All parThe CONNECT study methods and participant recruitment processes are detailed elsewhere . In brieThe eHealth Literacy Scale (eHEALS) is one of the very few existing scales assessing eHealth literacy. It comprises 8 items scored on a 5-point Likert scale and aims to reflect the individuals’ own perception of their knowledge and skills at using eHealth information ,17. The The HLQ was used to assess health literacy. It comprises 9 independent scales that assess distinct aspects of health literacy and aims to measure an individual’s capacity at effectively using health information and services . Each scIn order to examine the relationship between health and eHealth literacy, we undertook a process of matching HLQ scales to eHEALS items by grouping related items with similar themes . eHEALS t test was used to compare the means between the 2 groups. To determine predictors of low eHealth literacy, multiple-adjusted generalized estimating equation logistic regression model was used. Independent predictors included in the model were gender , age (<65 or 65-70 or >70 years), education , income (<Aus $1000 or Aus $1000-2000 or > Aus $2000 per week), CVD-related polypharmacy (active consumption of >3 medications related to CVD), private health care (yes or no), main electronic device used (desktop or laptop or mobile phone or tablet), and time spent on the Internet on any device (≤1 hour or >1 hour per day). These variables were included regardless of the statistical significance in the univariable comparison due to their clinical significance in relation to eHealth literacy. This analysis adjusted for the clustering effect of primary health care practices. The derived odds ratios (ORs) and corresponding 95% CIs were plotted in a forest plot. An adjusted analysis using the eHEALS score as a continuous variable was also done to see whether other predictors emerged. To test for the correlation between eHealth literacy and health literacy for the 4 literacy themes, multiple-adjusted generalized estimating equation linear regression models were used for each of the themes. The dependent variable, eHEALS score, was in a continuous form, and the corresponding continuous HLQ score was included in the model with the aforementioned covariates. Data were analyzed using SAS version 9.4 for Windows (SAS Institute Inc).Univariate analyses were performed to compare patient demographic, socioeconomic, and technology use characteristics between the low (eHEALS<26) and high (eHEALS≥26) eHealth literacy groups. Chi-square test was used to compare the categorical variables, and independent In total, 453 participants were included in the analysis; 1 was excluded due to an incomplete eHEALS . The meaP=.02), had lower level of education (P=.007), and spent less time on the Internet  was associated with the level of eHealth literacy (P=.26), gender (P=.18), education (P=.19), income (P=.15), CVD-related polypharmacy (P=.22), private insurance (P=.47), and main device used to access the Internet (P=.30) did not achieve statistical significance.The univariate analysis showed that patients with lower eHealth literacy were older  and “critical evaluation”  and “skills to use resources” (P=.06), the 2 scales were not well correlated.After adjustment for demographic, socioeconomic factors, and technology use, a positive linear relationship was found between the eHEALS items and HLQ scales for the themes “usefulness for oneself” , white , and well educated , and all had access to the Internet. Furthermore, people who agreed to participate were likely to be more motivated and interested in their health management, which might have introduced an element of selection bias. Although this limited the generalizability of the findings to all populations with and at risk for CVD, an ongoing Australian CVD registry found 70% of male prevalence  and thatAs Internet-based eHealth interventions are increasingly being developed to facilitate patients’ health management, it becomes essential to gain an understanding of their eHealth literacy and identify its predictors. If users do not have an adequate level of eHealth literacy, certain Internet-based eHealth interventions could be compromised. This study has shown the importance of evaluating patients’ familiarity with the Internet, as reflected, in part, by the time spent on the Internet, to improve their eHealth literacy. It has also shown the importance of specifically assessing eHealth literacy in conjunction with a health literacy assessment in order to assess patients’ navigational knowledge and skills using the Internet. Although related, eHealth literacy requires knowledge of electronic resources and abilities to use them, which are distinct from purely an understanding of health or health literacy.