Cardiovascular diseases (CVD) are the largest cause of death and disability in Australia. Australian national guidelines for the primary prevention of CVD recommend that all adults without CVD and aged 45 years or more are screened for their absolute risk of CVD every 2 years. Despite the compelling evidence to address CVD risk, treatment gaps remain and evidence suggests that much of the shortcomings are attributed to the performance of primary care practices. To address this issue, a quality improvement initiative is being implemented in a large urban multidisciplinary primary care practice in the South West region of Victoria, Australia. The key outcome of this intervention will be to increase the use and acceptability of CVD risk assessment guidelines. To ensure the intervention is tracking toward its objectives, a robust monitoring and evaluation framework was established.A novel framework that assimilates key traditional and theory-driven evaluation practices was developed to assess the impact of the intervention. The framework approach is termed the integrated model of evaluation (IMoE). Researchers and stakeholders convened several times to discuss and develop the evaluation protocol and align it with the quality intervention. The main objective here is to explore the feasibility of an integrated approach to evaluating clinical quality improvement interventions. The sub-objectives are to test the alignment of the IMoE to clinical quality improvement projects and its ability to derive findings to the satisfaction of stakeholders. The design and establishment of the evaluation approach is discussed in further detail in this article.The novel feature of the IMoE is its emphasis on tracking ‘change’ in practices that lead to quality improvement. This emphasis suits the quality improvement theme of this initiative as identification of change elements and explanation behind change is necessary to sustain and promote quality improvement. The other principle behind development of this model, which emphasises practicality in implementation, is to ensure stakeholders gain greatest value from the commissioning of program evaluation. By incorporating practical components and leaving out esoteric concepts, this approach ensures evaluation can be undertaken in realistic timeframes.The quality improvement intervention and evaluation framework received approval from the Deakin University Human Research Ethics Committee . Australia’s Health 2016 advises that 13% of Australia’s deaths are due to coronary heart disease and 3% due to cerebrovascular disease [Cardiovascular diseases (CVD) are the largest cause of death and disability in Australia.  disease . Nationa disease , and theDespite the compelling evidence to address CVD risk, there is significant evidence that treatment gaps remain in the primary prevention of cardiovascular disease , 8. An eIn this context, a research study to investigate the feasibility of a quality improvement initiative to improve CVD screening has been intitated in a large urban practice  in the City of Geelong in the Barwon South West region of Victoria, Australia see Fig. . This reTo enable a robust monitoring and evaluation (M&E) framework that tracks the progress and the impact of the quality improvement intervention, the program will adopt an integrated evaluation approach termed ‘integrated model of evaluation (IMoE)’ to guide the M&E process. The integrated approach combines key aspects of traditional and theory-driven evaluation approaches. Traditional evaluation approaches focus mainly on before-after and input-output elements but have been questioned about their validity  because of this narrow focus –15. Theo▪ Test the alignment of an integrated evaluation approach, which emphasies quality improvement monitoring and program theory testing, to clinical quality intervention projects▪ Examine if the IMoE approach is adequate to derive findings to assess the outcomes of the intervenion and inform ongoing quality improvement planning in the practice▪ Explore the acceptability of the IMoE approach amongst the primary healthcare workforce as an appropriate tool to monitor and evaluate clinical quality improvement projects.The main objective of this study is to explore the feasibility of implementing an integrated approach to evaluating clinical quality improvement interventions. The sub-objectives of the study are to:Kardinia Health is a large primary health care practice, located in the City of Greater Geelong (Barwon South West region of Victoria) that incorporates the values of traditional general practice with multidisciplinary, team-based care and an academic research unit in general practice , 18. KarProgram theory: is a causal statement outlining the expected outcomes as a result of the program intervention in a particular context , 15. TheContext: describes the situation in which the program has been introduced and is operating. The context includes geo-political, economic and other scenarios that influence the program’s implementation and outcomes. The context component is often ignored in many generic evaluation approaches exposing the results to poor validity . ProgramIntervention: includes the resources and outputs being introduced through the program. Inputs and outputs are data collected through regular evaluations but the IMoE groups them under the ‘intervention category to distinguish it from the “Change” and “Outcomes”’ components in the model. This is necessary not only to highlight the intervention but also present a plausible sequence of events leading to the outcomes as depicted in the program logic and program theory.Change: incorporates variations that have occurred as a result of the program intervention. Change is an important outcome in health interventions especially so in quality-focused interventions –23. HoweOutcomes: are the end results of a program, i.e. the objectives or the goals the program set to achieve , 20, 24.The key components of the IMoE are as follows:The key elements in the IMoE approach are visually represented in Fig. To enable a governance process, Kardinia Health has a steering committee that will oversee the implementation of the IMoE model. The committee’s main role is to advise the evaluators about implementation and receive regular reports about the progress of the evaluation. The evaluation process will complement the quality intervention for the duration of the program. In parallel to the quality intervention, the evaluation will be implemented in three stages: formative, process and summative. The key emphasis of the IMoE model is tracking ‘change’ in practices that lead to quality improvement and this will be reflected in all three stages of the evaluation.The IMoE approach will commence with a formative evaluation to ensure the quality improvement goals and the resources required to meet these goals are in line with stakeholder’s expectations. The formative evaluation will take into account pre-intervention or baseline key performance indicators (KPIs) to track program progress and impact. In addition to this, a preliminary program theory to explain the change process will be developed. Over the course of the program and as part of the process evaluation, KPI data will be collected to track progress toward program goals. At the end of the program, a summative evaluation to assess the efficacy and efficiency of the program will be undertaken. Also, the preliminary program theory will be revisited to suitably explain the change that has resulted from this intervention. This comprehensive approach will ensure a thorough assessment of the intervention (pre and post) is undertaken.The percent of active patients aged 45–74 years with all necessary data for CVD absolute risk assessment collected over the past 2 years (with or without CVD)—age, gender, blood pressure, total and HDL cholesterol, diabetes, status smoking status.Percent of patients at high risk of CVD or with CVD who are treated appropriately with medications (lipid lowering plus antihypertensive medicines).Number of vulnerable patients with CVD risk being screenedThe availability of high quality, reliable and pertinent data underpins the ability to undertake quality improvement. The main KPIs for the intervention have been identified as follows:In addition to monitoring the collection of absolute cardiovascular risk factor data , the evaluation will be assessing if modifiable/behavioural risk factors  are being screened by clinicians. Literature has indicated complementing screening of cardiovascular absolute risk factors with modifiable risk factors followed by relevant intervention maximises the preventative potential of the intervention . InvolveIndicator data will be extracted from the practice’s patient information system. The practice will develop a register of patients requiring risk screening in accordance with national guidelines. Risk factor monitoring will also be performed with all patients that have been diagnosed with CVD. Data extraction, capturing a broad range of patient information, will also be used to conduct detailed examination of evidence treatment gaps across the practice relating to CVD risk assessment and management.The integrated program logic for this evaluation, based on the IMoE model, is outlined in Figure (Context), a quality improvement initiative involving a collaborative model has been introduced (Intervention) to improve CVD screening and identification of at risk CVD patients. The initiative will work by improving compliance of clinicians with national CVD guidelines (Change). The change will lead to improved patient data quality and CV health outcomes at Kardinia Health (Outcome).In the context of a research and quality oriented integrated health practice One of the important features of the IMoE approach is the development of and testing of a program theory . The preliminary program theory for the quality intervention is outlined here:To assess the program theory, semi-structured interviews of Kardinia Health staff and patients coupled with document analysis will be undertaken. A purposive sampling strategy will be used to identify interview patients who are 45–75 years. All staff involved in the strategy group (steering committee), co-design group and project management group (there are three distinct groups) will be invited to participate in the semi-structured interviews. Interviews will be conducted in the formative and summative evaluation stages to assess the validity of the program theory and revise as necessary.To assess the efficacy of the quality improvement intervention, data from over 9000 patient records will be extracted for analysis by the new software. The KPIs mentioned earlier will be derived from this phase of analysis. For the semi-structured interviews, it is expected that a total number of 22 participants (10 staff and 12 patients) will be available. The 12 patients will be derived from the co-design patient group that has already been established for the clinical quality improvement intervention and the 10 staff will comprise a mix of doctors, nurses, allied health and managers working in the practice. From a qualitative research perspective, this number of participants seems adequate to help with the testing of the program theory . The dat2 test for equal proportion  and will be reported as either percentages (n) or percentages (95% CI). Continuous normally distributed variables will be compared using Student t tests and reported as means (95% CI). Nonparametric data will be compared using Wilcoxon rank-sum tests and reported as medians (interquartile range). Pairwise differences between pre- and post-intervention values will use paired t tests for normally distributed data, Hodges-Lehmann median difference estimate for non-normally distributed continuous variables and related-samples McNemar’s test for changes to proportions. Statistical process control techniques will be investigated for appropriateness in confirming real improvements to performance. In essence, this attempts to distinguish any change in performance as a result of quality improvement measures, from underlying trends in changing performance.To analyse the KPIs pre and post-intervention, univariate comparisons between groups will be conducted using χThe quality improvement intervention and evaluation framework received approval from the Deakin University Human Research Ethics Committee .Program evaluation is a well-established methodology to assess the effectiveness and efficiency of programs , 27, 28.To address inherent limitations of both traditional and theory-driven approaches, the IMoE utilises essential components from both evaluation approaches. The model includes traditional aspects of program evaluation such as program logic components: inputs, outputs, outcomes and KPIs. In addition, it includes theory-driven elements such as contextualisation of the research and formulation of program theory, which are often ignored in traditional program evaluation models at the detriment of the validity of the results. The model also includes innovative elements such as the emphasis on change. This approach we consider is particularly relevant to evaluating this quality intervention. It is known through literature that primary care clinicians find it difficult to incorporate CVD guidelines and risk assessment tools in their practice , 9, 10. The main principle behind development of the integrated model is to ensure stakeholders gain greatest value from the commissioning of program evaluation. By incorporating practical and useful components and leaving out esoteric concepts, the integrated model ensures deployment of this model can be done in realistic time frames. The emphasis of the IMoE is not the methods or academic interests of the evaluator but usefulness of the evaluation results to stakeholders, i.e. are the results valid and can they be acted upon? This is achieved through incorporation of the program theory and emphasis on the context, intervention and change. Also, as there is emphasis on program theory formulation, stakeholders are involved from the very onset. This provides an opportunity for their assumptions to be tested through a rigorous approach. As the IMoE incorporates context in the construction of the program theory, it ensures the results are tailored to the particular program, organisation and scenario. Further, by incorporating program logic elements, the IMoE ensures the implementation is practical, results are understandable to stakeholders and the model is not restricted to impact assessments only.The main limitation with this study is the first time an integrated approach is being considered to monitor and evaluate clinical quality improvement interventions. It is not known if the model is suitable to such interventions. Hence, one the objectives of this study is to test the alignment. The other caution is about the generasibility of the study findings to other clinical quality improvement interventions. While there is every intention on part of the authors to promote the wide use of the IMoE approach per say, the findings of this evaluation should be reviewed in the context of the Kardinia Health practice environment. Hence, the contextual description in the program theory and program logic to emphasise the prevalent conditions in Kardinia Health. In spite of these limitations, the authors consider the IMoE approach to have several strengths. The IMoE brings together the best of the traditional and theory-driven approaches of program evaluation. While it incorporates several components from both the approaches, the enjoining does not result in a complex or unwieldy model. In fact, a streamlined stakeholder centric process is constructed. To enable wider use of the model following completion of the study at Kardinia Health, the researchers involved in this study will be pursuing the implementation of this approach at other healthcare sites.


The American College of Surgeons National Surgical Quality Improvement Program (NSQIP®) developed a surgical risk calculator using data from 1.4 million patients and including 1557 unique Current Procedural Terminology (CPT) codes. Although this calculator demonstrated excellent performance in predicting postoperative mortality, morbidity, and six surgical complications, it was not developed specifically for use in older surgical patients who have worse surgical outcomes and additional unique risk factors compared to younger adults. We aimed to test the ability of a simple self-reported mobility tool to predict postoperative outcomes in the older surgical population compared to the NSQIP.We used data from a prospective cohort study that enrolled 197 older surgical patients (≥ 69 years) undergoing various elective surgeries and assessed 30-day surgical outcomes. Statistical models included data from the Mobility Assessment Tool-short form (MAT-sf) alone, covariates alone, and MAT-sf data and covariates. We used leave-one-out (LOO) cross-validation of the models within our cohort and compared their performance for predicting postoperative outcomes against the NSQIP calculator based on receiver operating characteristic area under the curve (ROC AUC).p > 0.1), with AUC between 0.604 and 0.697 for predicting postoperative complications and 0.653 and 0.760 for predicting nursing home placement. All models also predicted a length of stay (LOS) similar to the actual LOS.Patients with poor self-reported mobility experienced higher rates of postoperative complications and nursing home placement. There was no difference in performance between any of our models and the NSQIP calculator (Mobility assessment alone using MAT-sf can predict postoperative complications, nursing home placement, and LOS for older surgical patients, with accuracy comparable to that of the NSQIP calculator. The simplicity of this noninvasive risk assessment tool makes it an attractive alternative to the NSQIP calculator that requires 20 patient predictors and the planned procedure, or CPT code to predict the chance that patients will have 15 different adverse outcomes following surgery. The American College of Surgeons National Surgical Quality Improvement Program (NSQIP®) surgical risk calculator , and found it to be a good predictor for postoperative complications, hospital length of stay (LOS), and nursing home placement  and 0.58 (p < 0.001), respectively. It is also of interest to point out the standardized β weight for the MAT-sf was substantially larger than the PAT-D mobility subscale in both analyses  who were undergoing elective, noncardiac surgery from July 2012 to February 2014 , wound disruption, pneumonia, unplanned intubation, thromboembolism, on ventilator > 48 h, urinary tract infection, progressive renal insufficiency, acute renal failure, cerebrovascular accident, cardiac arrest, myocardial infarction, and sepsis  were compared using chi-square tests. Length of stay was compared across MAT-sf tertiles using the Wilcoxon rank sum test.Sex differences in MAT-sf scores were compared using Student’s To compare our risk models to the ASC NSQIP surgical risk calculator, we first calculated predicted surgical outcomes using risk factors obtained at enrollment by the NSQIP calculator  and nursing home placement (p = 0.009), as well as longer LOS . Men had higher MAT-sf scores (58.3 [IQR 48.3–65.5]) than women (49.9 [IQR 42.2–55.5]), p = 0.54). Figure p = 0.58). Table We developed and cross-validated models using MAT-sf only, covariates only , and MAT-sf plus covariates to predict postoperative outcomes. Table We set out to develop a surgical risk calculator that was simple and specific to the older surgical patient population. We developed three models using data from our prospective cohort of 197 older surgical patients and cross-validated the models to provide robust probabilities of risk. The model using only the MAT-sf score had similar ability for predicting postoperative complications and nursing home placement as the NSQIP surgical risk calculator. Thus, the MAT-sf score provides a simple, easy to use tool to predict surgical outcomes in older patients. The model performed as well as the NSQIP surgical risk calculator in predicting LOS.The NSQIP risk calculator was developed using data from more than 1.4 million patients and has demonstrated excellent performance in predicting postoperative outcomes (American College of Physicians p < 0.0001). However, the stair climbing task caused physiologic stress, especially in those patients who were slower climbers, causing a 16.6 and 21.8% increase in heart rate and mean arterial pressure, respectively, compared to 7.4 and 7.9% for faster climbers  or reflect the risks fully , the predictions for LOS using the simple MAT-sf model were remarkably similar to the universal calculator. Adequately incorporating procedure complexity and risk into future studies that use the simple MAT-sf could solidify its efficacy in forecasting surgical outcomes.n = 197), which may not accurately reflect surgical risk.The limitations of the study include its single-center design and use of a majority white patient cohort, features of the study design that compromise the external validity of the findings. We were unable to include the full list of measures included in the NSQIP calculator due to our small sample size and lack of variability in some measures such as cancer and chronic heart failure, among others. Since this issue could have accounted for the similarities observed between the universal calculator and the MAT-sf models, the results must be interpreted with caution. Nonetheless, we expect that our estimation of postoperative risks and LOS with the MAT-sf tool may have been better if we had a larger sample with increased variability. Although we used the LOO cross-validation method, our findings still need to be tested in a separate cohort. Lastly, the models were developed from relatively small number of older patients  in older surgical patients with predictive performance comparable to the NSQIP surgical risk calculator. Given the simplicity and noninvasive nature of the MAT-sf tool, it can be easily adopted into clinical practice for preoperative risk assessment. Future studies in a larger cohort of patients undergoing a single type of surgery are warranted to validate this new method to predict postoperative outcomes.


Although program directors judge residents’ performance for summative decisions, little is known about how they do this. This study examined what information program directors use and how they value this information in making a judgment of residents’ performance and what residents think of this process.Sixteen semi-structured interviews were held with residents and program directors from different hospitals in the Netherlands in 2015–2016. Participants were recruited from internal medicine, surgery and radiology. Transcripts were analysed using grounded theory methodology. Concepts and themes were identified by iterative constant comparison.When approaching semi-annual meetings with residents, program directors report primarily gathering information from the following: assessment tools, faculty members and from their own experience with residents. They put more value on faculty’s comments during meetings and in the corridors than on feedback provided in the assessment tools. They are influenced by their own beliefs about learning and education in valuing feedback. Residents are aware that faculty members discuss their performance in meetings, but they believe the assessment tools provide the most important proof to demonstrate their clinical competency.Residents think that feedback in the assessment tools is the most important proof to demonstrate their performance, whereas program directors scarcely use this feedback to form a judgment about residents’ performance. They rely heavily on remarks of faculty in meetings instead. Therefore, residents’ performance may be better judged in group meetings that are organised to enhance optimal information sharing and decision making about residents’ performance.The online version of this article (10.1186/s12909-018-1446-1) contains supplementary material, which is available to authorized users. Competency-based medical education (CBME) has been introduced over the past decades to ensure that residents in postgraduate training programmes attain the high standards and competencies that are required to become medical specialists –4. New mTo monitor growth in competencies, programmatic assessment has been introduced , 8. ProgProgram directors in postgraduate medical education are responsible for making decisions about residents’ overall performance. They are supposed to do this by aggregating information from various tools, which are completed by different supervisors . In pracLiterature provides a number of relevant insights that can be taken into account when studying how judgments of residents’ performance are established. Oudkerk Pool et al. studied how assessors integrate and interpret different tools from students’ portfolios . Their fLittle is known about how program directors form judgments of residents’ overall performance and how they feed this back to residents. It is relevant to obtain this knowledge because program directors have the ultimate responsibility for the performance of their residents , 13. TheThe aims of our study were twofold. The first aim was to investigate what information program directors use and how they value this information in making a judgment of residents’ performance. The second aim was to investigate how residents think that program directors do this, to find out whether there is a gap between what they think is important and what program directors actually value as important. Therefore, we sought to answer the following main research questions: (i) What information do program directors use to make a judgment of residents’ performance and how do program directors value the different sources of information? (ii) What do residents believe about the manner in which program directors make judgements of their performance?We conducted our study in the Netherlands, where all postgraduate medical training programmes are competency based. In the Netherlands, residents are medical doctors who work under the direct or indirect supervision of medical specialists. They are specialty trainees who are trained in order to obtain a license to practise a chosen specialty. They are trained and supervised by many faculty members, but the program director is the ultimate arbiter of whether progression in residents’ performance is adequate. Program directors must hold evaluation meetings with each resident at least twice a year. They are then expected to give residents an official judgment of their performance (below/at/above expected level) and provide them with sufficient feedback to set goals for further development. Program directors, however, are not necessarily the ones who work together with residents routinely in the clinic. They need to rely on various information sources to know how residents perform at the workplace.Residents are expected to collect feedback from multiple assessment tools in a portfolio to provide evidence for their progress and competency development. These assessment tools are, for example, results from in-service exams, OSATs and mini-CEXs. The assessment tools contain ratings as well as narrative feedback. The inclusion of the assessment tools in the portfolio is resident-driven with the exception of the in-service exam. Residents decide when to ask a supervisor to complete an assessment form and provide feedback on their performance. Program directors are supposed to make a robust judgment on residents’ progress based on these different data points. Some programmes also hold faculty group meetings to discuss the content of the tools in relation to residents’ performance. These meetings are not mandatory in the Netherlands; program directors can decide whether or not to organize a meeting and if so, how to set up these meetings.The goals of this study were to gain insight into the process of how program directors gather information about residents to judge their performance and to explore how residents think about this process. We used a constructive grounded theory approach to do so . GroundeN = 8) and residents (N = 8). The interviews lasted between 45 and 60 min. Program directors differed in years of experience and residents in their year of training  conducted all the interviews. Program directors and residents were interviewed separately immediately after they had an evaluation meeting together. Interviews were recorded and transcribed verbatim. All identifying data were omitted. We used two separate but similar interview guides for program directors and residents  separately coded the first four interviews (two interviews with program directors and two with residents), meaning that they organized the data into initial key concepts and themes . After dAn important principle of constructivist grounded theory is that the construction of concepts and themes arises through interaction with the participants and other researchers in the team . It is iThe results of the two main research questions are presented successively. The results are supported by quotes from program directors (O) and residents (A).We identified three sources of information that program directors predominantly use to form a judgment of the residents’ performance: assessment tools in the portfolio; faculty; and their own experience and personal connection with a resident.O3: There’s a lot of irrelevant information in the portfolio; I have to search for little pieces of relevant information.Most program directors mainly saw the portfolio as a tool to confirm their judgment already formed by their own experience and remarks made by faculty members. Program directors noticed that the portfolio was almost always filled with good results and positive feedback. If resident’s performance was adequate according to the program director, the portfolio was seen as proof that the resident was indeed performing well. If their judgment on resident’s performance was not in line with what the portfolio’s content appeared to imply, the portfolio was considered an inadequate tool and interpreted as providing an overly positive image of the resident.One cause of the portfolio being more positive than reality was the fact that faculty seemed hesitant to give negative feedback.O6: I don’t think the portfolio is always that correct. It mostly says “this is good”, while you may later hear that it wasn’t that good at all and that it was a lot less pretty than the portfolio suggests.Another reason program directors gave to explain why the portfolio was more positive than reality was that residents often only ask feedback after well-performed tasks and this influences their representation.O5: The really good residents have many assessment tools filled in their portfolio. They ask for a lot of feedback, also on difficult tasks. But the average ones don’t ask for much feedback, so staff won’t notice that they’re just average residents. They only ask feedback after they performed a task well.Program directors saw faculty meetings as an important source of information about residents because they knew that faculty found it difficult to put negative feedback or points for improvement down on paper, while acknowledging that not all information that was shared in these meetings was useful. There was a tendency to follow the first or the loudest opinion expressed during such meetings. Program directors considered this in the process of judging residents’ performance and took into consideration who said what and how. They were also aware that they put a greater value on some faculty members’ opinions than those of others.O8: When this person says something, I really see it as a red flag. This person always keeps his opinion to himself and his comment is really astute. When someone else says something, I may think: yeah sure, I hear that three times a week.Once faculty members had formed an opinion about a resident, they were not prone to change their minds, not even when the resident changed his/her behaviour. Program directors were aware of this and took it into consideration when they valued information about residents. If they noticed that the group followed one member’s opinion or based their opinion on one incident, they did not take it seriously.O1: I’m annoyed by many of my colleagues. They may have their negative opinion ready in just one second. They don’t say anything for a long time and then suddenly they say “this resident is worth nothing” and they won’t ever change their mind. I don’t like that, you know. But well, my colleagues are all different personalities […] and if I’ve learned anything in the past years as a program director it’s that I know how they judge people. Some of them draw conclusions too soon. I have to take this into account and not take it too seriously. I must be neutral as a program director.In general, program directors felt that faculty did not put education and training first. Some faculty members were more interested and involved in teaching and training than others and, as a result, their feedback was more appreciated by program directors.O1: My personal opinion about my colleagues is important. You know about one person that he’s not interested and doesn’t put any effort in residents’ training. And you know about another person that they take time and think about the feedback they’ll give before they write it down. Yes, that’s why I don’t take everyone seriously.O3: We have certain subgroups that never complete the assessment tools, or if they do, they do it weeks later. You cannot rely on them. We try to change this, but it’s a persistent problem.Program directors who felt more supported by their colleagues took their feedback and suggestions about residents seriously:O2: I don’t teach them [residents], but the whole group does. I think that’s because I give them [faculty] the responsibility: they feel responsible. I couldn’t do it all by myself. I need the interaction with my colleagues; they need to think with me, also about the really good residents.Program directors explained that their own opinion was an important influence on their overall appraisal. They got to know residents during meetings, nightshifts, and shift-to-shift handovers. Program directors were aware that the personal connection between them influenced their judgment.O8: There’s something in the assessment process that has to do with the personal bond you feel with this resident, something like a “personal preference”. So there’s a danger that I don’t judge all residents by the same standards.The program directors’ beliefs about teaching and learning seemed to influence how they valued information about residents’ performance and seemed to affect their judgment, as well as the feedback they provided to the residents in the evaluation meeting.O4: The good residents don’t need much feedback; if all goes well, it can be summed up in one or two sentences, like, erm, “everything goes well, no problems”.Some program directors expressed their belief that a resident’s level of performance would never change: residents who are not so good will never reach high standards and residents who perform well do not need much feedback.O5: With a somewhat dysfunctional resident, I don’t think that everything will stay bad, but it’ll never be totally good.O6: You know immediately how a resident performs. A resident who performs less than the others will never become a really good one.Other program directors thought that residents’ competencies could grow through training, learning, and applying feedback. They put great value in their judgments on whether or not residents applied constructive feedback and changed their behaviour.O1: He’s a resident with weaknesses, but he handles critical feedback very well and tries to apply it. I’ve seen him develop and grow in his performance. I value this more than residents who perform well but do nothing with the feedback I provide.Residents thought that program directors put great value on the portfolio and they believed they could influence the program director’s assessment through the assessment tools that they collected in their portfolio. They saw these as tools to demonstrate their performance and therefore often only asked supervisors to complete an assessment tool after a well-performed task.A1: All the good mini-CEXs in my portfolio are a reassurance to me. The program director can do nothing but give me a good appraisal. It is stated in black and white.A4: As a resident, you can influence your appraisal because if you only put OSCEs on the table after you’ve done something really well, then… well, I mean, what negative things could the program director say about you?Residents knew that faculty talked about them in faculty meetings and in the corridors, but they thought that program directors did not take these comments very seriously, as illustrated by the following quote:A1: I don’t think they put so much value on these things, especially when it concerns some vague email or some vague comment from a colleague.Residents were confident that program directors base their judgment on the assessment tools from their portfolios.Like program directors, residents noticed that some faculty members do not put teaching and training first. They were critical about the way faculty gave feedback; they felt that their supervisors were hesitant to present points for improvement, did not take enough time to give feedback, and made vague comments on overall performance.A6: Faculty don’t always think that training residents is interesting or important. They don’t let you get involved in research or they don’t like to teach things. […] I specifically ask for points for improvement, but if the only thing I hear is “keep up the good work”, then I give up. I think to myself “this is hopeless; no matter how many times I ask for constructive feedback, they won’t give it.”A5: They don’t have the time or don’t like to teach you things. I miss getting feedback. […] I don’t want only positive feedback, but I also want to hear things I can improve. I try to ask for this. I always ask them what they think of my work. I always ask for this. But I miss the spontaneous feedback and I want to learn from the things that didn’t go that well.Residents felt that once faculty had formed an opinion about them, it was very hard to make them change their minds. Furthermore, they noticed the same group dynamics as program directors did: faculty followed the opinion of the loudest mouths.A1: When something happens, faculty tend to bear this in mind forever … and I think there’s a minority that talks very loudly. The majority may think differently, but they keep quiet and don’t say a word.A1: I think that this has been my rescue. I think that having a good relationship with him has really been my rescue.Residents thought that their personal connection with the program director was of importance and could even alter their appraisal of them.A4: He [peer resident] has a really good relationship with her [program director]. They just have a really good connection, and he talks his way out of things.Our findings show that program directors scarcely use feedback from assessment tools to form a judgment on residents’ overall performance, but heavily rely on remarks of faculty in meetings instead. Contrarily, residents think that the feedback in the assessment tools is the most important proof to demonstrate their performance.The usage of formative assessment tools to facilitate learning, as well as the aggregation of multiple formative assessment tools to come to a summative decision about resident’s performance , both seThere is a contradiction in what residents wish for and how they act. On one hand, they are disappointed in hardly receiving meaningful feedback, because they feel it does not support their learning process. But on the other hand, residents seem to perceive the assessment tools as summative instead of formative data points. They are under the impression that program directors put much value on the feedback in the assessment tools (both ratings and narrative comments) and seem to think that program directors also perceive the tools as summative assessments. As a result, they only ask feedback after well-performed tasks because they are afraid to receive negative feedback. This is in line with previous research showing that assessment tools are frequently used for summative assessment but not for formative assessment to facilitate learning –32.Moreover, program directors point out that narrative feedback in assessment tools is often vague and predominantly positive on overall performance. This finding is consistent with previous research on feedback in medical education; faculty hesitate to give constructive or negative feedback –36. HoweAs a consequence of the above-mentioned issues, program directors cannot rely on feedback in formative assessments to make summative judgments. Instead, they are forced to turn to other sources to get feedback on residents’ performance. They rely on comments of faculty members during faculty meetings. Residents do not know what is discussed in these meetings and they do not receive feedback based on these meetings. At the same time, program directors acknowledge the problem that the information sharing process during faculty meetings is suboptimal. Literature tells us that ineffective information sharing endangers a good group decision and therefore jeopardizes the integrity of the evaluation of residents’ performance . It is tFurthermore, as the main objective of assessment within CBME is to support residents’ learning and development, it is problematic that some of our interviewed program directors seem to have a fixed-mindset   becomes obligatory and transparent. In the United Stated, the Accreditation Council for Graduate Medical Education (ACGME) already requires Clinical Competency Committees to determine residents’ competence . To arriA strength of our study is the diversity of our sample: we included participants from different hospitals, medical specialties, and experience levels in training and teaching. A limitation of our study is that we used a rather small sample in the specific postgraduate medical context of the Netherlands and we do not know if our results would apply to other program directors, programs or other countries.We believe, though, that our study lays a foundation for future research in other settings, so to further our understanding on how to optimize the process of forming robust and acceptable judgment on residents’ performance progress.Residents think that the feedback in the assessment tools is the most important proof to demonstrate their performance, whereas program directors scarcely use this feedback to form a judgment about residents’ performance. The objective of aggregating formative assessment tools to form a summative judgment is difficult to reach in practice. Formative assessment is perceived as being summative and faculty provides poor quality feedback. Program directors rely heavily on remarks of faculty in meetings instead. They acknowledge that there is no optimal environment for good decision making during these meetings. We suggest that group decision making concerning residents’ performance becomes obligatory, provided that these meetings are set up according to guidelines that support an environment for optimal information sharing and decision making. Furthermore, they should accommodate high quality feedback to residents, in order to facilitate their learning.Additional file 1:Appendix 1 Semi-structured interview guide program director. The semi-structured interview guide we used for the interviews. (DOCX 54 kb)Additional file 2:Appendix 2 Semi-structured interview guide resident. The semi-structured interview guide we used for the interviews with the residents. (DOCX 56 kb)


The endobronchial ultrasound is a minimally invasive technique that simultaneously associates ultrasound and bronchoscopy, to visualize lung nodule or masses, airway wall, and structures adjacent to the tracheobronchial tree. Endobronchial ultrasound has been incorporated into clinical practice all over the world because of its low risk and high diagnostic yield in neoplastic and non-neoplastic disease. Since it was introduced by Hurter and Hanrath, in 1992, endobronchial ultrasound (EBUS) has been a useful technique that allows the bronchoscopist to see beyond the airways, including airway wall, lung and mediastinum. Endobronchial ultrasound is performed during bronchoscopy and permits sample collection for different thoracic disease. There are two types of EBUS: convex-probe EBUS (CP-EBUS) and radial-probe (RP-EBUS). These technologies will be discussed in the following topics.1The currently available CP-EBUS is a dedicated ultrasound equipment, placed on the tip of a flexible bronchoscope to obtain images of the airways by direct contact of the probe with the tracheobronchial wall. The equipment has a working channel, Doppler function and a dedicated needle to perform a safe transbronchial needle aspiration, with ultrasound images, in real-time. The procedure is called EBUS transbronchial needle aspiration (EBUS-TBNA). All lesions near or in contact with the tracheobronchial tree can be accessed by EBUS-TBNA. In this way, the most frequent indications for EBUS-TBNA are diagnosis of mediastinal and hilar lesions of any etiology, mediastinal staging and restaging of lung cancer, staging of extra-thoracic cancer.The needle size varies from 19 to 25G, for histological and cytological, respectively. The needles available in Brazil are for cytological sampling only .2 established a mediastinal map and the lymph node stations were numbered from 1 to 14. The EBUS-TBNA can assess the stations 2, 4, 7, 10, 11 and 12. The assessment of each lymph node station depends on clinical indication, computed tomography (CT) or positron emission tomography CT (PET-CT) findings, and etiology of the tumor. For example, during mediastinal staging of lung cancer, it is mandatory to start from N3 station , followed by N2  and N1 nodes  at the end of exam. As we have a large number of lymph node stations to be evaluated, it is necessary to choose the suspected lymph node to be sampled by EBUS-TBNA according to ultrasonographic aspects. Malignant nodes tend to be round, with well-defined margin, heterogeneous, without central hilar structure. Furthermore, it is important to add this information to tomographic or PET-CT analysis. On Especially for mediastinal staging of lung cancer, EBUS-TBNA requires a systematic assessment. In this context, in 2009 the International Association for the Study of Lung Cancer (IASLC)3 The remaining material is placed in formalin for cellblock analysis. Molecular analysis for cancer can be carried out in EBUS-TBNA samples.5 If non-cancer is suspected during ROSE, the sample can be sent to microbiological analysis, flow cytometry or polymerase chain reaction (PCR). The sensitivity of EBUS-TBNA ranges from 84 to 96%.7Other important factor during EBUS-TBNA is the rapid on-site evaluation (ROSE) by a pathologist, to assess the representativeness of the sample and conduct the detailed specific analysis.osurrounding view of the parenchyma. It also helps to identify the correct bronchus of the lesion, based on echogenicity differences between bronchus, lesion and normal tissue.8 In this way, the development of RP-EBUS has emerged to improve bronchoscopy diagnostic sensitivity and accuracy for pulmonary nodules and masses.8Radial-probe EBUS is performed using a 20mHz flexible delicate probe , which is inserted through the working channel of the conventional bronchoscope towards to the target pulmonary lesion. It allows a 3609 For instance, Radial-probe EBUS should be performed under fluoroscopy guidance to help evaluating the target lesion. Cytologic examination and transbronchial biopsies  can be performed during the procedure, and the samples can be sent for microbiological, cytological and histological analysis, depending on clinical, radiological and cytological evaluation.8Some factors, such as nodule size and capacity to visualize and locate the probe inside the lesion may affect the diagnostic yield of RP-EBUS. Our preliminary experience with RP-EBUS in Brazil showed good sensitivity for both nodules and masses .10EBUS-TBNA and RP-EBUS are safe procedures with low complication rates. The most common complication in EBUS-TBNA is damage to equipment during needle manipulation. Other complications like bleeding and infectious are rare. In RP-EBUS the complications are pneumothorax and bleeding, ranging from 1 a 4% and 3 a 5%, respectively, in most series.Radial and convex endobronchial ultrasound is a safe procedure, with low complication rate and good sensitivity for both nodules and masses. endobronchial ultrasound ) tem se mostrado técnica útil, que permite ao operador visualizar além das vias aéreas, incluindo a parede das vias, o pulmão e o mediastino. A EBUS é realizada durante a broncoscopia e permite a coleta de amostras para diversas doenças torácicas. Há dois tipos de EBUS: a EBUS com transdutor convexo (CP-EBUS - convex-probe EBUS) e com transdutor radial . Essas tecnologias são discutidas nos tópicos a seguir.Desde sua introdução por Hurter e Hanrath, em 1992, a ultrassonografia endobrônquica . Todas as lesões próximas ou em contato com a árvore traqueobrônquica podem ser acessadas por EBUS-TBNA. Por isso, as indicações mais frequentes para EBUS-TBNA são diagnóstico de lesões mediastinais e hilares de qualquer etiologia, estadiamento mediastinal e restadiamento de câncer de pulmão, e estadiamento de câncer extratorácico.O tamanho da agulha varia entre calibre 19 e 25G, para análise histológica e citológica, respectivamente. As agulhas disponíveis no Brasil servem apenas para coleta de amostras citológicas .International Association for the Study of Lung Cancer (IASLC)2 estabeleceu um mapa mediastinal, e as cadeias de linfonodos foram numeradas de 1 a 14. A EBUS-TBNA consegue acessar as cadeias 2, 4, 7, 10, 11 e 12. A avaliação de cada cadeia de linfonodos depende da indicação clínica, dos resultados da tomografia computadorizada (TC) ou da TC por emissão de pósitrons (PET-TC), e da etiologia do tumor. Por exemplo, durante o estadiamento mediastinal do câncer de pulmão, é obrigatório começar pela cadeia N3 , seguida de N2  e N1  no fim do exame. Como é grande o número de cadeias de linfonodos para avaliar, é necessário escolher o linfonodo suspeito do qual deve ser coletada a amostra para EBUS-TBNA, de acordo com aspectos ultrassonográficos. Linfonodos malignos tendem a ser arredondados, com margens bem definidas, heterogêneos e com estrutura hilar central. Além disso, é importante acrescentar essas informações à análise por tomografia ou PET-CT. A Para o estadiamento mediastinal de câncer de pulmão, em particular, a EBUS-TBNA precisa incluir uma avaliação sistemática. Nesse contexto, em 2009, a 3 O material restante é colocado em formol para análise de blocos de células. É possível realizar análise molecular para câncer em amostras de EBUS-TBNA.5 Caso haja outra suspeita que não seja câncer durante a ROSE, a amostra pode ser enviada para análise microbiológica, citometria de fluxo ou reação em cadeia da polimerase (PCR). A sensibilidade da EBUS-TBNA varia de 84 a 96%.7Um outro fator importante durante a EBUS-TBNA é a avaliação rápida on-site (ROSE) por um patologista, para avaliar a representatividade da amostra e conduzir uma análise específica mais aprofundada.oda região e ajuda a identificar o brônquio da lesão, com base nas diferenças de ecogenicidade entre este e o parênquima normal.8 Dessa forma, a RP-EBUS surgiu para melhorar a sensibilidade e a exatidão diagnóstica da broncoscopia na investigação de nódulos e massas pulmonares.8A EBUS com RP-EBUS usa um transdutor delicado e flexível, de 20mHz , que é inserido pelo canal de trabalho de um broncoscópio convencional, em direção à lesão pulmonar a ser investigada. A técnica permite uma visão de 3609 Por exemplo, a A EBUS com RP-EBUS deve ser realizada sob fluoroscopia para auxiliar na avaliação da lesão-alvo. É possível realizar exame citopatológico e biópsia transbrônquica (escovado e punção aspirativa transbrônquica) durante o procedimento, e as amostras podem ser enviadas para análise microbiológica, citológica e histológica, dependendo da avaliação clínica, radiológica e citológica.8Alguns fatores, como tamanho do nódulo, capacidade de visualização e localização do transdutor dentro da lesão, podem afetar o rendimento diagnóstico da RP-EBUS. Nossa experiência preliminar com RP-EBUS no Brasil mostrou boa sensibilidade para nódulos e massas .10A EBUS-TBNA e a RP-EBUS são procedimentos seguros, com baixos índices de complicações. A complicação mais frequente na EBUS-TBNA corresponde aos danos aos equipamentos durante a manipulação da agulha. Outras complicações, como sangramento e infecções, são raras. Na RP-EBUS, as complicações são pneumotórax e sangramento, que variam de 1 a 4% e 3 a 5%, respectivamente, na maioria das casuísticas.A A ultrassonografia endobrônquica dos tipos radial e convexa é um procedimento seguro, com baixas taxas de complicações e boa sensibilidade para nódulos e massas.


Few studies have examined the sustainability of family planning program outcomes in the post-program period. This article presents the results of a natural experiment where the Nigerian Urban Reproductive Health Initiative Phase I programming ended in early 2015 and Phase II activities continued in a subset of cities. Using data collected in 2015 and 2017, we compare contraceptive ideation and modern family planning use in two cities: Ilorin where program activities concluded in 2015 and Kaduna where program activities continued. The results demonstrate that exposure to program activities decreased in Ilorin but for those individuals reporting continuing exposure, the effect size of exposure on modern family planning use remained the same and was not significantly different from Kaduna. Modern family planning use continued to increase in both cites but at a lower rate than during Phase I. The results are useful for designing family planning programs that sustain beyond the life of the program. There has been a growing emphasis on understanding the meaning and measurement of program sustainability in public health interventions –4. The dWhile much of the literature on the sustainability of health programs comes from developed countries,5, a groInterestingly, Scheier’s systematic review on the sustainability of health-related programs in developed nations  indicateIn the field of FP, one of the few examples of a rigorous examination of sustainability involved the Promoting Change in Reproductive Behavior (PRACHAR) program in Bihar, India that sought to delay marriage, delay first birth, and encourage contraceptive use within three months of a birth . EvaluatAnother analysis of PRACHAR program sustainability examined whether young people who participated in a three-day training from intervention communities differed from matched young people from comparison communities four years after the training . The autAnother example of a FP program that had a follow-up evaluation of program sustainability is the Better Life Options (BLO) Program in India that examined intervention and comparison areas with another post-test only design . The BLOThis paper begins to fill this gap by examining the sustainability of the Nigerian Urban Reproductive Health Initiative (NURHI) program on FP attitudes and behaviors two-years after the initial program end date. It takes advantage of the fact that the program was completely terminated in one site and continued in a modified form in the other site. This study is unique not only because it contributes to the knowledge base about sustainability of FP programs but also because it uses novel data sources and methods to inform the key outcomes of ideation toward FP and modern contraceptive use. The results provide crucial evidence regarding the sustainability of the NURHI program on FP attitudes and behaviors as well as whether there are diminishing returns on the continuation of the same activities over the longer-term in the two settings: where the program continued and where the program terminated. The hope is that this study will be important for informing strategies to spread and scale-up FP programming in urban Nigeria and elsewhere in sub-Saharan Africa.The Nigerian Urban Reproductive Health Initiative (NURHI), funded by the Bill & Melinda Gates Foundation (BMGF) and implemented by the Johns Hopkins Center for Communications Programs, aimed to improve FP access and modern contraceptive use in select urban areas through comprehensive demand and supply side programming. The NURHI program utilized demand generation activities to encourage interpersonal discussion about FP, reduce barriers, myths, and social stigma, and increase approval of FP methods. Strategies included community-level outreach events, distribution of information, education, and communication (IEC) materials at public and private health facilities and in the communities, and through mass media, including television and radio programs. In addition, the NURHI program worked to improve the supply of FP services by training providers on provision of quality care and new methods; ensuring commodity supplies; and improving the overall clinic environment in target, high volume facilities. Phase I (2009–2014) of the NURHI FP program was implemented in six cities: Abuja, Benin City, Ibadan, Ilorin, Kaduna, and Zaria. For details of the NURHI program see Krenn et al. . FollowiThe Measurement, Learning & Evaluation (MLE) project , conducted the evaluation of the NURHI Phase I program with Data, Research and Mapping Consult (DRMC) and the National Population Commission as the local implementing partners . The MLEIn 2017, another cross-sectional representative survey was conducted by DRMC in both Kaduna and Ilorin to examine the sustainability of the NURHI program two-years after program operations terminated. We compare two cities: one where program activities concluded at the end of Phase I (Ilorin) and one where activities continued in Phase II of the NURHI program (Kaduna). All clusters surveyed in Kaduna and Ilorin in 2015 were included in the 2017 survey. As part of the 2017 data collection, we undertook a census of all households located in study clusters in Ilorin and Kaduna. All women ages 15–49 were eligible for interview after providing informed verbal consent. A total of 4,673 women from Kaduna and 2,209 women from Ilorin were surveyed in 2017. The study protocol and all consent procedures and consent forms for both the 2015 and 2017 surveys were approved by the Institutional Review Board at the University of North Carolina at Chapel Hill and by the National Health Research Ethics Committee of Nigeria (NHREC) in Nigeria.The census of all households in study clusters in 2017 allowed us to match households and women from 2015 (that included a sample of households) to create a longitudinal sample of women from the two cities. During 2017 fieldwork, households that were interviewed in 2015 were identified and flagged for matching using information on the name of the household head and the household composition. In addition, a small number of households (n = 22) were matched after data collection using stringent criteria for matching . Matching of women within matched households rested primarily on age and consistency of reported parity as of the 2015 round. A total of 1,535  women were matched across the two time periods for longitudinal analyses. It is worth noting that data are also available from the larger cross-sectional samples in 2015 and 2017 for the two cities . Multivariate analyses of the longitudinal sample are used to estimate NURHI’s continuing effects after Phase I activities formally concluded in 2015.At both time periods, all surveyed women were asked questions on fertility and family planning behaviors as well as information on exposure to the NURHI program, among other things. Questionnaires used came from earlier rounds of the study (baseline and midterm) and most of the variables included in this paper were validated as part of the impact evaluation of the NURHI program . The firThe second outcome of interest is women’s FP ideation, a representation of knowledge, values, beliefs, and communication related to FP and modern contraceptive use. Because of the large number of factors that relate to ideation, principal components analysis (PCA) was used to construct a single, continuous ideation index. The index is based on the same set of underlying factors used by NURHI to examine ideation in an earlier analysis . The NURThe key explanatory variables for these analyses are exposure to NURHI program related factors. The NURHI program had many demand side activities and our analyses focus on indicators for eight of them. In earlier longitudinal impact evaluation analyses, we estimated the causal impact of each of these activities separately . HoweveIn the NURHI impact evaluation, supply-side variables were included . In thisA migration related variable was also included as an explanatory variable in the models. We had no a priori belief about the direction of the effect of this regressor—whether a woman travelled to another city or town in the last year could increase or reduce NURHI exposure and contraceptive use. Control variables included age, education, union status (married or living with partner), religion, primary language spoken at home, and a household asset score (see j) from community j  at time t  is a current user of modern contraception at the time of the survey. ijCity is a dummy variable indicating city of residence for woman i from community j, ijtE represents program exposure or the ideation index, ijtX represents individual level control variables such as age and education that are assumed to be exogenous, and ijμ represents time invariant unobservable individual and community level variables such as an individual’s level of motivation to limit family size or attitudes of community leaders towards modern contraceptive use. We do not specify separate time invariant errors for community and individual since, by design, we only have individuals from the 2015 survey who were found to still be living in the same community in 2017 and our estimation method is a variation of fixed effects because we suspect that program exposure is potentially correlated with this time invariant error.We exploit the longitudinal sub-sample to estimate models that control for the potential endogeneity of exposure or ideation caused by correlation with unobservables at both the community and individual levels. We use two dependent variables: modern contraceptive use estimated using a logit framework and the ideation index using an extension of the linear regression model. We illustrate our methods using the more complicated, nonlinear logit model:α4 and α5 will be identical to the fixed effects estimates or within estimates [λ's are equal to zero is a robust version of the Hausman test of the null hypothesis of no correlation between program exposure and the time invariant error term. In this logit form, we estimate Eq  examines the relationship between exposure to NURHI program components and ideation, controlling for demographic factors; model (b) examines how ideation is related to modern method use; and model (c) examines how NURHI exposure relates to modern method use. The multivariate results  for modeIn model (b) with modern contraceptive use as the dependent variable, we see that the sign of the Kaduna dummy is negative (p<0.10). Further, the effect of the ideation index is strongly significant and positive. From model (a) we know that the Kaduna dummy has a strong positive effect on ideation and thus the ideation effect is likely capturing some of this higher ideation in Kaduna. Modern contraceptive use continued to increase in 2017, and recent travel had a significantly negative effect on modern use. We have no a priori expectations for the sign of the travel variable since women who move between cities may be more likely to use contraception if they have greater access to methods from their travel or may have less need for contraception if they are moving around and are less likely to be having regular sex.Finally, model (c) shows the direct effects of exposure on modern contraceptive use. The effects that are positive and significant are the time dummy and the exposure index where over time, use is higher and women who have greater exposure are more likely to use. The Kaduna dummy has a positive point estimate; however, it is far from significant. The effect of recent travel is basically the same as model (b).Results for the control variables from the three sets of models are presented in The major conclusion one can draw to this point is that exposure to NURHI programs has continued to have strong positive effects on both ideation and modern contraceptive use and that ideation has strong positive effects on modern contraceptive use and so it is a major pathway through which exposure has its effect. However, the exposure index, while it was quite useful when we tested for many interaction effects, does not allow us to examine each program component separately. Since we found almost no significant interaction effects and a joint test of the interaction terms failed to reject the null that the effects were jointly zero  (results not shown). Note that the logo exposure measure was dropped because it tended to be found as part of many of the other exposure measures. We then calculated marginal effects for each program component. Since all exposures are binary, the marginal effect is the difference between predicted modern contraceptive use with the exposure set to zero and then set to one and all other variables kept at their actual empirical values. The marginal effects for individual exposures were not significant probably due to collinearity. However, when we examine the effect of groups of similar exposures , we see some significant effects . Print mFinally, the marginal effect for the 2017 dummy variable is also presented in This paper takes advantage of a natural experiment that occurred in two cities in Nigeria where the NURHI FP program received funding to continue operations in one city and its surrounding state (Kaduna) and not in another (Ilorin). We use data gathered in 2015, which coincided with the termination of the program in Ilorin, and in 2017, two years after termination, to see how the two cities fared.As we point out in the introduction, there have been very few studies that have examined the sustainability of FP programs anywhere in the world and none that we are aware of from sub-Saharan Africa that had data at the time of program termination and data from a later point in time. We not only have this type of data for two repeated cross-sections, but we also have a subset of the sample where we have the same individuals at two points in time and so we are able to use the powerful tools associated with longitudinal data in this analysis.An interesting descriptive result is that respondents in Ilorin reported continued exposure to the different aspects of the program even though it was terminated but at a level that was reduced from the peak exposure experienced in 2015. In contrast, exposure continued to rise in Kaduna. Because of the large number of exposures, we used principal components analysis to develop an exposure index that then could be used to test for interactions of exposure with city, time, and religion. Surprisingly, we found little evidence of interaction effects. Thus, the effect size of exposure in Ilorin was not statistically different from the effect size in Kaduna. The time and religion interactions also turned up little of substance. It is also notable that ideation had a similar pattern to exposure such that it decreased over time in Ilorin while it increased in Kaduna. However, the effect of ideation on contraceptive use remained positive for both cities and exposure to NURHI program activities was significantly related to ideation for both cities as well.The descriptive statistics provide some interesting perspectives on observed changes in the sample. Notably, in both Kaduna and Ilorin, modern contraceptive use increased significantly. Further, as mentioned above, there was significantly more exposure in Kaduna in 2017 relative to 2015 while exposure significantly dropped in Ilorin. The demographic factors worked in favor of increasing use in this longitudinal sample with more respondents “aging into” ages where we tend to see higher use and marital status having little change over time. This demonstrates the importance of undertaking the multivariate analyses.The results of the analyses that teased apart the exposure items demonstrated that a number of the NURHI program activities, when grouped by similar exposure methods, were related to continued contraceptive use in the follow-up period. Mass media and print media had significant effects in the final analyses. Notably, mass media increased significantly in Kaduna but decreased in Ilorin over the 2015 to 2017 period. Similarly, exposure to print media significantly increased in Kaduna but significantly decreased in Ilorin. These types of activities may continue to have an effect in Ilorin through sustained effects or diffusion if, for example, the outreach workers continue to use messages learned earlier under the NURHI program or if the television shows or advertisements are seen through the national media or through local cable channels. In an analysis that used the 2015 Nigeria data, authors  demonstrOur results can be compared to the earlier studies on sustainability of FP programs in India and for other health outcomes in sub-Saharan Africa ,21. LikeThis study is not without limitations. The first limitation is that it was difficult to examine the exposure effects with the granularity that we would have liked since there were a large number of program components and there was correlation across the NURHI program activities. With that in mind, we created an exposure index which demonstrated the importance of NURHI exposure in both Kaduna that continued activities as well as in Ilorin where activities ended; overall exposure declined in Ilorin over time. Second, the longitudinal sample used for the multivariate analysis, while providing a more rigorous analysis approach, was not large enough to allow us to stratify by city which would have been helpful for understanding city-level differences. Third, women are self-reporting their exposure to the program activities and while they were asked about exposure in the last year for many of the components, the radio and television measures were broader and may capture remembered exposure from earlier in the NURHI Phase I programming. With the data available, it is not possible to control for this potential retrospective reporting bias. Finally, discussions with the NURHI team have indicated that in NURHI Phase II, a focus of the program was to spread to the state-level. This meant that efforts were less intensive in the city of Kaduna in the two-year follow-up period. This may have attenuated the effect of program sustainability in Kaduna from what might have been expected if Kaduna city had remained the primary target area of the program.This study also has important strengths. Data were collected to examine the sustainability of program activities after the program ended; these types of data are rarely available post-program funding. The results of this study demonstrate important results to inform future FP programming. In particular, the program effects sustained two-years out such that exposure to the program continued to be related to ideation, a key intermediate factor for the NURHI program, and modern contraceptive use. Further, there may be some diffusion effects such that women continue to discuss program-related factors  and this discussion may help to spread the effects of the program post-implementation. Finally, this study showed that in Kaduna, the city where the program continued, there was less increase in modern method use than in the earlier phase. This may reflect diminishing returns of program activities, or alternatively may reflect the increased focus of program activities on other parts of the state.To conclude, the effect of the NURHI Phase I program sustained and was associated with continued increases in modern contraceptive use, even in the city where the program ended. It is notable that the NURHI Phase I program activities did not have an explicit plan for sustainability and therefore, sustained exposure, whether through diffusion or remembering earlier messages on the radio and television, is promising. That said, exposure in Ilorin did decline significantly, indicating a risk that with more time, the effect of NURHI programming on ideation and modern contraceptive use may fade. As part of NURHI Phase II, greater attention is focused on identifying strategies that are sustainable and engaging the government for long-term support. That said, in this analysis, the exposure effects in Kaduna were no greater than those in Ilorin; this might reflect the program focus in NURHI Phase II on spreading to rural areas and thus not continued attention to urban areas. Funders and program planners need to consider undertaking activities that can be sustained from the launch; this likely involves engaging the government and public sector throughout program design and implementation, as done under NURHI Phase I and more strongly done under NURHI Phase II. That said, there is also a need to identify strategies to insert new energy into FP programming to help affect greater increases in modern method use in a place like Kaduna that has had FP programming for a number of years. This study demonstrated sustainability of program activities; however, questions remain on what will happen over time given observed declines in exposure in Ilorin. Future studies are needed to examine the long-term impact of these declines on modern family planning use.S1 TableFactor loadings obtained using full representative cross-sectional sample for Kaduna and Ilorin in 2017 were used to calculate 2015 values.(DOCX)Click here for additional data file.S2 Table(DOCX)Click here for additional data file.S3 Table(DOCX)Click here for additional data file.S1 File(PDF)Click here for additional data file.S2 File(PDF)Click here for additional data file.


Much of vesicle traffic at the Arabidopsis  plasma membrane is subject to the Sec1/Munc18 protein SEC11, which, along with plasma membrane K+ channels, selectively binds with the SNARE SYP121 to regulate its assembly in complex. How SEC11 binding is coordinated with the K+ channels is poorly understood, as both SEC11 and the channels are thought to compete for the same SNARE binding site. Here, we identify a second binding motif within the N terminus of SYP121 and demonstrate that this motif affects SEC11 binding independently of the F9xRF motif that is shared with the K+ channels. This second, previously unrecognized motif is centered on residues R20R21 of SYP121 and is essential for SEC11 interaction with SYP121. Mutation of the R20R21 motif blocked vesicle traffic without uncoupling the effects of SYP121 on solute and K+ uptake associated with the F9xRF motif; the mutation also mimicked the effects on traffic block observed on coexpression of the dominant-negative SEC11Δ149 fragment. We conclude that the R20R21 motif represents a secondary site of interaction for the Sec1/Munc18 protein during the transition of SYP121 from the occluded to the open conformation that leads to SNARE complex assembly.SNARE (soluble  N-ethylmaleimide-sensitive factor attachment protein receptors) comprise a highly conserved superfamily of proteins in all eukaryotic cells and play important roles in membrane fusion for targeting and delivery of membranes, proteins, and soluble cargos. They form a core complex to bring vesicle and target membrane surfaces together, thereby driving secretion as well as the traffic of vesicles between endosomal compartments. SNAREs are classified as either target- or vesicle-SNAREs, reflecting their functional localization, and as either Q (Gln)- or R (Arg)-SNAREs, based on the conserved amino acid contributing to the central layer of the core complex. The Q-SNAREs are further divided between Qa-, Qb-, and Qc-SNARE forms, one of each contributing to the core complex with the cognate R-SNARE. In plant cells, SNAREs are important for biotic and abiotic stress resistance, homeostasis, cell expansion, and growth  is the best-known example. SYP121, and its close homolog SYP122, have similar structures and overlapping functions in facilitating secretory traffic in most tissues in the plant  of Arabidopsis . SEC11 is a member of the Sec1/Munc18 (SM) protein family that is now recognized to regulate SNARE complex assembly . SM protr SYP121 .Δ149 fragment, which includes the SM minor cleft and retains Qa-SNARE binding in vitro and in vivo  of SYP121 for binding with the SEC11 minor cleft will likely clarify the mechanics of SEC11 and the K+ channels within the SNARE cycle of SYP121.SEC11 binding distinguishes secretory traffic mediated by SYP121 from that mediated by SYP122. Notably, SEC11 binds preferentially with SYP121 and rescues traffic block by dominant-negative fragments of the SNAREs when these fragments are expressed, but only in plants that also express the native SYP121 . Further in vivo . The pro in vivo , includi20R21 of SYP121. We show that this motif is essential but distinguishable in its SNARE-binding characteristics from the F9xRF motif of SYP121. Moreover, we demonstrate that the R20R21 motif is important for the selectivity of SEC11 for SYP121 in secretory traffic and for the link between vesicle traffic and osmotic solute uptake during plant growth.Here, we identify a second, N-terminal SEC11-binding motif associated with residues R+ channel  assay for protein-protein interaction analysis , previou channel  and SNAR channel  interacth SYP121 , 2015.To verify this selectivity in SEC11 interaction between the two SNAREs in vivo, we used Förster resonance energy transfer (FRET) analysis  after su9xRF motif, previously identified also with K+ channel binding . These results demonstrate a requirement for residues R20 and R21 in SYP121 to interact with SEC11. Thus, we conclude that there are two distinct SEC11-binding motifs within the SYP121 N terminus.To validate these findings in vivo, we performed the FRET analysis using the corresponding n1, n2, and n3 segment chimeras as well as the site mutants. SEC11, like other SM family proteins, consists of four structural elements  that ass20R21 motif in SYP121 affects binding of SEC11 through this N-terminal domain, we performed the SUS assay in yeast using the SEC11Δ149 peptide as bait. To anchor the peptide to the membrane, we fused SEC11Δ149 with a glycosyl-phosphatidylinositol signal peptide derived from glucan 1,3-β-glucosidase2 (Exg2), a fusion that successfully recapitulates previous pulldown studies with the SEC11Δ149 peptide and the cytosolic domain of SYP121  promoter . Secreted YFP and GFP-HDEL were expressed in these lines after transient transformation using the pTecG-2in1-CC vector both without and with a cassette encoding the SEC11Δ149 peptide, much as before  that this unique residue pair is critical for the selective binding of the SM protein; (2) eliminating this site alone blocks SYP121-mediated traffic, much as does the expression of SEC11Δ149, and in a manner that is not additive with SEC11Δ149 block; and finally (3) we show that mutation of the R20R21 residue pair, unlike the SYP121F9A mutation, retains a substantial element of the link between vesicle traffic and osmotic solute uptake, blocking traffic without uncoupling solute and K+ uptake. These findings distinguish functionally the R20R21 SEC11-binding motif from that of the F9xRF motif for binding of both SEC11 and the K+ channels, and they suggest a mechanism by which the SEC11 binding exchange may facilitate the transition to SNARE complex assembly together with the channels.Although it might be anticipated that SEC11 and the Ke SYP121 , 2015. TSupplemental Fig. S3), notably the conserved F9 residue. From sequence alignment of SYP121 with SYP122, which does not bind SEC11 as strongly, we identified three regions , suggesting a specialization in the interactions engaged by its binding with SEC11. This motif also separates SYP121 binding with SEC11 from that of the K+ channels  for overnight growth at 180 rpm and 28°C. Liquid cultures were harvested and resuspended in yeast peptone dextrose (YPD) medium. Yeast mating was performed in sterile PCR tubes by mixing equal aliquots of yeast containing bait and prey constructs. Aliquots of 5 µL were dropped on YPD plates and incubated at 28°C overnight, and 1-mL aliquots were pelleted, ground in a ball mill, and frozen in liquid N2 for later immunoblot analysis  on CSM-AHLMTU plates with added Met. Plates were incubated at 28°C, and images were taken after 3 d. Yeast was also dropped on CSM-LMTU control plates to confirm mating efficiency and cell density, and growth was imaged after 24 h at 28°C. To verify expression, yeast was harvested and extracted for protein immunoblot analysis using commercial HA antibody for prey and commercial VP16 antibody (Abcam) for the bait.mbSUS assays were performed as described before , 2010a. analysis . ColonieArabidopsis thaliana) wild type (Columbia-0) as well as syp121-1 (syp121) and syp122 mutants . Homozygous T4 lines were verified for expression of the transgene by immunoblot analysis, and at least two independently transformed lines were chosen for experiments.Seedlings of Arabidopsis . Ten microliters of the same supernatant was used to determine the osmolality using a vapor pressure osmometer .For root phenotypic analysis, plants were grown on vertical 0.7% (w/v) agar plates and in liquid culture with defined minimal salts medium including 1 mm K+ . Plants EZ-Rhizo . The samw method . Whole shttp://rsb.info.nih.gov/ij/). No statistically significant differences were found between lines with the same construct; these data were therefore pooled.For hypocotyl elongation analysis, seeds were sterilized and placed on 0.5× Murashige and Skoog medium containing 0.8% (w/v) agar and 0.5% (w/v) Suc. After 3 d of stratification at 4°C, plates were placed under a red filter for 3 h to promote germination and left vertically in the dark for 6 d. Plates were scanned and hypocotyl lengths measured using ImageJ v.1.52 (Nicotiana tabacum) plants were grown in soil at 26°C and 70% relative humidity on a 16/8-h (day/night) cycle for 4 to 6 weeks. Plants with three to four fully expanded leaves were selected for transformation  in homogenization buffer that contained 500 mm Suc, 10% (v/v) glycerol, 20 mm EDTA, 20 mm EGTA, 1× Protease Inhibitor (Roche), 10 mm ascorbic acid, 5 mm DTT, and 50 mm Tris-HCl, pH 7.4. Samples were centrifuged at 13,000g for 10 min at 4°C, supernatants were diluted 1:1 in loading buffer (http://rsb.info.nih.gov/ij/).Leaf tissue was harvested, weighed, flash frozen, and ground in liquid Ng buffer , and prose as appropriate, with significance determined by Student’s t test or ANOVA in SigmaPlot v.11.2 (Systat Software).Statistical analysis of experiments was reported as means ± Sequence data from this article can be found in the GenBank/EMBL data libraries under TAIR accession numbers At3g11820, At3g52400, At1g08560, and At1g12360.The following supplemental materials are available.Supplemental Figure S1. The mutated Qa-SNARE SYP121R20A,R21A localizes to the cell periphery.Supplemental Figure S2. The expression of SYP121 chimeras and Ala substitution in the R20R21 motif in the syp121 line.Supplemental Figure S3. Alignment of the n1 and n2 segments of SYP111, SYP121, SYP122, SYP131, and SYP132.