The purpose of this study was to investigate which factors affected the implementation of a multicomponent Responsible Beverage Service (RBS) program in 290 Swedish municipalities and whether the amount of such factors influenced the level of implementation of the program.This study used variation in the presence of implementation-promoting factors to predict the level of implementation of the RBS program in municipalities throughout Sweden. The presence of such factors and the level of implementation of the program were studied by means of two surveys in all Swedish municipalities (N=290). Logistic regression and Spearman’s correlation analyses were used to analyze the relationship between implementation-promoting factors and the level of implementation of the RBS program.The response rates of the two surveys were 96% and 98%, respectively. One main finding was that program fidelity was low. Only 13% of the municipalities surveyed had implemented the RBS program as a whole, as stated in the specification of requirements. In municipalities reporting a higher amount of implementation-promoting factors, a significantly higher level of implementation of the program was shown. Evaluation and feedback was the only factor that correlated significantly with the level of implementation of the RBS program as a whole.Evaluation and feedback constitutes an important implementation-promoting factor also in complex programs like the RBS program. Program fidelity is significant for the outcome of an intervention and must be a major focus of the implementation processes. Alcohol consumption constitutes a main risk factor for health and contributes substantially to the global burden of disease. PreviouPopulation-based strategies that focus mainly on general affordability and availability and interventions directed toward consumers already at risk have proven to be cost-effective means of reducing alcohol-related harm,11. InteHowever, very few multicomponent RBS programs that aim to reduce violence associated with consumption of alcohol at on-licensed premises have been implemented and evaluated. One example, apart from the program implemented in Stockholm described below, is the Safety Action Projects in Queensland, Australia, where significant effects were found on observed aggression,16. SingThe main purpose of the Swedish multicomponent RBS program is to reduce violence associated with the consumption of alcohol at licensed premises by means of reducing the level of customer intoxication. The program was set up as a research project with an experimental area and a control area in the central part of Stockholm, the capital of Sweden. An evaluation of the RBS program showed a significant decrease in the number of police-recorded violent crimes, primarily assaults committed between 10:00 p.m. and 6:00 a.m., with 29% in the intervention area as compared to the control area. The stuAfter the initial evaluation, the RBS program grew in popularity, and the number of municipalities using the program increased during 2003 and 2004. In 2004, the Swedish National Institute of Public Health was commissioned by the government to spread the RBS program to municipalities throughout the country. Two years later, Sweden’s 21 county administrative boards were appointed to assist in implementing the program. One project coordinator was employed at each county administrative board to oversee implementation. Dissemination of the RBS program culminated in 2006 and 2007 when 84 and 60 municipalities, respectively, adopted the program. In 2008, representatives from 263 of Sweden’s municipalities (90%) stated that they had implemented the RBS program, in full or in part.A new evaluation was conducted in 2010 after dissemination of the program throughout Sweden; again, a significant effect was found on police recorded assaults. For eacThe RBS program consists of several components, of which this study focuses on the three main ones: education of restaurant personnel (RBS training), the presence of a community-coalition steering group, and the supervision of on-licensed premises. The RBSThe Swedish Alcohol Act specifies that municipal licensing boards are responsible for the handling of licensing matters. Both the municipality and the police department are responsible for the supervision of licensed premises, while the municipality alone is responsible for sanctions regarding infractions committed at these premises. The purpose of the act is to ensure that sales of alcohol are conducted in such a manner as to prevent harmful effects in a broad sense as much as possible. The focus is on overserving and sales to minors.implementation-promoting factors, however, have been suggested in the literature[Although research on implementation strategies has increased during the past decades, the evidence to support practice is insufficient,28. Factterature-33.The purpose of this study was to analyze the implementation of a complex, multicomponent public health program—the RBS program—in Swedish municipalities. Two specific questions were addressed:1. Are there any correlations between the presence of implementation-promoting factors and the level of implementation of the RBS program?2. Is there a dose-response effect, i.e., does a higher amount of implementation-promoting factors result in a higher level of implementation of the RBS program?The analyses in this study are based on data from two surveys conducted in all Swedish municipalities (N=290). The first, conducted at the turn of the year 2008-2009, studied to what degree municipalities had implemented the RBS program. The secThe study was approved by the Regional Ethical Review Board in Stockholm (EPN) (protocol no. 2008/1041-31/5 and 2010/736-32/5).Two criteria for municipalities to be included in the analysis part of this study were formulated: First, the municipality must have initiated the RBS program no later than 2008. Second, the municipality must contain on-licensed premises open during evenings. With the assistance of Sweden’s 21 county administrative boards, every municipality in the country was examined regarding fulfillment of these criteria. Of 290 municipalities, 235 qualified for inclusion.Although no manual has been developed to describe the details of the RBS program, a brochure has been published in which the main features of the program are described. These f1. RBS training should have been conducted for employees at on-licensed premises in the municipality.2. The duration of the training should have been at least two days.3. The training should have included the following five instructional components: the medical effects of alcohol consumption, the Alcohol Act, alcohol-related violence, drug problems at licensed premises, and the handling of conflict at the premises, followed by an exam.4. At least one training session with employees from licensed premises in the municipality should have taken place during the past two years.5. The number of on-licensed premises that sent employees to training sessions had to amount to at least 30% of the total number of on-licensed premises in the municipality.1. A community-coalition steering group for the concrete work conducted in accordance with the RBS program should have been present in the municipality or in a group of municipalities.2. The municipality, the police department, and owners of on-licensed premises should have been represented in the group.3. The community-coalition steering group should have conducted at least two meetings during the past year.To ascertain whether or not supervision in a given municipality was in line with the main features of the program, supervision was divided into three categories: supervision visits that were carried out (a) after 11:00 p.m., (b) by the municipality and the police department in collaboration, and (c) solely by the police department.The supervision component of the RBS program had to satisfy three requirements:1. At least two of the three categories of supervision described above had to be conducted during a year.2. The number of on-licensed premises visited during one year, for each category, had to correspond to at least 20% of the number of licensees in the municipality.3. The number of supervision visits at on-licensed premises had to have either increased during the course of the program or remained on the same level as it was when the municipality started to work in accordance with the RBS program.Municipalities that had satisfied the requirements for implementation obtained 1 point for each program component. To measure the degree to which each municipality had implemented the RBS program as a whole, the number of points was summarized in the form of an index, with each municipality receiving from 0 to 3 points.The second survey was used to ascertain the presence of certain implementation-promoting factors at the time of implementation of the RBS program in the municipalities. Factors related to perceived local needs, charactEight variables  were used to capture the presence of five implementation-promoting factors  in the municipalities . Nagelkerke’s RSpearman’s correlation coefficient was used in the univariate analysis of the correlation between the indices measuring the level of implementation-promoting factors and the level of implementation of the RBS program.The response rates of the two surveys were high, 96% and 98%, respectively. Of the 235 municipalities that satisfied the inclusion criteria for this study, only 13.2% had implemented all three main components of the RBS program, and 17.0% had implemented none of these components  among municipalities that had implemented all three components, compared to 25.2 among those that had not implemented any components of the RBS program  had a significant effect on the level of implementation of the RBS program. This was apparent in analyses of the full program as well as in separate analyses of two of the three main components, namely RBS training and supervision (see Table The factor early involvement of practitioners (another implementation process factor) was significantly associated with the level of implementation of the community-coalition steering group component, but not with implementation of the RBS program as a whole . This was probably due to repeated municipality-specific reminders that were distributed by the project coordinator at each county administrative board.Of the 235 municipalities that reported having worked fully or partly in accordance with the RBS program, 17.0% had implemented none of the program components. Furthermore, only 13.2% had implemented the program as a whole, as stated in the specification of requirements. Thus, program fidelity was low. One reason for this may be that the information provided was unsatisfactory when implementation took place, since only a simple brochure describing the RBS program was available.Evaluation and feedback showed a significant correlation with implementation of the RBS program as a whole, as well as with the two main program components RBS training and supervision. This result is in accordance with a Cochrane review based on 140 studies from the health care sector, where Ivers et al. conclude that “audit and feedback generally leads to small but potentially important improvements in professional practice”.Early involvement of practitioners was significantly associated with the presence of a community-coalition steering group, but not with the other two program components (RBS training and supervision of on-licensed premises). This finding is not surprising since involvement in a steering group often implies involvement in the planning phase of an implementation process.A dose-response effect was shown in this study in that higher amounts of implementation-promoting factors in a municipality were correlated with higher levels of implementation of the RBS program. In a study of over 500 health-promoting and disease-preventing programs, Durlak and DuPre showed that higher levels of implementation-promoting factors lead to better implementation results. On the The findings of this study should be considered in the light of some limitations. First, it is possible that errors in recall or social desirability biases crept into the surveys of key informants, meaning that levels of implementation of the RBS program could be even lower than those estimated in our study. Second, some implementation-promoting factors were not included in this study because of the difficulty of capturing those factors through surveys. For example, one factor left out of the survey, after much consideration, involved the extent to which the program conformed to the values of the implementing organization,33. AnotOne final question is whether it is worth the effort to implement such a complex, multicomponent program as this RBS program? The answer is, in all probability, yes. As mentioned in the introduction, two evaluations in Sweden have shown significant effects on police-recorded violence, and the cost-effectiveness ratio based on the results of the first evaluation was large.The most important conclusions of this study are these:•Program fidelity was weak. All the 235 municipalities in this study stated that they had implemented the RBS program, but only 13% applied all three main components of the program.•Evaluation and feedback seems to be an important implementation factor in multicomponent and complex interventions like the RBS program.•There seems to be a dose-response effect as regards implementation-promoting factors. That is, the more implementation-promoting factors there are, the higher the level of implementation will be, even when it comes to a highly complex program such as the RBS program.The authors declare that they have no competing interests.The study design was developed by all the authors. BT and UH carried out the surveys, and BT performed the statistical analysis. All the authors contributed to drafting the manuscript. All the authors have read and approved the final manuscript.


On May 12, 2008, a destructive earthquake registering 8.0 on the Richter scale struck Sichuan Province, southwest China. Beichuan County was the epicenter which was one of the areas nearly completely destroyed by the earthquake. In Beichuan, about 15000 people died and 3000 people were missing. Specially, the earthquake took 1587 students' and 214 teachers' lives from the elementary and middle schools there. The main purpose of the study was to provide a better understanding of mental health problems and associated risk factors among children after earthquake.Three hundred and thirty grades 3–5 children completed the questionnaire of disaster –related experience and the Trauma Symptom Checklist for Children-Alternate Version (TSCC-A). The first survey was carried out six months after the earthquake, and the second one was carried out six months later. The measurements and methodology applied in the two sessions were identical.The prevalence rates of the problems at two time-points were 23.3% and 22.7% for anxiety, 14.5% and 16.1% for depression, and 11.2% and 13.4% for PTSD, respectively. Among demographic variables, no significant age difference existed, while it was found that 6 months after the earthquake, symptoms of anxiety, depression and PTSD were significantly more common among students in grades 4 and 5 than those in grade 3, Initial exposure to death, bereavement and extreme fear were significant predictive factors for the occurrence of anxiety, depression and PTSD.Findings of this study suggest that posttraumatic mental health problems after natural disaster in children may have reached epidemic proportions and remain high for a long period. Psychologist and social workers should pay more attention to children who experienced more traumatic stresses and provide appropriate mental health interventions. Implications and limitations of these findings were discussed. Beichuan County  was the epicenter. The county was one of the areas nearly completely destroyed by the earthquake. Almost all the buildings, including houses, working places, schools, hospitals were ruined. In Beichuan, about 15000 people died and 3000 people were missing. Specially, the earthquake took 1587 students' and 214 teachers' lives from the elementary and middle schools there.On May 12, 2008, a destructive earthquake registering 8.0 on the Richter scale struck Sichuan Province, southwest China. The earthquake caused extensive damage and heavy losses; it took 69227 lives, injured 374,643, led 7,923 missing, and made about 4.8 million homeless , panic, phobias and depression Although some information on the effects of traumatic events on mental health in human beings worldwide is available, few follow-up studies regarding mental health problems in children in Asian following disasters have been published. Systematic investigating traumatic symptoms in children after such a disaster may provide critical information for rational managements of public mental health. Current study aimed to provide a better understanding of mental health problems and associated risk factors among children after earthquake.Sampling was conducted in the largest camp elementary school in Beichuan County. Students in the camp school were from Qushan Town (a town in Beichuan County) School that was completely destroyed. Three hundred eighty-nine (37%) students in the original Qushan Town School died in the earthquake.2χ = 0.000, p = 1.000); the mean (sd) age was 10.36 (0.98) years. The sample consisted of 97 (29.4%) students in grade 3, 83 (25.2%) in grade 4 and 150 (45.5%) in grade 5.Three hundred and thirty students ranged from grade 3 to grade 5 participated in both of the investigations, which were performed six months and one year after the earthquake. Gender of the sample was approximately evenly distributed  whether they were trapped during the earthquake; (b) whether they were injured during the earthquake; (c) whether their family members were killed during the earthquake; (d) whether they witnessed a death of someone during or immediately after the earthquake, and (e) whether they felt intense fearful during the earthquake.Trauma Symptom Checklist for Children-Alternate Version (TSCC-A) This study was approved by the ethics committees of the Institute of Psychology, Chinese Academy of Sciences. Informed written consent was obtained from children's guardians, together with oral approvals from participants, before the testing session according to the Declaration of Helsinki. A trained research assistant administered the tests in the school.The investigators included clinical psychologists and psychiatrists. As mentioned, the first survey was carried out six months after the earthquake, and the second one was carried out six months later. The measurements and methodology applied in the two sessions were identical.t tests were used to evaluate differences in continuous variables. Bivariate associations between traumatic symptoms  and each of the other variables were evaluated with regression analyses. Those variables with p values of <0.05 in bivariate analysis were included in a simultaneous multivariate regression model to evaluate the significance of each predictor after controlling all the other predictors. All analyses were conducted with SPSS version 15.0.Univariate descriptive statistics were computed for sample characteristics , trauma exposure indicators , and prevalence rates for symptoms of anxiety, depression and PTSD, Chi-squared tests were used to evaluate differences in categorical variables and p = 0.915), probable depression (p = 0.783) and probable PTSD (p = 0.721) between the two time points were not statistically significant. Although the number of children who became clinically symptomatic was higher than those who shifted out of this status during this period, these differences are not significant.As recommended by the manual of the TSCC-A, a T score of 65 or higher on a subscale was considered having clinical significance. This was used to identify whether a participant has anxiety, depression, or probable PTSD. According to this criterion, participants were classified as probable positive cases or probable negative cases. To further examine the changes of the severity of traumatic symptoms, paired sample t test was conducted. Tests of significance were also accompanied by effect-size statistics, which approximate the size and thus the practical importance of the difference. The results are summarized in Information on trauma exposure was based on the self-reported experiences about the earthquake: 108 (32.7%) subjects were trapped during the earthquake; 138 (41.8%) reported having physical injuries; 258 (78.2%) experienced family members' death due to the earthquake; 138 (42.2%) witnessed dead bodies; and 194 (58.8%) had extreme fear in the earthquake.In order to examine the effects of demographic and trauma exposure variables on symptoms of anxiety, depression, and PTSD at 6 months and 12 months after the earthquake, the bivariate analysis was conducted. The results have been showed in In the first survey (6 months after the earthquake), symptoms of anxiety, depression and PTSD were significantly more common among students of grades 4 and 5 than those in grade 3. Children who experienced physical injuries and witnessed death had higher prevalence rates of anxiety symptoms, while those who had physical injuries, bereavement, witnessing death and extreme fear had higher prevalence rates on the symptoms of depression and PTSD.At the time of follow-up survey, children who witnessed death and experienced extreme fear had higher prevalent rates on the symptoms of anxiety; those who had physical injuries, bereavement and extreme fear had higher prevalent rates on the symptoms of depression; and those who had extreme fear had higher prevalent rates on the symptoms of PTSD.th:2.08 , 5th: 2.55, p = 0.029), depression (OR(95%CI): 4th:2.55, 5th:3.49, p = 0.035) and PTSD (OR(95%CI): 4th:4.15, 5th:3.57, p = 0.045). Children who had physical injuries (OR(95%CI): 1.87, p = 0.032) had higher prevalence rates of anxiety symptoms; those who experienced bereavement (OR(95%CI): 5.14, p = 0.029)and extreme fear (OR(95%CI): 2.31, p = 0.033)had higher prevalence rates on the symptoms of depression; those who experienced bereavement (OR(95%CI): 9.30, p = 0.003)and witnessed death (OR(95%CI): 2.43, p = 0.018)had higher prevalence rates on the symptoms of PTSD.In the multivariate analysis for the first survey, the variable of grade was still significantly associated with the prevalence rates of traumatic symptoms. Students of grades 4 and 5 had higher prevalence rates than those in grade 3 on the symptoms of anxiety (odds ratio [OR](95%CI): 4p = 0.000), depression  and PTSD . Demographic variables have not significant effects on the prevalent rates of the mental health problems.The results of multivariate analysis for the second survey showed that, among trauma exposure variables, only extreme fear has significant effects on symptoms of anxiety , the severity of the injuries . Fourth, there may be several additional trauma exposure indictors that were not included in this study.Even with these limitations, this study may have made several contributions. First, this study expands our knowledge on posttraumatic symptoms and related risk factors in children following a very destructive natural disaster in non-Western communities. Second, we found that the prevalence rates of depression and PTSD stayed relatively similar in 6 months and 12 months after a very destructive earthquake. Third, significant schooling effect indicated that learning experience in school could provide more information than age. Finally, in children, disaster-related experiences, especially exposure to death and initial extreme fear are important on initiating and maintaining mental health problems. We should pay more attention to this in future research.


General qualitative data analyses utilizing intra and interrater reliability techniques were performed. Results showed that the descriptors used to describe the program and the metaphors named by the informants that could stand for the program were generally positive in nature. Program participants also perceived the program to be beneficial to the development of the students in different psychosocial domains. The present study further supports the effectiveness of the Tier 1 Program of the Project P.A.T.H.S. in Hong Kong based on the perspective of the program implementers.An integration of the qualitative evaluation findings collected from program implementers conducting the Project P.A.T.H.S.  in different years ( In the process of program evaluation, understanding the client's perspective is usually the primary focus. One example is the use of the client satisfaction approach in capturing the views of the program participants. Comparatively speaking, the viewpoint of the program implementers about the program is not adequately explored in the evaluation literature . There aThe proposal to evaluate the view of the program implementers is also highlighted in the existing evaluation frameworks. Although different evaluation emphases exist in the evaluation literature in the international context, there are common evaluation frameworks and standards that are maintained by researchers in the mainstream scientific community. For example, the Centers for Disease Control and Prevention  suggesteAlthough the experimental/quantitative approach is the dominant approach in the field and it is commonly regarded as the gold standard, it is not the only option, and there are alternate approaches. For example, according to Patton , quantitHow can the views of program implementers be assessed? There are different ways to capture the views of the program implementers. For example, rating scales or single-item open-ended questions are used to understand the viewpoints of the program implementers in subjective outcome evaluation. Although qualitative subjective outcome evaluation is good, its method to assess implementers' views by some open-ended questions in paper form lead to a lack of contact between the implementers and researchers. Therefore, it would be desirable to use other means, such as in-depth interviews and/or focus groups to collect qualitative data.Reviews of the literature show that there is a remarkable surge of interest in using focus groups in program evaluation in western countries. For example, Nabors and colleagues , 11 usedThe focus group method has been used successfully to assess client satisfaction and quality assurance in a variety of fields. It has also become a popular method in program evaluation in many research contexts, such as health settings , 14. FocFocus groups can be particularly helpful for the discovery of service problems and suggestions for fixing those problems . MoreoveIn the Project P.A.T.H.S. , the Tier 1 Program is a universal positive youth development program provided for secondary 1 to 3 students in Hong Kong. There were 52 schools that joined the experimental implementation phase 2005–2008) and more than 200 schools that joined the full implementation phase (2006–2009). Several studies have already documented the positive program effects based on the students' objective and subjective outcomes collected from survey questionnaires 05–2008 a. QualitaFrom 2005 to 2009, the total number of schools that participated in the Project P.A.T.H.S. was 244, with 669 schools across all grades. Among them, 46.27% of the respondent schools adopted the full program , whereas 53.73% of the respondent schools adopted the core program .Instructor focus groups were conducted for the secondary 1 level in the 2005/2006 and 2006/2007 school years, for the secondary 2 level in the 2007/08 school year, and for the secondary 3 level in the 2007/08 and 2008/09 school years. A total of 36 schools were randomly selected in the study of focus group evaluation . Among them, 28 schools joined the full program and eight schools joined the core program. Thirty six focus groups consisting of 138 teachers and 39 social workers in total were conducted. The average number of classes per school was 4.83 (range: 3–6), and the average number of respondents per school was 5.11 (range: 1–14). The characteristics of the schools joining this process evaluation study can be seen in As data collection and analyses in qualitative research are very labor intensive, it is the usual practice that small samples are used. In the present context, the number of focus groups and instructor participants can be regarded as respectable. In addition, the strategy of randomly selecting informants and schools that joined the Tier 1 Program can help to enhance the generalizability of the findings. An interview guide  was usedA total of 36 focus groups designed to elicit implementers' perceptions of the Project P.A.T.H.S. were conducted. All focus group interviews were jointly conducted by two trained colleagues. During the interviews, the respondents were encouraged to verbalize their views and perceptions of the program. In the interviews, the interviewers adopted the role of facilitators and were conscious of being open to accommodate both positive and negative experiences expressed by the informants. As the interviewers either had training in social group work and/or substantial group work experience, they were conscious of the importance of encouraging the informants to express views of a different nature, including both positive and negative views. The interviews were audio recorded, with the respondents' consent. The audio recordings were then fully transcribed and checked for accuracy.The data were analyzed by two trained research assistants. After initial coding, the positivity nature of the codes was determined, with four possibilities . The coding and categorization were further cross-checked by another trained research assistant. To enhance the reliability of the coding on the positivity nature of the raw codes, both intra and interrater reliability were carried out. For intrarater reliability, two research assistants who had been involved in the coding individually coded 20 randomly selected responses for each question. For interrater reliability, another two research assistants who had not been involved in the data collection and analyses coded 20 randomly selected responses for each question without knowing the original codes given at the end of the scoring process with reference to the finalized codes.In qualitative research, it is important to consider ideological biases and preoccupations of the researchers. As program developers, the author might have the preoccupation that the implemented program was good and it was beneficial to the students. Additionally, the researchers might have the tendency to focus on positive evidence rather than negative evidence. Thus, several safeguards against the subtle influence of such ideological biases and preoccupations were included in the present study. To begin with, the researchers were conscious of the existence of ideological preoccupations  and conducted data collection and analyses in a disciplined manner. Second, both inter and intrarater reliability checks on the coding were carried out. Third, multiple researchers and research assistants were involved in the data collection and analysis processes. Fourth, the author was conscious of the importance and development of audit trails. The audio files, transcriptions, and steps involved in the development of the coding system were properly documented and systematically organized.In this paper, qualitative findings on the following three areas are presented: (1) descriptors that were used by the informants to describe the program, (2) metaphors  that were used by the informants to depict the program, and (3) implementers' perceptions of the benefits of the program to students.For the descriptors used by the informants to describe the program, there were 270 raw descriptors that could be further categorized into 133 categories . Among tFor the metaphors that were used by the informants that could stand for the program, there were 72 raw objects involving 128 related attributes . ResultsThe perceived benefits of the program to the program participants are shown in n = 177 in 36 focus groups) participated in the study. Second, different datasets collected at different points of time were included in this integrative study. Third, implementers of the program in different grades were invited to participate in the study. Fourth, this is the first known scientific study of focus group evaluation of a positive youth development program based on program implementers in China. Finally, this is also the first focus group evaluation study based on such a large sample of program implementers in the global context.As Donnermeyer and Wurschmidt  pointed Based on the integrative analyses, two salient observations can be highlighted from the findings collected from different cohorts of students. First, the program was perceived positively from the perspective of the program implementers Tables  and 4. TThere is a growing trend for using focus group methodology in order to understand the views of stakeholders in the field of evaluation, and the number of qualitative evaluation studies is increasing in the field. For example, Chen et al.  employedIn qualitative studies, it is important to examine alternative explanations . The firThere are several limitations of the study. First, although the number of schools and workers participating in the study can be regarded as on the high side according to the common practice in mainstream qualitative evaluation studies, it would be helpful if more schools and workers could be recruited. Second, besides one-shot focus group interviews, regular and ongoing qualitative evaluation data could be collected. Third, although focus group interview data were collected, inclusion of other qualitative evaluation strategies, such as in-depth individual interviews, would be helpful in order to further understand the subjective experiences of the program implementers. Despite the above limitations, the present qualitative findings based on the experiences of program implementers showed that the respondents had positive perceptions of the program and implementers, and they perceived benefits of the programs throughout the years.


Biology, biomedicine and healthcare have become data-driven enterprises, where scientists and clinicians need to generate, access, validate, interpret and integrate different kinds of experimental and patient-related data. Thus, recording and reporting of data in a systematic and unambiguous fashion is crucial to allow aggregation and re-use of data. This paper reviews the benefits of existing biomedical data standards and focuses on key elements to record experiments for therapy development. Specifically, we describe the experiments performed in molecular, cellular, animal and clinical models. We also provide an example set of elements for a therapy tested in a phase I clinical trial.We introduce the Guidelines for Information About Therapy Experiments (GIATE), a minimum information checklist creating a consistent framework to transparently report the purpose, methods and results of the therapeutic experiments. A discussion on the scope, design and structure of the guidelines is presented, together with a description of the intended audience. We also present complementary resources such as a classification scheme, and two alternative ways of creating GIATE information: an electronic lab notebook and a simple spreadsheet-based format. Finally, we use GIATE to record the details of the phase I clinical trial of CHT-25 for patients with refractory lymphomas. The benefits of using GIATE for this experiment are discussed.131Iodine labeled CHT-25 chimeric antibody cancer therapy. As future work, we will expand the set of GIATE tools to continue to encourage its use by cancer researchers, and we will engineer an ontology to annotate GIATE elements and facilitate unambiguous interpretation and data integration.While data standards are being developed to facilitate data sharing and integration in various aspects of experimental medicine, such as genomics and clinical data, no previous work focused on therapy development. We propose a checklist for therapy experiments and demonstrate its use in the  Recording and reporting experiments -- including their context, design, methods and results -- in an unambiguous manner is crucial for the advancement of biological and biomedical research. Systematic reporting enables data sharing and reuse, thereby avoiding repetition and inefficient use of resources. Unambiguous data recording allows for well-grounded comparisons and aggregation of experimental results. Analysis of the aggregated data as a large dataset is more likely to produce statistically significant results. It is also expected to support new hypothesis testing, simpler and better systematic reviews and meta-analyses. Moreover, the data could be used for teaching and training purposes . In summMinimum information (MI) checklists, in general, advocate reporting transparency, better access to the data and support for effective quality assessment [The development and use of guidelines containing key information required to describe different kinds of biological and biomedical data are becoming widespread. For example, the practice of recording microarray data to the Minimum Information About a Microarray Experiment (MIAME) has been successfully adopted by the transcriptomics community. Many journals  and fundsessment . They hasessment .The Minimum Information for Biological and Biomedical Investigations (MIBBI)  project The EQUATOR   is controlled. An ontology, on the other hand, is a formal representation  of the knowledge in a particular domain as a set of concepts and their relationships. An ontology provides greater interoperability than a controlled vocabulary. Continuing with the examples from the microarray community, the MGED society also produced an ontology based on the MIAME guidelines -- the MGED ontology ,20 (MO) Thus, the three basic components of a reporting structure are ,15:• Minimum information (MI) specifications or checklists• Data formats: capturing MI in standard, non-proprietary formats• Controlled vocabularies or ontologies: using unambiguous, standard termswhat should be reported, the data standards specify the format or syntax to be used, and the terminologies or ontologies ensure that the meaning (or semantics) of the different elements is unambiguous.In summary, the checklists indicate Therapy development involves activities ranging from target discovery, the design of a therapeutic agent, through to investigations of the effects of the agent in molecular, cellular, animal and clinical models. Consequently, therapy development involves interpreting and integrating information from heterogeneous domains.While communities within the biological, biomedical and health research fields have developed guidelines to report various experimental data, none have specifically addressed therapy development. As in other areas of biomedical research , publicaA standard for therapy experiments would bring together different types of information, using, where possible, existing standards corresponding to the relevant sub-domains. The linkage of data from each sub-domain would produce a unified view of the different stages of therapy development. The immediate advantages of utilising a standard for this linkage are the avoidance of misinterpretations and repetition of time consuming tasks, as well as the minimisation of the risk to early phase clinical trials due to missing or misinterpreted data.In this paper, we describe the Guidelines for Information About Therapy Experiments (GIATE) as a consistent information framework for linking diverse data types that can be applied to all the major types of therapy. Using GIATE, complex data sets can be linked to facilitate the understanding of the therapeutic system as a whole, contributing to optimise the efficiency and safety in the development of new treatments. One objective of the GIATE framework is to improve the communication between basic and clinical research by relating data from both ends of the development spectrum. Thus, with this translational medicine approach, we expect that more therapeutic insights may be derived from new scientific information.® (caBIG®) infrastructure [GIATE originated from a collaboration among members of the Antibody Society , who wortructure ,26, whostructure .At a later stage, GIATE was extended to support other therapeutic approaches , while fIn this paper, we review and extend previous GIATE developments. The extension of previous work includes making GIATE objectives, scope, audience, design and structure explicit. As regards GIATE structure, we identify the modules that compose GIATE. We also introduce an extended GIATE checklist and spreadsheet-based data format. Finally, we consider a particular cancer therapy approach as a use case (CHT-25), and this is made available in GIATE structured spreadsheets.GIATE (Guidelines for Information About Therapy Experiments) is a set of guidelines for the key information that should be reported about a therapy experiment, so that it can be properly understood, analysed and reproduced.The objectives of the development of GIATE are as follows:• To provide a consistent information framework for reporting therapy experiments in a transparent way• To support efficient data-sharing of therapy experiments' methods and results• To guide the description of therapy experiments, enabling reproducibility, data reuse and re-purposing to avoid duplicated effort, support comparisons between experiments and increase the quality of the data• To facilitate the integration of data coming from more than one experiment in a machine-processable way• To allow for the possibility of obtaining aggregated and new knowledge coming from the statistical analysis of the data or mining of the aggregated information (data and knowledge mining)• To develop tools to help the different GIATE stakeholders to use the GIATE guidelines• To facilitate semantic publication of therapy experiments The scope of the GIATE guidelines is therapy experiments, such as small molecules targeting molecular pathways, engineered protein therapeutics, radio-immunotherapy, anti-vascular therapies, cellular therapies and others. For the design and development of new and improved therapies, it is crucial to be able to integrate laboratory and clinical data. Thus, GIATE aims at providing an information framework to record the properties of the target, the agent and models, the therapy investigation design, context and outcomes from the different studies.researchers designing a therapy experiment, to assist them in collecting consistent data elements. Moreover, they are intended to act as a reference in order to form a coherent description of an experiment.These guidelines are mainly intended for use by journal editors and referees to help confirm that certain data elements are uniformly presented in publications, and that their interpretation is unambiguous, allowing for the verification of the conclusions obtained.GIATE is also intended to be used by Other potential beneficiaries of GIATE are the communities that regulate and fund therapy experiments. Within these communities, the Antibody Society  has adopAdditionally, GIATE is intended to be used for educational or training purposes: we endorse the view that good research reporting habits should be introduced as early as possible in a researcher's career, to both students and young researchers . The strTo benefit the different stakeholders, the GIATE project aims at providing tools that will help in making the extra effort of recording the experiments minimal and worthwhile.GIATE  is part GIATE is structured as a set of modules, most of which are specific to therapy experiments. Others, such as the citation module, are generic and could be reused within other guidelines. GIATE establishes the relationship between its constituent modules. Each module encapsulates the information related to a specific sub-domain of therapy development. A modular design guarantees that changes in a specific module will be local and will not affect the guidelines and tools related to other modules.'meta-guideline' because it deals with high-level information from studies ranging from the lab to the clinic. These other guidelines form independent modules in GIATE, where appropriate. Users are therefore presented with domain specific guidelines if a higher level of detail is required to record any aspect of their data. GIATE, then, is the 'glue' linking diverse data to support translational research.In comparison to guidelines that focus on specific assay results, such as MIAME ,34 or thAnother important feature of GIATE is that it is not static. As with other checklists such as the Minimum Information for Molecular Interaction experiments (MIMIx) , it is eFigure investigation refers to 'a self-contained unit of scientific enquiry' [studies and assays. Core GIATE modules can be seen as a tree, including the therapeutic investigation description at the root and more specific studies including data on the therapy development at the branches and leaves.As per MIBBI guidelines, we consider the distinction between Investigation, Study, and Assay. An enquiry'  that is Therapeutic Investigation, whose design is determined by the Therapeutic Target and the Therapeutic Agent. In turn, the Agent may be composed of one or more Components.The main GIATE module is the Therapeutic Investigation module, with sub-modules describing the target and the agent with its components. Supplementary modules for citations about the target, agent and components are also included, together with links to external resources.Figure Models  and Clinical. A particular Investigation may have been applied only to some of the models, for example to Cellular and Molecular models but not to the rest. Progress made from bench-side to bed-side can be tracked with the information in GIATE element types. Common characteristics of all the models are grouped into a generic Model module. Each of these models may have one or more assays. For example, a cellular model might contain information about cellular assays, which are reported to the MIACA checklist. When describing each of the modules, we discuss some of the relevant guidelines that researchers should consider for each sub-domain. Figure The studies are represented by the different Citations, which is described in more detail when the GIATE checklist is introduced. In the future, other supplementary modules such as Imaging will be considered, given their role in therapy development [In addition to the GIATE core modules, we designed a module representing elopment .classification scheme, as they are grouped by the common characteristic of representing GIATE information.As described in the background section, GIATE was initially designed as a set of Common Data Elements (CDEs), as per the ISO/IEC 11179 standard for metadata registries . A regisObjectClass, a Property and a Value Domain.Figure Thus, a CDE involves simultaneously the three aspects of a reporting structure as seen before: what to record, how to record it, what is the meaning of the information recorded. Additionally, what to record (checklist) and how to do it (format) are intertwined between the conceptual and representational levels.ObjectClass related to Animal while others had an ObjectClass related to Organism: Animal Cancer Model Phenotype Description java.lang.String and Organism Species Name java.lang.String [what to record, having to use these two CDEs might be confusing, as in both cases we are referring to properties of the Organism used in the Animal Model.When identifying the sets of CDEs for a particular domain, it is recommended to reuse existing CDEs as much as possible, as this results in increasing interoperability of new data resources based on the new classification scheme with existing data resources. When developing the GIATE checklist we found that making this effort to reuse CDEs could impose constraints on the content of the key elements. For example, when dealing with the animal model some of the existing CDEs had an g.String . HoweverIn this paper, then, we present the key information elements independently of the CDEs, which can be associated a later stage. Thus, we divide GIATE in the three levels as determined by a reporting structure, and present the content to be reported independently of any data format.One of the tools developed to support GIATE is the GIATE Notebook-a piece of software that can be used as an electronic lab book to capture data on therapy experiments. The interface is composed of three panels: one for the GIATE elements, another one containing the CDE details showing associated terminology for each element and the third for data entry. The data produced with the GIATE Notebook can be exported as an eXtensible Markup Language (XML) document or in Portable Document Format (PDF).More details on the GIATE notebook and its use for the ADEPT therapeutic investigation applied to an animal model  were preWe have developed a checklist with the key information that should be recorded about therapy experiments. The checklist main modules are as in Figure This module involves some general information such as the goal and a brief description of the experiment, with an indication of the therapy type (e.g. antibody therapy), and a set of keywords and experimental factors. It also includes two sub-modules specifying the target and the agent, including possible components and their properties.For the target, agent and components, it is required to specify their identifiers as available in public databases.As in other minimum information specifications, such as MIMIx , we emphA database accession number identifies a unique molecule. In the case of a gene, giving its database accession number not only indicates its name but also the species from which the gene originated, which cannot be known by providing the gene name alone. It is noted that the annotations of proteins may change over time, for example when the coding sequence prediction programmes are updated . These cTable Therapeutic Investigation module will be included when reporting any kind of therapy experiment, regardless of which models are included.It is expected that the This module describes the experiments that study the strength of bonds between the target molecule and agent (or between components of the agent) as well as the distribution of the agent. Bond strengths are described in terms of affinity and avidity. Distribution is described in terms of concentration, volume and stability. This module also includes information about dose regimens.A relevant minimum information specification that could be use to complement this module is MIMIx .This module describes studies at the cellular level. GIATE recommends describing the genetic and epigenetic profiles of the cell lines in terms of:• The germline and somatic mutations• Epigenetic silencing• Gene expression fold changes.This module is also used to record the distribution of both target and agent, in relation to specific dose regimens and the concentration and duration of drug exposure required for efficacy and toxicity.This module lists information elements relevant to therapy experiments in animal models. Some key elements are also present in the cellular model: e.g. genetic and epigenetic profile, and target distribution study. GIATE recommends recording details about the organism , as well as pharmacokinetics, pharmacodynamics and therapy outcomes in relation to different dose regimens.For a clinical model, GIATE recommends recording information such as the name of the trial, its phase, the number of patients, their medical conditions , the tyConsidering endpoints, i.e. measurements that can demonstrate the clinical benefit of the trial, some possible values are: overall survival (OS), time to tumour progression (TTP), objective overall response (ORR), complete response (CR) and time to treatment failure (TTF) .Safety is a very important factor in the clinical model. In particular, for first-in-human trials, both the safe starting dose and higher dose levels or dose escalation criteria is paramount. Usually, the dose selection is based on specifically designed preclinical pharmacology and toxicology studies, ex vivo or in vitro experiments with human and animal cells and pharmacokinetics/pharmacodynamics (PK/PD) studies . Thus, tAs in the animal model, GIATE recommends to include the genetic and epigenetic profiles, target distribution, PK/PD studies, and therapy outcomes.If applicable, GIATE recommends considering existing guidelines on health research for the clinical model.clinical model is a randomised controlled trial (RCT), the Consolidated Standards of Reporting Trials (CONSORT) should be used [For example, if the  be used . The CON be used . The CON be used .GIATE specifies a citation module. This is a supplementary module, as it is not specifically related to a therapy development. A citation is a reference to another entity.When referring to bibliographic citations, the reference points to a publication such as a journal article, a book, a chapter, or a web page . On the Citation module is generic and allows to link any module or any of its elements to an entity, which can be a journal article, a database, a database record, a web page, a multimedia item, and so on. In Figure Therapeutic Investigation or each of the Models can be linked to one or more citations.Our We have developed a simple spreadsheet-based format for recording information about GIATE: GIATE TABular (GIATE-TAB). The advantages of using a spreadsheet are two-fold: researchers are usually familiar with this format and it gives them some freedom on how they report the experiment. We believe that this is particularly important when the guidelines are in the first phases of development, as this will allow scientists to add information elements they consider relevant and feed back so that these can be incorporated in subsequent versions.GIATE-TAB includes not only metadata about the therapeutic investigation, as described in the GIATE checklist, but also some generic input metadata for each module  to the IL-2 receptor in refractory lymphomas [In this section, we present the use of the GIATE checklist and GIATE-TAB for a therapy experiment described in . This exymphomas . The maiymphomas  and it wIn the therapeutic investigation module  and its two components (CHT-25 and 131Iodine) are specified. CHT25 is a chimeric monoclonal antibody with murine variable regions and human constant regions. CHT25 was radiolabelled with 131I, an appropriate radionuclide for radio-immunotherapy as it has β emission length of 0.8 mm and γ emissions for imaging purposes.Additionally, the target , the agent  in which standard therapies had failed or were not tolerated .• The study design or type, which in this case is single centre, open label, non-randomized, multiple dose escalation phase I trial.• The conclusions: it was found that CHT25 has an important clinical activity in CD25 positive refractory lymphomas; it is relatively non-immunogenic with low toxicity at a non-myeloablative dose. Further studies are required to assess clinical effectiveness and these will be carried out in a Phase 2 trial.General Information is linked to a citation module referring to the article [This sub-module on  article .For this therapy, information was included about individual patients. Elements referring to the type of lymphoma and treatment history are considered, including treatments such as chemotherapy, Autologous Stem Cell Transplant (ASCT), radiotherapy, time since last therapy, stage at therapy, and bone marrow involvement.genetic/epigenetic profile, target distribution and pharmacodynamics are not relevant for this particular investigation, and that is indicated in GIATE-TAB.The sub-modules for dose regimen is included for the investigation and for individual patients. The study consisted of a dose escalation using a standard dose of 10 mg of CHT25 antibody, with escalation of the radioactive iodine from 370Mbq/m2 to 2960Mbq/m2.Information for the 2 with grade 4 myelosuppression in one patient. The patient failed stem cell re-engraftment and died of infection. The dose was reduced to obtain the maximum tolerated dose and 3 patients were treated at 1200 Mbq/m2 with recruitment completing at 1480 Mbq/m2. Other toxicities were mild.CHT-25 was administered to 13 patients in 24 cycles. The dose limiting toxicity was determined at 2960 Mbq/mdistribution study was performed to analyse radioactivity uptake in target and non-target tissues. The main conclusions are summarised in GIATE-TAB.A radiation dosimetry study are also included. It is noted that this GIATE sub-module is only relevant for radioimmunotherapy experiments.Details of the Pharmacokinetics (PK) studies, i.e. how a drug or substance is absorbed, distributed, processed and eliminated in animals and humans. In order to study the PK for 131I in the CHT-25 investigation, blood samples were taken into EDTA blood tubes at the following time points, when possible: 1, 3, 6 and 24 h, then on day 2, 3, 6 and 9. The data presented in the paper [GIATE-TAB also includes information about he paper  has beenoutcome is also included, at the investigation and patient levels. Elements included are best response, Common Toxicity Adverse Events Grade and survival status. As the Cheson criteria [et al. 's article [outcome study sub-module.Finally, a sub-module indicating the criteria  were use article  was linkIn this section, we indicate how GIATE has contributed to knowledge about CHT-25 and how having the data elements in the GIATE-TAB spreadsheet will help in understanding the different components of the therapy as well as facilitate secondary use of the data.overall view of the CHT-25 therapeutic investigation, which highlights the main points and their relationships. This process is simplified by the spreadsheet in comparison with the more time consuming task of reading the paper, the protocol, and if necessary, contacting the authors of the trial.Firstly, the spreadsheet provides an links to external resources that are available neither in the scientific article nor in the protocol. For example, the GIATE-TAB format for CHT-25 makes clear that the therapeutic target is CD25-Interleukin-2 receptor subunit alpha, accessible in UniProt . The specific link to UniProt allows users to uniquely identify the molecule that is mentioned in the paper. Thus, scientists wanting to analyse the CHT-25 trial could navigate to extra information about the CD25 molecule. Similarly, additional information about the agent component CHT-25 is accessible through the DrugBank database (at http://www.drugbank.ca/drugs/DB00074).Secondly, the spreadsheet provides re-use of the information in a dynamic fashion. For example, while the pharmacokinetics analysis is available in the paper as Table Thirdly, the paper provides static information presented in diagrams and tables. On the other hand, the GIATE-TAB for CHT-25 makes answering queries about the therapeutic investigation much easier than having to go through the whole paper or protocol. For example, GIATE-TAB allows users to identify quickly what were the studies performed for the CHT-25 therapy and compare the Cheson score for the outcome of each patient at a glance. Additionally, GIATE-TAB is a step towards answering queries about therapeutic investigations in a machine processable way. As part of our future work, we intend to build a therapeutic investigations knowledge base, which will support to retrieve this kind of information.Finally, having the CHT-25 data in GIATE-TAB format facilitates The development of therapy experiments involves activities ranging from target discovery to therapeutic design, and experiments to study the therapeutic approach performed in molecular, cellular, animal and clinical models. Interpreting this heterogeneous information in an unambiguous fashion is fundamental to draw new conclusions that interrelate data from the different models. GIATE has been presented as a set of guidelines divided into several modules, each dealing with one of the aspects or stages of the therapy development process. We have introduced the key elements of each of the modules and a use case for the CHT-25 therapy, focusing on collating GIATE information about the target, the agent and the molecular model and the phase I trial. As demonstrated in other areas of biological and biomedical research, producing a guideline to record experiments is the first step towards being able to report them transparently, compare them and integrate data coming from different experiments. We discussed the benefits of describing the CHT-25 therapy following GIATE.As future work, we will develop an ontology associated with GIATE to facilitate both data annotation and data integration, by making the recorded elements unambiguous. Additionally, we will provide a machine-processable format to store information elements and facilitate automated data integration. This format will support building a knowledge base of therapeutic investigations with rich querying capabilities and links to other relevant data repositories.We welcome feedback from the scientific community to help improve our proposal for recording therapy experiments. The GIATE project's email address is giate@ucl.ac.uk.®: cancer Biomedical Informatics Grid(R); ASCT: Autologous Stem Cell Transplant; CDE: Common Data Element; CiBEX: Center for Information Biology gene Expression; CONSORT: Consolidated Standards of Reporting Trials; CR: Complete Response; DC: Dublin Core; EQUATOR: Enhancing the QUAlisty and Transparency Of health Research; GEO: Gene Expression Omnibus; GIATE: Guidelines for Information About Therapy Experiments; GIATE-TAB: GIATE TABular; MABEL: Minimal Anticipated Biological Effect Level; MAGE-ML: MicroArray Gene Expression Mark-up Language; MAGE-TAB: MicroArray Gene Expression Tabular; MGED: Microarray Gene Expression Data; MI: Minimum Information; MIACA: Minimum Information About a Cellular Assay; MIAME: Minimum Information About a Microarray Experiment; MIBBI: Minimum Information for Biological and Biomedical Investigations; MIMIx: Minimum Information for Molecular Interaction; MO: MGED Ontology; NOAEL: No Observed Adverse Effects Level; ORR: Objective Overall Response; OS: Overall Survival; PD: Pharmacodynamics; PDF: Portable Document Format; PK: Pharmacokinetics; RCT: Randomised Controlled Trial; SNOMED CT: Systematized NOmenclature of MEDicine-Clinical Terms; TTF: Time to Treatment Failure; TTP: Time to Tumour Progression; XML: eXtensible Markup Language.caBIGThe authors declare that they have no competing interests.RB, MY and AGB contributed to the GIATE checklist and presented it as a MIBBI checklist, extending the previous guidelines developed by RB, MY and others and initially expressed as a classification scheme for ISO/IEC 11179. MY had developed the GIATE notebook, including the initial set of guidelines. MY and AGB developed the GIATE-TAB format, based on the GIATE checklist. RB and GD reviewed the GIATE checklist and GIATE-TAB format. MY, AGB and GD completed GIATE-TAB for the CHT-25 use case. RB and GD verified that the information about the CHT-25 use case was correct. AGB wrote the initial version of the manuscript, which was revised and edited by the rest of the authors. RB provided vision, scope, and requirements analysis. All authors participated in revision and have read and approved the manuscript.The GIATE checklist version 0.1 is included as an additional file. It is noted that the file contains links to terms belonging to different ontologies. Some of these terms belong to the NCI thesaurus and were extracted from the GIATE classification scheme. These terms are provided as a guideline but are not considered part of the GIATE checklist. The terms will be exploited during the development of the GIATE ontology. -- GIATE checklist version 0.1.Click here for fileThe second additional file is the GIATE-TAB spreadsheet to be used in conjunction with the GIATE checklist, when compiling metadata about therapy experiments. -- GIATE-TAB spreadsheet.Click here for fileThe third additional file is the GIATE-TAB spreadsheet completed with metadata about the CHT-25 phase 1 trial. -- GIATE-TAB spreadsheet for the CHT-25 cancer therapy.Click here for file


Neglected tropical diseases (NTDs) cause significant morbidity and mortality worldwide and impose a large economic burden on endemic countries The implementation of integrated NTD programs at the full national scale remains an important objective in many endemic countries To allow governments to more easily enumerate costs and funding commitments for NTD control and elimination, the NTD Control Program developed the Tool for Integrated Planning and Costing (TIPAC). The TIPAC, a versatile planning and costing instrument, is designed to be used by members of a NTD program at the national level. For countries with decentralized political structures, the TIPAC can also be implemented at a subnational administrative level. NTD program and financial managers are the primary users of the tool; the involvement of other personnel, including representatives from partner organizations and ministries of education, improves the accuracy and completeness of the TIPAC data.The TIPAC implementation process includes four main phases:Background data collection: country coordinators compile, review, and verify demographic, epidemiologic, and cost classification data. In countries where individual disease control programs are not integrated, this phase may generate fruitful discussions among stakeholders and stimulate collaboration on program planning, outreach, implementation, and monitoring.Data entry: a focal person appointed by the national program enters the planned activity costs. The national strategic plans for NTD control and elimination, also referred to as master plans, serve as the guiding documents for data entry. As the tool is populated, integration opportunities and areas of overlap and duplication are identified. After the costs are entered, stakeholder meetings are convened to identify drug and funding commitments.Finalization and approval: the entered data is reviewed by all stakeholders and approved for use by ministry of health representatives from the national NTD control and elimination program.The aim of this feasibility study is to assess whether the TIPAC effectively informs and facilitates country program decision-making and the integration of program activities. This study presents excerpts from data collected in two African countries , one Asian country , and one Latin American country . The lessons learned from implementing the TIPAC in these four countries can guide the planning and costing of annual NTD control and elimination activities in other NTD-endemic countries.The initial 2009 tool was a one-page spreadsheet in Microsoft Excel which allowed users to enter cost and funding levels for planned NTD program activities. A subsequent multisheet Excel workbook was developed, which enabled users to more accurately specify activity costs . In 2009 and 2010, the tool was implemented in 14 countries  by technical advisers in collaboration with ministries of health. Based on these experiences, it was revised in 2012 to strengthen its functionality and user-friendliness, increase the transparency of cost inputs, and enable more advanced customizable reports. Prior to 2012, the tool was known as the Funding Gap Analysis Tool (FGAT). The newest version of the tool, available in Bahasa Indonesia, English, French, Portuguese, and Spanish, facilitates program planning and budgeting. The main improvements include a timeline of activities, an expanded capacity to enter information for non-PC NTDs, and a five-year projection of activity costs and PC drug needs.The current TIPAC version was developed in Microsoft Excel 2007. The tool applies an iterative approach to guide users through data entry, with user forms designed in Visual Basic for Applications (VBA). It includes four data entry modules and one reports module:Base data: stores demographic, epidemiologic, and program information for use throughout the tool.Activity costs: collects costs of NTD program activities.PC drug acquisition: estimates PC drug demand and costs .The planning and costing of activities are based on World Health Organization (WHO) treatment guidelines for implementing integrated PC disease control programs and the scope of the national strategic plans This paper examines the feasibility of the TIPAC to support country program decision making and the integration of program activities. Feasibility is assessed based on the five tool objectives:Estimate the costs of implementing integrated NTD programs in accordance with international guidelines and national plans of action.Quantify the existing resources of governments and donors.Identify the funding gaps of the national NTD programs.Encourage the rational allocation of resources and coordination between governments, implementing partners, and donors.Facilitate the identification of integration opportunities and annual planning of NTD programs.The countries represented in this study  were selected to provide a diverse geographic distribution and varied disease landscapes for analysis. Of the four, Tanzania is the only country treating all five diseases, while Haiti, Nepal, and Sierra Leone treat two, three, and four diseases, respectively. Haiti and Sierra Leone targeted 100% of the endemic districts for treatment. Nepal targeted all districts for STH treatment, but is continuing to scale up treatment for LF and trachoma. Tanzania only provided data for 25 districts that were targeted with USAID funding, or 19% of the total number of districts. An additional 36 districts in Tanzania were supported by the African Programme for Onchocerciasis Control (APOC), but were not included in the TIPAC. Tanzania is using a phased approach to scale up NTD treatment and has not yet reached 100% geographic coverage. Key information on the national NTD programs is summarized in All monetary values entered into the TIPAC were converted to US dollars ($) using the exchange rate provided by a government employee or technical adviser at the time of data entry . All the data were initially entered into an earlier version of the TIPAC and later transferred to an updated version. The cost and funding data were obtained from the perspective of national NTD programs.The level of time and effort required for data entry varied across the four countries depending on the size, scope, complexity, and maturity of the programs. In most cases, ten working days were needed to enter the vast majority of the data; additional input and refinement was possible as new information became available. Each country program received in-country assistance from a trained facilitator who introduced the TIPAC to program managers and provided technical support during implementation. Programs were encouraged to appoint a national focal person to maintain and update the TIPAC for subsequent work planning. In all four countries, data were successfully generated that contributed to the following key objectives.The epidemiological data captured in the Base Data module were used by countries to set targets and estimate annual costs to operationalize national plans of action. In addition, countries were able to calculate the average projected cost per person for PC NTDs. To illustrate the ability to quantify existing resources from government and donor budgets , In Sierra Leone, staff salaries contributed to 15% of the total program funding, while government resources allocated through the national budgeting process accounted for less than 1% of the total. This data helped the Ministry of Health and Sanitation (MOHS) better understand the annual costs of implementing NTD activities, the financial commitments of donors, and the resources required for scaling up to reach national coverage. It motivated the MOHS to increase their financial commitment for drug distribution, drug storage, and personnel salaries.The TIPAC provided program managers with a valuable advocacy tool and an opportunity to communicate to funders how available resources aligned with planned activities and where additional assistance was needed. For example, in the four countries described in this study, the results of the TIPAC were used during annual planning to identify funding gaps and to guide the budget requests for USAID support. Notably, the implementation of the TIPAC in Sierra Leone reaffirmed that morbidity control and surgery  was underfunded. Johnson & Johnson therefore recommitted funding for hydrocele surgeries and physician training on novel hydrocele surgery techniques. Thus far, more than 150 patients have received free operations and 50 physicians have obtained supplementary training. In Nepal, the data entry process enabled the program to identify and quantify the funding gap, which corresponded to 10.3%  of the total budgeted program costs.Potential gaps in the quantity of drug units required for each targeted PC NTD and the associated costs were automatically calculated for each country based on their respective epidemiologic statuses and the entered program goals for the time period . This information was used to check against drug procurement records to confirm whether quantities of donated and procured drugs were sufficient to cover targeted districts. In Haiti, populating the TIPAC allowed the various partners, including departments within the Ministry of Public Health and Population (MSPP), to share information on the NTD activities they were conducting or planning. These activities included morbidity management, vector control through the use of long-lasting insecticide treated nets (LLITNs), and a new proposal to implement a second round of mebendazole for STH in targeted districts. Although the second round of mebendazole was not approved by the MSPP, further discussions led the Inter-American Development Bank (IDB) to instead finance a second round of albendazole in five districts. Through this process, the MSPP gained an enhanced understanding of the contributions of its partners and adjusted implementation strategies when appropriate.Given that the desired number of PC treatments administered is fixed, program managers were able to determine the costs of alternative PC approaches  and revise policies according to specific cost drivers. The implementation of the TIPAC in Nepal, alongside other strategic planning discussions, highlighted potential integration opportunities across the LF, STH, and trachoma programs. For example, the national NTD program in Nepal is now integrating information, education, and communication (IEC) materials for the three diseases. The program also considered integrating training procedures. However, due to the disease-specific training strategies and reports of serious adverse effects (SAEs) from previous LF PC rounds, the national program decided not to integrate training. Nepal's experience highlights the usefulness of the TIPAC for identifying integration opportunities, and it emphasizes the need to make country-specific decisions on which activities are feasible and appropriate to integrate.Health systems worldwide operate under relentless fiscal pressure to curb escalating costs. The emphasis on cost containment is heightened in low- and middle-income countries where scarce resources necessitate improved resource allocation for NTD programs. Previous studies calculating the costs of implementing NTD activities have mostly consisted of retrospective analyses of PC costs AdvantagesCollaborative stakeholder engagement: improves accountability and coordination; encourages programs to set realistic objectives.Comprehensive data collection: tool use promotes the collection of programmatic, epidemiologic, and financial data; these data can also be used by countries for other purposes .Itemized costing: enhances program transparency, encourages activity integration, and helps inform funding requests.Long-term outlook: multiyear planning improves the accuracy and validity of the data and facilitates the identification of long-term cost efficiencies.Tool versatility: the data entry process is flexible, and most inputs can be adjusted based on the needs of individual programs, including those targeting non-PC NTDs; a generalized version of the tool is also available for use by other programs .LimitationsInter-user variability: the quality of the results depends on the accuracy and integrity of the inputs; in-country assistance from a trained facilitator is recommended to introduce the TIPAC to program managers and to provide technical support during the initial implementation round.Prospective planning exercise: may adversely affect the validity of data if a user believes that entering higher costs will increase funding; it is important to involve the national personnel responsible for finances and other stakeholders when estimating costs.Time-intensive: ten working days were usually needed to populate the TIPAC; however, the duration of the data entry process is shortened once users become accustomed to the tool and are able to use stored information from previous years.The ability to itemize activity costs in the TIPAC-enabled countries to generate detailed estimates of annual program costs. These cost estimates were used to inform funding requests and were helpful for countries when discussing funding strategies with stakeholders. However, like other costing tools, the TIPAC allows for some inter-user variability; the quality of the results depends on the accuracy and integrity of the inputs. As previously noted, Tanzania only provided data for 25 districts that were targeted with USAID funding, which suggests a narrow application of the TIPAC to plan and cost NTD control and elimination activities. The inclusion or exclusion of activities on the basis of the funding source biases results and represents a salient challenge when coordinating activities across multiple implementing organizations.It is also important to remember that the costs entered into the TIPAC reflect planned expenditures and not necessarily realized expenditures. In some cases, reliance on the tool may therefore establish expectations for funding. This may impact the validity of data if a user believes that entering a cost will increase the likelihood of future funding. To minimize the inflation of the budgeted costs, it is suggested that programs start planning based on program strategies from previous years and involve the national personnel responsible for finances. Additionally, a well-trained point person in the MOH who is in charge of managing the implementation process can help to set realistic goals and cost thresholds. In Nepal, data were entered by representatives with complementary technical, programmatic, and financial expertise. Overall, a participatory data entry process that involves multiple stakeholders improves the transparency and quality of the TIPAC results.The results of the TIPAC highlighted the commitments made by governments and implementing partners. In particular, the TIPAC's automated calculation of government salaries outlined national ownership of NTD programs. Countries with limited resources may not be able to allocate substantial funds through their national budgeting process, but can show their commitment through the employment of full- and part-time employees for NTD control and elimination. Goldman and colleagues noted the importance of capturing government choices of resource allocation but faced challenges estimating the proportion of staff time spent on LF-specific efforts Inter-country differences in cost allocations were expected due to variations in population sizes, disease burdens, relative purchasing power parities, levels of integration of sub-activities, and year-to-year fluctuations in program foci . This study did not analyze differences in costs across countries. As costs were entered in current dollars for a given year, comparisons over time and across countries regarding funding commitments and achievement towards closing gaps would need to be converted to a base year. It would be useful, however, to monitor the progress of programs towards narrowing funding gaps over time. Levels of funding support are expected to change as donors re-evaluate funding commitments and strive to eliminate funding gaps.For nationally-owned programs, it is important that ministries of health have a comprehensive overview of the activities that are implemented and the funding that is allocated to NTD control and elimination. This is necessary to ensure effective coordination between stakeholders and rational allocation of resources. For example, the Global NGO Deworming Inventory has noted that although many NGOs administer deworming treatments, the national programs and the wider deworming community are often unaware of these efforts During the implementation of the TIPAC, it was observed that the process can be time-consuming. If the involved personnel are not able to participate throughout the entire data entry period, it may decrease the accuracy of the cost estimates and limit the use of the tool for coordination. It is therefore recommended that countries accumulate the relevant data  prior to implementing the TIPAC. Also, after the first year's entry, users are able to populate the tool more quickly as they become accustomed to the tool and can use the stored information from previous years. The NTD control program members in all four countries expressed positive experiences collaborating with colleagues during the data entry process; this finding may be applicable to any process or tool that helps organize and share programmatic, epidemiologic, and financial data in a transparent way.The integration of program elements provides an important method to reduce the transaction and administrative costs of organizing program activities and sub-activities. The similarity of the strategic approaches for the five NTDs targeted through PC delivery and the epidemiologic overlap among affected populations generate significant integration opportunities While these efficiencies may lead to some cost savings, a consistent decrease in costs over time is not always expected for certain activities. For example, turnover rates among government staff and volunteers will generate a continuous need to retrain personnel. Programs also experience diseconomies of scale and scope when targeting hard-to-reach populations. In addition, costs may increase towards the end of a program's lifespan, when impact assessments are implemented to monitor the effectiveness of interventions and to measure progress towards control and elimination goals. It is therefore important to not rely exclusively on financing estimates based on average costs that assume that costs will monotonically decrease over time.Implementing the TIPAC in these four countries offered an opportunity to assess the feasibility of using a versatile costing instrument to inform and facilitate resource planning. In the study countries, the TIPAC provided results that were concordant with the objectives of tool use; the implementation process also helped identify key lessons to improve future use. Populating the tool promoted synergistic efforts between national NTD programs and partners to estimate costs accurately, coordinate activities, identify integration opportunities, and achieve program goals to control and eliminate targeted diseases. Although the TIPAC is not a substitute for the strategic process of developing a national plan of action, it should strongly align with this document to improve resource planning. Once the financial landscape is evaluated through the TIPAC, a program manager can better decide the rate at which scale-up is possible, what activities should be postponed until funding is available, and the quantity of drugs needed to support program activities.Since the development of the TIPAC for national NTD control programs, the tool has also been modified and generalized for wider applicability. It has been adapted as a platform to cost and plan malaria control activities  and HIV programs. Additionally, a generic version of the tool is being developed that can be applied to other health interventions. The impetus to apply the TIPAC process to other projects is due to the growing need for national programs and donors to demonstrate efficiencies and to collaborate when scaling-up activities and forecasting future needs.The TIPAC was recently recognized by WHO as a recommended planning and costing tool for national NTD programs Table S1Default tool PC activities and subactivities.(DOC)Click here for additional data file.Text S1PC target population and drug demand algorithms.(DOC)Click here for additional data file.


There are only a few process evaluation studies on positive youth development programs, particularly in the Chinese context. This study aims to examine the quality of implementation of a positive youth development program  and investigate the relationships among program adherence, process factors, implementation quality, and perceived program success. Process evaluation of 97 classroom-based teaching units was conducted in 62 schools from 2005 to 2009. Findings based on different cohorts generally showed that there were high overall program adherence and implementation quality. Program adherence and implementation process were highly correlated with quality and success of the program. Multiple regression analyses further showed that both implementation process and program adherence are significant predictors of program quality and success. Theoretical and practical implications of the findings are discussed. Positive Adolescent Training through Holistic Social Programmes), a large-scale positive youth development program in Hong Kong.Program evaluation is systematic assessment of the process and outcomes of a program with the aim of contributing to program improvement, such as deciding whether to adopt the program further, enhancement of existing intervention protocols, and compliance with a set of explicit or implicit standards . This paOutcome evaluation focuses mainly on the results of the program, whereas process evaluation is concerned with how the program is actually delivered , 3. ProcProcess evaluation consists of five components, namely, program adherence, implementation process, intended dosage, macrolevel implication, and process-outcome linkage . ProgramProcess factors are elements that are contingent to implementation quality or success and can be observed during the implementation process. There is a variety of process factors in terms of program characteristics and program developers' needs. Some programs are even designed with their own process measurements . Common Process evaluation can provide important findings with macrolevel program implications, such as the importance of the engagement of different community stakeholders , 38, cliThe components of process evaluation point towards its importance. First, outcome evaluation provides inadequate hints on the quality of program implementation. Process evaluation demystifies the “black box” of intervention and aids in the understanding of the elements of program success or failure . ProcessMany primary prevention programs and positive youth development programs have been developed in the West to address the growing adolescent development problems, such as substance abuse, mental health problems, and school violence . HoweverThe project commenced in 2004 and is targeted to end by 2012. There are two tiers of programs in this project. The Tier 1 Program is a universal positive youth development program where students from Secondary 1 (Grade 7) to Secondary 3 (Grade 9) participate in a classroom-based program, normally with 20 h of training in the school year in each grade. Around one-fifth of adolescents with more psychosocial needs join the Tier 2 Program, which consists of intensive training on volunteer service, adventure-based counseling camp, and other experiential learning activities.The project was designed according to 15 constructs conducive to adolescent healthy development : promotiThere are two implementation phases: the Experimental Implementation Phase and the Full Implementation Phase. The Experimental Implementation Phase aims at accumulating experience from trial teaching and administrative arrangement. Program materials are revised and refined during this phase. The Full Implementation Phase aims at executing the programs in full force. Several lines of evidence support the effectiveness of the Tier 1 Program, including the evaluation findings based on randomized group trials , 49, subProcess evaluation has also been carried out in both the Experimental and Full Implementation Phases for different grades , 56. TheStudent Interest. A successful program usually elicits the interest of the students.Active Involvement of Students. The more involved the students are, the higher the possibility that the program can achieve its outcomes.Classroom Management. The program implementer can manage student discipline during student activities. Students obey the requirements set by the program implementer and are attentive.Interactive Delivery Method. Interactive delivery is better than didactic delivery for positive youth development programs.Strategies to Enhance the Motivation of Students. The use of various learning strategies can enhance the engagement of the students and result in positive learning outcomes.Positive Feedback. The use of praise and encouragement throughout the lessons by the program implementers can promote the engagement of the students.Familiarity of Implementers with the Students. All other things being equal, a high degree of familiarity with the students is positively related to student learning outcome.Reflective Learning. The program implementer should engage the students in reflection and deeper learning. This can lead to growth and meaningful changes among the students.Program Goal Attainment. The achievement of program goals constitutes program success.Time Management. Efficient time management ensures that the majority of the program materials are carried out with high program adherence.Familiarity of Program Implementers with the Implementation Materials. Familiarity with the material ensures that the messages are conveyed effectively to the students.These eleven attributes can form a checklist for evaluating the implementation process.On the other hand, program quality is the subjective appraisal of the program implementation process by the observer. It can be reflected from the implementation atmosphere and the interaction between program implementers and students. Program success refers to the extent of unit objective attainment and the subjective evaluation of the response of the students to the program.By conducting secondary data analyses on all datasets collected over the past 4 years regarding process evaluation, the current study has two purposes: (1) to understand the program implementation quality across different cohorts in terms of program adherence, process factors, program quality, and success and (2) to explore factors contributing to the overall quality and success of the Tier 1 Program. Correspondingly, there are two research questions: (1) what is the overall implementation quality of the Tier 1 Program of the Project P.A.T.H.S. in Hong Kong? and (2) how are program adherence and other indicators of process evaluation related to the implementation quality and success of the Tier 1 Program?From 2005 to 2009, the total number of schools that participated in the Project P.A.T.H.S. was 244. Among them, 46.27% of the respondent schools adopted the full program , whereas 53.73% of the respondent schools adopted the core program .A total of 62 schools were randomly selected to participate in the study of process evaluation ; 25.80% of the participating schools adopted the core program, while the remaining 74.2% adopted the full program. Around 65% of schools incorporated the program into their formal curriculum , 27.42% used the class teacher's period to implement the program, and less than 8% used other modes. The average number of students in the participating schools was 33.91. The characteristics of the schools that joined this process evaluation study can be seen in Process evaluation was carried out in each participating school using systematic observations of actual classroom program delivery. For each school that joined the process evaluation, one to two program units were evaluated by two independent observers who are project colleagues with master's degrees. A total of 97 units were observed for this study. During the observation, observers sat at the back of the classroom and evaluated the method by which the units were actually implemented through completing several evaluation forms.r = 0.84,   P < 0.001), suggesting that the assessment of program adherence was consistent.Observers were requested to rate program adherence in terms of percentage . Aggregating all data across studies, Pearson correlation analyses showed that the ratings of program adherence were highly reliable across raters (P < 0.001).Observers were requested to report their observations using a 7-point Likert scale ranging from 1 (extremely negative) to 7 (extremely positive) on the items. Aggregating the data across studies, the internal consistency of the scale, as shown by Cronbach's alpha, was 0.93. The interrater reliability of IPC, as shown by Pearson correlation, was 0.72 (P < 0.001), whereas that for implementation success was 0.64 (P < 0.001). In short, the aggregated data across studies suggest that the observations on process outcome were consistent across raters.Two items were used to evaluate the process outcome: implementation quality and implementation success. Observers were requested to indicate their observations using a 7-point Likert scale ranging from 1 (poor) to 7 (excellent). A higher score represents better implementation quality or success. The interrater reliability for implementation quality, as shown by Pearson correlation, was 0.74 (M = 5.62) and familiarity with students (M = 5.43) had the highest scores, whereas reflective learning (M = 4.96) had the lowest score. Apart from reflective learning, all scores were on the high side.As the interrater reliabilities of the scores across all units were high, the ratings of each item by the two observers were averaged to form a combined indicator. r = 0.93) and between process and quality (r = 0.81) were high, whereas the correlations between process and adherence (r = 0.40) and between quality and adherence (r = 0.49) were relatively low. Multiple regression analyses were further performed, in which program adherence and implementation process were entered as predictors, and implementation quality and implementation success as two separate dependent variables. R2 = 0.68). The effect size of the variance explained (Cohen f2), as calculated by (R2/1 − R2), is 2.13, which is large. R2 = 0.67). The effect size is 2.03, which is also large.This study attempts to integrate the process evaluation findings based on multiple studies via secondary data analyses. There are several unique features of this study. First, a large number of teaching units and schools was evaluated. Second, two observers who were independent raters conducted the assessment. Third, a structured and reliable measure of program implementation was used. Fourth, inter-rater reliability analyses showed that the observations were basically reliable. Finally, this is the first large-scale process evaluation of positive youth development programs in the Chinese context.Despite the discrepancy in the ratings of program adherence on different units, the overall degree of adherence to the program is on the high side. This observation is generally consistent with previous findings generated from separate process evaluation studies conducted by observers , 56 and The findings on program adherence are very encouraging because program adherence is generally low in the international context. In a meta-analysis of evaluation studies of primary and early secondary prevention programs published between 1980 and 1994, Dane and Schneider  showed tThe present study found that different aspects of the program delivery were perceived to be positive, highlighting the fact that the Tier 1 Program of the Project P.A.T.H.S. was well received by both program implementers and students. Moreover, the implementation was regarded as successful by the observers, although relatively low average ratings were reported on time management and reflective learning. These findings are similar to those based on the Experimental Implementation Phase , 56. TheThe current study also found that program adherence and implementation process are closely associated with implementation quality and success. For positive youth programs, an interactive program delivery is the key milestone for program quality and success . This exFindings of the present study have two implications. The first implication is on the conceptual level. When we are concerned about program implementation regardless of external environment , we can focus on three variables: program adherence, implementation process, and context. These variables are all related to implementation quality or success. The present findings provide conceptual insights for understanding program quality or success. The second implication is on a practical level. The process variables covered in the study can actually be used in other social work or health science contexts, especially in educational and developmental groups. All these measures are important for positive youth programs and should be brought to the fore in the training. Youth workers, social workers, and teachers should be aware that implementation process is critical for classroom-based psychosocial intervention programs.This study has several limitations. First, only 62 randomly selected schools participated in this study. Although the number of schools can be regarded as respectable, inclusion of more schools with diverse backgrounds would be helpful. Second, process evaluation with reference to macro-level implication, dosage issues  and inte