Universidade Federal do Rio Grande do Sul in Brazil as we celebrate the 30th anniversary of its first course in 1988. Paired with the beginning of the postgraduate psychology program, our journal was launched, and we would like to take this opportunity to talk about its path as well. In Brazil, many scientific journals have been launched and continue to be associated with higher education institutions. Psicologia: Reflexão e Crítica is one of these journals, affiliated with the Postgraduate Program in Psychology of Universidade Federal do Rio Grande do Sul (UFRGS). The journal actually appeared in 1986 with a modest regional proposal to discuss current ideas on the relations between psychology as a profession and science, hence the original title in Portuguese of reflexão (reflection) and crítica (criticism). In 1985, after 31 years of military dictatorship, democracy was reestablished in the country, and universities were discussing the best options for their administrative and academic policies. Thus, Psicologia: Reflexão e Crítica emerged, in the midst of these hearty discussions, as a precursor to the Postgraduate Program in Psychology created in 1987 and beginning its activities in March 1988. In its modest proposal, the journal adopted a scientific editorial style and soon attracted articles from researchers from leading Brazilian universities. The growing development of the Postgraduate course in psychology at UFRGS and the recognizable importance of the journal Psicologia: Reflexão e Crítica can be related, as the journal played a great role in documenting the history of the program, the diffusion of the research and the discussions regarding the advances in postgraduate studies in psychology in Brazil. Through time, Psicologia: Reflexão e Crítica became a singular periodical and a model in our context of pioneering initiatives towards internationalization . This interview was recorded and published in the first issue of the Psicologia: Reflexão e Crítica, entitled “Clinical Practice and Academic Work: An Interview with Psychologist Terezinha Féres-Carneiro”  and how to reconcile these demands. In addition, they asked about the experience of Professor Féres-Carneiro as coordinator of postgraduate studies in clinical psychology at They were also intrigued with what would be suitable conditions for an institution to organize a postgraduate course in psychology at the master’s and doctoral levels. At that time, the national requirements were that at least 50% of the faculty should teach in the main research concentration area, with the other part teaching in related areas. Today, the requirements are not so rigid, and the emphasis is placed on teaching and research production with the curriculum structured to suit the program’s lines of research.Pontifícia Universidade Católica do Rio de Janeiro. In the 1970s, master's degrees were created in the cities of São Paulo, Brasília, Porto Alegre and João Pessoa. In 1978, the evaluation system was instituted by Coordenação do Aperfeiçoamento do Pessoal do Ensino Superior (CAPES) in order to better evaluate and monitor the expanding programs, and in 1982, the CAPES agency became consolidated as a government postgraduate studies coordinator. In the 1980s, master’s programs in psychology were implanted in Recife and Natal. In the following decade, Belém, Florianópolis, Belo Horizonte, and Campo Grande added programs. In the 2000s, Salvador, Vitória, Goiânia, and Manaus implemented postgraduate studies. More recently, programs were implemented in Porto Velho and São Luís. Regarding the doctoral level, until 1970, there was no program in the area. In the 1970s, the first doctoral programs were created in São Paulo.Recalling the history of postgraduate psychology training in Brazil, we see that, in the 1960s, the only existing master's degree in psychology was at Rio de Janeiro. In the 1990s, doctoral programs were organized in the states of Rio Grande do Sul, Pará, Rio Grande do Norte, Pernambuco, Espírito Santo, and Brasilia DF. In the 2000s, the states of Santa Catarina, Minas Gerais, Bahia, Goiás, and Ceará created their doctoral programs. Thus, 20 Brazilian states already had postgraduate programs, of which only three did not offer a doctorate degree: Rondônia, Amazonas, and Maranhão.From the 1980s, doctoral programs in psychology emerged in several states, such as the state of The goal of implementing psychology postgraduate programs in all Brazilian states has been longstanding , São Paulo (three), Pernambuco (two), and Rio Grande do Norte (one).In addition, as noted by Féres-Carneiro et al. , the creUniversidade Federal do Rio Grande do Sul, which is one of the oldest program, the course has emerged as a center of excellence in research and the training of teacher-researchers, with extensive international placement, since being established. In 1988, the program had only five faculty members. Six years after the implementation of the master’s program, in 1994, the doctorate program was started and had 12 permanent faculty members and two collaborators.Returning to the history of the Postgraduate Program in Psychology of The program’s continued growth was recognized by the excellent evaluations obtained from the CAPES agency. In the triennium 1998–2000, the Postgraduate Program in Psychology of UFRGS earned a score of “five;” in the following period, it earned a score of “six;” and since the triennium 2004–2006, it has maintained a score of “seven,” the maximum score of the CAPES evaluation.Universidade Federal do Rio Grande do Sul” in Psicologia: Reflexão e Crítica; the article documented information of value about the program’s growth. At that point, the course had 18 faculties, 17 with the distinction of having received a Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq) researcher productivity scholarship  psychological assessment and measurement; (2) human development; (3) health, prevention, and intervention; and (4) cognitive and behavioral neuroscience. The journal To expand the lines of research, the curriculum is now organized around four major axes: thematic topics, methodology, academic, and complementary skills. The disciplines are offered in three modalities—obligatory, alternative, and elective—, a structure that bears a resemblance to the course’s initial proposal, despite the difference in relation to the multiple and innovative current developments.Universidade Federal do Rio Grande do Sul followed the growth of the country. In 2012, the number of programs evaluated by CAPES was 69; this number jumped to 84 in 2016. As a result, there was also an increase in the number of faculty members, which went from 1212 to 1558 in the same period. The postgraduate programs in the area have from 9 to 43 faculty members, with an average of 18.5 faculty members per program. Permanent faculty members represent 77% of the professors in Brazilian programs, with the remaining being visiting or collaborating professors. The intellectual production of the area has also greatly increased, increasing by 80% in the last 4 years, compared to the previous triennia, the period of years previously used in the CAPES evaluation . Currently, the Postgraduate Program in Psychology of UFRGS has 23 faculties, which is higher than the faculty average per program in Brazil. The stability of the faculty staff is worth noting. Three of the founders, professors Claudio Hutz, William Gomes and Cesar Piccinini remain in the program to this day. Others hired in the 1990s and 2000s remain in the program. The program has also sought to renew its faculty by incorporating young, productive researchers.Thus, the Postgraduate Program in Psychology of Universidade Federal do Rio Grande do Sul has the highest average H-Score among all the programs in the area: 25.8 . It was also assessed to have a remarkable academic maturity and high qualification, a quality common to its professors. In addition, several professors hold important positions in the decision-making committees of development agencies such as CAPES and CNPq, in the directorates of scientific societies, with publishers of national and international journals, and in the coordination research groups at the Associação Nacional de Pesquisa e Pós-graduação em Psicologia (ANPEPP). In addition, the majority of its professors are involved in some form of international exchange linked to their research projects.In the last CAPES assessment, the overall H-Score (Google Scholar) for the programs in the area was 9.26, and programs with H-scores above 10.5 are considered “Very Good” or as having a faculty with high maturity . The average quality of articles, books, and chapters was 96, which is equivalent to the top classification for high-quality chapters and books. This production is fairly well distributed among the faculty, and 85% of the faculty members produced above average for programs at the top of the CAPES assessment scale. In addition, production is highly internationalized. Approximately 84% of the articles were published in international journals or in English, in higher nationally evaluated journals . Thus, the last CAPES evaluation noted that the Postgraduate Program in Psychology of Universidade Federal do Rio Grande do Sul had the most outstanding scientific production in psychology in Brazil. With all these highlights, even in relation to other programs awarded a score of 7 by CAPES , we can say that the Postgraduate Program in Psychology of UFRGS is one of the best programs in the area, standing out from so many others, including some traditional programs, getting the highest score in the evaluations since 2004.Moreover, according to the 2017 Quadrennial Report , Ana Maria Nicolaci, and Teresinha Féres-Carneiro. For Universidade Federal do Rio Grande do Sul, they were Ângela Maria Brasil Biaggio (1940–2003), William Gomes, and Claudio Hutz. At the time, the symposia were scheduled to take place every year. In 1989, professors Claudio Hutz and William Gomes, organized the second symposium in Gramado, RS, proposing an innovative format that is maintained to this day and is implemented for other scientific events in the country. The format of working groups  has been very successful and continues to this day. At the second symposium, there were 10 working groups, who gathered with great eagerness in Gramado. In 1990, the third symposium took place in Águas de São Pedro, SP, and from then on, they were held every two years (even-numbered years). The number of ANPEPP working groups can easily evidence the growth of the area over the last 30 years. The second symposium held in Gramado, RS, in 1989 had 10 registered working groups. For the seventeenth symposium (2018), in Brasilia, DF, 81 working groups registered, and 79 have currently been approved.Another historical dimension of postgraduate studies in psychology in the country where the Postgraduate Program in Psychology of UFRGS played a very important role was the collaboration with ANPEPP. In 1988, the first symposium of the ANPEPP took place in Caruaru, PE, organized by Ana Lucia Schliemann, who was at that time a professor at the Psicologia: Reflexão e Crítica has represented all these changes and advances as a kind of public notary, documenting the facts as they occur. As an example, the journal published a special issue in 2015 that included 13 articles and documented the dialogs with researchers from other countries from the ANPEPP’s Forum around issues facing Brazilian scientific policies, including analysis of international relations on research and publication by scholars from different countries, evaluation of English language in articles of the top five Brazilian psychology journals, a detailed description of the Brazilian postgraduate education system for the non-Brazilian world, and a description of ANPEPP’s Symposium from the perspective of an observer from the American Psychological Association  have been consistently increasing. Along with an increasing interest from international researchers to publish in the journal. Recent data provided by Google analytics (June 2018) indicate a 35% increase in visits to our main website (https://prc.springeropen.com/) from 2016 to 2017. The ten countries with the highest access to the website in 2017 were (data within parenthesis indicates proportion in comparison with other countries) Brazil (26.7%), the United States (15.9%), the United Kingdom (7.5%), India (5.4%), Portugal (4.9%), Spain (2.6%), Australia (3.2%), Canada (2.2%), Philippines (1.9%), and the Netherlands (2.0%).In recent years, with the full professionalization of the journal Psicologia: Reflexão e Crítica over its 31 years of existence is an important piece of its success. This journal has been one of the most important publications on psychology in the country, presenting Brazilian research in psychology to the world abroad and being one of the few Brazilian journals ranking in the first quartile (“A1”) in the area’s Qualis .The tireless work of the current and former editors of Universidade Federal do Rio Grande do Sul, Professors Claudio Hutz, William Gomes, Cesar Piccinini, and the retired professors Tânia Mara Sperb and Maria Alice de Mattos Parente, who also contributed so much to the research lines on human development and neuropsychology, respectively. In particular, we would like to mention the dear teacher to many among us, Ângela Maria Brasil Biaggio. Professor Biaggio was a pioneer in developmental psychology research in the country. Unfortunately, she is no longer with us, but she gave us, like so many other great masters, an inheritance. She, like the great masters, taught us that this heritage can be transformed, leading us to be unafraid to dare and innovate, a legacy that we too must leave to our students and to the new generations, in the exercise of our gratifying profession to prepare them well for teaching and for research.Finally, we would like to pay tribute to the founders of the Postgraduate Program in Psychology of Additional file 1:30th Anniversary Ceremony of the Postgraduate Course in Psychology at the UFRGS (Brazil), March 12, 2018. (DOCX 35 kb)


This systematic review assessed the current evidence base of substance use prevention programs for Indigenous adolescents in the USA, Canada, Australia and New Zealand. The authors investigated (a) the outcomes, type, setting and context of prevention programs; (b) the common components of beneficial prevention programs; and (c) the methodological quality of evaluations of included prevention programs. The authors searched eight peer-reviewed and 20 grey literature databases for studies published between 1 January 1990 and 31 August 2017. Data extracted included type of program , the setting , delivery , context  and common components of the programs. Program evaluation methodologies were critically appraised against standardised criteria. This review identified 26 eligible studies. Substance use prevention programs for Indigenous youth led to reductions in substance use frequency and intention to use; improvements in substance-related knowledge, attitudes and resistance strategies; and delay in substance use initiation. Key elements of beneficial programs included substance use education, skills development, cultural knowledge enhancement and community involvement in program development. Five programs were rated as methodologically strong, seven were moderate and fourteen were weak. Prevention programs have the potential to reduce substance use among Indigenous adolescents, especially when they are developed in partnership with Indigenous people. However, more rigorously conducted evaluation trials are required to strengthen the evidence base. Some governments have officially acknowledged and apologised for their roles in the disruption and abuses inflicted upon Indigenous peoples , Canada, Australia and New Zealand have a comparable history of colonisation and dispossession of culture and land by English settlers, resulting in predominantly English-speaking countries in which Indigenous People are a marginalised minority. The centuries following early colonisation have continued to harm Indigenous peoples through This history combined with contemporary issues such as continued policy failures in social services, education and health care systems have resulted in loss of cultural knowledge and language in Indigenous communities and poorer outcomes in many of the social determinants of health compared with the non-Indigenous population, and significant trauma for Indigenous peoples with lasting inter-generational effects Cornell . The impSubstance use prevention strategies have shown to be effective for non-Indigenous adolescents, including school-based, community-based and family-based programs  the outcomes, type , setting  and context  of prevention programs; (b) the common components of beneficial substance use prevention programs; and (c) the methodological quality of evaluations of substance use prevention programs.This systematic review followed the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) guidelines  evaluated a prevention program aiming to reduce substance use and related outcomes, (b) compared an experimental group to a control group and/or assessed changes over time, (c) included participants aged 10 to 19 years, (d) comprised at least 50% of participants who identified as Indigenous and/or there was a specific sub-analysis for Indigenous participants, (e) were published between 1 January 1990 and 31 August 2017, and (f) full text was available to the authors.BL screened all titles and abstracts based on the eligibility criteria, and a random selection of 25% of studies was independently screened by MS to ensure accuracy in the study selection. There was an agreement on 90% of studies, and consultation was held between the two authors to reconcile the disagreements. MS and BL independently assessed the eligibility of 67 full-text articles, with high inter-rater agreement between the two authors for this assessment . Again, consultation was held between the two authors to reconcile differences of opinion.Data relating to the program and the evaluation of the program were extracted from the included studies. Program types were culture-based, culturally adapted or unadapted . Common program components were determined by identifying the components of these programs that showed broader beneficial impacts.The methodological quality of quantitative studies was assessed using the Quality Assessment Tool for Quantitative Studies from the Effective Public Health Practice Project (EPHPP) , followed by tobacco (n = 12) and cannabis (n = 10). Other substances included stimulants (n = 1), inhalants (n = 2) and analgesics (n = 1). Appendix Table Of the 1483 articles screened, 26 publications reporting results of evaluations of substance use prevention programs for Indigenous youth were eligible. Reasons for exclusion are detailed in Fig. Outcomes of the evaluations are listed by substance-related outcome type in Table Keepin’ it REAL program: two were a cultural adaptation of this program called Living in 2 worlds  studies evaluated a culture-based program, ten (38%) studies evaluated a culturally adapted program and one (4%) study evaluated an unadapted program. Cultural adaptation of non-Indigenous programs included translation of concepts into local Indigenous language and concepts, developing cultural images and the adaptation of activities to include cultural activities such as prayer, dancing and circle conversations. Three studies evaluated the CONNECT), a community-based program (CMCA) and a program where CONNECT and CMCA were combined. These studies found the multi-setting programs had a smaller effect on substance use than the school- or community-based program on their own  evaluated programs were school-based, five (19%) were community-based, five (19%) combined school and family, one (4%) combined community and school, two (8%) combined family and community-based programs, and four (15%) were delivered in community, family and school settings. Family involvement in programs was primarily through one-off workshops or pamphlets given to parents and was not specifically evaluated in any study. Two studies evaluated programs that were delivered in a single setting compared to multiple settings: Schinke et al.  comparedIn terms of the context in which prevention programs were implemented, 12 (46%) were implemented in reservation/discrete Indigenous communities where all community members identify as Indigenous, ten (38%) were implemented in an urban setting, two in Indian territory and two in rural communities. Programs implemented in non-Indigenous specific areas still had 100% Indigenous participants in the evaluation study, except for five studies which had between 16 and 90% Indigenous participants and were all school-based  and Indigenous program facilitators (5 programs). Four programs used health education strategies, which included providing healthy alternatives to substance use, incorporating a holistic concept of wellbeing and media campaigns about healthy lifestyles. Three programs included a booster session 3 to 6 months following program implementation and three programs included recreational activities, such as sports, festivals, painting, going out bush and filmmaking.SmokingZine and Boy and Woman Bear. SmokingZine was an adaptation of a non-Indigenous Web site and included educational modules with culturally relevant content and imagery. It was found to reduce intention to use tobacco, reduce positive beliefs about tobacco smoking, and increase likelihood to help others quit smoking  were provided in two of the four (50%) studies  measured four outcomes, whereas beneficial studies  measured three outcomes. While this is not ideal, it was a useful method of narratively summarising outcomes of the included studies. However, as a consequence of this approach, conclusions about effectiveness of drug prevention for Indigenous youth could not be made. Finally, this review included some studies with a small sample size and these findings should be interpreted with caution as they may lack statistical power to show a statistically significant or clinically meaningful result.Overall, the results of this review indicate that substance use prevention programs have the potential to produce beneficial substance-related outcomes for Indigenous adolescents, especially when they are developed with Indigenous people and include components of skill development, cultural knowledge enhancement and substance-related education. However, methodological quality of the included evaluations lacked the rigour required to draw conclusive statements about the effectiveness of substance use prevention programs for Indigenous adolescents. There is an urgent need for more financial and time investment in conducting rigorous evaluations using practical and alternative research designs, such as multiple baseline designs and cluster RCTs, to create a strong evidence base of what works to prevent substance use among Indigenous youth.


Os acromiale is a developmental defect which results from the lack of an osseous union between the ossification centers of the acromion, leading to the fibrocartilaginous tissue connection. The prevalence of os acromiale is 1% to 15%, and is quite common in the African American population. Os acromiale in adults is easily diagnosed by symptoms and X‐ray, particularly on the axillary view; however, the differential diagnosis of adolescents may require MRI or SPECT–CT. Generally, nonoperative therapy for symptomatic os acromiale should be started, including physiotherapy, nonsteroidal anti‐inflammatory drugs, and injections. Surgical treatment is indicated after failed conservative treatment. In symptomatic patients with fixable acromiale, the tension band technique should be used to make the anterior aspect of the acromion elevated from the humerus head. In patients with small fragments which are unsuitable for reattachment, excision might be the best therapeutic option and lead to good outcomes. Whether using internal fixation or resection, the arthroscopic technique results in a better outcome and fewer complications, especially in older patients or athletes with overhead movement, because of the high incidence of shoulder impingement or rotator cuff tears which can be treated concurrently. Os acromiale represents an unfused accessory center of ossification of the acromion of the scapula. It is regarded as one of the reasons for rotator cuff tears and shoulder impingementThe inclusion criteria was studies related to os acromiale. Exclusion criteria were: (i) papers not in English; and (ii) commentary or letters to the editor.The purpose of this study is to review reports on topics related to os acromiale that have been published in PubMed (from 1984 to May 2018). We used the key word “os acromiale” to query the PubMed database of the US National Library of Medicine. From the resulting list, we reviewed 101 published papers. We categorized, summarized, and included 52 studies finally.The acromion is normally formed by the fusion of several ossification centerset al.n = 494) with a medieval Danish archaeological sample (n = 532). The results showed that the South African frequency (18.2%) was significantly higher than the medieval Danish frequency  among the South Africans (P = 0.013). However, sex and age biases were not found. Kumar et al.The frequency of os acromiale has ranged from 1 to 15% in radiographic and anatomical studieset al.et al.et al.P = 0.015, Fisher exact test; OR = 5.4). In addition, grades of rotator cuff tears were significantly higher in patients with acromial apophysiolysis compared with control patients (P = 0.03).Some previous studies have demonstrated a high incidence of full thickness rotator cuff tears in os acromiale patients (approximately 50%)In general, the most common os acromiale is the large, relatively triangular mesoacromion, which forms an interface with the acromion in proximity to the acromioclavicular joint (ACJ), while a less common os acromiale, the preacromion, is noted at the distal tip of the acromionOs acromiale can be easily diagnosed with plain X‐rays with at least two views . As mentioned above, the frequency of os acromiale has ranged from 1 to 18.2% in radiographic and anatomical studies. If these figures are correct, clearly, most of patients with os acromial are asymptomatic.Patients with symptomatic os acromiale, especially in young people and athletes with overhead activity, may have pain at the superior aspect of the shoulderet al.Although the os acromiale usually can be seen on an axillary lateral radiograph Fig. 28, it caet al.The radiologic diagnosis of os acromiale remains challenging in young patients. In an adolescent patient presenting with a painful shoulder, it may be difficult to distinguish a normally developing acromion with a secondary ossification center from the early formation of an os acromiale on the basis of age alone. Therefore, MRI and SPECT–CT are receiving more and more attentionNonsurgical management: Most cases of os acromiale are asymptomatic and, thus, require no specific treatmentOnce all conservative means have failed, surgical treatment should be considered. Numerous surgical procedures have been introduced, including open or arthroscopic excision of the os fragment, ORIF with or without bone grafting, arthroscopic subacromial decompression with acromioplasty, and arthroscopically assisted reduction–internal fixation. Depending on the individual situation, various techniques have different indications. Basically, surgical techniques should focus on the os acromiale itself when the non‐healing site is unstable and painful only; however, the concomitant pathology, including rotator cuff tears or shoulder impingement, need to be solved together in some cases.As to the unstable and painful os acromiale, internal fixation can fix the fragment and relieve the pain from the pulling of deltoid; therefore, its primacy and significance are widely recognizedet al.In addition to these open surgeries, the arthroscopic technique is intended to preserve the blood supply to the os acromiale, to minimize deltoid muscle injury by avoiding its detachment, to improve the cosmetic results, and to preclude the need for hardware removal. Atoun et al.et al.et al.It is generally recommended that small acromial fragments should be excised, and that large ones should be fusedet al.et al.Arthroscopic excision has the possible benefit of less periosteal and deltoid attachment injury, potentially lending to better results than open excisionet al.et al.et al.et al.Although the surgical management of simple os acromiale and its results have been accepted, the ideal treatment for os acromiale‐related diseases is remains controversial. In some studies, os acromiale with shoulder impingement or rototar cuff tear has been described as a depressing event. Hutchinson and VeenstraIn addition to the deltoid, the articulation of os acromiale with the lateral end of the clavicle is another area of interest. Based on the review, there is only one published paper in the English literature: it investigated 211 volunteers (control group) and 33 subjects without or with os acromialeet al.As previously mentioned, hardware‐sourced problems were often occurred after tension band fixation. Depending on the fixation technique, the incidence of nonunion after internal fixation was from 0% to 100%Os acromiale is not a rare finding in patients with painful shoulders and the meso‐acromion type is found in clinic most frequently. There is a strong relationship between os acromiale and race, except for Korean patients, while the correlation between os acromiale and rotator cuff tears is controversial. Os acromiale in adults is easily diagnosed by X‐ray, particularly on the axillary view; however, the differential diagnosis of adolescents may require MRI or SPECT–CT. Generally, nonoperative therapy for symptomatic os acromiale should be commenced, including physiotherapy, and administration of nonsteroidal anti‐inflammatory drugs and injections. Surgical treatment is indicated after failed conservative treatment. In symptomatic patients with fixable os acromiale, the tension band technique should be used so that the anterior aspect of the acromion is elevated from the humerus head. In patients with small fragments which are unsuitable for reattachment, excision might be the best therapeutic option and can lead to good outcomes. Whether internal fixation or resection, the arthroscopic technique results in a better outcome and fewer complications, especially in older patients or athletes with overhead movement, because of the high incidence of shoulder impingement or rototar cuff tears which can be treated concurrently. In reviewing the literature, it is found that extended arthroscopic subacromial decompression, successful rotator cuff repair, and RSA could also achieve a good result in subacromial impingement, rotator cuff tear, or end‐stage cuff tear arthropathy, respectively, even without osteosynthesis of the os acromiale. In the future, a prospective study is necessary to answer this question of whether the os acromiale is related to those patients with high incidence of subacromial impingement or rotator cuff tears.


The school system is one setting in which children’s physical activity levels may be increased through daily physical activity (DPA) policies and initiatives. Adherence to DPA policies is typically poor and results are limited in regard to the associated benefits for participating children. Therefore, the purpose of this study was to evaluate a range of psychosocial outcomes following a community-led, in-school DPA initiative for 9–14 year old children and youth.This program evaluation examined the impact of a DPA program consisting of 20 min of teacher-led DPA for 20 consecutive weeks. Student outcomes were measured using a questionnaire administered at three time points: baseline, mid-intervention, and post-intervention. A teacher questionnaire regarding program adherence and student behaviour was completed at post-intervention. Mixed effects models were used to test for intervention effects, with random intercepts for students, classes, and schools, as well as fixed effects for age and sex. Due to the large number of outcomes measured, we first conducted an omnibus test of the intervention effect followed by three exploratory analyses examining each outcome separately, associations between outcomes and program adherence, and results from the teacher survey.N = 19 experimental) from 7 schools participated in the study, with a total of 362 children . There was no significant overall effect of the intervention  and the exploratory analyses demonstrated significant differences only for self-esteem and subjective happiness, with the control group slightly increasing relative to the experimental group. Teacher-reported adherence to the program was poor with only 21% of teachers adhering to the program. There was no association between overall adherence and student reported outcomes; however, positive correlations were present between adherence and teacher-reported student behaviour.Thirty classes (The DPA program evaluated here did not improve the psychosocial well-being of elementary school-aged children more than usual practices. However, adherence to the program was poor and it did not have a negative effect on the students. Future work is needed on how best to support DPA implementation in the context of the school day and how student well-being may be positively impacted through school-based physical activity.NCT03618927, August 6, 2018. Retrospectively registered.Program Evaluation of an in-school Daily Physical Activity Initiative  There is considerable evidence regarding the impact of daily physical activity (DPA) on youth development and health , 2. ChilOne way in which we may positively influence the physical activity patterns of children and youth is through the school system, as a large portion of each child’s day is spent in that setting. The evidence, however, is mixed in regard to the benefits of school-based physical activity interventions , 18. ForMost of this evidence stems from programs designed and implemented by research teams rather than program evaluation (and collaboration) of new or existing practices within the school . ResearcIn Ontario, Canada, there has been an attempt to increase school-day physical activity through a 2005 DPA Policy released by the Ministry of Education . DespiteThe recent evaluation of Ontario school teachers and administrators also identified a number of factors related to adherence to the DPA policy . They foGiven low rates of compliance, some school boards in Ontario have partnered with external organizations and/or have developed in-school programming to improve compliance with the policy. Our involvement in this study was on behalf of an organization who has been working with local schools proximate to our university to increase DPA using a structured, teacher-led program of in-class exercises. Given the literature on the impacts of physical activity to psychosocial outcomes in children and youth –14, bothThe intervention consisted of a DPA program designed by a national organization with expertise in school-based physical activity programming and delivered in school by teachers. The program was offered to students in grades 4 through 8 and consisted of 20 min of structured DPA in school for 20 consecutive weeks. The DPA activities included jumping jacks, squats, running and other body weight exercises. School teachers and student leaders attended a one-day workshop on how to deliver the program as part of regular school activities and were provided with instructional materials to take back to their school for program delivery. This workshop was intended to increase teachers’ confidence to implement DPA through the use of the manual and supporting resources . At the conclusion of the 20 weeks, participating students were invited to participate in a 5 km fun run/walk with students and teachers from participating schools, and community leaders from within the geographical region in which the program took place. Schools and teachers were encouraged to approach the program as a “team,” emphasizing pro-social aspects of team work and collective impact on behavioural  and social change. This was reinforced at the annual fun run/walk where schools run together as a group, along with children from across participating schools. Moreover, the organization promotes the idea of physical activity and participation to both teachers and students as means to not only improve physical health, but also mental and social well-being. Given the developmental benefits associated with engagement in regular physical activity, and the positive developmental environment that the program strives to achieve, it is possible this program had the potential to positively impact children’s social and psychological well-being through increased DPA.Mage = 11.7y, SD = 1.3y, n = 176 females). All students (n = 783) in the participating classes were approached to participate in the study. We received written parental consent and child assent from 362 students (46.2%). Teachers had previously chosen whether or not their class would participate in the program or to act as a control class, and this determined the final groups. This program evaluation study therefore included 19 classes and 265 students participating in the DPA program, and 11 classes and 97 students participating in their typical classroom routines. Only teachers who had chosen to participate in the program received the DPA program training. The remaining teachers were however still expected to provide DPA to their students, as per the Ontario education curriculum [Seven schools from a local Catholic school board were chosen by the school board to participate in this program evaluation. Teachers were asked to participate by their school administrators. Between one and seven classes participated in each school, for a total of 30 participating classes. All classes were between grade 4 and grade 8 , and post-intervention (within 2 weeks post-intervention). Teachers of classes receiving the intervention completed a brief questionnaire post-intervention to assess fidelity to the intervention, while teachers of control classes completed a questionnaire about usual classroom routines.All questionnaires were administered in class using Android NeuTab Pro™ tablets provided by the research team. Participants who preferred to complete the questionnaire on paper were able to do so. The student questionnaire took approximately 30 min to complete, was composed of subscales drawn from existing questionnaires, and was identical at each time point, with the exception of demographic questions, which were asked only at baseline. The teacher questionnaire took approximately 10–15 min to complete. Research assistants were on hand during administration to assist with completion and answer any questions students may have had. Students and teachers were provided with a small incentive  for their time in the study.A 92-item questionnaire was developed for this study. It assessed demographics, aspects of psychosocial well-being, well-being associated with school, and engagement in physical activity in school and outside of school. Scales and subscales were drawn from existing validated measures.Participants were asked for their birth date, gender, ethnicity/cultural heritage, and postal code, as well as about the composition of their household.the extent to which people see themselves as being in control of the forces that importantly affect their lives,” [Strongly Disagree) to 4 (Strongly Agree). An example item is “I can do just about anything I really set my mind to.” One of the seven items (“What happens to me in the future mostly depends on me”) was excluded for this study, and the first item was modified from the negative to the positive to ease comprehension for younger children. The average internal consistency of the measure across the three time points was Cronbach’s α = 0.70.Mastery which refers to “ lives,”  was asse lives,” . Pearlin“When I’m sad, I can usually start doing something that will make me feel better.” An example item from the long-term self-regulation scale is: “If I really want something, I have to have it right away.” Items are scored on a five-point Likert-type scale ranging from 1  to 5 . The scale has demonstrated concurrent validity, being positively associated with pro-social behaviour and negatively associated with externalizing and internalizing problems [α = 0.61.Self-regulation was assessed using a subset of 7 items drawn from the Adolescent Self-Regulatory Inventory , a 36-itproblems . The aveNot True) to 3 (Certainly True). An example item is “I get a lot of headaches, stomach-aches, or sickness.” The self-report SDQ has demonstrated good psychometric properties with diverse groups of children and adolescents [α = 0.80.Emotional well-being, behavioural difficulties, and pro-social behaviour were measured using the self-report version of the Strengths and Difficulties Questionnaire (SDQ) , 28. Thelescents –31. The Strongly Disagree) to 3 (Strongly Agree). An example item is “On the whole, I am satisfied with myself.” The measure’s average internal consistency was α = 0.86 across the three time points.Self-esteem was measured using the Rosenberg Self-Esteem Scale , a 10-it“perseverance and passion for long-term goals,” [Not at all like me) to 5 (Very much like me). An example item is “New ideas and projects sometimes distract me from previous ones.” The average internal consistency of the measure was α = 0.71.Grit, defined as  goals,”  was asse goals,” , 35. The“In general I consider myself …” with 1 anchored as Not a very happy person and 7 anchored as A very happy person. The scale has demonstrated good psychometric properties in children and adolescents [α = 0.74 across the three time points.Global happiness was measured using the 4-item Subjective Happiness Scale . The scalescents –38. The Strongly Disagree) to 4 (Strongly Agree). A subset of five-items was selected, scored on a five-point Likert-type ranging from 1  to 5 . An example item is “You try really hard at school.” The average internal consistency of the measure was α = 0.80.Commitment to school was assessed with 5 items drawn from the 10-item Commitment to School scale . The scaI feel comfortable at my school.” The measure’s average internal consistency was α = 0.86.Sense of belonging at school was measured using the 5-item short form version of the Sense of Belonging Scale . An examIn the last 7 days, what did you do most of the time at recess?” The average internal consistency of the measure was α = 0.82.Physical activity was assessed with the Physical Activity Questionnaire for Older Children (PAQ-C) . The PAQI am confident in my ability to engage in …” The individual items were “Physical activity ,” “Sports ,” and “Activity play .” Following guidelines provided by Bandura [= not confident, 10 = totally confident). A generalized task self-efficacy overall score was computed by averaging the ratings for each item to produce a scale value out of 10. The measure’s average internal consistency was α = 0.86.Self-efficacy for engaging in physical activity, sports, and active play was assessed using a three-item scale adhering to recommendations by Bandura , 44 for  Bandura , 44, par“On average, how many days per week did your students participate in the program?” Questions regarding student behaviour were completed by all participating teachers and included behavioural ratings of student behaviour  were completed by all participating teachers.A 21-item questionnaire was developed for this study. Completed by the homeroom teacher at the last measurement point, it included 3 sections: adherence to the program, student behaviour, and physical activity opportunities. Questions regarding adherence to the program were completed by the experimental classroom teachers only and included questions such as, We first produced descriptive statistics and tested for overall group differences using t-tests (for continuous variables), chi-square tests , and a Wilcoxon test (for number of siblings).To test for intervention effects, we used mixed effects models. Participants were assessed on three occasions, and the intervention was administered at the class level. We therefore included random intercepts for students, classes, and schools, as well as fixed effects for age and sex. To test for intervention effects, we included a group*time interaction in each model.Given the large number of outcomes, we conducted an omnibus test of the intervention effect using the approach suggested by O’Brien , which iX2 = 0.57, df = 1, p = 0.45). The characteristics of the sample at baseline are shown in Table The final sample included 362 children with consent. Of these, 265 received the intervention, while 97 served as controls. Attrition by the study end was 16% in the intervention group and 20% among the controls (p = 0.38). Results of the exploratory analysis of individual outcomes  below 0p = 0.52) or with any of the individual outcome variables. There were no significant differences in the quantity of physical education classes in the experimental and control classes over the intervention period (t(27) = − 0.23, p = 0.82). Further, all control classes also reported providing their students with opportunities to be physically active throughout the school day, beyond physical education and recess, in order to align with curricular mandates. The teachers in the control classes reported that these opportunities mainly consisted of 10–20 min of DPA  completed on days when physical education classes did not take place.Within the experimental group, teachers reported engaging in 30 to 100 min of exercise per week, with only 4 experimental classes  adhering to the mandated 100 min. The amount of exercise performed in the experimental classes (according to teacher report) during the program averaged 70.7 (SD = 23.0) minutes per week and was not associated either with our omnibus outcome , emotional regulation , and social interactions .In the experimental group, teachers reported a relatively high level of self-efficacy (mean = 7.3 ± 1.9 out of 10) in their ability to lead the program, and positive correlations between adherence to the program and teacher reported perceptions of change in student behaviours were present. More specifically, significant correlations were evident between program adherence and teacher-reported student attentional control  teachers reporting daily adherence and most only engaging in the program 3–4 days per week. The low rate of adherence is alarming as 20 min of DPA is the standard that should be implemented by all classroom teachers [Although 20 min of DPA is the expected standard in Ontario , in addiPoor adherence to these interventions may be due to the competing interests and pressures facing elementary school teachers. Allison et al.  reportedOne means to achieve these goals is through professional development activities for teachers. According to the literature on best practices in this area, in addition to the aforementioned considerations, other strategies should include active learning strategies for teachers , by making DPA coherent with student learning  and by building a community of practice within school boards on physical activity  . Indeed,Another possible reason for our lack of significant findings may be due to the structure of the program, which consisted of fitness-based exercises . It is possible that the program was therefore less enjoyable than more game-like, or otherwise more engaging, activities. If students did not enjoy the sessions, they could have been less motivated to participate, and may not have been participating at an optimal aerobic intensity to elicit positive affective experiences associated with various changes in psychosocial outcomes . A shiftOne type of physical activity that is gaining popularity for school DPA is the use of cognitively-engaging physical activity, or physically active academic lessons, which have been found to both increase activity levels and improve academic outcomes . SpecifiThere are a number of limitations to the current study. First, slightly less than half of eligible students consented to participate in the program evaluation. It is possible that those students who did not consent had lower levels of psychosocial well-being at baseline and thus, would have benefited the most from the program. Second, the unbalanced nature of our groups will have reduced statistical power; however, we did not have control over the poor response rate from the control group. Third, we did not measure baseline DPA practices prior to the onset of the intervention. Fourth, that we relied on teacher self-report for adherence to the program and changes in student behaviour post-intervention is also a limitation. However, that we found such low adherence is alarming given that recall or other biases  may have resulted in teachers reporting higher than actual adherence. We also employed a rather lengthy survey over three-time points which may have resulted in response bias, social desirability, dissimulation or participant fatigue, any or all of which could have unintentionally contributed to our findings (see ). FinallAt the outset of the paper, we noted the limitations of investigator-led interventions in school based settings. Seeking to evaluate a program designed and implemented outside the research context offered the chance to overcome some of the implementation challenges  noted in investigator-driven research. This approach is consistent with so-called effectiveness trials, where an intervention is evaluated in a real-world setting, typically without the controls normally in place for experimental research . In thi


PRAME or PReferentially expressed Antigen in Melanoma is a testis-selective cancer testis antigen (CTA) with restricted expression in somatic tissues and re-expression in various cancers. It is one of the most widely studied CTAs and has been associated with the outcome and risk of metastasis. Although little is known about its pathophysiological function, PRAME has gained interest as a candidate target for immunotherapy. This review provides an update on our knowledge on PRAME expression and function in healthy and malignant cells and the current immunotherapeutic strategies targeting PRAME with their specific challenges and opportunities. We also highlight some of the features that position PRAME as a unique cancer testis antigen to target. Conventional cancer care, including surgery, radiotherapy, and chemotherapy, considerably prolongs the survival of patients; however, a significant proportion of cancer patients do not respond to standard systemic therapy or relapse after treatment. The need for alternative, well-tolerated first-line treatments has been partially met by the development of targeted therapies involving a number of novel therapeutic drugs that target specific molecules. For example, antibody-based targeting of the Epidermal Growth Factor Receptor (EGFR), Human Epidermal Growth Factor Receptor 2 (Her2), and Cluster of Differentiation 20 (CD20) are currently used as standard targeted treatments in lymphoma, breast, colorectal, and head and neck cancers ,2,3. DesPRAME, also known as CT130, MAPE  or OIP4 (OPA-Interacting Protein 4), is encoded over a region of approximately 12 kilobases on chromosome 22 at locus 22q11.22. It was first identified as a tumor antigen that could be recognized by human leukocyte antigen (HLA)-A*24 restricted cytotoxic T lymphocytes in metastatic cutaneous melanoma . PRAME iPRAME could restore sensitivity to retinoic acid treatment in RA-resistant melanoma cell lines, thereby reducing cancer cell proliferation by de-repressing the expression of RAR target genes [The cellular and molecular functions of PRAME in normal and neoplastic cells remain largely unclear and may differ depending on tissue-specificity and/or subcellular localization of PRAME. PRAME was identified as a dominant repressor of retinoic acid (RA) signaling, thereby inhibiting differentiation, proliferation arrest, and apoptosis . Its oveet genes . Howeveret genes . In thisIn recent years, however, it has become apparent that PRAME could promote tumor development and progression via different mechanisms. Its nuclear localization has been implicated in transcriptional regulation, thereby possibly driving the expression of genes that are involved in tumor promotion. More specifically, analysis of the PRAME interactome revealed that PRAME facilitates the recruitment of Cullin2 ubiquitin ligases to the EKC/KEOPS complex in the nucleus, which is involved in telomere maintenance, transcriptional regulation, and threonylcarbamoyladenosine (t6A) modification of transfer RNAs (tRNA) for accurate decoding of A-starting mRNA codons ,55,56.Furthermore, the increase in PRAME expression with more advanced disease and a higher risk of metastasis suggests that PRAME plays a role in the acquisition of various cancer hallmarks, including replicative immortality or stemness, and invasion and metastasis ,41,57,58PRAME, including nuclear factor kappa B (NFĸB), interferon regulatory transcription factors (IRFs), and signal transducers and activators of transcription (STATs) [In addition to supporting tumor cell features, PRAME has been implicated in the regulation of the immune response. Homology sequence analyses revealed that PRAME contains 21.8% (iso)leucine residues, and hence belongs to the leucine-rich repeat (LRR) family of proteins . Further (STATs) . PRAME h (STATs) . This raThe restricted re-expression of PRAME across malignancies renders it a candidate target for cancer treatment and has sparked numerous studies on the development and evaluation of immune-based interventions targeting PRAME. A variety of immunotherapy strategies are currently developed with the common goal of eliciting durable and potent anti-tumor immune responses. To date, efforts in PRAME immune-based targeting have focused on cancer vaccines and adoptive T cell therapies, as described below. Ongoing clinical trials of cellular immunotherapies targeting PRAME are summarized in Clinical trials on PRAME cancer vaccines have reported no major toxicities in patients with various solid cancers; however, thus far, the therapeutic benefits have been rather limited, hence discouraging the further development of PRAME vaccines ,64. A fiCellular immunotherapy with PRAME-specific T cells has largely been explored using two different approaches involving either the isolation of pre-existing circulating PRAME-specific T cells for ex vivo expansion and selection, or the genetic engineering of high-affinity PRAME-specific TCR T cells. As aforementioned, PRAME was originally identified through the detection of specific tumor-reactive HLA-A*24–restricted T cell clones in a metastatic melanoma patient . AnalysiFollowing the identification of these candidate epitopes, patients’ blood samples were screened to evaluate each epitope’s ability to induce specific T cells, thereby providing a rationale for cellular immunotherapy using ex vivo expansion of circulating autologous antigen-specific T cells. In vitro priming of isolated peripheral blood mononuclear cells with peptide-pulsed autologous antigen presenting cells demonstrated the presence of PRA100-108 and PRA300-309-specific T cells in 36% of melanoma and 70% of AML cancer patients, respectively ,69. FurtIn comparison to the research efforts on cellular therapy using pre-existing autologous PRAME-specific T cells, there is very limited published information on PRAME genetically modified T cell therapy. Notwithstanding, adoptive therapy using TCR transduced T cells has shown encouraging results in generating durable responses in selected cancer types, resulting in a surge of clinical trials targeting various tumor-associated antigens, including the cancer testis antigens NY-ESO-1. For instance, NY-ESO-1 specific TCR-T cells are currently being evaluated in numerous clinical trials following encouraging results with improved clinical response rates and prolonged survival of patients with metastatic melanoma, synovial sarcoma, and multiple myeloma . A recenIt is important to note that the efficacy of adoptive T cell therapy is not only determined by the avidity of the T cell clones but also by the expression level of the target throughout the tumor. As mentioned earlier, tumors often show heterogeneous tumor-associated antigen expression patterns so that distinct tumor cell clones with absent or relatively low target expression might escape T cell killing. In order to tackle this, it would be beneficial to induce re-expression of the target in all tumor cells using demethylating agents (DMA), histone deacetylase inhibitors (HDACi), or a combination of both. These epigenetic modulators have been successfully used to upregulate tumor PRAME expression, resulting in an increase in cytolytic activity of ex vivo expanded autologous T cells ,73. The PRAME has emerged as a potential candidate target for immunotherapy, and we are currently awaiting the results from numerous clinical trials evaluating PRAME-specific cancer vaccines and adoptive T cell therapies. Although preclinical studies have been promising, we should be vigilant and anticipate, as for any novel treatment approach, that PRAME-targeted immunotherapy might face challenges in the clinic. As depicted in PRAME in the latter [As a first challenge, the heterogeneity of PRAME expression could likely present a bottleneck for the efficient targeting and complete eradication of tumor cells. Firstly, PRAME expression shows a wide range of inter-tumor heterogeneity among patients carrying tumors of the same tissue origin, thereby limiting the overall response rate. This is of particular importance for breast cancer where only 27% of tumors display PRAME expression . Secondle latter . Moreovee latter . In the e latter . TogetheAlthough increasing target expression represents one approach to improve the efficacy of targeted immunotherapy, positive tumor cells can still evade the anti-tumor immune response through several mechanisms. For example, dedifferentiated liposarcoma and leiomyosarcoma tumors with positive PRAME expression have been found to express reduced levels of antigen presentation molecules, such as HLA-B, HLA-C, beta-2 microglobulin (B2M), latent membrane proteins 2 and 7 , and transporter associated with antigen processing 2 (TAP2) . This suInterestingly, antibody-based targeting of PRAME has emerged as a novel approach, which could offer a new avenue to induce potent anti-tumor responses. PRAME was, until recently, not considered a candidate target for antibody therapy, as monoclonal antibodies exclusively bind to the cell surface and extracellular antigens, while cancer testis antigens are intracellular antigens. However, a research group from the Memorial Sloan Kettering Cancer Center developed a TCR mimic human antibody that can recognize the PRA300–309 peptide in a complex with HLA-A2 . The afuAlthough PRAME is widely believed to be an intracellular antigen, one study detected a membrane-bound form of PRAME in chronic lymphocytic leukemia and mantle cell lymphoma . This reGiven the widespread re-expression of PRAME among malignancies and the novel developments in the immunotherapy field, we believe that there is great potential to target PRAME using a variety of immune-based interventions, either as single modality or as a combinatorial treatment. We discussed some of the major challenges for PRAME-based immunotherapy that need to be addressed, including the intra- and inter-tumor heterogeneous expression of PRAME and the potential role of PRAME in the upregulation of immune evasion mechanisms. Furthermore, we described the current countermeasures and innovative technologies that are being exploited to improve the potency of the PRAME-specific anti-tumor response.Among all cancer testis antigens, PRAME is in a unique position as it is not only widely expressed in various cancers but also elicits specific cellular immune responses against numerous neoplastic cells as well as against leukemia progenitor cells, plays a role in the innate immune response and possibly also the adaptive anti-tumor immune response, could enhance the ATRA treatment response as a combinational therapy, and could be targeted using a wider range of approaches including antibody- and CAR-based therapies.


Background: The RE-AIM framework has been widely used in health research but it is unclear the extent to which this framework is also used for planning and evaluating health-related programs in clinical and community settings. Our objective was to evaluate how RE-AIM is used in the “real-world” and identify opportunities for improving use outside of research contexts.Methods: We used purposive and snowball sampling to identify clinical and community health programs that used RE-AIM for planning and/or evaluation. Recruitment methods included surveys with email follow-up to funders, implementers, and RE-AIM working group members. We identified 17 programs and conducted structured in-depth interviews with key informants (n = 18). Across RE-AIM dimensions, respondents described motivations, uses, and measures; rated understandability and usefulness; discussed benefits and challenges, strategies to overcome challenges, and resources used. We used descriptive statistics for quantitative ratings, and content analysis for qualitative data.Results: Program content areas included chronic disease management and prevention, healthy aging, mental health, or multiple, often behavioral health-related topics. During planning, most programs considered reach (n = 9), adoption (n = 11), and implementation (n = 12) while effectiveness (n = 7) and maintenance (n = 6) were considered less frequently. In contrast, most programs evaluated all RE-AIM dimensions, ranging from 13 programs assessing maintenance to 15 programs assessing implementation and effectiveness. On five-point scales, all RE-AIM dimensions were rated as easy to understand , but obtaining data was rated as somewhat challenging . Implementation was the most frequently used dimension to inform program design (M = 4.7 ± 0.6) relative to the other dimensions (3.0–3.9). All dimensions were considered similarly important for decision-making (average M = 4.1 ± 1.4), with the exception of maintenance (M = 3.4 ± 1.7). Qualitative corresponded to the quantitative findings in that RE-AIM was reported to be a practical, easy to understand, and well-established implementation science framework. Challenges included understanding differences among RE-AIM dimensions and data acquisition. Valuable resources included the RE-AIM website and collaborating with an expert.Discussion: RE-AIM is an efficient framework for planning and evaluation of clinical and community-based projects. It provides structure to systematically evaluate health program impact. Programs found planning for and assessing maintenance difficult, providing opportunities for further refinement. With the proliferation of evidence-based programs for improving population health, there is a greater concern with promoting the dissemination and implementation (D&I) of health programs . It is iWe define “non-research” projects as programs and interventions intended for local quality or health improvement rather than generalizable knowledge . Research may use random assignment to control conditions and be intended to establish efficacy or effectiveness. In contrast, non-research uses –which as discussed here, include quality improvement (QI), program or product evaluation, and demonstration projects. QI and program evaluations may have greater opportunity for adaptations and iterative refinement of protocols and intervention delivery. In both research and non-research contexts, the value of RE-AIM is in adding information on issues that are often not considered, such as outcomes involving representativeness and generalizability. Yet it is not known the extent to which and how RE-AIM is used for non-research purposes.RE-AIM is a planning and evaluation model that addresses five dimensions of individual- and setting-level outcomes important to program impact and sustainability : Reach, www.re-aim-org) has been addressing  are evaluated and reported more often than other dimensions —with the primary underreporting in describing the representativeness of individuals and settings, maintenance of effects, costs, and sustained program implementation . To our dressing , 15. AddTo understand the application of RE-AIM in non-research settings across different content areas, populations and settings, we designed a study to assess experience (understandability and usefulness) and methods and resources used to apply all five RE-AIM dimensions for program planning, evaluation, decision-making and improvement in non-research settings.This is a cross-sectional, retrospective mixed-methods evaluation of use of RE-AIM for program planning and evaluation in clinical and community settings. We identified eligible programs and representatives using purposive and snowball sampling. We administered structured interviews to program representatives to assess both qualitative perceptions and quantitative ratings of the usability and usefulness of RE-AIM. Open-ended questions were used to elaborate upon and explain quantitative ratings, consistent with an explanatory concurrent mixed methods design. Study procedures were approved by the Colorado Multiple Institutional Review Board in June 2018.Our goal was to interview 15–20 representatives from projects or programs that used RE-AIM for planning and/or evaluation in clinical or community settings. Eligibility criteria for interviewees included a lead or supportive role in the local planning, implementation, and/or evaluation of a health-related program or intervention  and self-reported use of RE-AIM.It is challenging to review and evaluate non-research use of models and theories . There aWe used multiple strategies to identify and recruit interviewees. First, we developed a brief survey sent to contacts at agencies that we had reason to believe had funded, conducted, or organized health-related programs designed primarily for local QI or community health impact (purposive sampling). Agencies included health foundations, health systems, state health departments, and national health agencies in the U.S and internationally. We targeted both agencies and individuals known to us (experienced RE-AIM scholars) to have required or encouraged use of RE-AIM for planning or evaluation of supported projects, as well initiatives in which use of RE-AIM was unknown. The identified contact persons from target agencies were sent an email with a survey link. Those that did not reply after 10 days were sent a reminder e-mail.The survey asked, “Have any of your grantees ever actually used RE-AIM, in whole or in part, as a program planning or evaluation framework for one or more health-related programs or initiatives?” Respondents indicated how many programs or initiatives had used RE-AIM, and described projects that “Primarily aimed to improve the health or well-being of a specific community or population” and had used RE-AIM. With respondents' permission, we followed up via email to request introductions or contact information for representatives from relevant projects.Names of representatives from relevant projects were also obtained through personal contacts, PubMed and Google Scholar searches, emails to the RE-AIM working group, and nomination by RE-AIM researchers and interviewees . Literature search terms included “RE-AIM,” “health” and “evaluation”; we reviewed abstracts to identify potentially eligible projects, and emailed lead authors.In total, we emailed 95 people with invitations to complete the survey (for agency representatives) and/or participate in the interview . Many were multiple potential contacts for the same program. Of the 35 surveys completed, 19 reported their grantees had used RE-AIM; 14 indicated they would provide introductions to project representatives. Of the 17 interviews conducted, 8 resulted from contacts identified by those who were sent the survey, 1 resulted from nominations from other interviewees, 2 resulted from emails to the RE-AIM working group, and 2 resulted from personal contacts. Four of the interviewees were co-authors on this paper, who are experienced evaluators and had used RE-AIM for eligible projects. All but one interview was conducted by a trained research assistant; one was conducted by the lead author.Data were collected using a structured interview guide . AdditioFor each RE-AIM dimension, participants reported whether they used the dimension for initial planning or program design, evaluation, or both, and described how they measured or otherwise operationalized the dimension. Additional details about measurement/operationalization were gathered from written reports or documents provided by the interviewee. Interviewees rated usability and usefulness of the five RE-AIM dimensions using 5-point Likert-type scales: “Dimension was easy to understand” (1 very difficult – 5 very easy); “Getting the data to assess this dimension was easy” (1 very difficult – 5 very easy); “Did consideration of this dimension inform initial program design?” ; “Was consideration of this dimension important for decision making during the program?” .Open-ended questions in the interview guide addressed the following topics:Description of the health program, project, or initiative, its intended audience and outcomes, and its origin and funding;Rationale, purpose and funding for use of RE-AIM;Experience with and methods used to apply the five RE-AIM dimensions;Receipt and nature of consultation, published guidance, online resources, training, and technical assistance; andRecommendations for improvement in the model itself and guidance materials.For quantitative ratings, we calculated descriptive statistics including means, medians, standard deviations and ranges for 16 (of 17) interviews. One interview was excluded from the quantitative analysis as it covered use of RE-AIM for a series of projects, rather than a single project, and hence was not comparable for quantitative analyses.For qualitative data, the research assistant who conducted the interviews organized her notes into a case-based matrix by topic . Another team member coded the data within each topic area and identified themes. We used a qualitative descriptive approach to content analysis . All 17 www.re-aim.org (9 programs) or formal training or consultation (7 programs).As shown in We identified themes related to the usefulness of RE-AIM for real-world projects, including its impact on planning and usefulness for evaluation and program refinements over time, challenges in using RE-AIM and strategies for overcoming those challenges, resources used for learning about and applying RE-AIM, recommendations for improvement, and how use of RE-AIM is funded.RE-AIM was reported to be a useful organizing framework or “roadmap” for planning (impact and usefulness theme 1). An interviewee explained, “[RE-AIM] provides a nice roadmap with all the components of program development that you need to consider.” RE-AIM helps to ensure consideration of adoption, implementation and sustainability, reaching the right audiences, and ensuring you have been clear about who will benefit and expected outcomes, as well as potential unintended consequences . An interviewee described the value of the “reach” dimension for attending to health equity, “It was a moral imperative to check in on what we were doing…. We wanted to use this to inform, how can we reach those vulnerable populations, how can we reach rural populations, how can we reach those disadvantaged or untouched sectors of the community.”RE-AIM was reported to help with focusing on context and setting, and implications for what works in “real life” (impact and usefulness theme 2). Specifically, understanding who is or isn't adopting or delivering the program well, what's required for sustainability, and making refinements to the program over time to ensure overall quality. Finally, RE-AIM was described as a practical, familiar, easy to understand and well-established D&I framework (impact and usefulness theme 3). One interviewee indicated, “if you are wanting a framework or conceptual model, this is a good one” since many people are familiar with it and it's fairly straightforward, and helpful for coordinating and communicating with program implementers. Furthermore, “I'm from the public health field and it's always nice to have a framework, theory or model on which you base your decisions on. So that was a big value-add for me was to have an established D&I framework that was known in the field and highly applicable to our project.”adoption and parents contacted the measure of reach. Second, there can be difficulty with data acquisition, as in a source of data exists but access was slow or limited . Specifically, it was especially hard to get electronic health records (EHR) data that allowed linking patients over time to track clinical outcomes longitudinally. One person summarized the problem as, “Electronic health records are not set up to extract the data for these [RE-AIM] measures.” As in research and evaluation in general, interviewees struggled to schedule interviews with busy staff and providers, and often found participants did not want to spend a lot of time answering questions. A parallel data challenge theme concerns when a source of data doesn't exist ; this includes getting denominator data to estimate reach and adoption, knowing who does not get into a program, getting demographic data for characterizing sites and participants, and measuring maintenance. For example, “Often there is no denominator known, so you want to describe our reach but don't know your full population so you cannot say if you did well on reach.” Finally, interviewees reported practical and logistical issues with RE-AIM evaluations—likely not specific to RE-AIM itself, but to the nature of evaluation . These issues included few resources, changing organizational priorities, staff turnover, and frustration that it can take a long time to see impact at the patient level.Interviewees reported challenges with understanding the differences among the RE-AIM components . Notably, respondents reported that there are “fuzzy boundaries” between adoption and implementation, adoption and reach, and reach and effectiveness . A respondent noted, “I just think that distinguishing between when does somebody move from being an adopter to an implementer, I find that a fuzzy boundary.” There can be lack of clarity on the unit of analysis , as well as defining terms and figuring out relevance to a specific program. For example, in a program focused on recruiting staff members to deliver a program to parents, the staff participation would be the measure of To overcome challenges, interviewees reported that being flexible and adapting the approach over time to best fit your needs and purpose was helpful  but also resulted in modifications to RE-AIM definitions. One participant described this as “the way we're interpreting it (adoption) is probably not pure RE-AIM so we might be taking some liberties because … you're usually looking at adoption as sort of the analog to reach, what sort of percentage of all the eligible health systems that you approached actually signed on to do it. Respondents noted the liberal use of RE-AIM while focusing on one's purpose: “The way we used some of the reach data as a metric for taking a look at fidelity and seeing if, for example, one clinic is really falling short.” Some reported using a trial and error approach to methods and measures, being open to changing methods over time, and going back and forth with sites to figure out what will work from a measurement perspective. For instance, “In terms of maintenance … we had to expand beyond the cancer center to include all hospital inpatients.” In addition, interviewees suggested putting in the effort for careful planning up front . Specifically, spending time at the beginning to define your terms, develop relationships with sites and participants, and figure out what's realistic at the outset for both evaluation and for the program . For instance, “Getting the data all depends on the previous steps. How you use it, how you define it, and what methods. It's worth spending a lot of time on designing that part so it's easy to collect.” Finally, interviewees advised learning more about the RE-AIM dimensions and how they have been applied in other real-world projects . This can help clarify definitions—such as one interviewee who reported that reading examples helped to clarify that adoption refers to the provider or setting level or that maintenance can refer to the organization level as well as the individual level.Interviewees reflected upon their use of the website, trainings or formal education, consultation or technical assistance, and publications to learn about and guide application of RE-AIM. Most interviewees were familiar with the website and had used or visited it in the past . They foTo help promote ease of use of RE-AIM for community and clinical programs, interviewees had several recommendations for RE-AIM developers. First, highlight real-world projects on the RE-AIM website that do a good job linking measures to the constructs and giving concrete examples. Describe tools and measures that can be adapted—especially for measuring and distinguishing reach , adoption and maintenance. For instance, it is helpful “To have some good examples of tools people use to capture some of the dimensions especially those that are a little bit harder to capture like the maintenance piece for institutions or the adoption aspect.” Emphasize that RE-AIM can be used pragmatically by only assessing constructs that are a priority for your stakeholders and/or aligning RE-AIM constructs with metrics stakeholders care about. This allows practice settings to plan strategies that can address each RE-AIM dimension without feeling obligated to collect data across all dimensions. Users of the framework can benefit from clarifying when RE-AIM (vs. another framework) may be most applicable. For example, “I think now that there are so many frameworks out there I think it's just helpful to understand…how is RE-AIM different, when is it appropriate to use RE-AIM, when is appropriate to use other frameworks…. It's still pretty driven toward the research community, but it can be used outside of the research community.” Interviewees asked for better placing RE-AIM in context by explaining how RE-AIM factors influence and relate to each other, and formally integrating contextual factors and consideration of facilitators and barriers to adoption, implementation and maintenance into RE-AIM.Finally, interviewees described funding sources for applying RE-AIM, which sometimes differed from the funding for the program itself. Sources included federal grants, national agencies, internal health system funds , foundations, state health departments, and commercial companies. The type or degree of support included small allocations from internal funds or carved out from larger grants, seed grants and case studies specifically geared toward evaluation, “leftover money” at the end of the year, or work done as part of regular employment or job duties.Although challenging to identify programs using RE-AIM in non-research settings, our multiple recruitment approaches identified 17 non-research programs in “real world” clinical and community settings that had used RE-AIM for planning and/or evaluation. These programs included funders as well as universities, and government and community organizations. Our mixed methods assessment of RE-AIM use revealed that most RE-AIM dimensions have been used and found to be useful for planning and evaluation across diverse content areas, programs and settings. Qualitative findings show that RE-AIM is a well-known and easy to explain organizing framework or “roadmap” for planning, especially with regard to encouraging consideration of context and setting, and implications for what works in “real life.” These results are complementary to the more quantitative reviews that have been conducted on the formal research literature on use of RE-AIM. For instance, Vinson et al. found it to be one of the most frequently used implementation science frameworks in research grant proposals .However, in contrast to the literature on research using RE-AIM, the program members interviewed for this report tended to use all RE-AIM dimensions for evaluation , 9, 21. Some of the most frequent challenges to applying RE-AIM identified in these real-world applications were the same as those identified in the research literature, namely difficulty distinguishing Reach from Adoption, and specifying denominators for measures of Reach , 22. Desn = 27 states, with multiple delivery channels in each state) that were all part of one large national project to enhance physical activity. Similar to that report, we found that most RE-AIM dimensions were rated as relatively easy to understand and that RE-AIM was used consistently for both planning and evaluation. This stands in contrast to the research literature, in which RE-AIM has been used much more often for evaluation, despite documented successes in applying it for planning  is an emerging D&I science framework that focuses on multiple factors related to context and setting , 24 thatThis study has limitations including lack of a tightly defined sampling due to lack of searchable databases of non-research applications of RE-AIM (or other D&I frameworks). Although our records show we attempted to contact 95 people, which led to 17 interviews (16 of which reflected distinct programs), it was not possible to calculate a true response rate. There is no clear denominator for those eligible to participate. While this study includes a relatively small sample of informants, the interviews spanned 16 distinct programs and initiatives across a variety of health domains and clinical contexts. As is typical of qualitative research, we focused more on depth of understanding experiences with RE-AIM rather than breadth . StrengtRE-AIM can be a useful organizing framework or “roadmap” for planning and evaluation of implementation of health programs in clinical and community settings, especially given it helps focus on contextual and setting factors that have implications for what works in “real life.” As a practical, familiar, and easy to understand D&I framework, it is a good choice for a planning or evaluation framework in real-world settings. RE-AIM is generally not seen as an alternative to more traditional program evaluation methods, focusing on effectiveness, but as a way to broaden and contextualize results, with a focus on population impact . ProjectTo address these challenges, those planning to use RE-AIM may benefit from reading key papers on pragmatic application of RE-AIM –10 and nThe datasets for this study can be requested from the corresponding author with appropriate approvals.The studies involving human participants were reviewed and approved by the Colorado Multiple Institutional Review Board. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.BK, RG, MO, PE, and JW contributed to the conceptualization of the project, study design and development of data collection materials, and recruited participants. BK and HM prepared IRB documents and collected data. BK, HM, and RG analyzed the data. BK and RG drafted the manuscript, and all authors edited and approved the final draft.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.