Akaike information criterion (AIC) has been recently adopted to identify possible earthquake precursors in ionospheric total electron content (TEC). According to the authors of this methodology, their technique allows finding abrupt increases (positive breaks) in vertical TEC rate of change 25–80 min before the occurrence of large earthquakes, highlighting a promising implication of AIC method in Mw > 8 earthquakes alert strategies. Due to the relevance of this matter, a lively scientific debate ensued from these results. In this study, we carefully evaluate AIC method potentiality in searching earthquake TEC precursory signatures. We first investigate the dependence of the detected breaks number on the adjustable AIC method parameters. Then, we show that breaks occurrence clusters around specific local times and around moderate and high solar and geomagnetic activity. The outcome of this study is that AIC method is not concretely usable for issuing large earthquakes alerts. An effective method for earthquake prediction requires the identification of reliable and reproducible precursory signals and low false alarm rates. Any method candidate for identifying precursors should be supported by concrete evidence of its reliability15.The search for precursory signals of earthquakes is aimed toward the monitoring that could allow predicting the occurrence of the earthquake, someday. Despite the great economic and scientific effort, and the publication of numerous papers reporting the observation of pre-earthquake anomalies in ionospheric data19. Differently, a direct correlation between the observed TEC changes and an incoming earthquake has still to be validated by the scientific community. Recent studies24 have rather shown that many observed pre-earthquake TEC changes were not associated with the incoming earthquake but were actually driven by either changes in solar and geomagnetic activities, or artifacts related to the adopted procedure of analysis.Many papers report of earthquake-induced ionospheric effects in total electron content (TEC) observed shortly after the shock. These coseismic ionospheric disturbances generated by the earthquake are well understood and described in scientific literature3 and Heki and Enomoto4 analyzed, respectively, slant TEC (sTEC) and equivalent vertical TEC (vTEC) recorded around Mw > 8 earthquakes. These studies showed the occurrence of an anomalous TEC increase that systematically starts 3 and Heki and Enomoto4 has been seriously questioned by Masci et al.23, Kamogawa and Kakinami25 and Utada and Shimizu26. According to Masci et al.23, the pre-earthquake TEC increase identified by Heki3 and Heki and Enomoto4 should be regarded as an artifact induced by their analysis rather than a precursor of the incoming earthquake (see Supplementary Discussion S1 for details).Among the authors reporting pre-earthquake changes in ionospheric data, Heki27 proposed a new numerical approach based on Akaike information criterion28(AIC) to detect abrupt increases (positive breaks in their terminology) in the vTEC rate of change shortly before earthquakes, associating them to the onset of pre-earthquake anomalies. Considering the extreme relevance and complexity of this matter, and the potential societal relevance of the results found by Heki and Enomoto27, i.e. the possibility to be able to detect signs of an incoming strong earthquake in real-time monitored vTEC data, is of uttermost importance to rigorously check the reliability of their results.Following the above-mentioned criticalities, Heki and Enomoto27, but differently from them we use a much wider vTEC database to build a more reliable and consistent statistics: one solar year (the year of the specific earthquake) of vTEC values also far from the earthquake instead of a few months around it. The four earthquakes are: 2011 Tohoku-Oki (Mw 9.1), 2012 North Sumatra mainshock (Mw 8.6), 2012 North Sumatra largest aftershock (Mw 8.2) and 2014 Iquique (Mw 8.2) .Heki and Enomotosee Fig. . In the mentclass2pt{minim27, Fig. n of points falling in this window is equal to 121. Let us suppose to compare the performance of the two models 27 call positive a break. An amplitude is associated to each positive break, defined as the difference between the angular coefficients of the lines that fit vTEC data in the first half ; it displays 27 for their Fig. 6: Before applying AIC method to the extended datasets, we validate our AIC algorithm on vTEC analyzed in Heki and Enomoto27, supplementary Fig. S2 replicates Heki and Enomoto’s (2015) Fig. S6. As a test, in Supplementary Fig. S2 we display vTEC calculated considering all the elevation angles of the satellite. Once again, our algorithm is able to reconstruct all the breaks identified in Heki and Enomoto’s27 Fig. S6. Actually, our Supplementary Fig. S2 shows additional breaks outside the time interval they considered, at the edges of vTEC curves, when the GPS satellite has low elevation angles. These breaks are very likely related to multipath effects that become important when vTEC is calculated using signals from satellites that are low on the horizon. With regard to this issue, it’s worth highlighting that after comparing our results obtained using different values of the elevation angle threshold to those obtained by Heki and Enomoto’s27, we found that the threshold to use to reproduce their breaks and to limit multipath effects resulting in flawed breaks was of 2027.Comparing our Fig. The successful validation of our algorithm allows performing a stress test on AIC method. Open questions on AIC method, to which we attempt to answer, concern: 1) the compatibility of breaks frequency with alert issuing for incoming strong earthquakes, 2) the possibility to univocally set AIC method parameters and 3) the possibility to relate detected breaks to sources other than earthquakes.For each of the four considered earthquakes, we compute a 1-year vTEC time series from two GPS receiver-satellite pairs. Each time series has undergone the analyses shown below. Due to the coherence and similarity of results, here we include only those obtained using vTEC estimated from GEONET receiver 3009 and satellite 15 for the Tohoku-Oki earthquake, and from IGS receiver IQQE and satellite 25 for Iquique event. Results for the remaining six time series are displayed in the supplementary information.27, who used values of AIC parameters for the different earthquake case studies, without clarifying the criterion for such a choice for each parameter. On the contrary, we explore a wide range of values to include all the cases they investigated and go even beyond, by testing AIC method for 72 different terns of To begin, we investigate the dependence of positive breaks frequency as a function of Figure 27 for their Fig. 6). For this value of 2 for three case studies of Mw 27 before Tohoku-Oki earthquake disappears in our analysis for Observing both sides of Fig. Right side of Fig. Supplementary Figs. S3 and S4 are equivalent to Fig. Figure 27, i.e. 0.05 hOur analysis shows that relevant positive breaks are observed with a frequency by far greater than that obtained by Heki and Enomoto27 should be the precursor of the coming earthquake, vanishes. This means that when raising the thresholds to reduce frequency, in most cases AIC method is no more able to pick the breaks proposed as precursors. Differently, in the case of Iquique the break that Heki and Enomoto27 indicate as the precursor of the event is always present, independently of the tern used. However, this cannot be considered as evidence to support the seismogenic origins of the Iquique break, an interpretation of this finding is given in the following.Once assessed the way breaks frequency depends on the values of the adjustable parameters, next would be to tune them, if possible, to reduce AIC method’s sensitivity decreasing the breaks frequency to a reasonable value, without losing those breaks that are claimed to be precursory to strong earthquakes. We therefore record, for the eight analyzed vTEC time series, the terns  for which the break observed prior to the earthquake is visible. We find that in most cases for To investigate the relation of positive breaks to non-lithospheric physical processes, we study the possible dependence of breaks occurrence on local time, solar and geomagnetic activity. To unveil this point, we follow an approach based on three steps, starting from the investigation of the dependence of the occurrence of positive breaks on local time (LT). For each break, we calculate its local time and estimate the related probability density function (PDF). Results are shown in Fig. 32. To test this hypothesis we estimate the LT of the solar terminators (sunrise and sunset) at the time of each break. The first step (Step #1) in the identification of breaks due to Sun-Earth relation processes, consists of removing those breaks occurring at a LT falling within 1 h from that of sunrise and sunset; Fig. Figure 33. Figure 34, in 2011 the solar cycle was just beginning, it reached its maximum in mid 2014. This further supports the hypothesis that part of the positive breaks that AIC method detects are due to both the regular (daily) and irregular variations vTEC undergoes because of the different exposure of different parts of the ionosphere to the Sun and by the different levels of solar activity.In Fig. In the second step (Step #2), we consider the daily F10.7 index to investigate whether some of the breaks are due to solar activity. In Step #2, we eliminate from the set of breaks shown in Fig. To complete the picture of a possible dependence on the solar and geomagnetic activity we consider also the global geomagnetic index ap, the linearized version of the Kp index with a time resolution of 3 h. In the third step (Step #3), we eliminate from the remaining breaks those occurring when ap > 22, corresponding to moderate-high geomagnetic activity. The updated PDFs are displayed in green in Fig. 27 as precursors of Tohoku-Oki and Iquique earthquakes? Both disappear, occurring respectively when F10.7 = 120.5 sfu and ap = 56 and when F10.7 = 153.1 sfu and ap =18. The analyses illustrated above are repeated also on the remaining six vTEC time series and shown in Figs. S7, S8 and S9 of the supplementary information.What happened to the breaks claimed by Heki and EnomotoTable 23; Kamogawa and Kakinami25; Utada and Shimizu26) have given rise to a lively scientific debate35 on the possible observation of precursory signatures shortly before the shock. The purpose of our investigation is to check the validity and robustness of AIC method in the detection of rapid variations in the vTEC rate of change hypothesized to be precursors of Mw > 8 earthquakes27.Meticulous tests on published pre-earthquake ionospheric anomalies  are not frequent enough. It is worth highlighting that to reproduce the work by Heki and Enomoto27 we analyzed data related only to a single satellite per receiver. We are confident that considering all the satellites in view we would have found a number of breaks by far larger, as recently found by Ikuta et al.36. Ikuta et al.36 reproduced Heki and Enomoto’s27 work and, differently from us, they considered two-month vTEC time series’ reconstructed for all the GPS satellites in view from a given receiver and for only one combination of AIC parameters , the question is “Can a warning system based on vTEC breaks be diagnostic of impending large earthquakes and then effective for public safety?”. The answer resulting from this study is that the AIC method detects too many breaks especially during high solar activity years, showing the breaks’ dependence on solar-related sources . This indicates that most, if not all of them, are not related to earthquakes. So, identifying possible ionospheric pre-seismic signals is extremely challenging, because too many false alarms would make the method unusable.Considering that according to He and HekiSupplementary information.


The awakened (14%) had a decreasing trend during the first years, but then increased the likelihood of program participation. The tail‐offs (20%) decreased in the likelihood of participating after the third year, and the drop‐offs (23%) already after the first year. The findings suggest that many schools need support during the initial years to launch and maintain the implementation of evidence‐based programs; yet a large proportion of schools manage to sustain the program implementation for several years. The logistic regression analyses showed that large schools persisted more likely than small schools. Lower initial level of victimization was also related to the sustainability of the program. Finally, persistent program participation was predicted by several school‐level actions during the initial years of implementing the program. These results imply that the sustainability of evidence‐based programs could be enhanced by supporting and guiding schools when setting up the program during the initial implementation.We examined the sustainability of the KiVa antibullying program in Finland from its nationwide roll‐out in 2009 to 2016. Using latent class analyses, we identified four different patterns of implementation. The  During past years, bullying at schools and its prevention has attracted major attention worldwide  and two non‐sustained trajectories of implementation (late abandoners and rapid abandoners). Based on the trajectories, they concluded that the first and the third year of implementation are the most likely time points for abandoning the program, whereas it may take three to five years for some schools to achieve an adequate level of implementation. Their study demonstrated that several years of data collection are required to capture the full range of different implementation trajectories.Studies of sustainability also differ in the time horizons taken , and available . Scaling up the program was successful, and by the end of 2011, the program had reached 90% of Finnish schools offering basic education  or 2010 (n = 818). After excluding schools that had been closed or combined with other schools during the years 2009–2016 (n = 400), and schools not providing any responses in either the student or the staff survey during the focal years (n = 106), our final sample size was 1,771 schools. Of these included schools, 64.3% had started to implement KiVa in 2009 and 35.7% in 2010; 68.8% were elementary schools, 14.2% middle schools, and 17.0% combined schools with both elementary and middle school students. Finnish was the official language in 89.9% of the schools and the rest provided education in Swedish, the other official language in Finland. School size ranged from 13 to 960 students, the average school size being 230 (SD = 168) students.The sample consists of Finnish basic education schools registered as KiVa program users during 2009–2016.The data consists of the responses to the annual online student and staff surveys that take place at the end of each school year between the end of April and the beginning of June. The student survey has been conducted since the spring of 2009. To provide a baseline against which to compare the effects of the program, schools start administering the student survey in the spring prior to the launch of the program implementation .The student survey was expanded on 2011 to include questions measuring the implementation of the KiVa program components, and awareness of the program .The staff survey was administered for the first time after the first year of program implementation . One person from each school responds to this survey on behalf of the school. This survey has also been expanded over the years, and although the survey has been available since 2010, all the questions used in the present study were added in 2011.We considered the initial years of implementation to be crucial for integrating the program to the school practices. Therefore, we focused on the implementation characteristics from the years 1 to 3 as predictors of sustainability. To do so, we averaged all predictors over the first three years, except for the school size and type, and the baseline measure for victimization.Annual monitoring of the level of victimization is a core component of the KiVa program. Therefore, participating in the student survey is a natural index of schools' program participation. Each year, participation was coded as 1 if students from the school responded to the survey and 0 if students did not respond.Upon the registration to the KiVa program, schools provided background information on the number of students enrolled and school type . To keep this background data up to date, the schools' KiVa contact persons were asked to update the information annually. We used the across years' average for school size, whereas the most recent value was used for school type in the few cases where there were changes. For the analysis predicting persistent participation, we created a binary variable: schools including middle school level, that is, being either middle schools or combined schools with both elementary and middle school grade levels (= 1) versus elementary‐only schools (= 0).not at all, 1 = once or twice, 2 = 2 or 3 times a month, 3 = about once a week, 4 = several times a week). These data came from the pre‐implementation survey and the responses were averaged at school level to produce the school‐level measure of baseline victimization.Baseline victimization was measured by one item from Olweus Bully/Victim questionnaire Olweus, : “How ofNo, 1 = Yes).Staff members were asked “Has the school appointed a person/persons who is/are familiar with the program and who coordinate/s the implementation of the program as a whole and guide/s others in matters concerning the program” (0 = Yes (= 1) or No (= 0). The four variables were averaged as one composite measure of implementation planning.Staff members reported whether the school had a written plan on: (1) which grade levels will be targeted by KiVa lessons/themes ; (2) when the KiVa team discussions to address the bullying cases are organized ; (3) how bullying cases are directed to the KiVa team; and (4) how to inform parents about cases tackled by the team. Each question was responded by selecting either No, 1 = Yes); (2) “Have the students in the school been informed about KiVa ” ; (3) “Have parents/guardians been informed about KiVa ?” with response options No; Yes, a copy of the online Parent's Newsletter has been send to homes; and Yes, we have organized a Back‐to‐School Night about KiVa. ; (4–6) “Have the survey results been presented to staff/students/parents?” (each responded (0 = No or 1 = Yes). All items were averaged.Staff members reported whether students, parents, and staff members were informed about the KiVa program and whether the KiVa survey results were presented to the same parties. The exact questions were: (1) “Has your school organized a staff meeting day/discussion about KiVa for the whole school staff?” . The responses were aggregated to school‐level by averaging student responses, thus representing the proportion of students being aware of the KiVa program each year.The students were asked “Is KiVa program used in your school?” . The responses were aggregated to school‐level by averaging.The students were asked “Have the recess supervisors in your school been wearing bright vests with KiVa logo?” . The responses were aggregated to school‐level by averaging.As the student lessons/themes were targeted only to the grades 1, 4, and 7, only the responses from students in these grade levels  were used. Students were asked “Have you had KiVa lessons/themes delivered in your class since last fall?”  were asked whether they have played online KiVa games since last fall?”  made by FIML was possibly violated. In practice, a few of the predictor variables could be missing not at random (MNAR), meaning that whether a value is missing depends on the value itself. This is particularly a problem with the staff survey; it is plausible that low levels of the specific program component implementation predict missing answers to these questions because both depend on the overall degree of commitment to the program by the school. To assess this possibility, we ran another set of analyses using the program participation variables from the student surveys as auxiliary variables. The results were virtually the same as the original FIML results. We further assessed the potential impact of missingness in the staff variables by running the analyses on subsets of the data where the lowest values  on these variables were eliminated. The results from these subsample analyses were very similar to the main analysis results. Therefore, our data does not indicate that there is a MNAR problem.While we do not have evidence of MNAR being a problem for our analysis, it is ultimately impossible to test for MNAR empirically. Thus, it is prudent to speculate what the effects of MNAR would be to our results if the data on the staff data were MNAR. Because the staff variables are only weakly correlated with the student and demographic variables Table , it is uet al. , bootstrapped likelihood ratio tests (BLRT), and information criterion indices Akaike's information criterion (AIC), Bayesian information criterion (BIC), sample‐ size adjusted BIC (aBIC) produced by Mplus  is labeled as persistents. The estimated probability of program participation in this group was high every year (range 0.86–0.97). The second largest group was the drop‐offs . These schools had high probability for participation in the baseline survey, but by the second year the estimated probability declined to close to zero. The tail‐offs was also a relatively large group  with a declining trend in participation starting from Year 3. The fourth group, the awakened , started lower than average, showed an initial decline in participation, but then increased in participation.We analyzed the classes by assigning each case to the most likely class based on the LCA analysis. The four implementation profiles obtained from LCA are shown in Fig. We examined whether the four groups differed in school size, type and language Table . Persistr = –0.26 and –0.36, respectively) is less common in middle schools than in elementary schools.To identify factors predicting sustainability of the KiVa program, more specifically, the membership in the persistent group identified by the LCA, we applied logistic regression analyses. Due to the probabilistic nature of the latent class variable, the most likely class variable can be tough to contain measurement error . Planning and informing are also weakly positively related to student perceptions of the program use  suggesting that all these variables may reflect an overall level of fidelity in implementation, but the weak level of the correlations indicate the different schools may emphasize different parts of the program. The correlations are small to moderate among the variables measuring implementation of the program as perceived by students (0.18–0.32). Stronger correlations are found between students' awareness of the KiVa program with having had KiVa lessons (0.56), having played the online games (0.25), and the recess supervisors' vests being used (0.35), which is quite natural because these elements are clearly visible to students as well as being explicitly designed to raise student awareness.The results from the logistic regression analysis including all predictors are shown in Table et al., While there is considerable evidence on the effectiveness of evidence‐based prevention and intervention programs, researchers have raised concerns on the sustainability of these programs  maintained a high likelihood of participation throughout the study period . The awakened (14%) started lower than average with a decreasing trend during the first years, but then again increasing the likelihood of program participation . Furthermore, two groups with a declining trend in participation were identified, the tail‐offs (20%) and the drop‐offs  The likelihood of participation declined either after the first (drop‐offs) or the third year (tail‐offs) of implementation, thus being in line with the suggestion by McIntosh, Mercer, Nese, and Ghemraoui , further encouraging the continuation of the program.sustaining the fidelity or quality of implementation  in charge to coordinate and maintain the program. Moreover, informing the whole school community about a prevention program is important for the process of integrating the program in the schools' everyday life.


Expectant and parenting students (EPS) at community colleges are an underserved and often under-resourced group. In a rural, Midwestern state, the department of public health was awarded the Pregnancy Assistance Fund (PAF) grant to assist this population. This paper outlines the results of the implementation evaluation and offers suggestions for programs and evaluators working with this population in the community college setting.We conducted a multicomponent evaluation utilizing quantitative and qualitative methods. Evaluation activities included tracking activities/services, surveys and interviews with participants, and interviews with community college staff implementing grant activities. The research team calculated frequencies for quantitative data and coded qualitative data for themes.Data from the community colleges and students’ self-reports revealed that EPS most commonly received concrete support from the program, often in the form of stipends or gift cards. Students reported that concrete support was beneficial and helped to relieve financial stress during the semester. Students’ major barriers to participation were lack of knowledge about the program and busy schedules that prevented them from accessing PAF services. Staff reported that difficulty identifying EPS and the short one-year project period were major implementation challenges.We recommend that community colleges work to identify EPS, use fellow EPS to recruit program participants, and implement programming that works with the students’ schedules. Very little is known about the needs of EPS at community colleges and how to develop beneficial programming for them. EPS commonly attend community colleges and all EPS face barriers to graduation that their non-parenting counterparts do not. This study outlines results of the implementation evaluation of a program serving EPS at community colleges. These results highlight the need to conduct research early in program planning to understand the needs of the population, use these data to inform programming, be flexible in implementation, and work with students to offer activities and services that fit within their schedules.Nationally, the percentage of college students with children continues to grow; currently, this population accounts for an average of 30% of the student body for community college  is designed to assist expectant and parenting teens, young adults, and their families by providing health, educational, social, and economic services  or access to free food, diapers, and personal hygiene supplies, and creating or promoting activities including workshops and educational support. These workshops covered topics including social media and internet safety, domestic and intimate partner violence prevention, parenting support, job preparedness, and childhood literacy. One college chose to implement a more formal mentoring system and required consistent meetings between mentors and students to help them navigate their academic and personal responsibilities. This paper presents the methods and results of our implementation evaluation of the PAF program in community colleges in a rural Midwestern state implemented during the 2017–2018 academic year. We focused on implementation evaluation because of the shortened project timeline and because we did not believe implementation would be long enough to measure any tangible outcomes. Instead, we wanted to understand the process and the factors that contribute to program success. In order to provide the most helpful information to the colleges, our evaluation focused on collecting data that could be used to improve future efforts  the voluntary nature of their participation and that information would be reported in aggregate and kept confidential. Evaluation activities consisted of developing and implementing tracking tools at each college (RQ 1) and collecting and analyzing the results of student participant surveys (RQ 1), student interviews (RQs 1–3), and program administrator interviews (RQs 2–4). We did not perform any analysis comparing results across colleges because the program was implemented differently in each college, and our research questions focused on broad facilitators and barriers to implementation rather than specifics of the program.The evaluation team developed a tracking tool for community college staff to record EPS’ participation in and receipt of PAF services and activities during the grant period. Using a Microsoft Excel spreadsheet, staff recorded demographic information about participants  and their participation in activities or receipt of services. Staff also recorded aggregate data on the number of students who attended on-campus events in situations in which they could not collect individual-level data. The evaluation team was available throughout the program period to provide technical assistance on using the tracking tool. Community college staff sent tracking data to the evaluation team monthly. At the end of the program period, the evaluation team aggregated tracking data from all three colleges to determine total counts of EPS receiving services/activities. The data contained records for 875 unique individuals, and we report both the number of unique individuals who received services and the total number of times a service was accessed.The evaluation team developed a 10-minute online survey to measure EPS’ satisfaction with and to elicit feedback on the PAF program. The survey asked about which types of services they received through PAF, how satisfied they were with those services, suggestions for potential changes to the program, and demographic information about themselves and their families. At the completion of the program period, community college staff distributed the survey link to students with a record of PAF participation from the tracking tool. Data collection occurred over the course of three weeks, and students who completed the survey were mailed a $10 gift card. A total of 202 EPS completed this survey, and responses were distributed among the three colleges . We can estimate that 23.1% of participants who received services captured by the colleges’ tracking sheets responded to this survey, however we are unable to calculate a definitive response rate because multiple people at each college were involved in the distribution process. Moreover, we are unsure if every student with a record of participation received a survey or if the survey was forwarded to others. We downloaded data from Qualtrics and used SPSS v. 25 to calculate frequencies and percentages for all variables.To complement the student participant surveys, the evaluation team conducted interviews with program participants. The team designed an interview guide to further understand the EPS’ participation, services received, satisfaction with services, and barriers to participation Appendix . CommuniThe evaluation team interviewed the key program administrators responsible for implementing the PAF program at all three community colleges Appendix . Nine inFollowing all interview quotes, we used notation to differentiate between quotes from each of the community colleges, the students, and the program administrators.RQ 1: Was the program implemented as intended?We present the results below in order of our four implementation evaluation research questions.Each college was expected to provide services in all five service domains and in total, community colleges initially projected 350 EPS would be served in the first year of the program. To assess implementation, we used the colleges’ tracking data and results from the student participant survey. Table While colleges were able to provide services in all domains, responses from the student participant survey indicated that not all students received services or participated in activities in each of the domains. Overall, EPS reported receipt of or participation in some of the services/activities in some domains more than others Fig. . For exaRQ 2: What were the barriers to program implementation and participation?Program administrators and PAF participants reported various barriers to program implementation and participation. A primary barrier reported by both EPS and program administrators was promotion of the program. EPS reported hearing about PAF from various sources across their campus. Half of the EPS interviewed reported they heard about the program from their advisor or another staff member at their college (n = 7), and a few had learned of it either from a classmate who was receiving services (n = 2) or from an email or other online information (n = 3). Additionally, even when EPS knew about some aspects of the program—primarily the availability of financial support—they reported other aspects were not well advertised. Several of those EPS interviewed reported they had not realized the extent of activities/services offered until the end of the program. One respondent noted not “know[ing] anything about any of the services” [CCA_ST_4] beyond the financial assistance they received, and another believed the college needed to “create more awareness to…the parents who are students.” [CCC_ST_3].Similarly, although the program reached more students than projected, administrators at all three colleges reported challenges in identifying and recruiting eligible EPS. The only mechanism for identifying EPS was if students had filed dependents on their Free Application for Federal Student Aid (FAFSA) record. However, pregnancy status is not included on FAFSA, and colleges knew these records were incomplete; to identify EPS, staff often had to specifically ask students. Administrators noted the “awkwardness” of asking students if they were expectant or parenting. Questions about eligibility for programming involving financial and other concrete assistance  also led staff to have uncomfortable conversations about finances.Additionally, EPS who were interviewed reported several barriers to being able to participate in PAF activities or access services provided by the program. These included class schedules and the demands of their personal lives. Four of the EPS interviewed reported that although they knew about many of the services offered on campus, they were unable to attend them because of conflicts with their class or work schedules. For EPS who took only online classes, the services available on campus were not useful. Overall, this population is very heterogeneous , some administrators noted that some students’ needs were not able to be met. Several administrators noted having the flexibility to better match monetary support and comprehensively address all of EPS’ needs would have been helpful.Program administrators from all three colleges reported that having a “program champion” within their college helped to facilitate the success of program implementation. These individuals, which included faculty and staff, helped to identify and recruit participants and organize and promote activities. Administrators from each college reported the program champion’s recruitment via word of mouth and his or her promotion of activities/services were most successful in reaching potential PAF participants. Beyond these program champions, colleges relied on various recruitment methods, including word of mouth, emails, posters, social media, and print media, to advertise the program. Although administrators discussed these varied forms of recruitment for the program, in interviews, students most commonly reported that they learned about the program from community college faculty or staff.Our evaluation of PAF programs implemented at three community colleges revealed findings that may be beneficial for those who work with EPS populations in this setting. First, all colleges experienced challenges identifying and recruiting EPS. No database of EPS exists, and although it may be possible for administrators to glean some information about this group from FAFSA records, those forms have limitations and provide no tracking mechanism for EPS. We recommend that colleges collect information specifically on their expectant and parenting student population to better be able to serve them and match them with available services. The colleges in this study relied on one-on-one outreach conducted by staff champions to identify and track EPS, but because such outreach can be time consuming, having processes in place to accomplish this task would be beneficial.The problem of identifying and reaching this population is compounded by its heterogeneity. Compared to the state’s population, demographics from all data collections revealed the sample that we have data for is ethnically and racially diverse and often under-resourced. All these characteristics make them a vulnerable population  was easy to track, participating in workshops and using services like lactation rooms proved difficult to track, particularly at the individual level. Therefore, the participation numbers do not fully represent all activities and services at the three colleges and undercount the actual number of activities and students served.Despite these limitations, our findings highlight important considerations for future program implementers and evaluators of programs designed to assist EPS at community colleges. This environment is an excellent setting for interventions with this vulnerable population, and further research could expand our understanding of ways to identify EPS, comprehend their needs, and be responsive to their interests so that they receive the optimal support for their educational efforts.


Despite evidence that treatment reduces addiction-related harms, including crime and overdose, only a minority of addiction-affected individuals receive it. Linking individuals who committed an addiction-related crime to addiction treatment could improve outcomes.The aim of this city-wide, pre-arrest diversion program, Madison Addiction Recovery Initiative (MARI) is to reduce crime and improve health  among adults who committed a minor, non-violent, drug use-related offense by offering them a referral to treatment in lieu of arrest and prosecution of criminal charges. This manuscript outlines the protocol and methods for the MARI program development and implementation. MARI requires its participants to engage in the recommended treatment, without reoffending, during the six-month program, after which the initial criminal charges are “voided” by the law enforcement agency. The project, implemented in a mid-size U.S. city, has involved numerous partners, including law enforcement, criminal justice, public health, and academia. It includes training of the police officer workforce and collaboration with clinical partners for treatment need assessment, treatment placement, and peer support. Program evaluation includes formative, process, outcome (participant-level) and exploratory impact (community-level) assessments. For outcome evaluation, we will compare crime (primary outcome), overdose-related offenses, and incarceration-related data 12 months before and 12 months after the index crime between participants who completed (Group 1), started but not completed (Group 2), and were offered but did not start (Group 3) the program, and adults who would have been eligible should MARI existed . Clinical characteristics will be compared at baseline between Groups 1–2, and pre-post the program within Group 1. Participant baseline data will be assessed as potential covariates. Surveys of police officers and program completers, and community-level indicators of crime and overdose pre- versus post-program will provide additional data on the program impact.By offering addiction treatment in lieu of arrest and prosecution of criminal charges, this pre-arrest diversion program has the potential to disrupt the cycle of crime, reduce the likelihood of future offenses, and promote public health and safety. Over the past two decades, the scope and impact of fatal and non-fatal drug overdoses has grown to epidemic proportions in the U.S., with both prescription-based and illicit opioids driving this crisis    in Gloucester, Massachusetts relied on police officers to assist individuals with a referral to addiction treatment . MARI emphasized a collaborative, police-community effort that creates linkages to addiction treatment from police officers as a way for reducing crime and overdoses, and improving lives of individuals with addiction and the local community. In 2016, the MARI proposal was submitted for funding consideration to the U.S. Department of Justice Bureau of Justice Assistance (BJA) Strategies for Policing Innovation (SPI) grant program, which has prioritized development and evaluation of innovative, evidence-based approaches to address ‘chronic’ crime  by offering treatment to eligible individuals who committed a drug use-related, minor crime in lieu of arrest and prosecution of criminal charges. MARI is a pre-arrest diversion program. Its participants are not initially arrested or charged. Therefore, the official charge is not documented and is held in abeyance through the duration of the MARI program. For program completers, the criminal charges associated with bringing these individuals into MARI are voided . Those who do not complete MARI have their formal arrest charges referred to the District or City Attorney’s Offices at the time they leave the program Fig. .The secoThe MARI grant award was announced in October 2016 , and another in July 2020 (due to the coronavirus pandemic’s impact). The funding agency approved these requests, extending the participant enrollment period through August 2020, and overall funding period through April 2021. The participants enrolled in August 2020 will complete the project’s six-month active follow-up phase in February 2021.The program is supported and approved by numerous city- and county-level organizations, enabling the City of Madison Police Department (MPD) to roll it out across the city, involving all MPD districts. Madison is a medium size city in Wisconsin, USA. The MPD employs 483 commissioned police officers, with approximately 226 assigned to first-responder or patrol duties. Approximately 28% of the MPD officers identify as women, and 86% of the incoming officers completed at least 4 years of college education.Because addiction is a multi-faceted problem, affecting multiple sectors, collaboration among a wide range of partners has been critical. The MARI Ops Team, responsible for the design, development, and implementation of the MARI program, is comprised of representatives from the MPD (recipient of BJA grant award), city/county public health and human services, a community non-profit injury prevention and advocacy organization, the MARI Assessment Hub, treatment providers, and academic partners.The program has sought, and been supported by, numerous other local agencies, including the Mayor, Sheriff, District and City Attorney Offices, Emergency Medical Services (EMS), Fire Department, Wisconsin Department of Health Services, and other stakeholders, such as diverse community advocacy groups, treatment providers/programs, and health plans. The perspectives of diverse collaborators and stakeholders have been included throughout the MARI projects. The broader stakeholder-partners and the public have been invited to the quarterly MARI meetings for sharing information, and discussing ways to optimize MARI’s implementation, progress, and sustainability.Individuals are eligible for MARI if they are adults (18 years old or older), reside in Dane County, and committed an “eligible” non-violent, drug use-related crime , they were not eligible to re-start it; should they re-offend, they could be eligible for drug court or other treatment-oriented diversion programs run by the District Attorney Office, outside of the MARI program.The arresting officers are responsible for conducting an assessment during their investigation to determine if the offending individual meets the MARI eligibility criteria. The MPD officers are trained to identify and assess individuals for MARI eligibility, including whether the committed crime represented a MARI-eligible offense, and was related to addiction. For individuals who are deemed eligible, the arresting officers are directed by the MPD protocols to offer a referral to MARI in lieu of arrest and criminal charges. For individuals who accept the offer and enroll in MARI, criminal charges are held in abeyance . Those who decline MARI are arrested or issued citations by the officer, and their criminal charges are referred to local prosecutorial offices .After discussing the MARI program, officers review with eligible, interested individuals the MARI referral/consent form, which outlines the program, and provides the program and the Assessment Hub contact information. The referral/consent form is filled out by the arresting officer and signed by the individual. The arresting officer issues and notes on the form all citations, then sends the form to the MPD MARI officer. The MPD MARI officer is then responsible for verifying each referred person’s eligibility. If the individual is deemed ineligible, the MPD MARI officer reaches out to the individual to explain it, rescind MARI participation, file charges, and offer information on existing addiction treatment resources. If a person is incapacitated , the arresting officer can send the filled out MARI form to the MPD MARI officer who then attempts to contact the eligible person within the next one-to-two business days to discuss the program and offer MARI. The MPD’s trained Sergeants and the MARI officer provide continued oversight and support to officers for MARI-related assessments, decisions, questions and refresher trainings as needed.Eligible adults who agree to participate in MARI have their charges held in abeyance, and are referred to the MARI Assessment Hub for clinical assessment toward substance use disorders (SUDs) and for peer support . The MPD MARI officer securely shares referral information with the MARI Project Coordinator and the Assessment Hub to facilitate the referral. MARI participants are requested to get in touch with the Assessment Hub within three business days to schedule their assessment visit.The Assessment Hub, a local certified addiction treatment program, was selected through a competitive application process. The Hub provides clinical assessments, facilitates treatment referral and links MARI participants to the Hub’s ‘recovery coaches.’ The Hub’s certified substance abuse counselors complete clinical evaluation to determine the scope and severity of each MARI participant’s SUD-related problems, and optimal level of care, in accordance with the American Society of Addiction Medicine (ASAM) Placement Criteria  have their initial charges referred to the City or District Attorney’s Offices at the time of their departure from the MARI program Fig. .A).Formative Evaluation: Information about the program development and implementation will be collected throughout the project to identify the facilitators, barriers, and steps taken to overcome identified barriers to program implementation. This knowledge will enable learning from the project experiences toward optimizing of future implementation on a larger scale, or in different communities. Documentation of community and stakeholder support and engagement, media coverage, and community/system changes associated with progression of the project are indirect measures of the community-level interest in MARI, and can inform sustainability of the MARI approach beyond the grant funding.B).Process Evaluation: Assessment of participant engagement in the program, the scope of program services, and participant and MPD officer experiences with the program will help better understand factors contributing to program effectiveness, cost, and, ultimately, sustainability and reproducibility.The evaluation plan for the MARI program has focused on answering questions vital to the program itself, the stakeholders, and the long-term program sustainability .Outcome Evaluation (participant level): This evaluation will assess MARI’s impact on individuals. The main goal of the MARI project is to test the hypothesis that facilitating addiction treatment, instead of pressing criminal charges, will lead to reduced crime (primary outcome) and overdose (secondary outcome) among eligible adults who committed minor, eligible crime. The outcome evaluation will be completed by comparing crime-related data of the MARI participants 12 months before and 12 months after their ‘index crime’ date (program enrollment), and by contrasting outcomes between different subgroups of the MARI participants , and those in a Historical Comparison group, comprised of adults who would have been eligible for MARI, should this program had existed Fig. . In addiProgram engagement  will help evaluate the willingness to participate and interest among the target population in the MARI program. The Assessment Hub counselor tracks each participant treatment adherence (‘yes/no’) during the program. The MARI MPD Officer monitors each participant’s criminal record for any new offenses; should a new offense occur, the MPD determines if it breeches the MARI program requirements and warrants a participant discharge from the MARI program. Descriptive statistics will describe participant engagement in MARI.Crime-related data are available for all MARI participants and those in the Historical Comparison group for the 12 months before, and 12 months after the ‘index crime’ Fig. . They arMPD MARI officer and the MARI Assessment Hub’s clinical staff securely share the data with the MARI Project Coordinator who links the crime and clinical datasets on a participant level, using the unique identification number, assigned to each participant upon the program entry. The Project Coordinator enters the de-identified data into a password-protected Excel database prior to sharing this de-identified dataset with the academic partners for analyses.D).Impact Evaluation (community level): Through improved access to addiction care, the MARI approach, over a longer period of time, has the potential to improve community health and safety, as assessed by the community-level reduced rates of crime, overdose, and overdose-related death, and to reduce related cost. Within the limitations of the MARI project , the impact of MARI on the community-level indices of safety and health will be of an exploratory nature. Aggregate community-level data will be obtained from the project collaborators who collect such data as a part of their routine duties .This article describes the protocol for an innovative, law enforcement-led, city-wide pre-arrest diversion-to-treatment project, which aims to reduce crime and criminal justice involvement, and improve treatment engagement and outcomes among adults with addiction who committed an “eligible” minor, drug-use related crime. Those who enter the program have their charges held in abeyance; after they successfully complete the six-month program by complying with its requirements , their initial criminal charges are voided . The protocol described here offers details on the program structure, so it can be considered by others and improved upon, and serve as a model for implementing similar pre-arrest diversion programs.MPD has incorporated the MARI diversion protocol city-wide into its standard operating procedures, making it a permanent part of its MPD services, and absorbing the cost to do so. Engaging throughout the project of “key players” and diverse stakeholders who are routinely involved in the prosecution as well as treatment and social services for adult offenders with addiction has been a deliberate step viewed by the project team as essential for the program’s success and long-term sustainability. If effective, the MARI program model could be expanded beyond the city limits, and extended to other individuals who are currently not eligible for MARI but whose untreated addiction might be fueling their criminal activity. For example, MARI focuses on individuals with drug addiction, especially when it involves opioids, which have driven the overdose epidemic. As such, those who committed an alcohol (but not other drug) related crime are not currently eligible for MARI. Similarly, those who are on probation or parole are currently not eligible to enter the MARI program; this may unintentionally negatively impact individuals of color and other groups, which have been disproportionately involved with criminal justice , can benefit the communities and their trust in, and relationship with, law enforcement officers.Traditional law enforcement, criminal justice-based approaches have been largely ineffective in dealing with criminal offenses committed by individuals who suffer from the disease of addiction (Chandler, Fletcher, & Volkow, The described pre-arrest diversion program, which relies on law enforcement to help link adults who committed minor, drug use-related crimes to addiction treatment, has the potential to improve the health and safety of individuals affected by addiction, and their communities.


For studying the effectiveness of treatment, it is important to check whether a new treatment is performed as originally described in the study-protocol.To evaluate whether an interdisciplinary graded exposure program, for adolescents with chronic musculoskeletal pain reporting pain-related fear, was performed according to protocol, and whether it is feasible to implement the program in rehabilitation care.A process evaluation where quantitative and qualitative data on participant characteristics , attendance and participants’ opinion on the program were collected, by means of registration forms, questionnaires and group interviews. To evaluate treatment fidelity, audio and video recordings of program sessions were analyzed.Thirty adolescents were offered the program, of which 23 started the program. Adolescents attended on average 90% of the sessions. At least one parent per adolescent participated in the program. Analysis of 20 randomly selected recordings of treatment sessions revealed that treatment fidelity was high, since 81% of essential treatment elements were offered to the adolescents. The program was considered client-centered by adolescents and family-centered by parents. Treatment teams wished to continue offering the program in their center.The interdisciplinary graded exposure program was performed largely according to protocol, and therapists, adolescents and their parents had a favorable opinion on the program. Implementation of the program in rehabilitation care is considered feasible.NCT02181725 (7 February 2014).Clinicaltrials.gov ID:  Chronic pain in children and adolescents can substantially impact their health-related quality of life . The paiIt has been demonstrated that pain-related fear is an important factor in explaining the severity of pain complaints and pain-related disability in adolescents , 14. WheBecause of demonstrated effectiveness in adults and supporting evidence for the applicability of the underlying theoretical model in adolescents, an interdisciplinary graded exposure program was developed to improve adolescent functional disability by reducing pain-related fear and catastrophizing of both the adolescent and the parents. This is accomplished by challenging catastrophizing thoughts and performing feared and avoided activities in order to improve physical, psychological and social functioning. To our knowledge, graded exposure in vivo has not been studied for its effectiveness and feasibility among adolescents before.This paper describes a process evaluation according to the framework of Saunders and colleagues . The objThe graded exposure program aims to restore the adolescents’ age appropriate level of functioning and decrease functional disability by activating and invalidating catastrophizing thoughts about feared and avoided movements or activities. The exposure program consists of an adolescent module, and a parent module that was offered in parallel with the adolescent module. For a subgroup of adolescents the adolescent module was combined with physical training if they were diagnosed with joint hypermobility syndrome. The modules of the program are described in Fig. The exposure program was offered by an interdisciplinary rehabilitation treatment team that is led by a consultant in rehabilitation medicine, and further consisted of a psychologist, and a physiotherapist or occupational therapist . When anOnce randomization indicated that an exposure program had to be started , first an intake was performed by a psychologist together with a physiotherapist and/or occupational therapist. During this interdisciplinary intake, with the adolescent (and parent(s)), a biopsychosocial perspective on pain and the role of fear were further explained and were placed within the context of the family system. The Photograph Series of Daily Activities – Youth (PHODA-Youth) was used to systematically build a personalized hierarchy of activities that were feared and avoided by the adolescent . SubsequAdolescents with additional joint hypermobility syndrome received physical training prior to the exposure sessions, to improve their physical fitness and muscle strength , 27 Fig. This phThe parent module consisted of three meetings where parents were educated on the treatment rationale, the interpersonal fear avoidance model, the role of pain within the family system, and the parents’ role in providing support and preventing relapse Fig. .This process evaluation used a multimethod approach including prospective and retrospective qualitative and quantitative components. Data were collected between February 2014 and February 2017, before the start of the exposure program, during the program and after the program had finished. Ethical approval for this study was provided by the Medical Ethics Committee Academic Hospital Maastricht/Maastricht University, the Netherlands, NL47323.068.13.The population who participated in this process evaluation consisted of the adolescents and their parents who attended the program, and the treatment teams who offered the program. Adolescents, 12–21 years old, were eligible for participation in the study if they were referred to outpatient rehabilitation treatment for chronic musculoskeletal pain in one of the four participating rehabilitation centers in the Netherlands . Adolescents were excluded if there was a suspicion of a medical  disease that could fully explain the current level of severity of pain complaints, if there was a suspicion of an underlying psychiatric disease that would hamper rehabilitation treatment, or in case the adolescent was pregnant.The treatment teams of three of the four participating treatment centers received a 4-day training in the exposure program and study procedures before the start of the intervention period. In one center the team was already trained in the exposure program and worked with the program protocol for 2 years already.Training was conducted by experienced behavioral therapists and experts in graded exposure in vivo treatment, and members of the study team. During the training, the program modules and study procedures were explained and practiced by the treatment teams by means of role plays. During the study period, supervision was offered through telephone or email contact with the trainers. Treatment teams received a detailed treatment manual (protocol) with instructions for each program session.The description of the data collection is in chronological order. Table Performance according to protocol is an integrated measure of treatment fidelity , and dose delivered  dose to the adolescents, and protocol deviations). Treatment fidelity was assessed for the adolescent module and was expressed as the percentage of essential treatment elements executed by the treatment teams. To enable analysis, the adolescent module was divided into phases: a preparation phase, which included the interdisciplinary intake and PHODA-youth, an education phase, which included the education session, and a treatment phase, which included the graded exposure in vivo sessions. Audio and video recordings (AV) were made by the therapists, during the three phases of the program. Specifically, the intake, education and graded exposure in vivo treatment sessions of the adolescent module were recorded. Physical training, explanation on the chronic pain syndrome, screening, and parent meetings were not recorded. Information about attendance of adolescents and parents, and protocol deviations by the treatment teams was noted on registration forms .The adolescents’ opinion on the treatment they received during the program was measured with the Giving Youth A Voice questionnaire and for the parents’ opinion, the Measure of Processes of Care questionnaire was used –33. AdolA group interview (GI) was planned in each of the four treatment centers after all participants completed their exposure programs. For each meeting, the complete treatment team that offered the exposure program was invited. A topic list with predetermined open-ended questions was used as interview guide. The interview was conducted by two members of the study team, and audio recorded after consent. The evaluation focused on the performance of the program according to protocol, reasons for protocol deviations (including context factors), attendance and participation of the adolescents during the program, and opinion of the therapists on the exposure program protocol they had been working with.The quantitative data from the questionnaires and registration forms were analyzed using descriptive statistics in IBM SPSS Statistics for Windows, version 25 .Treatment fidelity of the adolescent module was analyzed with an adapted version of the Method of Assessing Treatment Delivery, described by Leeuw and colleagues . Five stThe group interviews were recorded and transcribed. Analysis entailed open, axial and selective coding with the aim of proving an exploratory description of the treatment teams experiences with the program and their opinion about the program.n = 5), or the adolescent was 18 years and decided to participate without parents (n = 1). Therefore, questionnaire data was available for 31 parents . The mean age of the parents was 48 years . Seven out of the 30 adolescents (24%) and their parents did not start the program, because the complaint resolved (n = 2), the situation deteriorated so that outpatient treatment was no longer indicated (n = 1), adolescent refrained from rehabilitation treatment before start of the program (n = 1) or loss of contact (n = 2). In addition, one adolescent started after the study period. Of the 23 adolescents (76% out of 30 adolescents) for whom process data was available, 18 adolescents received the standard three-phase adolescent module. The remaining five adolescents received the adolescent module with additional physical training.Between August 2014 and December 2016 30 adolescents were offered the exposure program, of which process data was available for 23 adolescents.  was 15.6 years . Twenty-two adolescents were female (96%). Almost half of the adolescents  had a lower education, being elementary school or lower vocational education (n = 2 missing). For the majority of adolescents (80%) their pain complaints started more than 1 year ago (n = 3 missing). Twelve adolescents  have a relative with pain complaints.The mean age of the adolescents analyzed n = 20) were 4 consultants in rehabilitation medicine, 7 psychologists, 5 physiotherapists and 4 occupational therapists. For the training in the exposure program, thirteen of the sixteen (81%) members of the treatment teams participated . After the intervention period, 14 (70%) of the 20 invited therapists were present at the group interviews. Here, one physician in rehabilitation medicine, four psychologists and one occupational therapist were unable to attend.The participating members of the four interdisciplinary treatment teams . In addition, it was recommended that training should immediately be followed by the opportunity to work with adolescents in practice so that treatment teams could develop their skills. Due to practical reasons, such as the duration of the procedure for ethical approval of the study, and the set-up of the intervention and study procedures in the treatment centers, this was not immediately possible. by using a treadmill or exercise bike. If a therapist was unable to provide the exposure sessions due to illness, the sessions were performed by another therapist of the individual treatment team.In all centers the adolescent and parent module were offered as intended. Two centers did not offer the physical training, because no adolescents fulfilled the criteria for additional joint hypermobility syndrome. Some protocol deviations were made in all centers due to practical reasons, such as illness of a team member or absence of a swimming pool for training. Also, 12 parent modules were offered in a group setting and 11 modules were offered to individual parents/parent couples. Reason to offer the parent module individually was due to a low number of included adolescents in two centers.n = 8, education phase n = 5, treatment phase n = 7, in total 16 audio and four video recordings) was scored by two independent raters. Interrater reliability between the two raters was Cohen’s kappa 0.72 for all three phases together. Disagreement between the raters was not resolved. The reported percentages in Table To analyze treatment fidelity, 105 of total 266 treatment sessions of the adolescent module (39%) were recorded. According to the therapists, the main reason for not recording sessions was that it was forgotten; either to bring the recorder to the session or to turn it on. A random selection of 20 recordings  score below 30%, indicating that these elements were seldom performed during the program sessions of the adolescent. Further, elements on the aim of the program (E15 and E23), and agreements and homework  scored relatively low, indicating that the aim of the treatment was not clearly communicated to the adolescent during the education phase and that agreements and homework were not clearly communicated during the treatment phase.n = 18) attended on average 12.7 of the 14 program sessions . When including the 7 adolescents who did not start the program, and therefore attended 0 program sessions of the standard module, this percentage drops to 65%, meaning an average attendance of 9,1 sessions of the 14 program sessions potentially offered. Adolescents who received the standard module including physical training (n = 5) attended on average 21.6 of the 24 planned program sessions . In the latter group 14.8 of the 16  of the physical training sessions and 6.8 of the 8  of the graded exposure sessions were attended. Of the 23 adolescents who started the program, 21 adolescents completed their program. Two adolescents did not complete their program because they changed treatment (n = 2). These two adolescents attended 8 of the 14 (57%) and 20 of the 24 (83%) planned sessions respectively.Adolescents in the standard module .SD = 5.1, on a scale of 3–27) and mean expectancy was 13.2 , scored by 20 adolescents (n = 3 missing). These scores indicate that adolescents assign relatively moderate credibility that the exposure program will increase their functioning and participation in social activities despite their pain. Furthermore, these scores indicate the adolescents have relatively high expectations with regard to the improvements the exposure program can cause in functioning and participation in social activities.The adolescents’ mean credibility score for the treatment rationale before the start of the exposure program was 17.7 , reflecting their opinion on the exposure program. Mean scores on the subscales are presented in Table The Giving Youth a Voice questionnaire was completed by 17 adolescents . Mean scores on the subscales are presented in Table The Measure of Processes of Care questionnaire was completed by all participating parents n = 1. Mean sAll participating treatment teams had a positive opinion about the exposure program, although most therapists perceived practice of the program challenging. The teams, however, intend to continue the use of the exposure program within their rehabilitation center, by integrating the program in their regular care. Multiple reasons were mentioned for the willingness to implement the exposure program. The exposure program was found to be suitable specifically for adolescents with chronic musculoskeletal pain reporting pain-related fear and avoidance behavior. According to the opinion of the treatment teams, the program could cause behavioral change in activity performance in a relatively short time frame. Furthermore, the program provided a clear outline for the adolescents in terms of planning and working towards their treatment goals, which worked well with this patient group. The parent involvement was much appreciated and considered essential to preserve treatment gains at home. However, due to the relative short duration of the intervention and the experienced relatively narrow focus of the intervention  concerns were expressed regarding the long-term effects of the program. Especially with regard to relapse into previous pain behavior, and treatment of  co-morbidity, treatment teams expressed their doubts regarding treatment effectiveness. Moreover, guidance on how to handle in case of relapse in previous pain behavior was considered to be too restricted in the program protocol.The psychologists and physiotherapists/occupational therapists also indicated they enjoyed and preferred working in a duo during the program sessions. They could support each other during program sessions; it enabled evaluation of the sessions (they felt better informed of what happened in the treatment because they were there), and it improved communication with adolescents and parents.This process evaluation assessed whether a graded exposure program offered in interdisciplinary rehabilitation care was performed according to protocol and whether it would be feasible to implement the program in rehabilitation care. Data for the process evaluation was available for 76% of the adolescents who completed the program. Treatment fidelity was high in all three treatment phases, since on average 81% of the essential treatment elements were offered to the adolescents. Adolescent participation was high, with attendance rates of 90% and with at least one parent participating in the parent module. Adolescents and parents considered the exposure program client-centered and family-centered respectively. Adolescents considered the program moderately credible and had relatively high expectations about attaining the treatment goals  after their first visit to the consultant in rehabilitation medicine. Furthermore, treatment teams had a favorable opinion about the content of the exposure program, and had the intention to implement the program in regular care in their rehabilitation center. Psychologists, physiotherapists and occupational therapists appreciated working jointly together in duo’s in the exposure sessions.It is difficult to compare these results with other process evaluations of rehabilitation treatment for adolescents with chronic pain, as no studies of this kind exist to our knowledge. Leeuw and colleagues  providedThis study has some strengths and limitations that need to be mentioned. Most important limitations concern the small sample size of this study and the problems with missing data. At the onset of the research project, four treatment centers were invited to participate in the project in order to reach the desired treatment capacity for a properly powered RCT. However, the number of adolescents that was included in the RCT did not reach the calculated sample size. A subgroup of adolescents identified as potential research participants declined participation in a scientific study due to various reasons, often related to extra efforts as compared to normal treatment outside the RCT. These patients did, however, follow (regular) treatment. As a result, the potential number of patients for whom the program may be a treatment option is higher than the actual number of patients that received the program within this study.Concerns may be raised about the relatively large proportion of missing questionnaire data presented in this manuscript. With the high drop out rates in this study, it has to be acknowledged that there is the possibility of differential drop out and especially with the small study population, small systematic differences between participants and drop outs can have a significant influence on the evaluation of the client-centeredness of the treatment program. In the total RCT, missingness of the data was assumed to be random since missingness could not be related to any measured outcome or other measured variable (such as age or education). Furthermore, reasons for not participating in the study were various and did not lead to the suspicion of volunteer bias.Concerning the data collected at the parent level, another limitation needs attention. Although data of two parents of the same child is correlated, this data was treated as independent in this study. Treating the questionnaire data of the parents as independent probably (mildly) inflated the high satisfaction rates of the treatment. Another limitation is the possibility that socially desirable answers were given by the adolescents, parents and treatment teams, when asked about the exposure program. To limit this risk, adolescents and parents received all an individual link to their questionnaires by email, which they could complete at home and results would only be published at group level. Furthermore, the group interviews with the treatment teams were performed by two members of the study team. The risk of obtaining socially desirable answers might have been lower if these meetings were led by independent interviewers, not involved in the study. The different opinions of the treatment teams about the program were derived from group interviews. Because data from group interviews was used, no quantitative data on the frequencies of the reported opinions on an individual level is available. The results on the treatment teams opinions should only be interpreted as an exploration of how treatment teams feel about offering the program.Moreover, treatment teams were aware that the recordings they made of their treatment sessions would be used to evaluate their performance according to protocol, which may have influenced their behavior. This raises another limitation. Only 39% of the treatment sessions were recorded which may have biased the available data. It might give the idea that treatment teams who were less experienced were afraid to record their treatment sessions.An important strength of this study is the analysis of treatment fidelity with the adapted Method of Assessing Treatment Delivery. Applying this method, allowed real insight into the degree to which essential protocol elements were applied by the members of the treatment teams. Together with the attendance registration, both the quantity (dose delivered), and the quality (treatment fidelity) of the adolescent module were assessed. A limitation to this approach in our study was that treatment fidelity was not evaluated for the parent module. The parent module was intended to be a group intervention and it was possible that parents who did not consent to participate in the study would be present. Therefore, it was considered not possible to record these parent meetings. Similar, treatment fidelity was not assessed for the physical training of the adolescents, so we lack insight in the fidelity of this part of the program. By all means, evaluation of treatment fidelity of all parts of the exposure program would have provided a more complete picture of the quality of delivery of the program. Furthermore, as in the approach of Leeuw and colleagues , our evaCompared to similar graded exposure interventions in adults with chronic pain, in which a psychologist is present at each exposure session, the psychologist was allowed to be present at least every second exposure session in the treatment phase in this study. Since the members of the treatment teams expressed several advantages of working as a duo, we recommend to abide by this minimum of duo sessions when implementing the program into rehabilitation care. Further, although treatment teams were trained for 4 days, they perceived good practice of the program challenging. Hence, teams indicated they needed to gain real world experience and wished for feedback on their individual functioning. The possibility to contact the trainers during the intervention was rarely used. In the future, the organization of regular intervision-sessions within the centers can help to overcome this. To ensure high quality program delivery, de Jong and Verbunt  also empBased on the results of this process evaluation, we conclude that the interdisciplinary exposure program has been performed largely according to protocol and that treatment fidelity of the adolescent module was high. In addition, the participants in general actively participated in the adolescent module. The delivery of the exposure program was perceived client-centered by the adolescents and family-centered by the parents. Treatment teams expressed a favorable opinion about the exposure program and intend to implement the program within their rehabilitation centers. Therefore, implementation of the program is considered feasible in specialized rehabilitation care.In case the effect evaluation of the exposure program reveals positive results on treatment effectiveness, implementation of the program in specialized rehabilitation care is recommended. However, for future research, investigation of treatment fidelity (including treatment receipt and participant enactment) of the currently not evaluated parent module and physical training is recommended, since analyzing both the complete adolescent module and parent module will provide the most complete picture of treatment fidelity of the program.Additional file 1 Figure S1 Adolescent flow through the RCT.


We performed batch incubation experiments with an anammox-dominated biomass to investigate nitrogen (N) and oxygen (O) isotopic fractionation factors during anammox and also examined apparent isotope fractionation factors during anammox in an actual wastewater treatment plant. We conducted one incubation experiment with high δ18O of water to investigate the effects of water δ18O. The N isotopic fractionation factors estimated from incubation experiments and the wastewater treatment plant were similar to previous values. We also found that the N isotopic effect (15εNXR of –77.8 to –65.9‰ and 15ΔNXR of –31.3 to –30.4‰) and possibly O isotopic effect (18εNXR of –20.6‰) for anaerobic nitrite oxidation to nitrate were inverse. We applied the estimated isotopic fractionation factors to the ordinary differential equation model to clarify whether anammox induces deviations in the δ18O vs δ15N of nitrate from a linear trajectory of 1, similar to heterotrophic denitrification. Although this deviation has been attributed to nitrite oxidation, the O isotopic fractionation factor for anammox is crucial for obtaining a more detailed understanding of the mechanisms controlling this deviation. In our model, anammox induced the trajectory of the δ18O vs δ15N of nitrate during denitrification to less than one, which strongly indicates that this deviation is evidence of nitrite oxidation by anammox under denitrifying conditions.Isotopic fractionation factors against  However, they lacked information on the 18O fractionation factor for anammox, and assumed that the 18O fractionation factor during NO3– production by anammox was similar to that for aerobic nitrite oxidation to NO3– by nitrifiers (nitrification). Thus, it is essential to investigate the 15N and 18O fractionation factors during anammox not only for the better use of the δ18O and δ15N of NO3– in anammox studies, but also to obtain a more detailed understanding of 15N and 18O fractionation.Studies on 18O fractionation factors during anammox. We calculated apparent 15N and 18O fractionation factors with data collected from a wastewater treatment plant (WWTP) at which anammox reactors were installed at the final stage of treatment .In the first experiment (Experiment A), the biomass in the reactor was sampled and incubated with the media used for the reactor, while the sampled biomass was re-suspended in fresh, chemically defined media in the second (Experiment B) and third (Experiment C) experiments. Difficulties were associated with performing incubation experiments with the anammox biomass for isotopic measurements, and, thus, we employed slightly different settings and operations to facilitate constant and active anammox reactions. In each experiment, 15 mL of the biomass suspension in media solution was filtered with filter paper  and differences in filter weights before and after filtration were used to calculate the suspended solid (S.S.) concentration after the filter had been oven-dried (at 105°C).2 gas to remove dissolved oxygen (DO). pH, DO, and the concentrations of NH4+ and NO2– were regularly measured to confirm the anammox activity of the biofilm, and pH (8.0) was maintained by adding KH2PO4 and Na2HPO4·12H2O solution. After the addition of NaNO2, (NH4)2SO4, and NaHCO3, we started the incubation and sampled 10 mL of media. Sampled media were filtered with a 0.20-μm syringe filter and then split into three; one for NO3– followed by the removal of NO2– (2– with high pH by the addition of 2M 2– andNaOH solution to prevent oxygen atom exchange between NO water (4+ with low pH by the addition of 4.8 M H2SO4 to prevent NH4+ from volatilizing. These subsamples were frozen (–30°C) until further analyses.The biofilm and incubation media solution  were sampled from the incubation membrane in the anammox reactor (SI Text 1.1). The incubation was performed anaerobically in the glovebox at room temperature (25–30°C) after the purging of media by N of NO2– , another2 purged) in the glovebox. Media consisted of NaHCO3 502 mg L–1; MgSO4·7H2O 603 mg L–1; CaCl2 180.5 mg L–1; KH2PO4 169 mg L–1; Na2HPO4·12H2O 282 mg L–1; trace elements of solution I  0.5 mL and solution II  0.5 mL. We added NaNO2 and (NH4)2SO4 to media (500 mL) with the anammox granules and then started the incubation at room temperature (25–30°C). We monitored pH (7.9 to 8.8) and NO2– to assess the progress of anammox. Sampling was performed as described in Experiment A.The granule biomass that accumulated at the bottom of the anammox reactor was sampled. Granules were rinsed anaerobically with new media (Nδ18O of water (δ18OH2O) was markedly higher (229‰) than that in Experiments A and B (–8‰). This “heavy” water was prepared by mixing 18O-labeled water (10% atom 18O) with Milli-Q water. During the incubation, media with biofilms were shaken at a constant temperature (30°C) and continuously purged with a gas mixture (95% Ar + 5% CO2) to maintain low DO levels. pH ranged between 7.1 and 7.5 and the monitoring and sampling scheme was identical to Experiment BWe incubated the biofilm collected from the incubation membrane in the anammox reactor with the same media used in Experiment B; however, the 4+ concentrations were measured during the incubation by the o-phthaldialdehyde (OPA) method    with USGS 32, 34, 35, and IAEA-2 as standards, and with the azide method  with TUAT-NO2-1 to TUAT-NO2-5  were evaluated using GC-IRMS with the denitrifier method after the conversion of NH4+ to NO3– by persulfate oxidation –1 and (R_SampleO/R_Oxygen)–1 where R_SampleN and R_SampleO are [15N/14N] and [18O/16O] of the sample, respectively, R_Nitrogen is [15N/14N] of atmospheric N2 and R_Oxygen is [18O/16O] of Vienna Standard Mean Ocean Water , residual NH4+ (δ15NNH4+_ANX), and the fraction of NH4+ reacting (NH4+f) in ANX reactor (15ΔAMX (The ammonium oxidation to N reactor  at a ste15ΔAMX = (δ15NNH4+_ANX – δ15NNH4+_NT) / NH4+f --- eq. (1)NH4+f = ([NH4+]NT – [NH4+]ANX) / [NH4+]NT4+]NT and [NH4+]ANX are the NH4+ concentrations in NT and ANX reactors, respectively.where ANX / ([NO2 – ]NT – [NO2–]ANX)b = ([NO3–]ANX – [NO3–]NT) / ([NO2–]NT – [NO2–]ANX)δ15NNO2–_NT is δ15NNO2– in NT reactor and the concentrations of NO2– and NO3– in NT and ANX reactors are [NO2–]NT, [NO2–]ANX, [NO3–]NT, and [NO3–]ANX, respectively.where The combination of eqs. (2) and (4) givesδ15NNO2–_NT = (1 – a – b) × δ15NN2_ANX + a × δ15NNO2–_ANX + b× δ15NNO3–_ANXa – b) × (δ15NNO2–_ANX – 15ΔAMXNIR) + a× δ15NNO2–_ANX + b × δ15NNO3–_ANX= (1 – δ15NNO2–_ANX – 15ΔAMXNIR – a × δ15NNO2–_ANX+ a × 15ΔAMXNIR – b × δ15NNO2–_ANX + b × 15ΔAMXNIR+ a × δ15NNO2–_ANX + b × δ15NNO3–_ANX= b × (δ15NNO3–_ANX – δ15NNO2–_ANX) + δ15NNO2–_ANX+ 15ΔAMXNIR × (a + b – 1)= b × 15ΔNXR) + δ15NNO2–_ANX + 15ΔAMXNIR × (a + b – 1)= – (15ΔAMXNIR = [δ15NNO2–_ANX – δ15NNO2–_NT – b × 15ΔNXR] / (a + b – 1) --- eq. (5)18O fractionation during nitrite oxidation to nitrate, we followed the approach described by 18EAMXcombined) because of the lack of detailed information on isotopic fractionation for the nitrite oxidation and oxygen atom incorporation during nitrite oxidation. Thus, we calculated 18EAMXcombined as follows:To calculate 18EAMXcombined = 2/3 δ18ONO2–_ANX + 1/3 δ18OH2O – δ18ONO3–_ANX--- eq. (6)δ18ONO2–_ANX, δ18OH2O, and δ18ONO3–_ANX are the 18O ratios of NO2–, water, and NO3– in ANX reactor, respectively.where 2– were then estimated from isotopic data.We developed an ordinary differential equation model as described by Fluxes regarding the anammox process  are defikAMO14N × [14NH4+] --- eq. (7)AMX = AMXNIR = x / (1 – x)) --- eq. (8)NXR = AMXNIR × (14N) fluxes of ammonium oxidation, nitrite oxidation, and reduction by anammox  .where AMX, NXR, and AMXNIR are the   in the a [NO2–])  and the 2–;Regarding NO14NO2–] = – NXR – AMXNIR --- eq. (9)d/dt [15NO2–] = – (R_NitriteN × NXR / 15εNXR)15εAMXNIR) --- eq. (10)– (R_NitriteN × AMXNIR / d/dt [16O2–] = – 2NXR – 2AMXNIR --- eq. (11)d/dt [N16O18O–] = – (R_NitriteO × 2NXR / 18εNXR)– (R_NitriteO × 2AMXNIR / 18εAMXNIR)– N16O18O–exch_OUT + N16O18O–exch_INd/dt [N18εNXR)– (R_NitriteO × 2AMXNIR / 18εAMXNIR)– kexch × [N16O18O–] + kexch×R_WaterO / 18εEQ--- eq. (12)= – , N16O18O–exch_OUT, and N16O18O–exch_IN  as described by 2– and H2O.where R_NitriteO, R_NitrateO, R_NitriteN, and R_NitrateN are the e and pH . We appl3–;Regarding NO14NO3–] = NXR --- eq. (13)d/dt [15NO3–] = (R_NitriteN × NXR / 15εNXR) --- eq. (14)d/dt [16O3–] = 3 NXR --- eq. (15)d/dt [N18O16O2–] = (R_NitriteO × 2 NXR / 18εNXR)+ (R_WaterO × NXR) / 18εH2ONXR) --- eq. (16)d/dt [N18εH2ONXR d/dt [15NH4+] = – (R_AmmoniumN × AMX/ 15εAMX) --- eq. (18)d/dt [15N / 14N of [NH4+] and 15εAMX is the N isotopic fractionation factor for NH4+ consumption by anammox (where R_AmmoniumN is the  anammox .2– and NH4+ to N2 and NO3– is as follows (The approximate stoichiometry of the anammox process converting NO2– + 1NH4+ → 1N2 + 0.3NO3– + 2H2O --- eq. (19)1.3NOx) together with kAMO14N with concentration data, which provided the AMX, AMXNIR, and NXR fluxes used in the calculation above . We assigned the range from 0 to 60‰  to estimate isotopic fractionation factors. We considered this 60‰ range for the curve-fitting estimate to be reasonable because isotopic fractionation factors larger than 60‰ are rarely observed  and consequently 18EAMXcombined , were not all successfully estimated for Experiments B and C. Based on these uncertainties in parameter estimations, we did not report these calculated values for Experiments B and C; however, we speculate that these calculated parameter sets support 18εAMXNIR as normal and 18εNXR being inverse isotope fractionation, as discussed below for Experiment A. The curve-fitting function (“multiple-fit” in BM software) (–6. BM codes for the anammox model for curve fittings with concentrations and isotopic data are provided in the Zenodo website (https://doi.org/10.5281/zenodo.3895346) and However, this stoichiometry between nitrite removal and nitrate production has been reported to vary d/dt [15NO2–] = – (R_NitriteN × NXR / 15εNXR)+ (R_NitrateN × NAR / 1518εNAR)– (R_NitriteN × DENNIR / 15εDENNIR)– (R_NitriteN × AMXNIR / 15εAMXNIR) --- eq. (21)d/dt [16O2–] = – 2 NXR + 2 NAR – 2 DENNIR – 2 AMXNIR --- eq. (22)d/dt [N1518εNAR + (R_NitrateO × 2 NAR / 1518εNAR) / 18εH2OBRNAR– (R_NitriteO × 2 DENNIR / 18εDENNIR)– (R_NitriteO × 2 AMXNIR / 18εAMXNIR) --- eq. (23a)d/dt [N18εH2OBRNAR is the 18O fractionation factor for the “branching effect”  + 1]d/dt  = NXR – NAR --- eq. (24)d/dt [15NO3–] = (R_NitriteN × NXR / 15εNXR) – (R_NitrateN × NAR / 1518εNAR) --- eq. (25)d/dt [18O16O2–] = 3 NXR – 3 NAR --- eq. (26)d/dt [N18O16O2–] = (R_NitriteO × 2 NXR / 18εNXR)+ (R_WaterO × NXR / 18εH2ONXR)– (R_NitrateO × 3 NAR / 1518εNAR) --- eq. (27)d/dt  and [NO2–] decreased during anammox (δ18ONO2– and δ18ONO3– did not change in Experiment A (δ18ONO3– increased by ~2‰ and δ18ONO2– decreased by ~3‰ in Experiment B (δ18O of H2O (229‰), δ18ONO2– and δ18ONO3– rapidly increased  for Kuenenia stuttgartiensis in batch incubation experiments  (Candidatus Jettenia” with the nirK gene ( 2 ± 2‰) . The simirK gene .15N and 18O fractionation factors during anaerobic nitrite oxidization to NO3– of –77.8‰ for 15εNXR and –20.6‰ for 18εNXR ; defined as  – (18ε / 15ε), where 18ε / 15ε is the ratio of isotopic fractionation for O and N during denitrification, respectively, and assigned as 1; see the inset in 2– was consumed. In the case of denitrification in which AMX / NAR is equal to 0 (indicating no anammox), δ18ONO3– vs δ15NNO3– was set to show a slope of 1 . To elucidate the relationship between Δ, AMX / NAR, and isotopic fractionation factors, we simulated the δ18ONO3– and δ15NNO3– trajectories along with the different AMX / NAR ratios and 15εAMXNXR ; however, Δ depended on AMX / NAR, 15εAMXNXR, and 15εNXR , our simulation exercise revealed that Δ, the offset from the 1:1 relationship between δ15N and δ18O, may be useful for detecting NXR (nitrite oxidation) in denitrifying systems in both freshwater and seawater.We developed an anammox-denitrification model with the estimated isotopic fractionation factors  denitrification with auxiliary Nap NO3– reductase may exhibit a 2:1 rather than 1:1 relationship between δ15N and δ18O, resulting in an offset from the 1:1 relationship  is not considered to be a major environmental sink for NO3–  may induce an offset from the denitrification trajectory . In practice, Δ may be evaluated with time-course samplings or short incubation studies to investigate the occurrence of anammox, similar to denitrification. This technique will be advantageous because of its potential in evaluations of the quantitative contribution in situ of anammox versus denitrification. Although the detection and quantification of functional genes in denitrification and anammox may be readily performed, difficulties are associated with detecting the in situ occurrence of denitrification and anammox. Although the isotopic fractionation factors used also need to be considered, Δ is a promising parameter to complement molecular data and the results from laboratory incubation experiments in the study of anammox.We estimated Microbes Environ 35: ME20031.Kotajima, S., Koba, K., Ikeda, D., Terada, A., Isaka, K., Nishina, K., et al. (2020) Nitrogen and Oxygen Isotope Signatures of Nitrogen Compounds during Anammox in the Laboratory and a Wastewater Treatment Plant. https://doi.org/10.1264/jsme2.ME20031Supplementary Material


This study targeted the association of program characteristics of 203 Doctor of Physical Therapy (DPT) programs in the United States (US) reported by the Commission on Accreditation in Physical Therapy Education (CAPTE) in their 2017 Annual Accreditation Report (AAR) with measures of core faculty research volume. The association of institutional, program, and faculty characteristics of an institution with core faculty research volume was investigated.This observational study analyzed data provided in the AAR about program research volume. Predictor variables included institutional, program and faculty characteristics. Research volume was measured as a ratio of 1) number of peer-reviewed publications, 2) National Institutes of Health (NIH) funding, and 3) faculty with grants, per number of core faculty. Research volume was stratified by quartiles and analyzed using logistic regression analyses. The highest 25% were analyzed against the lowest 75%.In the multivariate logistic regression analyses, research Carnegie classification was positively associated with NIH funding  and number of peer reviewed publications . Square footage of research space was positively associated with number of peer reviewed publications . Private status was negatively associated with NIH funding  and faculty holding grants .There is strong evidence that research culture  is related to research productivity in DPT programs in the US. Private status was indicative of a non-research intensive environment, which may be reflective of a current trend of small, non-research based private institutions initiating DPT programs. An entry-level physical therapy program of education prepares the students to the point of being able to practice as physical therapists in the country of education . Entry rThe Commission on Accreditation in Physical Therapy Education (CAPTE)  collectsScholarly activity is historically low among physical therapy programs , althougScholarly activity is a focus of both CAPTE and the American Physical Therapy Association (APTA) in order to provide evidence-based care , 14. EviThis study was an observational design. The REporting of studies Conducted using Observational Routinely-collected Data (RECORD) Checklist was utilized to guide the reporting of this study.The data used in the study, the most recently organized results from the CAPTE AAR, was obtained from CAPTE-accredited DPT programs that graduated students in 2017. The CAPTE AAR provided institutional and general information about the accredited DPT programs, including characteristics pertaining to the institution, program, and faculty. The Institutional Review Board of the University of North Carolina (ID #18–3059) determined that the data does not constitute human subjects research. Thus, informed consent was waived under federal regulations [45 CFR 46.102 (d or f) and 21 CFR 56.102 (c)(e)(l)].The institutional data comprised the 203 DPT programs within the US that were accredited at the time of the study. Identifiers were masked for the institutions in the data set to conceal their identities, such that investigators could not link an entry to a specific institution. Selective data about each institution was provided based on the core faculty for the DPT program, defined by the AAR as “those individuals appointed to and employed primarily in the program … with the responsibility and the authority related to the curriculum” . The AARThe CAPTE AAR provided a comprehensive list of variables that had the possibility to be included in this analysis. However, because we were interested in the variables that could be indicators of research productivity, only those reflective of institutional, program, and faculty characteristics were considered. An additional glossary describes these terms in more detail  reflective of the institutions, b) reflective of the institutions’ program format, and c) reflective of faculty characteristics. These variables were separated into institutional, program and faculty characteristics for interpretation of the analyses.The institutional characteristics included four variables: Carnegie classification , private status, traditional institution type, and student body size (whole institution). All variables were reported as categorical data, dichotomized as follows: a) Carnegie classification , b) private status (private versus public), c) traditional institution status , and Liberal Arts University] versus all others ) and d) student body size .The program characteristics included nine variables: > 6 year program format, number of terms, total program length, number of credits, classroom education hours, hybrid curriculum, operating budget, total number of courses, and square footage of research space. Program format, operating budget and type of curriculum were reported as categorical data, dichotomized as follows: a) > 6 year program format , b) operating budget  and c) curriculum model . Number of terms, total program length, number of credits, classroom education hours, total number of courses and square footage of research space were reported as continuous data by mean and standard deviation (SD).The faculty characteristics included four variables: total number of vacancies, faculty turnover, faculty to student ratio, and total full-time equivalents. All four variables were reported as continuous data by mean and SD.In order to describe the scholarly culture of the institutions in the dataset from CAPTE, the outcomes representing a proxy for productivity were considered, including peer reviewed publications, National Institutes of Health (NIH) funding, and total number of faculty within the DPT program who reported they had grants were each standardized per core faculty member. Stratification was performed for these three outcomes to separate the top 25% from the bottom 75%, with the intent to capture the highest performing research-intensive institutions in the upper quartile . To our r = 0.00 to 0.30), low positive or negative correlation (r ≥ 0.30 to 0.50), moderate positive or negative correlation (r ≥ 0.50 to 0.70), high positive or negative correlation (> 0.70 to 0.90) and very high positive or negative correlation (> 0.90 to 1.0) .