The collaborative project between the University of Las Palmas de Gran Canaria and the public University of Mozambique, UniZambeze, aims to strengthen the institutional and training capacities of its Faculty of Medicine located in Tete to provide new medical graduates. The essence of the program, training doctors, has the objective of improving the healthcare system and making it sustainable for the integration of new graduates into the staff of the faculty. In this work, we determine the cost of education for a new doctor and we evaluate the social benefit of the program in terms of the human capital. The program has led to the training of 199 new doctors in the 11 years of operation and is leading the way for 100 more in the next four years. The incorporation of some of them into the faculty’s staff will generate new doctor graduates in the near future with a cost below 6000 EUR each in normal circumstances. These results can help to determine how much traditional international aid healthcare programs can save when investing in the education of new doctors. This program is an alternative policy for the international aid financing budgets of donor countries. Supporting teachers and native doctors in the future with private and public patronage programs can raise the efficiency per EUR spent. The essence of the program between the University of Las Palmas de Gran Canaria (ULPGC) and the UniZambeze, training doctors, has the objective of improving the healthcare system in Mozambique and making it sustainable for the integration of new graduates into the staff of the Faculty of Medicine in Tete who will substitute the presently foreign teachers in the future. The project complies with the two goals proposed by the new millennium declaration of the United Nations to reduce poverty in developing countries through health and education. The program is in line with what the World Health Organization considers a priority for cooperation in Mozambique, strengthening and addressing workforce education . The intMozambique has a population of 25.8 million people with a life expectancy of 57.7 years old, which is below the average for African countries . The neonatal mortality rate is 27.79 per 1000 before the 28th day of life . Chronic malnutrition affects 41% of children under five. Mortality under five is 73.22 , with a maternal mortality ratio of 289 per 100,000 live births. Furthermore, 47.15% of the population is under 15 years old, and 70% of the population lives in rural areas. The illiteracy is 60%, and the gross domestic product (GDP) per capita is 387 USD. The health expenditure per capita is 13 USD. Lastly, 54.7% of the population lives under the umbrella of poverty [Whereas Africa has an average of 2–3 health workers per 1000 people , MozambiHealthcare is provided by the public sector, private companies, nongovernmental organizations, and religious entities. The country has 1277 healthcare centers distributed across four levels with 15,877 beds and 26,000 health workers. Level 1 offers essential services, urban and rural healthcare centers, and essential healthcare spots. This represents 96% of the healthcare centers in the country. Level 2 involves district hospitals that represent the reference for first aid care; they can attend to emergencies, lower and obstetric surgeries, and simple traumas. Levels 3 and 4 constitute eight province hospitals and four central and specialized hospitals, two in Maputo, one in Beira, and one in Nampula [In this paper, we present an economic evaluation determining, first, the cost performance of the Faculty of Medicine and the cost of education for a new doctor, as well as how both have developed with the implementation of the program. Second, we calculate the social benefit of the program in terms of the human capital created. Determining the cost performance of the faculty has also made it possible to anticipate the cost of training new doctors in the future when the faculty will be able to work with its own staff. After adjusting for salaries, these results could be applied to other countries with social and health conditions similar to Mozambique. This can help international aid agencies to identify new alternatives to carrying out healthcare programs focused on the education of new doctors and the costs required to develop them.At an international conference in Gran Canaria in 2010, the rector of the UniZambeze requested the support of specialists who could teach the second cycle as part of the Faculty of Medicine in Tete. Two years later, the first 13 teachers of the ULPGC moved to teach in the second cycle of the training program. In 2019, 199 new doctors graduated. Since the program was completed in 2020, the Faculty of Medicine has been functioning entirely with its own staff.The academic year at the Medical School of Tete runs from February to November in two semesters. The training process for new graduates takes seven years of studies, six years in the school and one year of rotation. Rotations involve the practical application of theory by students in hospitals and healthcare centers supervised by a doctor. The training program started in 2009 with 49 students enrolled in the first year. As of November 2019, more than 1000 students were enrolled in the process of training. During this period, the faculty staff was made up of Mozambicans, Cubans, and ULPGC teachers.Financed by both private and public Spanish institutions, between 2012 and 2019, more than 75 teachers migrated from the ULPGC to Tete. They taught 13,400 h of classes in periods of four and six weeks, depending on the specific teaching plan. During this time, five cohorts of students graduated, and the new doctors integrated into the public and private Mozambican healthcare system. As the Faculty continues working, the number of doctors graduating will continue to rise in the future.The faculty started with 21 Mozambican teachers staffing the first cycle. In the subsequent years, as the number of students increased, so did the number of instructors, firstly with Mozambicans and later with Cubans. In 2011, the first six Cuban teachers were incorporated to complete the template. In 2013, coinciding with the beginning of the fifth class of graduates, this number rose to 10. This number remained high until 2016, at which point the new graduates were incorporated as teachers into the Faculty, substituting Cubans. The philosophy of the program was to make the Faculty sustainable and be able to train new physicians internally. In 2018, the number of Cuban teachers was reduced to only two.There is extensive literature discussing how to measure costs of medical education through the use of cost-effectiveness calculations and analyses ,14,15,16Until 2019, the second cycle was exclusively covered by Spanish teachers; however, when the aid program finished in 2020, four Cuban teachers taught in the second cycle that year for the first time. In 2020, the second part of the program of the ULPGC started at the University Hospital of Beira, training new specialists. Some of these specialists are expected to substitute the Cuban teachers in the next four years, making the training process completely independent in the future.In the Faculty template, there are permanent and part-time teachers. Every permanent teacher provides 248 h of lectures per year, whereas part-time teachers provide up to 8 h a week. Cuban teachers are all permanent. For the last 11 years, 43,200 h were taught, with 28,800 h in the first cycle and 14,400 h in the second cycle. In 2020, by the end of the program, Spanish teachers will have taught 14,400 h. While the annual salary of a Mozambican teacher is 6000 EUR, the salary of a Cuban teacher is 28,500 EUR per year (2019 prices), equating to 24.19 and 114.92 EUR per hour, respectively.The teachers of the ULPGC did not receive any salary for their activity at Tete’s School of Medicine. Their travel expenses and insurance were paid for by the aid program, while the UniZambeze supported the expenses of an assistant and the house where they stayed in Tete. Considering all those expenses, the average cost for each Spanish teacher in Tete was 1698 EUR. Each Spanish teacher visited Tete once a year. Following , we addeThe collaboration between the ULPGC and the UniZambeze had effects firstly on the economic performance of the Faculty of Medicine in Tete, making it sustainable, as well as reducing the cost of education for new doctors. Second, there have been effects on the healthcare system. The third benefit is the contribution to the formation of the human capital of the country. In this paper, we only focused on the measurement of the cost performance of the Faculty of Medicine and on the estimation of social benefit derived from the increase in the human capital.Through an analysis of the cost performance and indicators of cost efficiency of the Faculty in terms of teaching resources, we were able to study how the efficiency has developed in terms of the performance of the Faculty per student enrolled and graduated as well as make an estimation of its expected development in the future, both in the short term, when the Spanish teachers leave the second cycle, and in the long term, when the Mozambicans become able to teach in the second cycle, definitively substituting the Cubans teachers.We also calculated how the implementation of the program has developed the cost per student, both enrolled and graduated, for every class of graduates. Differentiating between the first and the second cycle, we were able to measure how the incorporation of new graduate students as teachers in the Faculty changed the efficiency and cost per student enrolled and graduated and how this cost has developed with the composition of the staff. The calculation of the incremental cost-effectiveness ratio (ICER) reports how much the cost of teaching per student has increased, how it is expected to develop in the future, and how the incorporation of new Mozambican teachers benefits the cost-effectiveness per student.The socioeconomic implications of higher education are often assessed using the human capital method when considering the educational institution as a producer of individuals who acquire competences, and this contributes to the economy by increasing the productivity ,20,21, wIn this paper, we approximate the social benefit generated by the increase in the human capital as the difference, for each student graduating during their working life, between the income expected working as a doctor after completion of their higher education at the Faculty  and what they would receive working in the labor market with a lower qualification, i.e., the average salary .The highest total annual cost for the first and second cycles was over 494,000 EUR in 2014, coinciding with the maximum number of Cuban teachers recruited: 294,000 EUR for the first cycle and 67,554 EUR for the second cycle. Later, the replacement of Cuban teachers by Mozambicans reduced the cost of teaching in the first cycle. In 2019, the complete first cycle was covered only by Mozambican assistant teachers with a total annual cost for the first cycle of 69,677 EUR, whereas it was 152,407 EUR in the second cycle, including the salaries of Cuban teachers.As a consequence of the substitution of Cuban teachers, the savings in terms of the total cost between classes of graduates eight and seven was around 46%. The overall cost savings in the program budget became positive for classes of graduates six and seven. With only Mozambicans teaching, the cost of the first cycle for class of graduates nine was the lowest ever achieved, 107,177 EUR, which is also the level expected for subsequent classes of graduates. This reflects the success of the program in terms of the total cost of performance of the Faculty. However, there was a cost increase in the second cycle for class of graduates seven as a consequence of the Spanish teachers leaving the program, who were substituted by Cuban teachers. This will also be mitigated in the future, as soon as the new Mozambican specialists are fully trained and can incorporate into the staff of the Faculty.The minimum total cost per graduate student achieved for one of the five classes of graduates was 5368 EUR, coinciding with a higher number of students graduated and the increasing presence of Mozambican teachers as previously indicated. The cost per student for the fifth class of graduates, 13,839 EUR, presents a large deviation for several reasons. There was a high number of Cuban teachers initially contracted, while the number of students enrolled and graduated fell considerably for that class (26 students). Since class of graduates seven, the number of students has reduced progressively due to a decision by the government arguing that they could not afford new doctors in the public sector due to their budget.The number of students reached the maximum in class of graduates seven (79 students), while also achieving the maximum level in the Faculty in terms of total students. This will have consequences in later years in relation to the cost per student graduated until the effect of the government’s decision comes into play.Class of graduates 11 will be the first to be taught completely by Mozambicans in the first cycle and by Cubans in the second. This is a good representation of future scenarios in the short term at the faculty, at least in the four years after 2020, until new Mozambican specialists are trained and can be part of the staff of the Faculty. In this scenario, The long-term scenario was analyzed by assuming only Mozambicans would be teaching in both cycles, but with 50% extra salaries for teachers in the second cycle. In that case, the cost per enrolled student varies between 5574 EUR if 25 students are enrolled and 1548 EUR if 90 students are enrolled. This cost would be for the whole training program. The cost per graduate would vary between 10,719 and 3096 EUR in the same conditions.By assuming that the new doctors do not receive the training program, i.e., they would be receiving the average salary for a professional ,29, we wThe execution of the program has made it possible to train a new graduate in medicine for a little more than 5300 EUR by combining native and foreign resources. A total of 199 new doctors have been trained in the 11 years of intervention. An additional 100 are expected in the next four years. Although this represents a significant financial effort made by the Mozambican government, in the province of Tete, where there have previously been approximately 100 doctors for two million people, this represents an important impact for the healthcare system. The incorporation of some graduates into the Faculty staff will consolidate the sustainability of the system, as well as make it possible to educate new doctors with a cost below 6000 EUR in the future under normal circumstances and a cost of 3000 EUR in the optimistic long-term scenario.The collaboration program with the UniZambeze ended in 2020; thus, because the training in the second cycle of medical schools in Mozambique must be provided exclusively by specialist doctors, the work of Spanish teachers was substituted by Cubans, thereby increasing the cost of teaching. The new program aimed at training specialist doctors started in 2020, involving the Catholic University of Mozambique and the University Hospital of Beira. This step will be decisive in completely substituting the Cuban teachers, thereby closing the circle and rendering the Faculty completely sustainable and independent.The results of the program have been affected by political decisions during its execution. Once the program began achieving its goals, the Mozambican government decided that it would not be possible to support so many new doctors in the public sector. They decided to cut the number of students enrolled in the Faculty, thus affecting the cost per new doctor and student enrolled and, as a consequence, the efficiency of the project. This external political decision has had strong effects on the efficiency of the program. As the training of students in the Faculty is an important fixed component, and as the number of teachers required to provide teaching remains the same for each course, the cost per enrolled student has increased. However, the cost of teaching 25, 50, 75, or even 90 students in the same room remains fixed, which exposes the political decision of reducing the number of students enrolled as being unreasonable according to the costs involved. It would be desirable for this decision to change in the future, thereby maintaining the cost of education for a new doctor close to 3000 EUR in the long term.The social benefit generated in terms of the human capital will also be affected by the number of students. In the long term, the social benefit expected varies between 2.8 million EUR and 11.5 million EUR per class of graduates for the upper and lower ends of the spectrum in terms of the number of students. The rate of return per class of graduates would be over 120% depending on the class graduated, which can be considered very high in comparison to that expected for higher education in low-income countries (around 27%) .This paper has a limitation in that it does not evaluate the social benefit for the Mozambican healthcare system. Furthermore, the benefit calculated in terms of the human capital cannot be used to substitute this parameter. The social benefit for the human capital must be considered as an additional benefit of the project that would have an impact on the economy and the life conditions in Mozambique. Carrying out a cost–benefit analysis of the effects of this project on the healthcare system would be very complex. In addition to the difficulties in gathering information on treatments received by the population, the project has had a huge impact on the number of new physicians in the country who have found many different occupations, e.g., as teachers in the Faculty or as physicians in primary healthcare systems. Some have continued with their education as specialists at the University Hospital of Beira, while some have become part of nongovernmental organizations and some are working in the private sector. This aid program can be used as an alternative policy for the international aid financing budgets of donor countries. Supporting teachers and native doctors can raise the efficiency per EUR spent with private and public patronage programs in the future. The MEPI program grant of the United States (US) government represents one such example obtaining positive results ,19.The collaborative program between the ULPGC and the UniZambeze has generated a system that provides university teachers to the Mozambique education system and doctors to the healthcare system. Furthermore, the cost of education has been reduced, thereby promoting the access of new students that will sustainably provide new doctors to the healthcare system.The economic evaluation presented in this paper allows approximating the efficiency of such programs in training doctors compared with the cost of supporting physicians from a developed country working in a traditional aid program. Governments and nongovernmental organizations can use this as a reference to determine the efficiency of patronage programs for new students, teachers, and doctors, as a complementary alternative in the short and medium term which, in the long term, could be able to substitute traditional healthcare aid programs, i.e., teaching one how to fish instead of giving them a fish.This type of international aid collaboration generates long-term results; however, it cannot substitute short-term direct aid programs because the African population also currently needs help. This program provides a lower dependency on foreign human resources in the future, thereby extraordinarily improving the availability and regularity of healthcare for the population. The training of new national doctors allows the replacement of foreign doctors, mainly Cubans and Koreans, which also implies a higher cost for the country. The use of local employees for subsequent generations will allow developing the local economy and improving the country’s trade balance.The program presented in this paper has enabled the generation of new doctors in Mozambique, who will continue training and graduating future students. This type of program not only contributes to the education and healthcare system, but also has an important effect on building social and human capital, particularly in a country with the level of poverty seen in Mozambique. Thus, in the long term, the financing and execution of this type of program in higher education will produce additional benefits for the society in terms of economic and sociopolitical development of the country.


Double fortified salt (DFS) is efficacious in addressing iron deficiency, but evidence of its effectiveness is limited. The few published evaluations do not include details on program implementation, limiting their utility for programmatic decisions.We sought to characterize the coverage of a DFS program implemented through the Public Distribution System (PDS) in Uttar Pradesh, India, and understand the drivers of DFS adherence.After 8 mo of implementation, we surveyed 1202 households in 5 districts and collected data on sociodemographic characteristics, asset ownership, food security, and regular PDS utilization. We defined DFS program coverage as the proportion of PDS beneficiaries who had heard of and purchased DFS, and we defined DFS adherence as DFS use reported by households. We used principal component analysis to create an asset-based index of relative wealth, and we categorized households into higher/lower relative wealth quintiles. We conducted path analyses to examine the drivers of DFS adherence, particularly the mediated influence of household wealth on DFS adherence. The evaluation is registered with 3ie's Registry for International Development Impact Evaluations (RIDIE‐STUDY‐ID‐58f6eeb45c050).The DFS program had good coverage: 83% of respondents had heard of DFS and 74% had purchased it at least once. However, only 23% exclusively used DFS. Respondents had low awareness about DFS benefits and considered DFS quality as poor. Being in a lower household wealth quintile and being food insecure were significant drivers of DFS adherence, and regular PDS utilization acted as a mediator. Adherence was lower in urban areas.We observed significant heterogeneity in DFS implementation as reflected by high coverage and low adherence. Findings from this process evaluation informed the design of an adaptive impact evaluation and provided generalizable insights for ensuring that the potential for impact is realized. Efforts are needed to increase awareness, improve product quality, as well as mitigate against the sensory challenges identified. Addressing micronutrient deficiencies through food fortification is cost-effective , and oneIt is posited that countries with successful experience implementing universal salt iodization but struggling with high iron deficiency due to inadequate intakes may gain from adding iron to the salt. Double fortified salt (DFS) is the dual fortification of salt with iodine and iron that simultaneously targets both micronutrient deficiencies . The effOnly 2 studies have evaluated the effectiveness of DFS to date, highlighting a gap in evidence. Both studies were performed in Bihar, India. DFS distribution in 1 program was via school feeding programs , whereasIndia's National Family Health Survey 2015–2016 reported that in UP, 53% of nonpregnant women of reproductive age of 15–49 y (WRA) and 63% of preschool-aged children of 6–59 mo (PSC) were anemic . SeveralThe PDS is a social safety net program in India, and PDS shops in UP distribute rations that include subsidized rice, wheat, and kerosene to eligible households every month. PDS eligibility is determined by the state government, and the lowest income households are categorized as Antyodaya Anna Yojana (AAY) cardholders, whereas slightly better off households are categorized as Priority Household (PHH) cardholders. After a recent restructuring of the PDS, 25% of rural households and 50% of urban households are not covered by the safety net program , and endline assessments. Five of the 10 DFS program districts were chosen using simple random sampling for the evaluation. In conjunction with examining DFS effectiveness through baseline and endline assessments, the process evaluation specifically focused on assessing the implementation of this fortification strategy. We developed a program impact pathway and conducted a coverage survey. In this article, we present data on the coverage of the UP DFS program and assess the drivers of adherence. In addition, these data were used to determine whether to conduct the endline assessment for the DFS program, using an a priori evaluability threshold, decided based on similar fortification evaluations  in whichTwenty clusters of villages or wards (urban neighborhoods in India) were selected using stratified random sampling and population proportion to size for the baseline survey and revisited for the coverage survey, between November and December 2018, after 8 mo of DFS rollout. Of the 20 clusters, 5 each were randomly selected from villages and wards from within the entire district and 10 consisted of villages or wards randomly selected from border areas (within 20 km of the district border). Within each ward, 1 census enumeration block (CEB) was randomly selected using the 2011 Census of India data . After aThe eligibility criteria for the baseline survey were maintained for the coverage survey. PDS cardholder households with at least 1 nonpregnant WRA and a PSC were interviewed; the primary respondent was the nonpregnant WRA. If more than 1 eligible WRA lived in the household, 1 was chosen randomly as the respondent. An additional household each was interviewed in 2 of the 5 districts due to oversampling, and these interviews were retained for the analyses, resulting in a final sample of 1202 interviews.Supplemental Table 1), and we considered responses that mentioned “good health or nutrition,” “anemia prevention,” or “goiter prevention,” as having at least partial awareness of DFS benefits.We assessed sociodemographic characteristics, housing conditions, asset ownership, and food security. We identified different salt types in household kitchens, distinguished based on packaging information: DFS from the PDS, iodized packaged salt from retail shops, or loose crystal salt from informal markets. We categorized caste—a symbol of social status—based on the classification provided by the Indian constitution, and we further categorized them into groups of higher and lower caste . AdditioWe measured wealth using household assets as a proxy, and we used principal component analysis for variables representing 36 assets that included household goods and vehicles, livestock as well as property ownership, and attributes of housing structure such as construction quality and light and fuel sources to create household wealth quintiles. A dichotomous variable was created in which households in the lowest 2 quintiles constituted the lower wealth category, whereas the rest belonged to the higher wealth group.Although PDS rations were available every month, some cardholder households did not regularly utilize the PDS for monthly purchase of subsidized rations. We therefore asked participants whether they regularly utilized their PDS cards and categorized households of those who responded affirmatively as regular PDS utilizers. We measured food insecurity using the Household Food Insecurity and Access Scale  and cateAdapting the Tanahashi framework on evaluating health service coverage, we examined a cascade of varying degrees of DFS program coverage –33. ThisFor all path analyses, the outcome variable of interest was complete DFS adherence because it was most likely to capture the DFS program's potential to benefit. We examined the relation between household wealth and complete DFS adherence and theorized a path model . Except for 1 ordinal variable (household head education), all other variables in the path model were dichotomous.We used descriptive analyses to assess individual characteristics, such as respondent age and education levels, and household characteristics, including family size, primary income source, access to facilities, religion, and caste. We examined and retained all outlier values after scrutinizing related variables and determining plausibility. Examination of bivariate associations and confounding assessment helped build the final path model, adjusted for household religion and household head's education. We examined all variables in the path model for missing data and found that values for household head's education were missing for 11 interviews. We adopted a listwise deletion approach to account for missing data  after deWe performed preliminary analyses using SAS software version 9.4 and weighted path analyses in R version 3.5.3 . We utilized survey methods and structural equation modeling approaches , adopting the diagonally weighted least-squares method to account for both ordered and dichotomous variables to conduct a path analysis. All standardized coefficients were exponentiated to obtain the adjusted OR (aOR); indirect and total effects were calculated. We assessed goodness of model fit using chi-square, root mean square error of approximation (RMSEA), comparative fit index (CFI), and standardized root mean squared residual (SRMR). The RMSEA, CFI, and SRMR suggest reasonably good model fit when they are <0.08, >0.90, and <0.08, respectively , 36.Institutional review boards at Sanjay Gandhi Post Graduate Institute of Medical Sciences and Emory University reviewed and approved the data collection and analyses protocol. The evaluation is registered with 3ie's Registry for International Development Impact Evaluations (RIDIE-STUDY-ID-58f6eeb45c050).P < 0.001). Urban households were also more likely to have access to electricity, piped water, and gas stoves. Nearly half the rural households experienced moderate to severe food insecurity . We also noted that more households in rural areas regularly utilized the PDS compared with those in urban settings. Two-thirds of sampled households found DFS quality to be poor, but a higher proportion of urban households noted the quality of PDS rice and DFS to be poor.Sociodemographic characteristics of the sample are presented in Supplemental Table 4), with 2 districts indicating higher adherence especially in the rural areas.P value (chi-square) = 0.003, RMSEA = 0.07, CFI = 0.99, and SRMR = 0.01]  < 0.001, RMSEA = 0.35, CFI = 0.99, and SRMR = 0.02]. Subsequent model respecifications of the urban model based on theoretically vetted modification indices resulted in nonconvergence, and we therefore do not present the urban model in this article.After removing the 11 households with missing information on household head education, the final sample for path models was 1191 households. Findings from the path model are illustrated in  = 0.01] , 36, andn = 861), the effects in the total sample were qualitatively similar to the effects in rural models. We discuss only the rural effects here; the direct effects for the total model are presented in Supplemental Figure 1. Standardized direct effects in the path model for the rural sample  showed urban households to be largely concentrated in the higher 2 relative wealth quintiles. Compared with rural households, urban households relied less on subsidized rations and were more likely to self-select themselves out of the PDS. Our rural path model indicates that lower household wealth, regular PDS utilization, and DFS awareness were strong drivers of DFS use. Building awareness around DFS benefits worked in this context, in which DFS adherence levels were higher among those who positively perceived DFS.These findings reveal 3 lessons for the UP DFS program. First, there is a need to overcome DFS product limitations and address quality concerns raised by users. DFS production is complex, with formulations for iron compounds constantly evolving. Four iron formulations for DFS currently exist (10), and the UP program tested 1 of these, which showed promise of addressing the discoloration problems faced in other DFS trials. However, as the program rolled out, DFS quality was compromised due to the significant investments required to produce and blend the iron formulation with high-grade refined salt, high production costs, and a lack of standards for producing extruded iron compounds . In our Second, the DFS program should reinforce efforts to boost awareness regarding the safety and benefits of DFS. Awareness creation efforts in the New Delhi DFS sensory trial showed that >85% of the participants were willing to use DFS after learning about its benefits . AlthougFinally, it is important to consider strategies to expand DFS access and availability in urban areas, where 50% of the population is not covered by the PDS. Simultaneously, the need for complementary strategies, including non-DFS interventions , 43, shoThese findings must be interpreted within the limitations of this study. There could be a social desirability bias—in either direction—for questions regarding PDS purchases and product quality rating because we informed respondents prior to interviews that we were evaluating the DFS program to improve it. Second, due to the cross-sectional study design, it is difficult to establish temporal relations with household wealth and food security. Third, the urban model fit was poor, indicating the possibility of an unmeasured mechanism and pointing to the need for more research, including qualitative interviews, to understand other potential drivers of DFS adherence in urban contexts—for example, increased access to retail markets. Fourth, although bundling of DFS with PDS rations was reported by the sampled population, the coverage survey questionnaire did not adequately capture this information to quantify the exact prevalence of bundling. However, we were able to use this insight to further examine PDS bundling in our qualitative assessments. Despite these limitations, the path models and coverage analyses provide important lessons for program implementers to improve coverage and adherence of DFS. Although not generalizable to all DFS programs and contexts, this evaluation of the UP DFS program has broader implications for the design and implementation of any nutrient intervention, especially those that use social safety nets as delivery platforms.Insights on implementation fidelity are critical to interpret findings and inform the design of future evaluations. We used the results to inform programmatic course correction and assess the evaluability of the program . ExaminiOur process and findings provide important information for conducting and using implementation research for designing and evaluating nutrient interventions at scale. At this stage of the UP program, the potential for measurable benefits is constrained by the low rates of DFS adherence. This study reveals implementation issues that the UP program and DFS programs globally must address. Targeted efforts to improve adherence are needed: addressing product quality issues, investing in well-designed awareness campaigns that ensure behavior change , and strnzaa133_Supplemental_FileClick here for additional data file.


ARID1A-deficient GCT cells and demonstrate by mass spectrometry that ARID1A is putatively involved in regulating transcription, DNA repair and the epigenetic landscape via DNA Polymerase POLE and the DNA methyltransferase 1-associated protein DMAP1. Additionally, ARID1A/ARID1A deficiency or pharmacological inhibition increased the efficacy of romidepsin and considerably sensitized GCT cells, including cisplatin-resistant subclones, towards ATR inhibition. Thus, targeting ARID1A in combination with romidepsin and ATR inhibitors presents as a new putative option to treat GCTs.Germ cell tumors (GCTs) are the most common solid malignancies found in young men. Although they generally have high cure rates, metastases, resistance to cisplatin-based therapy, and late toxicities still represent a lethal threat, arguing for the need of new therapeutic options. In a previous study, we identified downregulation of the chromatin-remodeling SWI/SNF complex member ARID1A as a key event in the mode of action of the histone deacetylase inhibitor romidepsin. Additionally, the loss-of-function mutations re-sensitize different tumor types to various drugs, like EZH2-, PARP-, HDAC-, HSP90- or ATR-inhibitors. Thus, ARID1A presents as a promising target for synthetic lethality and combination therapy. In this study, we deciphered the molecular function of ARID1A and screened for the potential of two pharmacological ARID1A inhibitors as a new therapeutic strategy to treat GCTs. By CRISPR/Cas9, we generated  Testicular type II germ cell tumors (GCTs) are classified into seminomas and non-seminomas. Both entities are thought to arise from a precursor lesion, the so called germ cell neoplasia in situ (GCNIS), which itself is the result of a defective  germ cell development ,2. GCTs in vitro and in vivo, but not non-cancerous control cells ,20ARID1A 0.4 μM) .Of note, treatment of TCam-2 cells with 2.5 μM C63 or 10 μM VE-822 24 h before addition of 1 nM romidepsin reduced viability compared to mono-application only slightly , while cWe extended our analysis to the EC cell lines NCCIT and NT2/D1 as well as their cisplatin-resistant subclones  and the choriocarcinoma cell line JAR (+2.5 μM C63 and 5 μM VE-822) D. We conATR, ATM, ETAA1, TOP2A, TOPBP1, and CHEK1. These factors play an important role in the impaired DNA damage response related to the ATR pathway in connection to SWI/SNF complex deficiencies [ARID1A−/− compared to -ARID1A+/+ cells, treatment with VE-822 led to an increased number of γH2AX-positive cells (not significant), indicative of DNA double-strand-breaks  and 41 depleted   or depleted  between ARID1A-deficent and C63-treated cells  . The dep others) . Thus, ted cells . FA2H isARID1A-deficiency or -inhibition on the expression of pluripotency factors [ARID1A−/−, 2102EP-ARID1A+/- and NCCIT-ARID1A+/− cells showed no considerable deregulation of NANOG, OCT3/4 or SOX2/SOX17 . In GCTs, ARID1A mutations are extremely rare, so we asked, if a romidepsin-mediated ARID1A downregulation, a CRISPR/Cas9-mediated ARID1A deficiency, or a pharmacological inhibition of ARID1A might phenocopy these effects of the ARID1A mutation and sensitize GCT cells to the various previously mentioned inhibitors. While ARID1A deficiency had no effect on HSP90, PARP, HDAC6, and EZH2 inhibition, the efficacy of the HDAC inhibitor romidepsin and ATR inhibitor VE-822 was significantly enhanced /pharmaced cells  . All type I GCT samples were cryopreserved and anonymized during the MAKEI 96-study and provided by the MAKEI 96-biobank with no concerns raised by EC-HHU-D about analyzing these samples (ethic votes 837 and 2019-822).The ethics committee of the Heinrich-Heine-University Düsseldorf (EC-HHU-D) raised no concerns about using all analyzed cell lines for All GCT cell lines were cultivated as described previously ,31. The The ARID1A inhibitors compound 63 and BRD-K98645985 (Prof. Dr. Emily Dykhuizen), the EZH2-inhibitor GSK126 , the PARP-inhibitors olaparib and rucaparib , the ATR inhibitors VE-822 and AZD6738 , the HSP90 inhibitors PU-H71 and 17-AAG , the HDAC6 inhibitors YAK61 and KSK64  were diARID1A were generated by the CRISPR/Cas9 technique as described previously [5 TCam-2 cells were transfected simultaneously with 1 μg pX330 vector encoding for three different guideRNAs directed against the ARID1A gene locus  and 100 ng of a GFP-encoding plasmid (pEGFP-N3). Clones were screened for deletions within the ARID1A gene locus by PCR. For genotyping strategy, see TCam-2 cells deficient for eviously ,32,33. B3 cells were plated onto 96-well plates 24 h before inhibitor or solvent application. For in a total of 96 h, every day viability was screened by adding 50 μL XTT (1 mg/mL) plus 0.5 μL phenazine methosulfate   and measuring absorbance 4 h later in a UV/VIS spectrometer . Each time point/concentration was measured in quadruplicates.The XTT assays were performed as described previously . In summ4 cells using the MACSQuant Analyser and evaluated by the Flowlogic software (both Miltenyi Biotech).Flow cytometry analyses of apoptosis rates and cell cycle distribution were performed by AnnexinV/propidium iodide (PI) or PI-staining, respectively. Briefly, for PI staining, cells were washed once with PBS and fixed with 70% methanol dissolved in PBS. After washing, cells were re-suspended in staining buffer, containing 2 μg/mL PI (Sigma-Aldrich) and 100 μg/mL RNAse A  dissolved in PBS before measurement. For apoptosis measurement, cells were washed with Annexin V-binding buffer  before incubation with 2.5 μL Annexin V-FITC (Miltenyi) and 7.5 μL PI  dissolved in 70 µL Annexin V-binding buffer for 15 min at room temperature (RT) in the dark. Afterwards, samples were re-suspended in 500 µL Annexin V-binding buffer. Measurement of cell proliferation was performed using the EdU Flow Cytometry Kit detecting Eterneon-Red 645 Azide according to the manufacturer’s protocol . A flow cytometric analysis was performed by measuring at least 5 × 104 cells/well were seeded onto 96-well plates. After 24 h, cells were treated with either VE-822 or DMSO as a solvent control for additional 24 h. For staining of phospho-Histone H2A.X Ser139 , cells were fixed by 4% formaldehyde for 10 min. Afterwards, cells were permeabilized by 0.3% Triton-X in PBS for 10 min and blocked in PBS  for 1 h at RT. The primary antibody (1:50) was incubated in PBS staining solution  over night at 4 °C. The secondary antibody was added for 1 h in the dark at RT. Cells were counter-stained with 0.5 µg/mL 4′,6-Diamidine-2′-phenylindole dihydrochloride . See For immunofluorescent staining (IF), 5 × 103 cells were seeded out.Cellular growth rates were determined by counting cell numbers every day over eight days (each in triplicates). Initially, 4 × 10For RNA extraction, cells were harvested and RNA was isolated using the RNAeasy Mini Kit  according to the manufacturer’s protocol. For protein extraction, cells were incubated with RIPA buffer containing 10% protease- and phosphatase-inhibitors (Sigma-Aldrich). Protein concentration was assessed by BCA Protein Assay Reagent Kit .An amount of 20 µg of whole protein lysates was used for western blotting. Staining with Ponceau S-solution  confirmed a correct membrane transfer. Membranes were blocked in 5% non-fat milk in PBS + 1% Tween20 (PBST) for 1 h and then incubated with primary antibodies  over nigEscherichia coli bacteria were transformed according to the ‘TA cloning kit’ manual. Plasmid DNA was isolated from bacteria mini-preps using the ZR plasmid miniprep kit . An EcoR1-HF restriction enzyme  double digest of 1 μg plasmid DNA for 30 min at 37 °C, followed by agarose-gel-electrophoresis, confirmed insertion of the PCR product into the plasmid. Plasmids were sequenced by M13 primers at the ‘Genomics & Transcriptomics Laboratory’ .For Sanger sequencing of ARID1A-deficient clones, PCR amplified DNA was cloned into the pCR2.1 vector using the ‘TA cloning kit’ (Thermo Fisher Scientific) and following the manual guidelines (ratio insert DNA to vector: 3:1). For genotyping primers see g and 4 °C, supernatants were collected. Protein concentration was determined by means of Pierce 660 nm Protein Assay (Thermo Fischer Scientific) and 10 µg protein per sample were loaded on a SDS-PAGE for in-gel-digestion. The isolated gel pieces were reduced , alkylated  and underwent afterwards tryptic digestion . The peptides were resolved in 0.1% trifluoracetic acid and subjected to liquid chromatography.Proteins were extracted from frozen cell pellets as described elsewhere . BrieflyFor the LC-MS analysis, a QExactive plus (Thermo Fisher Scientific) connected with an Ultimate 3000 Rapid Separation liquid chromatography system  equipped with an Acclaim PepMap 100 C18 column  was applied. The length of the LC gradient was 120 min. The mass spectrometer was operating in positive mode and coupled with a nano electrospray ionization source. The capillary temperature was set to 250 °C and source voltage to 1.4 kV. In the QExactive plus mass spectrometer for the survey scans, a mass ranging from 200 to 2000 m/z at a resolution of 140,000 was used. The automatic gain control was set to 3,000,000 and the maximum fill time was 50 ms. The ten most intensive peptide ions were isolated and fragmented by high-energy collision dissociation.Proteome Discoverer  was applied for peptide/protein identification applying Mascot  as search engine employing the human database . A false discovery rate of 1% on peptide level was set as identification threshold. Proteins were quantified with Progenesis QI for Proteomics . Only proteins containing at least two unique peptides were taken into consideration. For the calculation of enriched proteins in the groups a 5% false discovery rate and a minimum fold change of ≥2 was used.The mass spectrometry proteomics data were deposited to the ProteomeXchange Consortium via the PRIDE partner repository with the data set identifier PXD017898 .A total amount of 1µg of RNA was in-vitro transcribed into cDNA by using 1 µL dNTP-Mix (10 mM), 1 µL Oligo(dT)18 Primer (0.5 µg/µL), 4 µL RT Buffer (5×), 1 µL Maxima H Minus Reverse Transcriptase (200U/µL), and 0.5 µL RiboLock RNase Inhibitor (40U/µL)  on a S1000 cycler (BioRad) at 52 °C for 0.5 h. Further, qRT-PCR runs were performed on a 384-well C1000 cycler (BioRad). In general, all samples were analyzed in technical triplicates using 7.34 ng of cDNA for each replicate and the SYBR-green-based Luna Universal qPCR Master Mix . At the end of each run, melting curve analyses were performed. Oligonucleotide sequences are given in SOX2 and SOX17 in seminomas  and ECs . The Illumina 450k DNA methylation arrays were generated in a previous study and re-analyzed in context of this study (GSE76709) [The Illumina and Affymetrix expression array analyses of GCT cell lines  and tissues  were performed previously and re-analyzed in context of this study ,37,38,39https://bioinfogp.cnb.csic.es/tools/venny/) [https://string-db.org) [https://david.ncifcrf.gov/home.jsp) [https://xena.ucsc.edu), FireBrowse (http://firebrowse.org) and the cBioPortal  [https://www.socscistatistics.com/tests/studentttest/default.aspx) using a Student’s two-tailed t-test.Venn diagrams were generated using Venny 2.1 (/venny/) . The STR-db.org) . Functioome.jsp) ,43. For tal.org) ,45. Testp-values of * < 0.05, ** < 0.005 and *** < 0.0005 were assumed as ‘statistically significant’. Additionally, p-values were corrected for multiple testing by the Bonferroni method . p-values below the significance threshold after correction are labelled in green.Where appropriate, two tailed t-tests were calculated with α = 5%. In this study, we deciphered the molecular mode of action of the romidepsin-effector and SWI/SNF-complex member ARID1A in GCTs. We demonstrated that ARID1A might be involved in regulating transcription and the epigenetic landscape, putatively via DNA polymerase POLE and the DNMT-, HDAC- and PrC-interacting DMAP1. Furthermore, pharmacological inhibition of ARID1A increased efficacy or romidepsin and sensitized GCT cells to ATR inhibition. In summary, we added new pieces to the puzzle of ARID1A’s complex molecular functions in GCTs and highlighted ARID1A as an interesting target for combinatorial treatment with HDAC class I and ATR inhibitors.This study also has some limitations we would like to summarize here.ARID1A knock-out: based on the detection of protein fragments smaller than the ARID1A wild-type protein (270 kDa) by western blotting, we cannot rule out leftovers of a truncated ARID1A protein with unknown functionality. Additionally, although the ARID1A guideRNAs target exon 2 and 4 off all ARID1A isoforms, we cannot fully exclude that remaining fragments of other ARID1A isoforms or related proteins, e.g., ARID1B are redundant to ARID1A and might take over its function.Regarding our CRISPR/Cas9-based Furthermore, seminoma-like TCam-2 cells might not be an ideal model to study DNA methylation levels of ARID1A in seminomas, since TCam-2 harbor untypically high levels of DNA methylation.To confirm regulation of DMAP1 and POLE by ARID1A, further functional experiments will be necessary in the future.ARID1A, were present to a certain degree. Although, expression of typical immune cell marker genes was very low in analyzed tissues and expression data of tissues was highly similar to cell lines (The expression microarray data re-analyzed in this study, has been gathered from GCT tissues. Although, all tissues were stained by IHC for related marker genes and checked by a pathologist, we cannot fully exclude that immune cells, i.e., lymphocytes that express ll lines A.To confirm that targeting ARID1A and ATR inhibition presents as a therapeutic option, testing of more cell lines of different GCT entities as well as xenograft experiments is mandatory.


Randomized controlled trials (RCTs) with vigorous study designs are vital for determining the efficacy of treatments. Despite the high internal validity attributed to RCTs, external validity concerns limit the generalizability of results to the general population. Bias can be introduced, for example, when study participants who self-select into a trial are more motivated to comply with study conditions than are other individuals. These external validity considerations extend to e-mental health (eMH) research, especially when eMH tools are designed for public access and provide minimal or no supervision.Clustering techniques were employed to identify engagement profiles of RCT participants and community users of a self-guided eMH program. This exploratory approach inspected actual, not theorized, RCT participant and community user engagement patterns. Both samples had access to the eMH program over the same time period and received identical usage recommendations on the eMH program website. The aim of this study is to help gauge expectations of similarities and differences in usage behaviors of an eMH tool across evaluation and naturalistic contexts.Australian adults signed up to myCompass, a self-guided online treatment program created to reduce mild to moderate symptoms of negative emotions. They did so either by being part of an RCT onboarding  or by accessing the program freely on the internet  between October 2011 and October 2012. During registration, RCT participants and community users provided basic demographic information. Usage metrics  were recorded by the system.P=.003), with community users being on average 3 years older  than RCT participants . Furthermore, frequency of program use was higher for RCT participants on all usage metrics compared to community users through the first 49 days after registration . Two-step cluster analyses revealed 3 user groups in the RCT sample  and 2 user groups in the community samples (2-Timers and 20-Timers). Groups seemed comparable in patterns of use but differed in magnitude, with RCT participant usage groups showing more frequent engagement than community usage groups. Only the high-usage group among RCT participants approached myCompass usage recommendations.Samples at sign-up differed significantly in age (Findings suggested that external validity concerns of RCT designs may arise with regards to the predicted magnitude of eMH program use rather than overall usage styles. Following up RCT nonstarters may help provide unique insights into why individuals choose not to engage with an eMH program despite generally being willing to participate in an eMH evaluation study. Overestimating frequency of engagement with eMH tools may have theoretical implications and potentially impact economic considerations for plans to disseminate these tools to the general public. Well-designed randomized controlled trials (RCTs) are widely seen as the gold standard for determining treatment efficacy as random assignment to either a treatment or control group allows for the isolation of the treatment effect from both known and unknown confounding factors . AlthougSuch external validity considerations may be even more justified when considering those RCTs that evaluate e-mental health (eMH) programs, which are designed to deliver effective, scalable mental health care in the community -6. For eDespite the need for eMH programs to be applicable to real-world conditions, information on the external validity of eMH treatment outcomes is not widely available. In a recent systematic review of digital interventions addressing comorbid depressive symptoms and substance use, only 1 of the 6 studies examined reported on the comparability of the sample used to the wider population . SimilarThis problem extends to eMH program design. A considerable body of research examining the effects of different eMH design features on behavioral changes has not addressed the question of how applicable these findings are once the programs are disseminated . Althoug, side by side. Our aim is to explore whether patterns of program engagement differ between RCT participants who receive usage recommendations as part of being involved in an evaluation study versus users in the general community who receive the same usage recommendations only on the myCompass homepage. The goal of this paper is to examine engagement rather than outcomes; therefore, we are presenting cluster analysis findings that help visualize behavioral patterns rather than quantify differences in health and well-being.This short paper attempts to deepen the discussion on the influence of the RCT environment on eMH engagement behavior by presenting engagement patterns of RCT participants and community users of an eMH tool, called myCompassThe present analyses are based on the first version of myCompass, an eMH program that was available to all Australians between 2011 and 2018. myCompass is hosted and run by the Black Dog Institute, and funded by the Australian Department of Health. Version 1 of this self-guided program offered online mental health resources and activities to address mild to moderate symptoms of depression, anxiety, and stress. Core functionalities of myCompass were the daily tracking and learning activities components. The tracking function allowed users to track up to 3 moods, behaviors, or cognitions  in real time. Users indicated their current states on an interval scale from low (0) to high (10). Another main function of myCompass was learning activities. Learning activities were a set of 14 modules which aimed at aiding beneficial behaviors such as goal setting, sleep quality, or managing fear and anxiety. Each module was split into 2 to 3 sessions to promote skill-building exercises over the course of several days and took about 10 to 15 minutes to complete online.Our study considered engagement data from 2 distinct samples: (1) an RCT participant sample and (2) a naturalistic community sample of general public users who freely adopted myCompass. All participants and users registered to version 1.0 of myCompass between October 2011 and October 2012. Trial participants and community users freely registered either to the research study or directly to the myCompass website. Trial recruitment took place through multiple avenues, such as social media posts on Facebook and announcements on the Black Dog Institute volunteer research register , while tThe RCT sample comprised 231 participants allocated to the eMH treatment group for the initial evaluation study of myCompass . ExclusiThe community sample comprised 8391 adults who registered for myCompass on their own accord. Community members needed to provide a valid email address and mobile phone number to verify their willingness to register to the program. As part of the profile setup within the myCompass program, RCT participants and community users alike completed assessments of common mental health symptoms, which formed the basis for tracking recommendations made by the program . If scorWe selected 4 usage metrics to assess user engagement. These were number of logins, number of logged trackings (irrespective of how many items were tracked at any one time), number of learning activities started, and number of learning activities completed.t tests. We then conducted 2 two-step cluster analyses to determine distinct usage groups within the RCT participant and community user samples. A two-step cluster analysis allows for the detection of naturalistic  groupings within a data set by way of examining distances between data points (step 1). Based on these distance calculations, an algorithm (step 2) determines the number of groupings or clusters for each data set. In the current analysis, we selected the log-likelihood distance measure and the Schwarz Bayesian information criterion statistic to determine the most appropriate number of clusters.All analyses were conducted in SPSS version 25 (IBM Corp). Participants using myCompass as part of the RCT and those in the community sample were initially compared using chi-square and P=.30), community users were on average 3 years older than RCT participants (P=.003). Notably, all average access and engagements statistics were higher for RCT participants than for general public users .Data were available for 8622 users of myCompass. P values <.001; see A number of RCT participants allocated to the myCompass group did not proceed to register to myCompass  and therefore did not provide any engagement data. To make the RCT group more comparable to the general public sample (where each person registered to myCompass and provided engagement data), we removed these participants from the sample used for the cluster analysis. The first and larger user group  were “10-Timers” who logged into their assigned program around 9 times  and used myCompass mainly for tracking  rather than completing learning activities . The second user group  were “30+-Timers”, who logged into myCompass every other day . When the 30+-Timers logged on, they used the program’s tracking function  in addition to starting and completing around three learning activities  over the course of 7 weeks.P values <.001). The vast majority of general public users  were “2-Timers” or individuals who entered myCompass approximately twice  and used the tracking function once . The average module completion neared zero in this group .The second and considerably smaller group were the “20-Timers”  who used myCompass consistently. Members of this group logged in on average 22 times  and used the tracking function the majority of the time  over a 7-week period. In addition, 20-Timers started about 2 modules  and completed 1  during this time.Graphical representations of the cluster solutions can be found in Figures S1 and S2 in This paper presents findings on individual usage behaviors for an Australian eMH tool, either as part of an RCT or as an open-access tool freely adopted by the general public. Exploratory findings reveal that the same number of usage groups emerged in both data sets: a large “lower-intensity” usage group and a smaller “higher-intensity” usage group. However, our findings revealed interesting differences between the 2 data sets that warrant consideration. First, the general community group tended to be older than the RCT group and overall used the eMH program significantly less frequently based on all usage metrics. Of further note, a considerable number of individuals registered to myCompass directly and were not part of the research trial. This could be because participation in a research study is more time intensive, as research volunteers are required to complete psychometric measures in addition to using the eMH tool. It is also possible that only a relatively smaller number of individuals were exposed to the research trial recruitment calls, while a greater number of interested individuals were able to discover the myCompass website using internet search engines.Second, although the cluster analytical findings revealed 2 behavioral groups across both samples, the magnitude of usage was higher in the RCT sample for both usage groups. For example, the low-usage group in the RCT logged in an average of 9 times, while the low-usage group in the general community logged in only about twice on average. The high-usage group in the RCT sample was, again, not only higher in magnitude , but also proportionally bigger than that in the community sample. Specifically, 27.8% (44/158) of RCT users were identified as frequent users (30+-Timers), whereas only 8.46% (710/8391) of general public users were identified as such (20-Timers). Accordingly, the 30+-Timers RCT group completed the recommended 2 or more learning activities and came closest to the tracking recommendations of 49 logged trackings, whereas the 20-Timers community sample group clearly did not meet the tracking or learning activity recommendations. Thus, only the high-usage RCT group could be described as “adhering” to the learning activity recommendations, and no group adhered to the tracking recommendations.Our findings add weight to Cavanagh’s  concernsOur study is unique in that we contrasted RCT and general public behavioral patterns for the same eMH program during the same time period, maximizing the comparability of users’ eMH experience while minimizing the influence of historical factors. However, some limitations of our analyses warrant consideration. First, our analytic techniques only allowed for a limited number of engagement variables, but many other variables, such as the number of tracking reminders a user sets, could have provided us with a more detailed picture of eMH engagement behavior. Second, we were only able to study those users who actually engaged with the core functionalities of the program repeatedly; therefore, our findings largely reflect individuals motivated to adopt an eMH program. Third, the data reported in this paper were collected between 2011 and 2012 and certainly would have made a timelier contribution then. Technological advances since this time include improvements in interface design and user experience features, such as chatbots, gamification, and virtual reality ,16. HoweIn summary, our findings suggest that eMH engagement in RCTs likely matches the type of eMH engagement in real-world users, but may overestimate the magnitude of such engagement. This could be an important consideration for eMH researchers, designers, and policy makers, as they implement eMH tools after efficacy is established. We recommend that future eMH trials examine whether participant selection and per protocol instructions affect usage behavior, and if so, that due consideration be given to this in implementation planning. Similarly, those planning a wide-scale rollout of new eMH tools should consider which aspects of the original trial may help in promoting usage and whether similar methods can be used in the real world. Ecological validity in eMH engagement science is also relevant to theoretical investigations of how eMH tools improve individuals’ well-being. Mechanisms of change established in RCTs must be practicable in the real world for eMH to deliver on its promise of effective and scalable mental health care. Ultimately, the goal of improving eMH engagement science is to set realistic expectations of eMH benefits—both health and economic—and understand how to maximize these. The more accurately we can speak to eMH engagement, the more fruitful both eMH science and policy will be going forward.


Childhood cancers previously considered to be incurable now have 5-year survival rates up to 84%. Nevertheless, these patients remain at risk of morbidity and mortality from therapy-related complications. Thus, patient education and self-management strategies for promoting a healthy lifestyle are of tantamount importance for improving short- and long-term health outcomes. A Facebook-based “Healthy Teens for Soaam” (a Korean term for childhood cancers) program was developed to help improve knowledge and self-management practices of teens with cancer related to their disease and treatment.The two-fold purpose of this usability study was (1) to describe the process of developing an 8-week Facebook-based intervention program for teens with cancer, and (2) to evaluate its usability to refine the program.Multiple phases and methods were employed to develop and evaluate the usability of the program. Study phases included: (1) needs assessment through focus group interviews and qualitative content analysis, (2) development of module content, (3) expert review and feedback on module content, (4) Facebook-based program development, (5) usability evaluation by heuristic evaluation, (6) usability evaluation by targeted end-user testing, and (7) modification and final version of the program. Usability of the final version was confirmed through feedback loops of these phases.Based on 6 focus group discussion sessions, it was determined that teens with cancer were interested in seeing stories of successful childhood cancer cases and self-management after discharge, and preferred multimedia content over text. Therefore, each Facebook module was redesigned to include multimedia materials such as relevant video clips tailored for teens. Usability assessed by heuristic evaluation and user testing revealed several critical usability issues, which were then revised. Potential end users tested the final program and perceived it to be usable and useful for teens with cancer.To our knowledge, “Healthy Teens for Soaam” is the first Facebook-based intervention program for teens with cancer. We actively worked with current childhood cancer patients and survivors to develop and improve this program, achieved good usability, and met the expressed needs and preferences of target end users. This 8-week Facebook-based educational program for teens with cancer, developed as the first step of an upcoming intervention study, will be useful for improving knowledge and self-management strategies of teens. Survival rates associated with childhood cancers continue to improve, partly due to advances in diagnostic techniques and treatment modalities along with clinical research . AccordiDespite improvements in survival, childhood cancer survivors still face a high risk of therapy-related complications or adverse effects that persist or arise after completion of treatment . ChildhoSome morbidities are modifiable through preventive healthy behaviors such as physical activity, good nutrition, obesity prevention, and avoidance or cessation of smoking, as well as identification of characteristics that may modify morbidities or adherence to healthy behaviors to better target interventions ,9. For eChildren currently undergoing cancer treatment, as well as survivors, require follow-up care for the rest of their lives; thus, the Children’s Oncology Group (COG) has emphasized the importance of regular medical follow-up and has developed the “COG long-term follow-up guidelines for survivors of childhood, adolescent, and young adult cancers” . In addiKnowledge about their disease, the importance of treatment adherence, and strategies to improve adherence could be improved through innovative health interventions designed specifically for teens . Among tSoaam” (soaam is the word for pediatric cancers in Korean), as the first step of an intervention that will deliver educational materials with the aim of increasing knowledge and improving self-care practices related to their disease and treatment. The purpose of this usability study was (1) to describe the process of developing this 8-week modular intervention program for teens with cancer and (2) to evaluate its usability to refine the program.Social media has been increasingly utilized as a platform for delivering health interventions in childhood cancer care. For example, Watson  reportedThis study was reviewed and approved by the principal investigator’s institutional review boards (1041078-201606-HRSB-122-01 and 20200026). The study was conducted in 7 phases  and was Semistructured focus group discussions were conducted with 12 teens with cancer. Participants were recruited from childhood cancer self-support groups in Korea using convenience and snowball sampling methods. A flyer was posted to the online platforms of self-support groups on popular Korean online platforms , and interested volunteers contacted the researchers by phone or email. Interested participants also suggested other potential participants to the researcher.A total of 6 in-person focus group discussion sessions were conducted, during which participants were accompanied by their parent or legal guardian. Each session included 2 to 5 participants, and each lasted for about 60 to 90 minutes. Focus group discussions were facilitated by the principal investigator (BP) and a research assistant. First, the principal investigator explained the overview of the program, distributed handout materials detailing program contents, and asked participants and parents to provide their opinions regarding the contents. For example, they were explicitly prompted to provide suggestions regarding the addition, removal, or emphasis of content. Besides answering questions, participants and parents were encouraged to share their personal experiences with childhood cancer using prompts such as “If you have anything you want to share with us even though it is not directly related to the contents, don’t hesitate to tell us.” Focus group discussions were recorded and transcribed verbatim for qualitative content analysis using NVivo 11 .Qualitative content analysis was conducted using a combination of the content analysis method suggested by Krippendorff  and the Qualitative content analysis was conducted by the two coders  to ensure reliability of the analysis. Coders independently analyzed the transcripts. Questions or disagreements regarding the themes or categories were discussed. Revision and refinement of themes and categories continued until content analysis was completed.Based on results from focus group discussions, the program module content was organized using the following usability design methods. First, we used group information together  after the focus group discussions. We defined the important themes from the focus group discussions and grouped them into several categories such as diagnosis, examination, treatment, social support, and return to society/school. Second, we developed a structure for 8 weekly modules. Third, we developed low-fidelity module prototypes such as paper prototypes. Finally, we established high-fidelity mockups using PowerPoint slides.Main topics for the program module content were developed based on the results from the focus group discussion “needs assessment.” Module content was written at a sixth-grade reading level using the Flesch–Kincaid readability test guidelines ,36. ContWe incorporated usability principles for teen-friendly websites  into theEach module included specific learning objectives, structured learning material for fundamentally important content with external web links for further information, multimedia such as video clips or games relevant to the topics, and discussion topics at the end of the modules. Participants were asked to respond to discussion topics by adding comments on Facebook. This served to monitor module participation and sharing of personal experiences for social support. Although not done during testing, in the implementation study, a program moderator (researcher) will provide feedback and encourage interactions among participants on discussion boards. Discussions included the following topics: “Let’s talk about blood transfusion experiences. Have you ever received a blood transfusion? If yes, please tell us about the specific blood product you received, your feelings, and side effects,” and “Have you had any pain while getting treatment in the hospital? Which one was the most painful, and how did you overcome the pain—pain medication, massage, and ice bag?”The module content was reviewed by four experts currently working in pediatric hematology and oncology units in tertiary hospitals, who each had more than 10 years of experience: two nurses with master’s degrees in pediatric nursing and two pediatric hematology and oncology physicians with PhDs. Reviewers were knowledgeable about the most recent evidence in this highly specialized field and reviewed the module contents for accuracy. They were asked to rate the ease of understanding and appropriateness of information for teens with cancer . Based on their feedback, module contents were revised and finalized for mockups of the Facebook-based program.Facebook Group was used to develop the 8-week intervention program, with the “private group” functionality utilized to protect participants’ privacy. With a private group, only people who are invited by the group creator  can join and see who is in the group and what they post . We chosTo deliver module content, we composed PowerPoint slides and then uploaded each slide to the group’s Photos section. Participants could easily navigate the slides by clicking the “next” and “back” buttons. Considering that the purpose of this program is to deliver accurate and reliable educational content, only group administrators were granted posting permission, but participants were allowed to add comments to each post to share their opinions or contribute to discussions.Two experts in the field of usability and human-computer interactions reviewed the Facebook-based program using Nielsen’s heuristics ,43 adaptUser testing was conducted using observation, the think-aloud method, voice and screen activity capture using Camtasia 9 , and surveys. Participants for this phase were recruited through parents of teens belonging to a childhood cancer self-support group in Korea using convenience and snowball sampling. A flyer was posted to the online platforms of self-support groups on KakaoTalk and Band, and interested volunteers contacted researchers by phone or email. Inclusion criteria were teens with cancer who were (1) aged 13 to 18 years old, (2) diagnosed with any type of childhood cancer, (3) received any type of cancer treatment , and (4) at any stage of treatment (newly diagnosed through completed treatment). Participants arranged appointments for screening, consenting, and user testing. As the participants were adolescents, both informed consent from their parents or guardians and assent from the teens were obtained prior to participation.A total of 11 face-to-face, 1:1 user testing sessions were conducted. Testing was facilitated by the principal investigator in a private room at the study site. Before conducting user testing, the researcher gave instructions about the program and usability evaluation methods such as the think-aloud method and Camtasia screen-capture software. Participants were then provided with an information page that included the Facebook program link, user ID, and password. They were assigned a user testing task to review 2 out of the 8 modules, including logging in, navigating through instructions on the first page to modules, reading content, watching video clips, and adding comments. They were assured that they could stop at any time for any reason. Each session lasted 30 to 40 minutes.During evaluation of the program, participants were encouraged to think out loud while their voices were recorded using Camtasia. During pauses from thought verbalization, participants were probed with the following prompts: “What do you think about this? What features do you think need improvement?” At the same time, the researcher documented comments from participants and her own observations using the user testing worksheet.Before user testing, participants’ demographic characteristics as well as their internet and Facebook usage habits were queried and documented. After user testing, participants filled out a short survey that evaluated electronic health  literacy and perceived usability. The eHealth literacy was assessed using the eHealth Literacy Scale (eHEALS) , which hPerceived usability of the program was assessed using the Perceived Health Website Usability Questionnaire (PHWSUQ) . This quQuantitative data from the questionnaires were analyzed with descriptive statistics using SPSS Statistics for Windows, version 25.0 . Audio recordings from user testing sessions were transcribed verbatim and analyzed employing the content analysis method described in Phase 1. Qualitative data, collected throughout observation and screen activity and recorded by Camtasia, provided a rich contextual background and a strong source of triangulation for developing themes and generating a comprehensive review of the program’s usability. Thus, screen recordings were reviewed when the transcript did not capture enough usable detail, which improved the accuracy of content analysis.Modifications regarding content and appearance issues were made on the PowerPoint slides, and the revised slides were uploaded to Facebook (the process was conducted in the order phase 2 to phase 4 to phase 7). If content revisions required expert opinions, relevant experts were contacted to confirm the accuracy of the new content (the process was conducted in the order phase 2 to phase 3 to phase 4 to phase 7).Participant characteristics are summarized in Participant feedback regarding the outline of the program  includedThis is the big problem. The reason I’m so supportive of this education program  is that there’s a limitation of internet and you narrow down the range a little bit…... I hope the contents go on in the future.Additionally, participants freely described and shared their experiences with cancer treatments, and the emerging themes and categories dealt with information needs, support systems, barriers to treatment, facilitators to treatment, return to social life, and health care system issues. Other representative comments from the focus group discussions are provided in An overview of the final version of the program is provided in Four childhood cancer experts reviewed the content and provided feedback on whether the information was accurate and current, including the pictures and video clips from movies and Korean dramas. Two nurses reviewed relevant medical terminology from academic journals and textbooks, and noted that it differed from the terminology commonly used by participants and parents. For example, granulocyte colony-stimulating factor injections for neutropenia were usually referred to as “count shots”  or “Grasin” among participants and parents. Thus, we also used these words in the program content. Two physicians contributed information on the most recent treatment practices for childhood cancers in tertiary hospitals. For example, a previous version contained only bone marrow transplantation, but we added information about cord blood transplantation and peripheral blood stem cell transplantation.Usability experts pointed out usability problems of the program among three categories of Nielsen’s heuristics tailored for children’s eLearning : (3) useAll four childhood cancer experts rated the content highly for its ease of understanding (4=strongly agree) and appropriateness for teens with cancer (4=strongly agree). They also commended the detailed instructions and the neat and consistent overall design of the PowerPoint slides.User testing participant characteristics  includedThe transcript of user testing content analysis identified 20 themes under four categories: program content, program appearance, navigation, and others, which mainly included feedback about the strengths of the program . ParticiSoaam” program on Facebook.Critical usability issues reported during user testing were reviewed, and, where necessary, revisions were made and confirmed through feedback loop of the phases . RegardiSoaam” revealed that the program was perceived as usable by our participants, who were representative of our target audience (teens with cancer). Seven phases of development and usability evaluation uncovered usability issues as well as areas to enhance user satisfaction, which were then modified accordingly.The aim of this study was to develop a Facebook-based intervention program for childhood cancer participants and to evaluate its usability to guide program refinements. Overall, the evaluation of “Healthy Teens for Childhood cancer participants and families wanted a comprehensive online information source where they could find childhood cancers–related information. Preexisting sources of information  were not reliable and potentially provided limited or outdated information. Therefore, focus group discussion participants greatly valued the objectives and content of this program. Moreover, participants supported the discussion topics at the end of each module. Considering the characteristics of adolescence, peer groups are important for their socialization; however, hospital admissions and principles to prevent infection limit their opportunities to participate in peer group activities. Online interactions among teenagers with similar diagnoses provide the next-best opportunity for social support.User testing by potential end users revealed that the program content was comprehensive. Previous research has found the two main predictors of noncompliance to chemotherapy to be child resistance and inadequate information . To overBy adapting the Facebook platform to this program, we took advantage of the following key points: (1) potential end users are familiar with navigation of the program, (2) researchers can reduce costs for developing new online platforms, and (3) researchers may be able to improve treatment fidelity, as teenagers access Facebook almost every day and are continuously connected via the smartphone app. By contrast, the limitations of using Facebook  include:Soaam” program will be delivering this 8-week Facebook-based intervention for teens with cancer in South Korea. The purpose will be to increase knowledge and provide accurate, up-to-date information about self-management strategies for improving health and wellness. Through this intervention, we will be able to determine the effectiveness and efficiency of the Facebook-based intervention program.The next step of this “Health Teens for A previous Facebook-based intervention program for young adult cancer survivors found that engagement with the Facebook program was variable, and investigators recommended that future research should explore how to promote sustained engagement in online social networking ,50. Our This study had several strengths. We used rigorous qualitative analysis methods, including focus group discussions and user testing transcriptions. Additionally, the use of multiple methods provided a strong source of triangulation and enhanced the reliability of our results, which were informed by both qualitative and quantitative data. We described all development and usability evaluation processes conducted in 7 phases to provide guidance to researchers who want to use Facebook as an intervention platform. According to a recent systematic review of the use of Facebook, only 10 studies have investigated this platform for such interventions . This coOur study also had several limitations. Regarding participant recruitment, focus group discussions and user testing employed convenience and snowball sampling methods. Sample sizes were relatively small and participants were from one metropolitan area. We did not apply strict inclusion criteria for user testing participants, instead including participants with any type of cancer and at any stage of treatment (ranging from newly diagnosed to completed chemotherapy). Additionally, the current health status of participants was biased toward teens who were in good health. These limitations could decrease the generalizability of the study findings. In addition, our findings may reflect a response bias, as participants who were already interested in Facebook may have been more likely to participate.


Type 2 diabetes (T2D) is a public health pandemic disproportionately affecting low- and middle-income countries. The purpose of this formative research was to adapt evidence-based diabetes self-management education programs to the context of Seguro Popular clinics in Mexico. A theory-based mHealth  component was developed.Our formative research and development of the program protocol consisted of six phases: (1) interviews and focus groups with stakeholders on the challenges to T2D management, curriculum content needs, and the use of mHealth as a supplement to a DSME program; (2) review of the theoretical underpinning, curriculum, and interactive strategies of four evidence-based DSME programs and modification to meet the needs of adults with T2D and systems of care in Mexico City; (3) development of theory-based illustrated text messages; (4) evaluation of text messaging acceptability and access in adults with T2D via focus groups; (5) development of program manual; and (6) development of a training program for health care providers.The ¡Sí, Yo Puedo Vivir Sano Con Diabetes! included 7 group-based weekly lessons; simple, interactive content; weekly empowerment messages; video novellas; group activities; and goal setting. Adaptations to the cultural context of Mexico included content/activities on diabetes etiology , nutrition (indigenous foods and plate method), self-blood glucose monitoring, and diabetes-related stress/coping. We used the Health Action Process Approach to guide the text message development, which posits that adoption, initiation, and maintenance of health behaviors require the development of intentions, plans, coping, and self-efficacy. Our final text message bank consisted of 181 messages. There were approximately 20–30 messages for each process of behavior change  and 30 messages for each content topic . There were 96 messages that were illustrated. Training materials were also developed.We used a systematic approach, collaboration with stakeholders, and a well-established behavior change theory to develop an evidence-based intervention to an international context and system of care. Collectively, this process has the potential to enhance the feasibility, acceptability, and efficacy of the program. Type 2 diabetes (T2D) is a global health concern with persons in low- and middle-income countries affected disproportionately . Based oAdults with T2D in Mexico may not have access to high-quality DSME programs and often lack knowledge and understanding about T2D etiology and self-management. Many Mexican adults with T2D believe that the primary etiology of T2D is extreme fright or stress –8. This Seguro Popular, was instituted to provide care to individuals not covered by the social security system. Adults who receive care in the Seguro Popular system have a mean age of 44.7 years, are 57% male, have low income (66.9% are in the 3 lowest socioeconomic quintiles), and 12.4% report a diagnosis of T2D , mean age 52.33 years [standard deviation (SD) = 10.47], 88.89% had less than high school education, 88.88% married or in cohabitation, mean A1C 9.13% (SD = 1.07), and had a median of three comorbidities. Overall, participants understood the intended meaning of the majority of the text messages and felt that the messages were positive and helpful. There were several text messages that they perceived did not apply to them or conveyed a misconception that needed to be addressed in the curriculum (see Table We subsequently conducted a focus group with adults with T2D (at the same Seguro Popular clinics as the previous focus group) to determine their ability to receive and view text [short message service (SMS)] and picture messages [multimedia messaging service (MMS)] on different types of cell phones. Multimedia messaging (MMS) became standard in cell phone technology in 2011, which allows pictures to be received even if the phone does not have a camera. Moderators of the focus group sent text and image messages to all participants to determine their ability to receive the messages. We sent both low and high-resolution images to determine if that made a difference in receiving MMS messages. Moderators then recorded the participant’s ability to receive SMS and MMS messages.There were seven adults with T2D who participated in this aspect of the text messaging evaluation with the following characteristics: 71.43% female, mean age 51.14 years (SD = 10.96), median age 50 years (IQR 20), and 100% had less than high school education and were married or in cohabitation, with a mean A1C 9.6% (SD = 1.13) and a median of three comorbidities. Participants had a wide variety of cell phone brands/models and used two different cell phone carriers. All participants were able to receive SMS messages without difficulty. However, many participants had difficulty receiving the MMS messages including not receiving the message, difficulty in downloading the message, or difficulty in viewing the image . Participants who were able to receive messages preferred the high-resolution image as the colors were brighter and easier to view words and images. The results of this focus group reinforced the need to provide all participants with a color booklet with each text message to assure that the content of the text message was received by all participants. We also determined that we would send high-resolution image messages to enhance the visual experience as there was no difference in the ability to download images based on image size.¡Sí, Yo Puedo Vivir Sano Con Diabetes! program. The program consisted of seven group-based sessions, daily text messages, and monthly booklets that had each text/image message numbered in the order that they would be received if accessible by text message. For the group-based sessions, we created a training manual for the leaders of the program, a participant workbook, colored posters to be used in sessions, and materials needed for sessions . We also developed a fidelity checklist for program leaders to complete after each session.At the conclusion of the focus groups, we finalized the protocol for the Our final text message bank consisted of 181 messages. There were approximately 20–30 messages for each process of behavior change  and 30 messages for each content topic . There were 96 messages that were illustrated. We also created the text message color booklets on 4 × 4 cards, with a single text message on each card. Cards were numbered to align with the text message of the day and collated with a binder ring, so that participants could easily flip from one message to the next. Daily text and image messages were provided via the iMessage app™ for IOS (Apple) because Web-based platforms did not provide services in Mexican territory.Sí, Yo Puedo Controlar mi Diabetes! program. Plans for monitoring program fidelity were also reviewed.We developed a training program for the facilitators of the program (group sessions coordinators) on the program protocol—a registered nurse and a social worker. This training entailed self-study, interactive sessions, and role-playing which required 4 days of training. There was one session of the training that was provided through distance learning by the original developer of the ¡The training program consisted of content on the program, the theoretical underpinnings of the program, the pathophysiology and treatment of type 2 diabetes, and the social determinants of health in Mexico City. Each session of the curriculum was reviewed in detail and strategies to enhance the success of group-based education were presented. Role playing was utilized to practice providing encouragement, empathy, active listening, and critical thinking. Case studies were used to allow class leaders to get feedback on addressing challenging situations . Time was allotted every day for questions and conducting one of the relaxation exercises used in the curriculum. During the last day of the training session, class leaders presented one session to the research team, and feedback was provided on aspects that went well and areas to improve.Plans for monitoring program fidelity during implementation were also reviewed during the training program. Class leaders were instructed to follow the program manual, providing the same content for all sessions and all participants. The fidelity checklist was discussed and class leaders were asked to complete this after every session. In addition, class leaders received ongoing coaching by a master trainer with the objective to maintain high levels of implementation fidelity, to perform and document adaptations to the class plan, and to keep class leaders motivated. The master trainer also attended some of the sessions to conduct passive observation and to provide positive feedback to class leaders. Based on these aspects, an implementation analysis will be performed and documented in the future.We used a systematic approach, formative research, and collaboration with stakeholders to adapt evidence-based DSME programs to an international context and system of care. We also used an established behavior change theory (HAPA model) to inform the development of illustrated text messaging to supplement the DSME program, an innovative aspect of this program. As we developed our program protocol, we specifically sought to address perceived challenges to T2D self-management, provide content that met the need of adults with T2D and providers, and develop a text messaging component that was accessible and acceptable to adults with T2D in Seguro Popular clinics. In addition, we addressed system challenges of delivering diabetes care in low- and middle-income countries, such as lack of administrative support, clinic staff shortages, lack of established guidelines, and limited resources , 36. We ¡Sí, Yo Puedo Vivir Sano con Diabetes! + mHealth or wait-list control condition with the hypothesis that clinical  and self-efficacy outcomes will be greater in ¡Sí, Yo Puedo Vivir Sano Con Diabetes! + mHealth participants compared to the wait-list control condition at 3 and 6 months of follow-up. We also hypothesize that the ¡Sí, Yo Puedo Vivir Sano Con Diabetes! + mHealth program will be feasible and acceptable to adults with T2D and providers and that fidelity of the program will be maintained.To evaluate the program, we will use a randomized, controlled pilot study design in which 40 adults with T2D will be randomized to the