Taking evidence-based interventions to scale is a challenge for prevention science. Mighty Girls is an evidence-based sexual health intervention program that combines classroom sessions with novel, cutting-edge technology . The program was developed for 7th grade Latinas, but US school and community demographics rarely allow interventions targeting a single ethnic group. Additionally, digital puppetry is costly to scale up, and parent disapproval often prevents successful dissemination of adolescent sexual health programs. Intervening steps along the scaling-up pathway are needed to adapt the program prior to scaling up for dissemination.The aims of this study were to create a multicultural adaptation of the Mighty Girls program using a mobile app that is less costly to disseminate and is acceptable to parents of 7th grade girls.t tests.This study used a three-phase process to adapt Mighty Girls into Mighty Teens. All phases used purposive (nonprobability) sampling of low-income, multicultural, urban metropolitan groups (7th grade girls and their parents) within central Florida. Phase 1 involved two videotaped implementations of a multicultural adaptation of the classroom sessions, one involving focus groups (N=14) and the other serving as a single-group pretest-posttest pilot study (N=23). Phase 2 involved development of a narrative cell phone app prototype, which was subjected to usability testing (N=25). App usability and engagement were assessed qualitatively  and quantitatively. Phase 3 used focus groups to assess parent support for the program (N=6). Qualitative data were analyzed using descriptive content analysis. Quantitative data were analyzed using descriptive statistics and paired P=.04). App usability and appeal were supported by a System Usability Scale score of 76 (exceeding 68 per the industry standard) and 83% (20/24) of participants agreeing they would recommend the app to friends. Parents (mothers) expressed only positive regard for program goals, and classroom session and app activities.Qualitative findings supported classroom sessions being multicultural, and identified simple changes to improve engagement and learning. Quantitative findings from the second classroom session implementation pilot study indicated a significant pre-post difference in intention to delay sexual intercourse (This study adapted Mighty Girls into an engaging, easier-to-disseminate, multicultural program, termed Mighty Teens, that uses a narrative-generating app to support behavior change, and is likely to be accepted by parents of 7th grade girls. This study also provides evidence of the preliminary effectiveness of Mighty Teens classroom sessions. The sampling method and sample size were appropriate for adaptation, but research involving a more representative US sample is needed to confirm multicultural fit, parent receptivity, and program effectiveness. Study implications include integrating app use throughout the classroom sessions to build narrative-generating skills across the program and increasing the number of narratives produced, which should in turn increase the program’s behavior change potency. Taking evidence-based interventions to scale is a challenge for prevention science ,2 becausThe purpose of this paper is to illustrate the steps taken to adapt Mighty Girls, an evidence-based sexual health intervention program , prior tSessions began with “Mighty Moments,” a kinesthetic learning experience of relationally competent resistance communication skills developed by the second author. Session objectives are listed with the original (Mighty Girls) program session title. Objectives were retained in the adaptation with minor modifications.Choices & Results Identify personal goals Understand the results of everyday choices (modified as “Choices & Results” in Mighty Teens)Relate choices and results to goals (modified as “Choices & Results” in Mighty Teens)What’s Risky? (maintained as “What’s Risky?” in Mighty Teens)Explain why some behaviors could be considered riskyIdentify potential results of choosing to engage in risky behaviorsDiscuss what increases or decreases the risk of a specific behaviorIdentify what behaviors are risky for oneselfThe Avoid Skill (modified as “Avoid & Leave” in Mighty Teens)Define Avoid SkillList 3 methods for avoiding an uncomfortable or risky situationAvoid the Mighty Girls way: being considerate, confident, and convincingThe Refuse Skill (modified as “Refuse & Explain” in Mighty Teens)Explain differences between aggressive, passive, and assertive communicationDemonstrate matching voice and body language using Refuse SkillRefuse the Mighty Girl way: being considerate, confident, and convincingMedia Influences Explain purpose of mediaThink critically about advertisements and TV show messagesIdentify positive and negative messages about girls and women in the mediaIdentify ways in which teens on popular TV shows are different from and similar to teens in real lifeWrap-up & ReviewModel results–based choicesDemonstrate Mighty communication: using considerate, confident, and convincing words and body language.Tech Component to Support and Sustain Behavior Change (modified as a cell phone app in Mighty Teens)Digital puppetry computer game, DRAMA-RAMAThe Avoid Skill and The Refuse Skill sessions were adapted from “keepin’ it REAL” . Other sMighty Girls has three features, which the nonadoption, abandonment, scale-up, spread, and sustainability (NASSS) framework identifies as likely to result in failure to scale up and disseminate . First, All three phases of program adaptation  used purPhase 1 used an iterative process involving consultation and two implementations of the classroom sessions, with the second implementation serving as a pilot study. After each iteration, the classroom sessions were revised.Three changes to program materials were made in preparation for phase 1. First, session slide sets were revised to ensure representation of a variety of cultural/ethnic groups. Second, content on narrative writing was added to the final “Wrap-up & Review” session. Third, ordering of the Avoid Skill and Refuse Skill sessions was reversed relative to their positioning in the original program. This last change was driven by insights from implementing the original program . Both seConsultants included a female middle school staff member and three adult women in their early 20s . The latter were recruited as research assistants through Indeed, a worldwide employment website for job listings. These young women all lived in the same low-income, multicultural, urban metropolitan area in which the study was being performed. We specifically empowered them by emphasizing during recruitment and task directions that they brought a unique and valuable expertise critical to the success of the research effort . They were tasked with reviewing the Mighty Girls classroom session program manual and slide sets for language, terminology, and image inclusivity  and making recommendations.Participants in both implementations were girls enrolled in the 7th grade (aged 11-14 years) at two multiethnic middle schools in an urban metropolitan area within central Florida. No participant had difficulty speaking, reading, or writing in English, and on average used only English, or English more than other languages, for these activities . Participants of implementation 1 were recruited at a low-income  middle school with 80% minority enrollment. Two girls dropped out after session 4 due to repeated disruptive behavioral issues. Participants of implementation 2 were recruited at a second low-income (52% reduced/free lunch) school with 69% minority enrollment (top 5% of state schools for enrollment diversity). Participants at both schools were recruited with: (1) flyers posted at the school; (2) a message sent out via phone by the principal to parents of 7th grade girls; and (3) the three research assistants discussing the study and handing out study packets during 7th grade lunch .Each implementation involved a two-person multicultural teaching team consisting of either the first or third author in the teaching role and one of the three research assistants in the teaching assistant role . This team also acted as participant observers who made postsession notes about the session they had taught, identifying what worked and did not work, and providing feedback on the cultural relevance of the wording and examples used to explain concepts.Teaching assistant role training involved completing session activities in the role of a participant, and provided an opportunity to try out a training approach that combined two 2-hour face-to-face sessions with a “homework” assignment similar to what might be used for training when the program is taken to scale. All three research assistants spontaneously reported how much they liked their training, particularly the “homework” (using communication techniques with friends and family).Both implementations were videotaped. The first implementation involved postsession focus groups, and the second was a one-group, pretest-posttest pilot study (not powered for significance).Participants at the first school completed a brief demographic questionnaire immediately prior to the first session that assessed age; birth country; race and ethnicity; acculturation (use of first language relative to English for those who spoke more than one language) ,23; and Each Mighty Teens classroom session was delivered and then followed by a short focus group led by the teaching team. Focus group questions were displayed on slides. Participants wrote responses on large pieces of paper posted on walls or on anonymous index cards, depending on the question. They then voted  on themes identified in comments written publicly and anonymously, and responded to probes for more details or new thoughts emerging in response to other teens’ comments.In lieu of focus groups, participants at the second school completed a paper-and-pencil pretest immediately prior to the first session  and completed a posttest immediately after the last session. All measures had been pretested or used with this age group previously ,24.The pretest and posttest contained a short self-efficacy scale, along with items assessing resistance and sexual intentions, and intention to postpone sexual intercourse. The 9-item self- efficacy scale comprised 5 items from the 12-item measure reported by DiIorio et al  , probably not (2), probably yes (3), and definitely yes (4). Higher scores indicate greater intention. Psychometric analysis argued for treating intention types as measures of separate constructs, not part of a single multi-item intentions scale. Resistance intentions were assessed by a question related to a particular response strategy when asked “to do something you do not want to do.” Response strategies included: not resisting , resisting by using relationally competent communication strategies , or resisting by using a nonrelationally competent strategy (tell them I don’t want to because it’s stupid). Kirby et al’s  3-item st test was used to assess pretest-posttest differences in self-efficacy, resistance and sexual intentions, and intention to postpone sexual intercourse (second implementation only).Demographic responses were summarized with descriptive statistics. A paired The first and second authors reviewed classroom session videos (both implementations) for signs of engagement, restlessness, and confusion. Engagement was defined as girls raising their hands to participate, smiling, looking at the teaching team or slide presentation when not engaged in a task, working on tasks, or easily redirected from chatting with peer(s) back to the main discussion and remaining attentive. Restlessness was defined as fidgeting, repeatedly talking with peer(s) and not easily redirected back to the task, complaining about the activity, being bored, or not having fun. Confusion was defined as a facial expression in which the nose and forehead were scrunched up in a type of frown, or complaints about being confused or not understanding task directions.The first author also reviewed and performed a descriptive content analysis  of the tPhase 2 began with app development that culminated in usability testing. The app prototype was initially developed for Android phones given Pew Foundation reports  indicatiTwo theories guided app development. First, the theory of fun  stressesBoth the app and a secure, password-protected dashboard (backend accessible to administration staff) were developed using a collaborative approach, involving the authors and Margaret Broucek, Principal of Tapp.technology who was a phase 2 technology partner . Tapp.teMighty Teens appIntuitive interface with minimal text or directionRecord oral story. Text input only required for naming storyPresent users with icons representing places, characters and their emotions, and Mighty Teens skills from which to selectProvide users with button to click to access definitions of individual Mighty Teens skillsProvide users with a button to click to record story while viewing selected icon places, people and their emotions, and Mighty Teens skillsAllow users to listen to their story, and submit, rerecord, or delete itSend user stories using secure encrypted transmission to Mighty Teens dashboardMighty Teens DashboardSecure, password-protectedAllow direct entry and edit of information into sortable fields with controlled access to such privilegesUsernames grouped by implementer and implementation school/clinic siteImplementation school/clinic site with street, address, state, and zip code information, and grouped by organizationStore usernames, app access codes, addresses, and story submissionsAssign various levels of access to all stored informationPhase 1 participants were invited to participate in usability testing as part of their study participation. However, some were unable to attend due to transportation issues or schedule conflicts , resulting in a sample of 25 participants. Almost all participants  created and recorded at least one story. Two participants interacted with the app but did not record (or deleted recordings of) their stories. One of these participants did not want anyone to listen to her story. It is unclear if the other had difficulty with the recording feature or felt similarly. Although 25 girls participated in app testing, only 24 completed written assessments of usability and engagement.Usability testing was performed at each school involved in phase 1 with hot spots, approximately 1 month apart. Participants accessed the app using either their own Android cell phone or a “testing” cell phone provided by the research team.When participants arrived at their respective school, they were split into two groups with each group assigned to a different classroom. Once in their classroom, participants were split into dyads and a research team member was assigned to observe each dyad. Those that finished early were free to record additional stories. Research team members took ethnographic field notes describing participants’ verbal and nonverbal behaviors during app use, but were unable to see the participants’ screens unless shown to them. Research team members debriefed immediately after testing. Notes from this meeting became part of the ethnographic field notes.After using the app, the girls completed written assessments of usability and engagement. Following this, a brief focus group discussion was held with the first or second author in the role of facilitator and a member of the research team in the role of cofacilitator/notetaker.Self-reported usability was assessed with an adaptation of the System Usability Scale  in whichr=0.60, P=.002), task difficulty , and flow .Consistent with the original scoring process , responsUsability was also assessed with an open-ended question asking what could be done to improve the app, and task difficulty was assessed with a closed-ended question: “It was hard to think of an idea to use for the Mighty Teens app.” Response options were strongly disagree (1), disagree (2), agree (3), and strongly agree (4).Self-reported engagement was measured with 6 items adapted from those used in DRAMA-RAMA development by replacing the word “game” with “app” and referring to writing stories instead of interacting with game characters . A singlEthnographic field notes were used to assess usability and engagement presence and issues. Research team members were trained to collect these notes and used a checklist as a guide to verify that they had commented on all assessment domains. The checklist prompted assessment of (a) verbal and nonverbal signs of positive and negative affect, and (b) problems with usability indicated by participants having completed the storytelling task .Focus group interviews consisted of open-ended questions that asked for general impressions, likes and dislikes, and recommendations for improvement. These questions were followed by requests for feedback on specific screens or parts of screens.Ethnographic field notes, focus group notes, and participant responses to the open-ended usability question were analyzed using descriptive content analysis , similarParticipant stories were coded for narrative structure and reference to Mighty Teens program skills using two dichotomous  variables. Narrative structure was defined as three complete components (1). A reference to a Mighty Teens program skill (1) required the participant to describe a character correctly using (or teaching another character to use) at least one skill. It was not necessary to use the exact program name for the skill. However, not referencing a skill or naming one without any application were both coded as no reference (0). Interrater agreement was 100%.Mothers and fathers of 7th grade girls involved in the second implementation were invited to participate in a focus group (a) by flyers included in the study packets given out to their daughters, and (b) during the parental consent process for their daughters’ participation. No fathers expressed interest in participating or were involved in the consenting process.Scheduling of focus group participation coincided with the first national lockdown in response to the COVID-19 pandemic. Parents were distracted, under distress, with other activities competing for their attention, complicating consenting and scheduling . Scheduling and COVID-19 accommodations resulted in two focus groups involving 6 mothers as participants.Focus groups were conducted using Zoom conference technology, which was challenging because the participants seemed unfamiliar with Zoom and video conferences in general. However, the facilitator (second author) was able to assist by phone, and familiarize participants with the process and importance of confidentiality and privacy once they opened Zoom.Focus group interviews consisted of open-ended questions about (1) what the parents had heard about Mighty Teen classroom sessions or the app from their daughters, and (2) the parents’ feelings about and challenges related to their daughters’ use of apps and cell phones. Information about classroom session topics and activities, and about the app was provided once parents shared what they had heard. Feedback was then solicited, followed by specific probing related to their daughters’ use of the app, her doing so privately, and supervision preferences and concerns.Descriptive statistics are used to summarize the mothers’ ethnicities. Focus group recordings were transcribed verbatim and analyzed using descriptive content analysis , similarParticipant demographic characteristics for all three study phases are provided in The consultants recommended taking more time in session 1 to explain the program, and making two changes to increase program language inclusivity: (1) replace “girls” with “teens” (including in the program title); and (2) use nongender-specific names and pronouns to avoid gender identity and sexual orientation biases. These changes were recommended to make teens feel included regardless of gender identity or sexual orientation. The only exception was in the Media Influences session, where the focus on how girls and women are depicted in advertisements was viewed as valuable. The three young adult consultants also reported (without prompting) how much they liked the program and wished it had been offered to them when they were in middle school, lending support for its multicultural potential.Review of classroom session videos and teaching impression notes from both implementations failed to identify instances indicating a lack of multiculturalism in terminology or directions, including those related to resisting peer pressure. For example, expressions of confusion tended to be made by multiple teens who were not more likely to be of a particular cultural/ethnic group than other participants. Moreover, the teaching teams of implementations 1 and 2 failed to report any instances of program content, directions, or examples feeling “off,” “needing to be translated,” and similar. Below, we discuss the implementation-specific findings.Analyses of focus group and classroom session videos, and teaching impression notes identified two pedagogical issues and one opportunity to fine-tune the content. First, participants appeared to be confused when identifying goals in session 1, indicating that more time was needed for teaching this content, resulting in the decision to split this session in two, with the first objective retained in the new Goals session and the remaining two objectives addressed in the new Choices & Results session . Second,The teaching team reported that splitting session 1 into two different sessions increased class discussion time and allowed for the goal content to be covered more slowly, improving participants’ grasp of this content. Review of videos indicated that any confusion initially present declined over the period in which the goal content was presented. Similarly, fidgeting and restlessness during the Refuse skill content presentation appeared to be eliminated by increasing the physical learning of this skill. Nevertheless, the teaching team reported that, overall, it was difficult to deliver all content within 45 minutes and still have sufficient time for discussion. This was particularly true for sessions 4 and 5, which had been revised to include content on Explain and Leave, respectively, and also for session 7 to which content on narrative writing had been added. In other words, all three sessions need to be revisited to streamline/reduce their content.Engagement was evident across all sessions. Any disengagement could again be linked to a preexisting peer conflict noted in the teaching impression notes.t14=–2.26, P=.04). No significant pre-post increases in resistance self-efficacy (P=.71) or resistance intentions (P≥.30), or decreases in sexual intentions (P≥.17) were observed.Quantitative analyses identified a significant pre-post increase in intention to delay sexual intercourse . The prototype did not track the time spent between opening the app and submitting a story, and teens were offered the opportunity to record their stories outside if they did not want to be overheard. Hence, the observed time to complete a task may be confounded with the time spent visiting with friends, a behavior observed at the same time the recording was presumed to be occurring, arguing for it being a “high-end” estimate of the actual time required.Only 24% (6/25) of the participants created a story with a beginning, middle, and end, and less than half of the participants  created a story in which Mighty Teens skills were used. The stories reflected both same and opposite sex orientations, consistent with the tailoring expected in self-generated narratives.The System Usability Scale score was 76 despite 55% (12/22) of the participants agreeing that it was hard to think of an idea to use for the Mighty Teens app. Content analysis of open-ended question responses and focus group notes indicated that about half of the participants wanted the app to have a tutorial, more directions, or prompts, whereas about half felt that it was “simple,” “easy,” or “very straightforward.”Analysis of ethnographic field notes described some participants as frowning and stopping, as if concentrating, and others clearly stuck at the initial step of identifying a title until told they could use “My Story” as the title if they wanted to. Regardless, these participants quickly transitioned to a rush of tapping and swiping with brighter affect, and focus group notes indicated that all but one participant (disgruntled by a preexisting peer conflict) were pleased with their experience. Nevertheless, many did not like how their recorded voice sounded.Slightly more than a third  of the participants stated in their written responses to the usability open-ended question that there was nothing needed to improve the app . Less than half  made one or more comments concerning actual or desired app features, and 17% (4/24) made no comments.Descriptive content analysis of responses to the usability open-ended question ethnographic field notes or focus group notes identified ways to improve usability . TrianguFour unsubmitted stories were found in the testing phones after the session supported participants’ suggestion to add confirmation of a successful story submission. Additionally, the focus group discussion helped clarify other usability improvements. For example, all participants wanted the character set to be more representative but some also felt that the numbers of icons in the current character set was overwhelming, arguing for customizing over expanding the existing set.Similarly, the importance of adding an “other” category for character emotions was emphasized by comments indicating that participants considered qualities such as courageous, shy, and determined to be emotions. In contrast, comments regarding places suggested adding both icons  and an “other” option to accommodate places that might be more regional or seasonal .Usability improvement categories and themesAllow both recording and text entry of stories.Label emoticons (cannot tell what some of the faces mean).Improve ability to select story elements:Allow multiple selections without exiting and reentering screen.Increase differentiation between skill versus skill definition selection (add a question mark for definition).Improve ability to delete characters.Make “controls”  on recording screen clear.Confirm story submission, so you know it worked.Increase choices and/or allow more customization:Current character set not sufficiently representative (nobody looked like me).Add selection of various different intensities of particular emotions .Add more places.Add “other” option for emotions and places similar to what is available for characters.Change to blue and red colors if app to be used by both boys and girls at some point; current colors are “girly.”Engagement improvement categories and themesMake possible to anonymously share and chat about stories to help other teens or obtain feedback on how story creator responded (or could have responded) to a particular situation.Increase the fun:Add emoticons floating across login page.Have app “read back” in a male or female voice, according to user’s choice.Animate stories (have characters act out story).Descriptive content analysis indicated smiling, intense focus, and constant interaction with the app for 23 out of 25 (92%) participants with these same participants asking if they could use the app to make multiple stories. At least 4 out of 23 (17%) participants who requested this option did so in the time available with one participant creating 4 stories.The word “fun” was used multiple times in all focus groups to describe the app; 83% (20/24) of participants reported they would recommend the app to their friends. Mean Fun and Like scores were 3.0 (SD 0.75) and 3.2 (SD 0.60), respectively, indicating that the participants agreed that the was app fun and liked using it. Additionally, both ethnographic and focus group notes indicated that participants found selecting icons for characters, character emotions, and places very appealing. Selecting emotions was stated as a “favorite part.”Analysis of open-ended question and focus group data identified changes that could increase engagement . One sucDescriptive content analysis of focus group transcripts indicated no obvious group differences, and no ethnic or racial differences. Hence, results for each group are combined with participants referred to as mothers because no fathers participated . There wScaling up culturally tailored sexual health interventions can be challenging and not always straightforward . This stPhase 2 used a combination of qualitative and quantitative methods to evaluate app usability and engagement potential. The usability score (76) exceeded the minimum industry standard (68) for minimum or average usability . AdditioFinally, phase 3 parent focus groups demonstrated not only parent receptivity but also positive regard for the Mighty Teens program. This argues for the program’s ability to surmount the political and ideological issues surrounding sex education that typically impede sexual health program dissemination and uptake -8.Looking across the findings from all three phases of this study, two themes stand out. First, the study findings suggest that Mighty Teens has high engagement potential. This argues for program success and impact because engagement predicts program participation , learninThis study did have limitations. Participants were purposively selected to represent various cultural and ethnic groups living within a low-income, urban metropolitan area within central Florida, and sample sizes were small, preventing us from statistical evaluation of possible cultural or ethnic differences. All parent focus group participants had previously consented to their daughters’ participation in a sexual health program, and may have been more likely to approve of sex education in general, as well as this program. Further, only mothers participated in the parent focus groups. However, research shows that fathers often defer to mothers when it comes to their daughters’ sexual health , and ourDespite these limitations, this study illustrates how program developers can adapt a program prior to scaling up for dissemination so that clear barriers to implementation by targeted user organizations  are addressed. Our three-phase approach models the use of specific guidelines and best practices for creating cultural  and multAdditionally, the study findings provide clear directions for three app-related program improvements. First, there are clearly two different teen subgroups that need to be accommodated: those that prefer to record a story and those that prefer to use text. Second, story creation can be a difficult task for this age group. More support for structuring stories, including a prompt to use a Mighty Teens skill, needs to be designed into the app. Our findings suggest that either narrative writing is not consistently taught in 7th grade, despite being part of the US 7th grade common core standards , or, if In conclusion, this paper describes a three-phase process for adapting an evidence-based monocultural sexual health program to increase the ease and success of scaling up and dissemination. Specifically, this process (a) created a multicultural version of the classroom sessions (phase 1); (b) replaced a costly technological component with a less costly easier-to- disseminate narrative-generating cell phone app (phase 2); and (c) obtained preliminary parent support for the adaptation, including their child’s use of the program’s app component (phase 3). Findings generated by this process provide preliminary support for Mighty Teens program efficacy, and insights for increasing program engagement, potency, and perceived value.


The purpose of this study was to gather opinions from experts via the Delphi method to inform the future development of a virtual reality based English language communication program for university level students in Korea. The participants, who consisted of a panel of experts and professors who majored in English language and multimedia education, completed three Delphi surveys based on Context, Input, Process, and Product evaluation, which is referred to as CIPP. In the first Delphi survey, the participants answered multiple choice questions and open-ended questions related to four areas relevant to the development of a virtual reality based program. Based on their answers, a second Delphi survey was designed to determine the participants’ level of agreement with the appropriateness of the questions related to the four areas. In the third Delphi survey, participants were shown the results  and were asked to confirm or modify their answers based on the other participants’ answers. According to the analysis of the Delphi survey results, need for the development of a virtual reality based English language communication program was suggested, and recommendations were made regarding the content and application of the program. The ability to communicate in English has increasingly become recognized as an essential skill due to globalization. This is particularly relevant when considering the growing importance of university-level English-language education programs that can train students for this globalized world in which the ability to communicate in English is required in most fields.To prepare students appropriately, Korean universities have attempted to implement more practical English as a Foreign Language (EFL) programs by focusing on communication skills rather than traditional teaching methods  in order to increase the students’ ability to communicate in English. In fact, in the last twenty years, the goal of the university-level EFL program has shifted to focus on strengthening the four English skills  and on developing communication skills  in a reaInformation and communication technology offer effective tools to create the desired meaningful learning environment. The widespread use of smart devices  has contributed to technological tools becoming widely adopted in pedagogical contexts . This haAlthough English-language programs enhanced with VR technology provide academic benefits , 11–13, Second, most research on these programs has taken place in elementary schools and has focused on the development of reading, writing, and vocabulary, as well as its effect on affective factors , 17 but Third, sufficient practical suggestions have yet to be provided on the application of virtual environments especially in educational settings at a school level regardless of the numerously claimed benefits of virtual environments in educational settings . This, KFinally, the existing research regarding the development of a software that combines educational content and new technology are mostly experimental, and thus, it is necessary to reflect on and expand the experts’ opinions in the early stages of development in order to reach educational effectiveness. To address this gap in research, this research study aimed to conduct a series of surveys via the Delphi method, allowing researchers to analyze the opinions of an expert group by collecting, combining, and quantifying the experts’ opinions on multiple occasions .Accordingly, there is a need for research into the development of a VR-based English-language communication program at a university level. Thus, the purpose of this study was to gather the opinions of experts to develop such a program. The purpose of this study was two-fold. The first purpose was to collect the experts’ opinions regarding the development of a VR-based English-language communication program using the Delphi method. The second purpose was to propose a plan to develop a VR-based program for university-level students in Korea based on the Delphi results.Korean universities have recognized the importance of English-language education programs and have made English a required course for graduation to improve the students’ level of English , 19. In Not only have teaching approaches evolved since the 1990s, but teaching tools have also evolved as a result of new technology. Research has been conducted to examine the potential of these new technological tools in pedagogical contexts by particularly with a focus on the potential of computers and smart devices . More recently, the development of VR, AI, and augmented reality (AR) software has led to an increase of research conducted to examine the educational potential of these software.University-level English-language education has become increasingly interested in teaching methods that foster a communicative approach based on the educational theory of constructivism . ConstruHowever, there is a lack of CALL tools that offer content that can be selected and adapted by professors according to the learning objectives and level of the students. Therefore, this research will investigate the need to develop a VR-based English-language communication program offering customizable content according to learners’ own needs for a variety of teaching environments.Virtual Reality (VR) is a technology that allows users to experience a virtually-simulated world that is similar to reality as it involves visual, auditory, and tactile senses, as well as to communicate with information from that world via a computer (desktop/hard VR) or a VR headset (3D VR) . The priSeminal research on VR consisted of pilot studies examining the new technology and its applicability in a variety of fields  –30. In tPrevious research has examined the integration of VR in English-language pedagogical contexts in Korea and around the world to determine the benefits of the technology and its potential in the classroom. Chung  investigBonner and Reinders  examinedThe use of a CMC tool in a VR program was also examined in a context where learners of English had the opportunity to have a conversation with virtual characters whose voices were provided by native speakers of English . The resResearch on VR has also examined the effect of the technology on affective aspects , 37. In VR-based English-language communication programs have also been developed to include AI algorithms and voice recognition. In fact, Cheon  developeAs seen, research on the implementation of VR-based English-language communication programs has been conducted to identify the potential of VR programs on the improvement of English communication skills. However, research on the use of VR in pedagogical contexts is still in its preliminary stage as most are exploratory studies that examine possible applicability or that investigate the students’ perspectives . In a siThe participants in this study consisted of a group of experts on English-language and multimedia education from Korea (OOU 2019-11-014-002). This study was approved by the OO UNIVERSITY INSTITUTIONAL REVIEW BOARD. Fourteen teachers and professors who majored in English-language education or multimedia education, and who teach English-language education at an elementary, secondary, or university level, were selected. The main purpose of this study is to verify the content validity of VR-based English communication content, and for this purpose, English education and multimedia education experts as content experts were recruited as the target population of this study. Based on Anderson’s  suggestiAll the participants agreed to participate in the research and written informed consent was obtained. The list of participants with their respective participation rates are shown in The Delphi surveys. As part of the Delphi method, Delphi surveys were created. The Delphi survey is a group facilitation technique, which is an iterative multistage process, designed to transform opinion into group consensus [onsensus . The Delonsensus . This DeEvaluation categories were selected. As well, the evaluation scale  was developed based on responses obtained from surveys administered to a panel consisting of teachers and an expert group. The teachers were experts in English-language education and multimedia education. The expert group mostly consisted of professors with expertise in English-language education at the university level and with experience in developing multimedia educational programs as they are familiar with the needs of the target end-users of the program. The development of the Delphi surveys began with the extraction of the four areas and associated content and was followed by the development of the evaluation scale.An expert group comprised of professors in the education department  revised the Delphi surveys to ensure the validity of the tool’s content. Some of the recommendations taken into consideration were the following: adding multimedia data and program prototype; increasing the total number of questions to ensure the reliability of responses; adding consideration of importance for decision making on development decision; subdividing learning objectives into understanding, application, and application to real situations; and providing instructions on how to use VR program.The Delphi method was used with the participants to obtain information related to existing VR-based English-language communication programs. In this study, a questionnaire tool with CIPP model was developed, and Delphi survey was conducted three times in total in order to collect basic data of VR-based English communication program development.In the first Delphi survey, experts’ opinions on four areas consisting of multiple-choice questions and open-ended questions were collected by mixing unstructured response forms and structured questionnaires. The main purpose of the first survey was to examine the divergent perception of experts, and then collect opinions from them. The first Delphi survey, which took place over a period of four weeks, consisted of both multiple-choice questions (MCQ) and open-ended questions (OEQ) to obtain the experts’ opinions regarding four areas: (1) the program’s necessity and academic diagnosis, (2) the education content, (3) the teaching method and evaluation, and (4) the results, applications of the four English skills, and other issues to consider for program design. To verify the validity of the questionnaire, two English education experts and one expert in Education participated to review the validity of the questionnaire. The content of the questions in the first survey, the number of questions, and the format of the questions are shown in The second Delphi survey took place over a period of seven weeks. 93 questions were created based on the responses from the first Delphi survey regarding the four areas in order for the experts to determine their level of agreement with the appropriateness of the questions. The responses were on a 5-point Likert Scale ranging from 1 (very inappropriate) to 5 (very appropriate). The median, mean, and interquartile range were computed based on the responses from the multiple-choice type questions from the first survey. The content of the questions in the second survey, the number of questions, and the format of the questions are shown in The third Delphi survey took place over a period of ten days. The participants were shown the results of the second Delphi survey, so that the respondents could compare their opinions with those of the other experts. The results consisted of the expert’s own responses, as well as the mean, standard deviation median, and quadrant range for the consensus items, which are statistical analysis results for the second questionnaire so that respondents could refer to the opinions of other experts in re-evaluating the importance of each item. After consulting the results, the participants were prompted to either agree or disagree with the opinions of other experts and to provide an explanation when their responses deviated from those of the others. In this study, as a result of the 3rd survey, most of the items showed a high level of consent and agreement between the panels, so the panel question was terminated with the 3rd survey.The summarized Delphi survey procedures and analysis stages are shown in Tool development. Evaluation categories were selected, and evaluation scale was developed through surveys done by teachers who are experts in English language education and multimedia education, as well as by an expert council. An expert group was mostly consisted of professors who currently have experience in English language education at universities and also have experience in developing multimedia educational programs because it is considered that the possible users of developing program in this study are university students, and the purpose of this study was to enhance availability of English language education and learning at a university-level. VR-based English communication development was done through the following procedures: Extraction of program development areas and criteria -> Selection of items by criteria -> Scale development.Verification and development of tool through revision . The developed tool went through the process of revision and correction of the content validity by professors in the field of education , and then the final Delphi survey tool was developed. The revised contents in the draft survey tool based on a consensus as follows: ‘Additional offering of multimedia data, and program prototype’, ‘Adjustment of the total number of questions in order to increase the reliability of responses’, ‘Additional consideration of importance for decision making on development decision’, ‘Subdivision of learning objectives into understanding, application, and application to real situations’, ‘Instructions on how to use VR program’.Quantitative data from the first Delphi survey consisted of the responses to the MCQs which were converted into numeric value and entered in the SPSS statistical program (25.0). Descriptive statistics  were computed. Qualitative data from the first Delphi survey consisted of the responses to the OEQs.In terms of the second and third Delphi surveys, the quantitative data consisted of the responses to the questions on a 5-point Likert Scale ranging from 1 (very inappropriate) to 5 (very appropriate) related to the level of agreement with the appropriateness of the questions. The descriptive statistics were also computed for the second and third Delphi surveys. In addition, consensus chart and convergent degree were computed to draw out the sum of the opinions of participants  and listening programs (M = 2.69).The results of the participants’ level of agreement with the appropriateness of the questions related to the first area, Necessity for a Program and Academic Diagnosis, for the second and the third Delphi surveys are shown in necessity of an enhanced program, there was a high level of agreement with the necessity to develop an English conversation program by the third Delphi survey (M = 4.77), which meant that recognition for the necessity was high. The need for content from native speakers obtained the highest level of agreement by the third Delphi survey (M = 4.85).Regarding the second criterion, expected educational effect, by the third Delphi survey, there was a high level of agreement for effects from multimedia learning (M = 4.31) and learning in VR situations (M = 4.31), and many responded that this program is expected to be effective in improving English-language abilities.When looking at the third criterion, problems of current programs, there was a consensus among experts by the third Delphi survey on the necessity of a program and expected educational effects criteria.Excluding the criterion of academic goal, the categories that obtained the highest level of agreement with the appropriateness of the content by the third Delphi survey included the goal to have authentic interaction (M = 4.15), the goal for improvement of competence (M = 4.31), and the goal for an increase in the interaction confidence level (M = 4.62). These results show that the respondents considered the development of English-language abilities which include learning about and understanding English-speaking cultures, as well as learning a language in a VR space, being an important objective.The results of the participants’ level of agreement with the appropriateness of the questions related to the second area, Education Content of the Program, for the second and the third Delphi surveys are shown in course content, the mean of the level of agreement with the appropriateness of the content for each category was above 4. Moreover, the participants reached an agreement on all categories of the criterion as the consensus level was above .75 and the convergence degree was below .50. On the second Delphi survey, the participants identified their level of agreement with the appropriateness of following topics in VR learning situations enumerated from highest to lowest level of agreement: travelling, interview in English, movies or English drama, public institutions, emergency situations, university campus, K-culture , sports, English-speaking cultures, cooking, shopping, beauty, fashion, time travel to the past or future, medical knowledge, Do-it-Yourself projects, reading.In terms of the second criterion, VR learning situations, the participants rated their level of agreement with the appropriateness of the conversation topics considering the difficulties  of the VR situations. There was a high level of agreement by the third Delphi survey in the situations related to the library: borrowing or returning a book (M = 4.46), requesting a book (M = 4.46), using library facilities (M = 4.31). In terms of situations related to tourist spots, there was a high level of agreement for the topic of asking for direction (M = 4.69), inquiring or purchasing a product (M = 4.54), and exchanging or refunding a product (M = 4.69). As seen in the responses of the experts on the learning objectives of the program that ‘it is important to learn a language while indirectly experiencing English-speaking cultures’, there is a high necessity in developing a virtual reality situation where it is hard to experience. There was a high level of agreement in the two remaining categories of topics related to a café (M > 4.50) and to a concert (M > 4).In terms of the third criterion, difficulty level factors, the participants rated their level of agreement with the appropriateness of the difficulty levels that were suggested by a group of experts who recommended that the program should be developed by subdividing into difficulties depending on the learner’s ability to communicate in English. The level of agreement with the appropriateness by the third Delphi survey was high regarding the topic or situation (M = 4.54) and sentence length (M = 3.23).The results of the participants’ level of agreement with the appropriateness of the questions related to the third area, Learning Methods and Assessment of the Program, for the second and the third Delphi surveys are shown in providing feedback, the category related to corrective feedback obtained the highest level of agreement of the appropriateness by the participants (M = 4.62). However, the category related to feedback during dialogue obtained the lowest level of agreement (M = 4.00). The results imply that it is necessary to provide them feedbacks that help them enhance communication abilities more practically rather than simply providing feedbacks with graphs showing accuracy on their communication ability.In terms of the second criterion, The results of the participants’ level of agreement with the appropriateness of the questions related to the fourth area, Program Results and Applications, for the second and the third Delphi surveys are shown in The purpose of this study was to lay the groundwork for a university-level VR-based English-language communication program. Using the Delphi method with experts of English-language and multimedia education, a preliminary investigation was conducted to examine existing English-language communication programs to inform the development of an enhanced VR-based program. The overall findings and implications of this study in terms of the four areas, which are 1) necessity for a program and academic diagnosis, 2) education content of the program, 3) learning methods and assessment of the program, 4) program results and applications, derived from the Delphi surveys are as follows.In terms of the first area, Necessity for a Program and Academic Diagnosis, the findings of the Delphi surveys indicate that the experts had a low level of agreement regarding the appropriateness of the existing speaking and listening programs. However, the level of agreement related to the necessity of an enhanced program was high, specifically when the program contained content from native speakers and VR-applied content. Moreover, according to the first Delphi survey, it was shown that students lacked experience in using conversation strategies needed for conversations with English speakers, and that learning from lectures did not necessarily transfer to real-life situations. These results suggest the need for the development of an English-language communication program where students can practice conversing with English speakers in authentic and realistic situations that simulate real life. The results of this study corroborate the findings of Joh  and ChanIn terms of the second area, Education Content of the Program, the academic goal criterion indicates that experts had a relatively high level of agreement regarding the appropriateness of the goal to have authentic interaction, the goal for improvement of competence, and the goal for an increase in the interaction confidence level. These results suggest that through an enhanced program, it is suggested that learners could overcome the fear of communicating with English speakers, increase their confidence when using English and increase their overall ability to communicate in English while learning about the cultures of English-speaking countries. Learning about and understanding other cultures is particularly valuable in the context of globalization. VR can effectively simulate situations that would not otherwise be available for the learners due to time, location, and monetary constraints. Therefore, VR can provide an enhanced educational experience that develops intercultural abilities through experiencing and interacting with people from different cultural backgrounds. Within the same area, the results of the Delphi surveys related to the course content criterion revealed that there was a high level of agreement with the appropriateness of content that allows for the learning of English with VR, the evaluation of speaking and listening, and the practicing of communication with VR. Furthermore, the responses from the experts indicate that the most appropriate VR simulation topics consists of travelling. Other VR simulation topics were also considered to be appropriate, such as interviews in English, movies or English dramas, public institutions, and emergency situations. The topics chosen for the third criterion, VR learning situations, consisting of library, café, tourist spots, and concerts along with their difficulty levels  received a high level of agreement of their appropriateness from the expert group. These VR situations are similar to those used in the VR-based program examined by Jin  in whichdifficulty-level factors, the level of agreement of the appropriateness revealed to be high for topics or situations, sentence length, and sentence structure. When considering the second criterion, providing feedback, the level of agreement for the method was high for corrective feedback and low for pronunciation or fluency correction. These results corroborate those of Kim et al. [In terms of the third area, Learning Methods and Assessment of the Program, Delphi surveys examined the factors that should be considered in terms of feedback . In terms of the first criterion, m et al.  where thIn the last area, Program Results and Applications, the Delphi surveys revealed a higher level of agreement of the appropriateness of a program in extra-curricular activities and intensive courses than in university teaching and learning situations. While Chung  introducThis study is meaningful in that it laid the foundation for developing a program that can improve English communication skills by applying VR technology to college students.As a result of SWOT analysis of the Delphi survey results to develop a VR-based English communication program in There are some limitations in this study that should be considered for future research. Indeed, the participants who participated in this study were university professors only from Seoul and Chungcheongnam-do, and there were only 14 participants. Additionally, they were experts in English-language education, and multimedia education, but they did not have experience in the development of VR-based programs. In this study, English education and multimedia education experts were selected as an expert group to verify the content validity of the VR-based English communication program. In selecting the panel, representativeness, appropriateness, professional knowledge ability, sincerity of participation, and the number of participants were taken into consideration, but it has a limitation in that it has not been able to include a wider group of experts with experience in VR-based education. Therefore, in the future, it would be required to compare and analyze results by organizing a group of experts in consideration of whether they have experience in performing VR-based classes and whether they have experience in developing VR-based educational contents. Therefore, the results of this study should be generalized with caution. Based on these limitations, further research should be conducted to survey more experts in different geographical areas and with more expertise on VR.Furthermore, future research should also be conducted to collect empirical data on VR-based English-language communication programs. For the purpose of this study, a group of experts was surveyed to gather their opinions related to VR-based English-language communication programs with the purpose of improving the quality of those programs and ultimately developing VR-based programs according to the recommendations. However, in the future, in order to verify the validity of the program design and interface, research should be conducted by forming a group of experts including educational engineering experts and VR program developers.Finally, this study revealed that it would be beneficial for VR-based programs to include content that could be customized by learners and professors to personalize the learning experience. Therefore, in the future, research should be conducted to examine learners’ needs for VR-based English communication programs targeting learners as well as instructors. In this regard, research that reflects the opinions of various expert groups and users in the program development by comparatively analyzing the needs of VR developers, instructors, and learners. In addition, more research should be conducted to collect and analyze quantitative and qualitative data to identify the specific needs of university students and professors.S1 Fig(TIFF)Click here for additional data file.S2 Fig(TIFF)Click here for additional data file.S3 Fig(TIFF)Click here for additional data file.S4 Fig(TIFF)Click here for additional data file.S5 Fig(TIFF)Click here for additional data file.S6 Fig(TIFF)Click here for additional data file.S7 Fig(TIFF)Click here for additional data file.S8 Fig(TIFF)Click here for additional data file.S9 Fig(TIFF)Click here for additional data file.S10 Fig(TIFF)Click here for additional data file.S11 Fig(TIFF)Click here for additional data file.S12 Fig(TIFF)Click here for additional data file.S13 Fig(TIFF)Click here for additional data file.


D) in the brain is required to formulate the definitive diagnosis of multiple system atrophy (MSA) and Parkinson’s disease (PD). We recently showed that αSynD can be detected in the olfactory mucosa (OM) of MSA and PD patients. For this reason, we have performed the first interlaboratory study based on α-synuclein Real-Time Quaking-Induced Conversion (αSyn_RT-QuIC) analysis of OM samples collected from PD and MSA patients with the parkinsonian (MSA-P) and cerebellar (MSA-C) phenotypes.Detection of the pathological and disease-associated alpha-synuclein , MSA-C , PD , and healthy control subjects (HS) (n = 11). Each sample was analyzed by αSyn_RT-QuIC in two independent specialized laboratories, one located in Italy (ITA-lab) and one located in the USA (USA-lab). Both laboratories have developed and used harmonized αSyn_RT-QuIC analytical procedures. Results were correlated with demographic and clinical data.OM samples were prospectively collected from patients with a probable diagnosis of MSA-P  between laboratories . In particular, αSyn_RT-QuIC seeding activity was found in the OM of 9/13 patients with PD  and 18/20 patients with MSA-P . Interestingly, samples collected from patients with MSA-C did not induce αSyn_RT-QuIC seeding activity, except for one subject in USA-lab. Therefore, we found that MSA-P and MSA-C induced opposite effects. Regardless of disease diagnosis, the αSyn_RT-QuIC seeding activity correlated with some clinical parameters, including the rigidity and postural instability.D may serve as a novel biomarker for accurate clinical diagnoses of PD, MSA-P, and MSA-C. Moreover, αSyn_RT-QuIC represents a reliable assay that can distinguish patients with MSA-P from those with MSA-C, and may lead to significant advancements in patients stratification and selection for emerging pharmacological treatments and clinical trials.Our study provides evidence that OM-αSynThe online version contains supplementary material available at 10.1186/s13024-021-00491-y. Multiple system atrophy (MSA) is a rare and rapidly progressing neurodegenerative disorder that can be classified into at least two main subtypes: the parkinsonian (MSA-P) and the cerebellar (MSA-C) types . MSA-P sD)  for 1 h at 37 °C (under shaking at 500 rpm). PK digestions were stopped by the addition of the LDS-PAGE loading buffer  and by boiling the samples at 100 °C for 10 min (under shaking at 500 rpm). Proteins were then separated by means of SDS-PAGE, using 12% Bis-Tris plus gels (Thermo Scientific), and transferred onto polyvinylidene difluoride (PVDF) membranes . PVDF membranes were incubated with paraformaldehyde (0.4% in PBS) for 30 min at room temperature (under shaking) and subsequently incubated with non-fat dry milk  for 1 h at room temperature. Membranes were then incubated with a rabbit polyclonal α-synuclein antibody  overnight at 4 °C (under shaking). After three washes in TBST-0.05, membranes were incubated with a secondary antibody  for 1 h at room temperature (under shaking) and developed with a chemiluminescent system . Finally, reactions were visualized using a G:BOX Chemi Syngene system. Densitometric analysis of Western blot bands was carried out using ImageJ software (1.51v).αSyn_RT-QuIC kinetics were represented using the Prism software (GraphPad v9.0.0) on a graph where mean fluorescence values were plotted against time. For each patient, we calculated the average times at which the positive replicates crossed the predefined threshold of 30,000 AU (time to threshold). The mean fluorescence intensity reached by each patient at 13 h at ITA-lab or 22.5 h at USA-lab was used to calculate the fluorescence values at the time threshold. Graphic representations of densitometric analysis were performed using the Prism software (GraphPad v9.0.0). Associations between variables were investigated through t-test or Mann-Whitney test, Chi-square or Fisher exact test, as appropriate. Dot blot results and PK resistance profiles of αSyn_RT-QuIC products generated by PD and MSA were analyzed through repeated measure analysis of variance (ANOVA). Given the exploratory nature of our study, the calculation of the power analysis was not performed. Kappa statistic with the corresponding 95% confidence interval (CI) was calculated to assess interrater agreement between ITA-lab and USA-lab.The USA-lab introduced some important modifications to the RT-QuIC protocol published in 2019  by some D strains  . Similarly, both laboratories did not observe differences in the average fluorescence values reached by PD and MSA-P at the time thresholds  of PD (9.31 ± 1.19 h) was statistically different from that of MSA-P (10.54 ± 1.17 h) , (ii) MSA-P and HS , and (iii) PD and HS  with a specificity of 91%. Notably, repeated cycles of freezing and thawing of the OM samples did not affect their αSyn_RT-QuIC behavior.With respect to the clinical diagnosis, we obtained a 90% sensitivity of αSyn_RT-QuIC seeding activity in OM samples of patients with MSA-P while that of PD patients was 69% (slightly higher than that published in 2019 ) Table . Surprisp = 0.014) and postural instability . Furthermore, there was a trend toward greater seeding activity prevalence, albeit not statistically significant, in patients with bradykinesia compared to those without symptom . Finally, an inverse, although not statistically significant, association was observed between seeding activity and disease duration . Similar results were observed considering separately PD and MSA-P patients groups.We then performed association analyses between αSyn_RT-QuIC seeding activity in PD and MSA-P patients and several clinical parameters   and 6/9 (67%) PD samples that induced rec-αSyn aggregation Fig. a. Notabl04) Fig. e. None o04) Fig. a.Fig. 3D strains responsible for PD and MSA-P. This may result in important differences in the PK resistance of the αSyn_RT-QuIC reaction products generated from OM samples of PD and MSA-P. Finally, the αSyn_RT-QuIC products obtained from the analysis of MSA-P and MSA-C samples at ITA-lab (collected at 13 h) were analyzed by Transmission Electron Microscopy (TEM) and confirmed the presence of amyloid fibrils only in MSA-P samples capable of inducing an efficient seeding activity (except for MSA-P_10 and MSA-P_11 that remained negative by RT-QuIC). The amyloid fibrils were not detected in all MSA-C seeded samples  with the aim to explore the interlaboratory reproducibility of RT-QuIC performed on OM samples collected from PD, MSA-C, and MSA-P patients. In contrast to CSF collection that can be painful and associated with adverse effects such as headache or cerebrospinal fluid leak, OM samples can be easily and periodically collected with a minimally invasive procedure. Both of our laboratories harmonized the analytical procedures by minimizing all the experimental variables that could lead to inconsistencies in the results or conflicting findings . The final αSyn_RT-QuIC protocol that has been developed enabled us to reach a very high IAR (96%) between ITA-lab and USA-lab. In particular, the OM of 69% PD patients and 90% MSA-P patients induced αSyn_RT-QuIC seeding activity with 100% of IAR. In contrast, none of the OM belonging to MSA-C patients and the HS induced αSyn_RT-QuIC seeding activity at ITA-lab, except for one patient per group at USA-lab, thus leading to 90 and 91% IAR, respectively. We decided to consider non age-matched HS (younger than the patients) for reducing the probability to include subjects at pre-symptomatic stages of α-synucleinopathy, eventually causing positive αSyn_RT-QuIC results difficult to interpret . Surprisingly, OM collected from MSA-P and MSA-C patients showed totally different behavior, with the latter being almost unresponsive to αSyn_RT-QuIC. The lack of seeding activity in MSA-C suggests that, although MSA-P and MSA-C belong to the same disease group, they may be caused by distinct αSynD strains that possess different tropism for peripheral tissues, including the OM. In very recent studies, it has been suggested that αSynD may originate outside the brain, including the nose, the gut and the urogenital tract and then spread to the CNS , (ii) Congo Red [5 μM] and (iii) Amytracker 630 (purchased from Ebba Biotech and diluted 1:800) for 30 min in the dark at room temperature. Fluorescent signals were measured with the appropriate wave-lengths  using the BMG LabTech CLARIOSTAR microplate reader. Regardless of the probe used, all MSA-P samples (except for MSA-P_10 and MSA-P_11) showed strong fluorescent signals while the MSA-C did not. These differences were statistically significant in the case of (i) ThT (t-test p = 0.0063), (ii) Congo red  and (iii) Amytracker 630 . Graphs show the mean values (± SEM) of MSA-P and MSA-C fluorescent signals obtained from (i) ThT (graph on the left), (ii) Congo red (graph in the middle) and (iii) Amytracker 630 (graph on the right) analysis (a). Eight μL of αSyn_RT-QuIC products generated by MSA-P and MSA-C samples were treated with PK [2.5 mg/mL] for 1 h at 37 °C under shaking (500 rpm) and immunoblotted with antibodies directed against three different epitopes of α-synuclein: (i) clone 5C2 , (ii) AB5038 , and (iii) clone 42 . No signals were observed in samples immunoblotted with the AB5038 and the clone 5C2, while with the clone 42, PK resistant bands were specifically detected in αSyn_RT-QuIC products seeded with MSA-P but not in those seeded with MSA-C samples (b).


Many programs exist to prevent bullying and cyberbullying. Nevertheless, despite evidence of the numerous overlapping risks of the Internet, programs that jointly and adequately address large sets of risks are not presently described in the scientific literature. This study’s main objective was to assess the effectiveness of the Safety.net program in a pilot sample. This program prevents eight Internet risks: cyberbullying, sexting, online grooming, cyber dating abuse, problematic Internet use, nomophobia, Internet gaming disorder, and online gambling disorder. The Safety.net program comprises 16 sessions and 4 modules . Each session lasts one hour, but the program has a networked instructional design to recall previous content in later sessions. For its assessment, a pre/post-test repeated measures design with a control group and an intervention group was used. The study sample was 165 adolescents between 11 and 14 years old . The intervention group demonstrated improvements compared to the control group concerning online grooming, problematic Internet use, Internet gaming disorder, and nomophobia. These results suggest that the Safety.net program is effective in preventing the increase of most of the assessed risks and that it reduces some of them with a small number of sessions. Heavy and continual use of information and communication technologies (ICTs) has provided adolescents with numerous ways to communicate and interact. In fact, 80% of European adolescents ranging from 9 to 16 years of age declare that they use their smartphones daily to connect to the Internet and use social networks . AccordiIn contrast to relational risks, other behaviors are more inherently linked to the dysfunctional use of ICTs. One of the most studied risks is problematic Internet use (PIU), which is characterized by the existence of negative consequences arising from inadequate self-regulation (compulsive use of the Internet and uneasiness when not connected), preference for online social interaction, and using the Internet to modify one’s state of mind . The preThere is a shortage of literature addressing the concurrence of the aforementioned relational and dysfunctional risks. However, some studies provide evidence of overlapping relational risks among adolescents ,17,18. MConsidering this plurality and concurrence of Internet risks, their prevention is essential, and it is necessary to promote the positive use of technologies among children and adolescents. Most previous programs intended to prevent specific Internet risks, particularly cyberbullying . In SpaiThere are also very few programs with indicators of empirical effectiveness in the field of dysfunctional Internet risks. In Spain, the most prominent programs are ADITEC , which iTherefore, the present study’s main objective was to assess the effectiveness of the Safety.net program in a pilot sample of adolescents. Due to the multi-risk approach of the program, with one risk addressed per session, the incidence of relational risks  and dysfunctional risks  were not expected to increase among the intervention group. In the post-test analysis, the prevalence of both relational and dysfunctional risks was expected to be higher in the control group than in the experimental group.n = 120) and a control group (n = 45). The selection of the educational centers was carried out using nonprobability convenience sampling. The pre-test was carried out in December 2019 and the post-test in May–June 2020, during the COVID-19 lockdown. This study participants were 165 adolescents between 11 and 14 years old  from 5 Spanish educational centers in three Spanish regions . To assess the effects of the Safety.net program, a repeated pre-posttest measure design was carried out with an intervention group  and dysfunctional  in 11–14-year-old adolescents. Specifically, this program aims to primarily prevent the appearance and increase of these problems in adolescence .Conceptually, this program is based on four theoretical frameworks: the theory of planned behavior , the socThe program has followed the general lines of an instructional design; that is, it has sought to create some learning experiences that enable the acquisition of knowledge of the risks and the skills required to prevent them attractively . In addiSafety.net program is made up of 16 one-hour sessions see , dividedTo evaluate the effect of the Safety.net program, the Spanish version of the following questionnaires for relational and dysfunctional risks were administered.Cyberbullying Triangulation Questionnaire (CTQ) . For thiQuestionnaire for Sexual Requests and Interactions with Adults . ThroughVictimization Scale of the Online Partner Abuse Questionnaire . This scAdditionally, three of the items of the Sexting Questionnaire created by  were useGeneralized and Problematic Internet Use Scale (GPIUS2) . It evalNomophobia Questionnaire (NMP-Q) . It has Internet Gaming Disorder Scale-Short Form (IGDS9-SF) . This scOnline Gambling Disorder Questionnaire (OGD-Q) . This scFirst, contact was made with an educational institution that groups together numerous educational centers in Spain. Once a collaboration agreement had been signed, several schools were chosen on account of their convenience. The necessary institutional permits were obtained, and the parents or guardians of the children who took part in intervention groups had to sign an informed consent form. For the students in the control groups, passive consent was chosen (since this group only participated in administering the questionnaires and not the program). This passive consent was sent through the official way established by each school and, when the families refused to give their consent, it was to be sent back signed to the school so that the minor would not take part in the study (less than 1%).www.programasafety.net (accessed on 1 March 2021). The program was implemented from December 2019 to March 2020 during tutorship hours. Due to the state of alert decreed by the Spanish government on March 11th on account of COVID-19, module 4 of the program could not be implemented. It is a supplementary module that reinforces the contents taught in the three previous ones.The teachers in each school were responsible for the administration of the program. All the teachers who took part received specific training (30 h) to implement the program. Such training was given on an online platform containing all the training resources . Everything was at their disposal on the website of the program The pre-test evaluation took place in an online format in the computer science classrooms of each school under the supervision of each course tutor. In the post-test phase, due to the lockdown situation, the questionnaires were filled out by the students from home during school hours under the teachers’ online supervision. The teachers reminded the students at all stages that there were no right or wrong questions and that they should answer honestly. The Survey Monkey©  survey platform was used.This study was approved by the Ethics Committee of the International University of La Rioja PI: 004/2019). The study was brought to the attention of any juvenile prosecution services involved to follow the legal procedure for such cases.®, Armonk, NY, USA) program. First, Cronbach’s alpha was analyzed to check the internal consistency of each of the instruments used in this study. The total scores of each construct were obtained by adding the scores of all of its items.The statistical analyses were carried out using the Statistical Package for the Social Sciences (SPSS) 25  was used with an inter-group factor (intervention group and control group) and an intra-subject factor (before and after the program: pre-test and post-test). A value of ndomized . The int2 > 0.14 .t = 1.60, p = 0.110) and sex  = 1.03, p = 0.369). Regarding the intervention group, no significant differences have been found in any of the pre-test study variables, except online grooming, where the intervention group has slightly lower scores .To evaluate the effects of the Safety.net program, 120 adolescents were assigned to the intervention group, where the Safety.net program was implemented during school hours, and 45 to the control group, where the program was not implemented. The distribution of both groups by sex, age and academic year is shown in F = 13.44, p < 0.01,) and the group  = 5.64, p = 0.02), with medium (η2 = 0.076) and small (η2 = 0.012) effects, respectively. In the online grooming variable, a significant group x time interaction effect was found  = 6.20, p = 0.01), with a small effect (η2 = 0.04).As regards sexting and cyber dating abuse, no significant main or interaction effects were found, as shown in F = 8.29, p = 0.005) and on Internet gaming disorder  = 5.39, p = 0.02), with a small effect on both Internet risks . As shown in Regarding the dysfunctional Internet risks, the results show a significant effect of time x group interaction on problematic Internet use  = 19.90, p = 0.01), with a medium effect (η2 = 0.12). In this case, the results evidenced that, whereas the intervention group’s scores are lower, the control group’s scores regarding this Internet risk have increased. No significant main or interaction effects have been found regarding online gambling disorder  and four dysfunctional Internet risks . Furthermore, the program is characterized by its minimal intervention (a one-hour session to address each risk) using a network instructional design that seeks to maximize the connections between risks and their prevention, generating significant changes between pre- and post-implementation. For these reasons, the program is a remarkable contribution to the field of prevention, especially considering the worrying growth and concurrence of many risks of the Internet ,17,19.Regarding relational Internet risks, the control group showed higher rates of cybervictimization compared to the intervention group. This was also true for previous programs, such as the cyber-friendly program, Prev@cib, and Cyberprogram 2.0 ,31,32. HThis program also served as a buffer for online grooming; the adolescents in the control group demonstrated significantly increased post-test online grooming rates, while these rates decreased slightly in the intervention group. These results are similar to those of other programs that have shown a reduction in reciprocity between sexual solicitation and sexual interaction with adults in one session ,33. The The results regarding problematic Internet use and Internet gaming disorder showed a buffering effect in the intervention group compared to the control group. It should be noted that the evaluation of Internet gaming disorder is carried out from a clinical perspective ; therefoIt should be noted that this study was considered exploratory, particularly because of the impossibility of completing the fourth module (focused on cognitions and attitudes changes) and the lockdown caused by the COVID-19 pandemic. The first three modules are essential for the program, but the fourth one (while it is an important complement) is unnecessary. Nevertheless, the pandemic may have turned out to be a very pertinent unintentional context for analyzing the program’s effectiveness. Lockdowns and social distancing measures have increased people’s exposure to the Internet in general. Since the post-test was evaluated during the lockdown in Spain, such evaluation may have been influenced by a situation of overexposure to the Internet and its potential consequences ,66. The This study is not without its limitations. First, the results were based on self-reports, which may have included a level of response bias. Second, the pandemic context may have added additional variables to the T2 sample collection process. Third, there was high experimental mortality due to the extraordinary circumstances and the fact that families and course tutors found it difficult to offer the required contact and attention during the lockdown. The study’s exploratory nature and the sample forces us to be cautious with the interpretation and generalization of the results. In addition, the impossibility of implementing the fourth module (which has a complementary nature) because of the closing of schools during the state of alert in Spain is a limitation that should be considered. To confirm the program’s effectiveness, future studies should increase the number of participants in both the intervention and control groups, implement the entire program, and include a random assignment and an additional measure after the post-test.This study has important implications. It offers a preventive program that addresses many risks of the Internet in minimal sessions. The program is intended to be implemented within the framework of tutorship action. The materials found in the program would provide any guidance department, course tutor, or teacher with numerous resources with which to promote its application in class. In the coming years, at least until society recovers from the COVID-19 pandemic situation, the need to prevent problems of this nature is likely to increase. This is particularly true in the potential case of future lockdowns .The preliminary finds of this study suggest that Safety.net can be an effective program, especially to prevent online grooming, problematic Internet use, nomophobia and Internet gaming disorder in early adolescence.


These short-term results suggest that the SCHH program has potential to positively affect healthy eating and to reduce sedentary behaviors, both of which are fundamental to good health and wellness.The community-based Small Changes, Healthy Habits (SCHH) program was developed to teach skills and techniques to help adults in Louisiana make and maintain small behavioral changes in their food selection, preparation, and consumption, and in physical activity routines. The content of this four-week program included habit formation and goal setting techniques; physical activity guidance; strategies for a healthier home food environment; a grocery store tour focused on label reading for healthier food selections; basic knife and cooking skills. The program was piloted at ten sites throughout the state. A survey with 14 core items was applied before and after the program to evaluate participant acquisition of skills and behaviors associated with topic areas. A total of 47 participants provided complete data sets. Post-program, these participants reported increased confidence in preparing healthy meals at home ( Healthy eating habits and regular physical activity are fundamental to achieve a healthier weight and healthier life. Diet and physical activity patterns are strongly linked to chronic health conditions such as hypertension, heart disease, diabetes, and obesity ,2. This The small-changes framework is a behavioral strategy that advocates “conscious small changes in lifestyle behaviors” . It was Findings from multiple studies on this subject were compiled in a key report, which concluded that small reductions in energy intake and small increases in physical activities have reduced excessive weight gain . CurrentTo be effective, healthy behaviors, whether diet or exercise related, must be consistently and independently maintained. A habit is an automatic behavior developed through repetition over time in a stable context . As an aHabits are relevant to maintenance of positive behaviors and a healthier lifestyle ,17. The Participants: The SCHH program was available to adults (≥18 years old) in Louisiana. Participants in the evaluation study were those who registered for the program and agreed to participate in the evaluation process, a convenience sample of 47 of the 56 registrants. This pilot program was conducted by 10 nutrition agents, each based in a different community throughout the state  were prearranged and practiced. Instruments: The primary evaluation tool was a participant survey containing 14 core items  to capture short-term changes made by participants. Originally, the design also included a 6-month follow-up. However, the COVID-19 pandemic hindered follow-up data collection and value, as access to participants as well as participant routines were significantly altered. Evaluation measures were selected or created to reflect the potential for influence of the core curriculum components on healthy eating and physical activity behaviors among participants. The participant survey and habit assessment were both tested prior to implementation, using a cognitive interview process. Revisions to some phrasing and response choices were then made to improve clarity. The pre-program survey was completed immediately before the first session began; the initial habit assessment was completed after the first session because it referred to a small goal determined by the participant during the first session. The post-program survey was completed immediately after the last session. A follow-up habit assessment was to be collected 6 months after the program. yes, no or not sure. Confidence in specific knife skills and cooking skills was measured as ratings on a 5-point Likert scale, from 1  to 5 (completely confident)  spent on physical activity on those days  from 338 min to 277 min per day. The pre to post mean differences in physical activity measured as 0.2 days/week and as 11.1 minuets/day were not significant  of sitting time. The mean time spent sitting significantly decreased (nificant . p = 0.03) and for salt (p = 0.01). A similar analysis was also applied to the items on confidence in food preparation skills and food selection skills. No discernible improvements were made by these participants in knife skills or selecting appropriate cooking methods. However, there were significant favorable improvements in confidence to prepare healthy meals at home, with 16 participants improving (p = 0.04); to determine healthier prepared food options, with 19 participants improving (p = 0.01); to determine healthier options between similar foods, with 23 participants improving (p < 0.01); and to distinguish “everyday” from “occasional” foods, with 21 participants improving (p = 0.01)  .Among these participants, 15 reported a favorable improvement, a reduction, in frequency of consumption of meals/week prepared away from home. There was a complementary favorable improvement among 15 participants who reported increased frequency of home meal preparation . There wThis SCHH pilot evaluation focused on participant changes in a variety of food and physical activity/sedentary-related factors. During the four-week program period, participants began to watch or reduce their salt and fat intake or change the types of fats they are using. Participants learned to use food labels or product information to select healthier foods in grocery stores and in prepared food outlets. Participants also increased their confidence in food selection and preparation skills, which coincided with an increase in home meal preparation. Participants reduced daily sitting time, though this did not translate to significant increases in physical activity. These results are similar to a recent six-week healthy lifestyle program in Washington State, targeting obese Hispanic women . After aCooking skills are another indicator. Individuals who lack cooking skills are more likely to eat away-from-home or purchase convenience foods on a regular basis; foods obtained away from home generally have higher sodium, kilocalories, saturated fat and cholesterol . In a coThe home food environment, the availability and accessibility of foods, influences food selection and consumption ,33. As aThe SCHH program focused on how to read and use the food label, which is recognized as a primary tool for assisting the public in making healthier food choices . UnderstPhysical Activity plays a critical and complementary role in health and wellness. For this reason, a recommendation to meet the Physical Activity Guidelines for Americans is included in the Dietary Guidelines for Americans 2020–2025 . A sedenHabit formation techniques to help transform small conscious behavioral changes to automatic behaviors are an integral component of this program. The evaluation attempted to measure habit strength progress. However, the habit assessment tool, found to be acceptable and functional during preliminary testing, did not work with the pilot evaluation. Post-program participant interviews (not described here) indicated that the habit assessment was confusing and complex, preventing participants from responding in a meaningful way. Alternative formats were discussed. A simplified version will be tested during the next program implementation. The evaluation was conducted with a relatively small and homogeneous group of participants, not necessarily representative of the state. These participants gave high ratings to the program, but they also suggested program adjustments. The participant survey and the habit assessment were formatted for a traditional face-to-face experience. However, an electronic version of the program and the participant evaluation will now be developed to increase reach and facilitate both short and longer-term follow-up, even during times of social distancing. As modified, the SCHH program will be further tested with a larger, more diverse audience. The Small Changes Healthy Habits pilot program was evaluated for effectiveness at influencing healthy eating and physical activity behaviors among participants. This evaluation utilized participant self-reports over a four-week period; too short to see large and/or significant changes in behaviors—but enough to see significant progress in many important areas. These short-term results suggest that the SCHH program has the potential to positively affect healthy eating and reduce sedentary behaviors, both of which are fundamental to good health and wellness. The improvements in healthy eating and physical activity-related behaviors presented here are precisely the type of small changes that could have a large health impact over a lifetime.