Participant swim level was evaluated at the start and end of the program. Linear mixed-effects regression was used to examine the effects of Time (start/end), Age, Dose of Swim Practice , and ADHD status on the overall swim level. Participants significantly increased the swim level from the beginning to the end of the program , and participants with ASD and co-occurring ADHD had greater swim levels regardless of Time than those without ADHD . Overall, iCan Swim is effective in improving the swim skills of children and teenagers with ASD.Drowning is one of the leading causes of death in children and teenagers. Individuals with autism spectrum disorder (ASD) are at increased risk for drowning. Improvements in swim skills have been observed in children with ASD participating in learn-to-swim programs. However, it is unclear if age, co-occurring conditions, and/or the dose of practice influence swim skills in this population. To this end, a secondary data analysis of iCan Swim program data was conducted to determine the efficacy of the 5-day adapted learn-to-swim program for a cohort of children with ASD ages 3–16 years ( Drowning is one of the leading causes of death in children and teenagers . In 2019Individuals with ASD exhibit lower physical activity levels than their peers ,7,8. ThiNot only do learn-to-swim programs promote lifesaving skills, but they also create the path to access swimming as a recreation and sport, which is an excellent source of physical activity and exercise across the lifespan. Furthermore, swimming as a nontraditional team sport, affords individuals with ASD opportunities for social interactions while performing in either individual or team competition. Beyond Special Olympics Swimming, organizations such as USA Swimming and the United States Masters Swimming both have statements of inclusion, promoting the participation of individuals with ASD in competition. Despite these opportunities for participation and competition, environmental  ,10 and sSeveral studies have shown an improvement in swim skills in individuals with ASD participating in adapted learn-to-swim programs ,21,22,23Group lessons may be an effective means of increasing program capacity. However, there is limited research examining the efficacy of group swim programs for individuals with ASD. Indeed, the feasibility and effectiveness of group lessons on swim skill improvements for individuals with ASD, in particular, are unknown. Therefore, this study examined the iCan Swim learn-to-swim program’s efficacy on the development of swim skills in children and adolescents with ASD. Additionally, this study examined the effects of individual factors  on swim level. https://icanshine.org/program-hosts/ican-swim-hosts/ (accessed on 22 November 2021).A secondary data analysis of anonymized program data from nine iCan Shine swim programs was conducted across the United States from April–August 2019. The program iCan Shine is a 501(c)(3) that works toward inclusion for individuals with developmental disabilities in biking, swimming, and dancing. In addition, iCan Shine hosts iCan Swim camps across the United States each year. The program was implemented by a trained adaptive aquatics swim instructor on 5 consecutive days (Monday–Friday), 45–60 min each day. All of the data were acquired by the instructors and program director to track the progress of each participant during the program. This information was shared with parents and volunteers during the program and was used for setting goals for each session. These data were not previously evaluated statistically for research purposes. Detailed information on the iCan Swim program including the training for volunteers and swim instructors is found at: Briefly, as part of the iCan Swim program, the hosts recruit at least one volunteer per swimmer to register for the program. Volunteers must: Be 15 years or older, be comfortable in water, attend a 30-min Parent and Volunteer Orientation session, and attend a 90-min Volunteer Training (of which 60-min is in the pool). During the orientation and training, volunteers learn how the iCan Swim program operates, the expectations of volunteers, and how to work with swimmers during the program. Volunteers are assigned to work with each swimmer in order to help them learn basic swim skills . They provide encouragement and physical support  as needed. Volunteers may be local community members, family members, local swim instructors or lifeguards, etc. Fifteen minutes before each session begins, the iCan Swim instructors and swim instructors in training meet with the volunteers to debrief from the previous session and provide additional, individualized lesson plans for that session. For our study, the swimmers were divided into small groups with a ratio of six swimmers per two instructors. The instructors were certified Red Cross Water Safety Instructors and received training by the iCan Shine staff on implementing adaptive swim lessons for individuals with disabilities. With respect to the instructors, the iCan Swim program has two delivery models: One in which iCan Swim provides the adapted instructors and one in which iCan Swim provides the adapted aquatic instructor training (a minimum of 20.5-h) to implement the adapted curriculum for certified learn-to-swim instructors. The adapted aquatic instructor training includes: An introduction to adapted aquatics, basics of adapted aquatic skills, creating an effective class, strategies for success working with individuals with disabilities, specific techniques by diagnosis, behavior management strategies, equipment use, safety considerations, effective communication, managing volunteers, identifying and meeting swimmer goals, tracking progress, planning sessions, overcoming common obstacles, and implementing adapted aquatics in the community. Instructors must complete 4 h of classroom instruction and a minimum of 16.5 h in the pool instruction working with participants.The iCan Swim program consists of 5 consecutive days with each session lasting 45-min (3–7 years old) or 60-min (8 years and older). Each session has a minimum of three swimmers. To maximize capacity and instructor training opportunities, the program is generally run Monday–Friday from 8:30 a.m.–4:30 p.m. with 30-min breaks between sessions and a 60-min lunch break. A maximum of five sessions may be run daily. The lessons consisted of a group game/warm-up , skill practice , group game, and cheer . Participants receive instruction on safe entry/exit, breath control , strokes , floating/gliding (front/back), rolling (back-to-front and front-to-back). Each skill progressed from full support from a buddy, then moved to independent skills on a pool noodle if needed, and finally full independence without support. Swimmers were taught safety skills, including identifying a lifeguard, holding onto the wall for support, and using a lifejacket. Behavioral supports included picture schedules, visual supports, and token systems, which were employed when needed by the swimmer. Additionally, instructors used skill development tools and skill modeling, including hand-over-hand instruction and out-of-water skill practice. Equipment aids were used when swimmers required additional assistance with a concept or skill. For example, to help swimmers practice blowing bubbles, participants were instructed to blow a plastic ping pong ball across the top of the water. Skill assessments were conducted on Monday, Thursday, and Friday by the group instructors to determine an improvement in key skills . For this study, the overall swim level on the first and last day of the program was examined. The Institutional Review Board at Auburn University approved the exempt status of this secondary data analysis of anonymized data from the iCan Swim program (IRB#21-555 EX 2111). Therefore, no consent/assent forms were required. As part of the iCan Swim program intake forms, the parents report the primary diagnosis, which was used to identify participants with ASD.The iCan Swim program includes any individual ages 3 and older with a disability  of all abilities. If a participant has a G-tube, the stoma must be at least 2 months old. Participants with a tracheostomy are not eligible, as it is not possible to guarantee a splash-free environment. Participants with behavior problems are not excluded from the program. However, if their behavior is harmful to themselves or others, they may be asked to leave the program.For the purpose of this secondary data analysis, participants were included if parents reported a diagnosis of ASD, had complete data for the first and last day of the assessment, and were between the ages of 3–18 years. Note: Four children with ASD did not complete the assessments from the first or last day of the program and were not included in the present analysis. Although iCan Swim does not restrict the upper age range of swimmers, only two swimmers were older than 18 years in the present data. Given the present interest in children with ASD, these two participants were excluded from the analysis.n = 17, 16 male/1 female). Additional details regarding ASD or ADHD diagnosis were not provided . In addition to ADHD (n = 17), parents reported the presence of the following as “secondary diagnoses”: Anxiety (n = 3), apraxia (n = 3), cerebral palsy (n = 1), epilepsy (n = 2), global delay (n = 2), hearing impairment (n = 1), intellectual disability (n = 1), language delay (n = 6), non-verbal (n = 5), obsessive compulsive disorder (n = 1), oppositional defiant disorder (n = 2), sensory processing disorder (n = 4), and speech impairment (n = 3).A total of 86 individuals with ASD , ages 3–16 years (M = 8.84 years) were included in the study. A parent-reported secondary/additional diagnosis was used to identify swimmers with attention-deficit/hyperactivity disorder  could be used. The swimmers were leveled into one of four categories at the beginning of the program. Non-swimmers (NS) needed assistance at all times, could not complete or refused to attempt a back float or front float. This group is typically fearful of the water. Beginner-swimmers (BS) needed some assistance, could complete a back float or front float with the help of a noodle or the instructor. This group could hold on to a noodle and perform skills independently. Intermediate swimmers (IS) could swim two body lengths independently using any stroke. This group could also swim back to the wall independently. Advance-swimmers (AS) could swim independently for more than two body lengths, take a breath on their own, and learn additional swim strokes. The swimmers could then use any kind of stroke or technique to advance to the next level. Assessments were completed on Monday, Thursday, and Friday of the 5-day program. Data from Monday and Friday were examined in the present analysis.n = 22), the swimmer was fearful (n = 16), the swimmer’s behavior stopped their progress (n = 2), the camp was stopped early due to the weather (n = 2) or the buddy prevented the success of the swimmer (n = 1). In addition, for each of the non-swimmers, the instructors provided details at the end of the 5-day program as to why the swimmer did not progress to the beginner level. The instructor selected one or more of the following reasons: The swimmer needed more time ), and within-subjects fixed effect (Time (first/last session)). These fixed effects and interactions amongst the fixed effects were added iteratively to the model and each model was tested to determine the best fit model based on the AIC, BIC, and Log Likelihood. The best fit model included Age, Time Out of the Water, ADHD status, and Time. There were no significant interactions amongst these variables in the final model. T-tests were used to evaluate differences between groups. The level of significance was set to p < 0.05 for all of the analyses. A sensitivity analysis was conducted for each significant fixed effect factor based on the standard error of the observed coefficients in the linear mixed effects regressions, degrees of freedom, and 80% power based on the procedure described in Judd, McClelland, and Ryan (2017) [All of the statistical analyses were conducted using MATLAB, version R2018a . Linear mixed-effects regressions (LMER) were conducted to evaluate the factors that influenced the swim level. Age, Time Out of the Water (a proxy for a dose of practice), and ADHD status were between-subject fixed effects. Time was a within-subject fixed effect. A random intercept was modeled for each participant. Follow-up n (2017) .n = 20) were: The swimmer needs more time (n = 2), the swimmer was fearful (n = 13), and no reason was provided (n = 5). There was a significant difference in the amount of Time Out of the Water for the group that remained as non-swimmers, compared to those that progressed to the beginner level (T(40) = −2.74, p < 0.01), but there was no difference in age between the two groups (p > 0.05). The mean start and end levels were m = 0.63 (SD = 0.72) and m = 1.17 (SD = 0.91), respectively. The mean Time Out of the Water was m = 14.77 (SD = 33.7718). Of the 36 participants that started off as beginner swimmers, 17 increased to the intermediate swimmer level , and one increased to the advanced swimmer level . Of the six participants that started off as intermediate swimmers, all of them increased to the advanced swimmer level. p < 0.001). T-statistic, degrees of freedom, and p-values for each factor examined. A significant main effect of Time (p < 0.001) and ADHD status (p = 0.03) were observed. Overall, participants significantly increased their swim level from the start to the end of the program (Linear mixed-effects regressions (LMER) were conducted to evaluate the factors that influence swim level. The model was significant and accounted for 83% of the variance in swim level  = 185.12,  program , left. n = 4), the swimmer was fearful (n = 2), and no reason was provided (n = 11). In comparison, for the group without ADHD, the primary reasons provided for not advancing in level included the swimmer needs more time (n = 11), the swimmer was fearful (n = 13), and no explanation was provided (n = 45). Although the ADHD status did not interact with Time Out of the Water or Age, the participants with ASD and ADHD spent slightly less time out of the water than those without ADHD (T(84) = 1.96, p = 0.05), but were not different with respect to age (p > 0.05). Participants with ADHD exhibited a higher average swim level than those without ADHD regardless of time , right. p < 0.001). p > 0.05 for all).In addition, a significant Time Out of the Water x Time interaction was observed . For the Time x Time Out of the Water analysis, the critical beta would need to be −0.003. Given that the observed beta was −0.005, we had sufficient power to detect this effect.This study aimed to examine the efficacy of the iCan Swim group learn-to-swim program in children and adolescents with ASD. We further examined individual factors that influenced swim levels. Consistent with previous studies examining adapted swim lessons for individuals with ASD with either one-on-one or two-to-one designs ,20,22,23It is possible that the success of this program is due to the use of behavioral supports, as well as the quasi-individualized programming provided by the “buddy” in the water, along with the trained adapted swim instructors. It is also possible that the blocked practice (5-days in a row) helped in maintaining motor skill gains for each day of the program. In comparison, Yilmaz et al. met for 10 weeks (3x/week) , and theInterestingly, age was not a significant factor in the development of swim skills. A previous study by Anderson and Rodriguez found that older typically developing children  develop swim skills at a faster rate than younger typically developing children  . AlthougIndeed, when evaluating the reasons provided by swim instructors for children not progressing from the non-swimmer category, most of the children were reported to “need more time” or were fearful. Therefore, a greater dose of practice  may be needed to compensate for time out of the water and enable more time for the children to decrease fear/anxiety to progress in their swim skills. Future programs should provide similar details to determine why children spend time out of the water and do not progress in their abilities.A quantitative comparison between the two groups found that the groups with ADHD spent significantly less time out of the water than those without ADHD. Participants with ADHD exhibited a higher swim level than those without ADHD regardless of time. In addition, a qualitative comparison between the two groups revealed a smaller percentage of non-swimmers in the group with ADHD (24%) compared to those without ADHD (55%). Moreover, the percentage of children with ADHD that were reported as having fear as a primary reason for not progressing in swim skills (12%) was lower than those without ADHD (19%). Therefore, it appears that the group with ADHD may have previous experience participating in swim programs, which may have resulted in less time out of the water and less fear compared to those without ADHD.While over half of the participants improved their swim skills in 5 days, only eight of the participants reached the highest level of swim proficiency measured. This means that many participants will need additional swim lessons in order to continue to gain the skills to be proficient and independently safe swimmers. The participants in this study did not disclose previous swim lesson experience. To this note, however, Caputo et al.  showed tn = 17/86). Therefore, these results would require replication with a larger sample of individuals with ASD and co-occurring ADHD. In the future, a waitlist control randomized design rather than a true control design would be appropriate for future studies, given the small number of programs and large demand for adapted aquatics. First, the present study evaluated the efficacy of a program developed by iCan Swim and only included the program participants. There was no control group in the present study, nor the randomization of participants to the intervention. This study is not a randomized control trial, and thus the following threats to the internal and external validity exist: Testing (test-retest improvement), selection bias , experimenter effects (due to different instructors), and testing effects (less anxious during the last session of testing). Future randomized controlled trials employing these effects will be necessary to exceed the critical beta value based on the 80% power. The observed beta value for the ADHD effect did not exceed the critical beta value based on the 80% power. These results suggest that this analysis was underpowered due to the relatively small sample of individuals with co-occurring ADHD in the present sample (n = 86 participants). However, this is a moderate to large sample compared to the previous studies evaluating adapted learn-to-swim programs or even other adapted physical activity programs [Second, relative to large-scale clinical or epidemiological studies, this study has a small sample size  based on the DSM-5, nor was there information regarding symptom severity. Moreover, there was no additional information regarding medication use. It would be useful for future iCan Swim or adapted swim programs to collect more detailed information regarding ASD and other diagnoses, ASD symptom severity, and medication use. These factors may affect the swim level and behavior during the program. The program asks about communication needs/abilities in their registration material, these data were not available in the dataset for which this secondary data analysis was conducted. However, participants that are non-verbal, minimally verbal or use alternative communication  are not excluded from program participation. Participants with behavioral problems are also not excluded from participation. However, if their behavior is harmful to themselves or others, they may be asked to leave the program. Therefore, it is likely that the program included participants with the full range of ASD severity. Importantly, this study includes the largest sample of children with ASD participating in a learn-to-swim program, and thus provides valuable information on the efficacy of group swim lessons for this population.Fourth, this study was conducted from a single year of programming from a relatively new program (onset 2015). Future studies are needed to replicate and extend this study with additional years of programming. Fifth, no follow-up swim skill assessment was completed to determine skill retention following program completion. Given the number of participants and the geographic spread of program participants across the US, it would be very difficult for iCan Swim to conduct retention assessments. To date, no studies of adapted swim programs for individuals with ASD have collected retention data. Additional studies are needed to determine the retention of swim skills following adapted swim programs in children with ASD. Sixth, this program is short. Future studies are required to determine the dose of programming (how many sessions), for which a non-swimmer with ASD on average becomes an independent and safe swimmer. Seventh, although iCan Swim conducts post-program surveys (via survey monkey) with all the parents and volunteers, there is no question that asks parents about their child’s diagnosis or asks volunteers about the diagnosis of the swimmer with whom they worked. In addition, it is possible that parents had multiple children participate in the program and volunteers may have worked with multiple participants in the program. The survey is completed once by the parents and volunteers, thus it is not possible to separate the responses with respect to each participant. Therefore, it is not possible to determine the unique responses of parents of children with ASD or the volunteers that worked with a child with ASD during the program. In the future, the parent survey would need to be completed for each participant. In addition, a question specifying the participant’s disability would need to be added to determine the satisfaction with the program, specifically for parents of children with ASD. Moreover, the volunteer survey would need to be completed for each participant and a question specifying the diagnosis of the swimmer with whom they worked would also need to be added. n = 17/86). Therefore, these results would require replication with a larger sample of individuals with ASD and co-occurring ADHD.Finally, although the observed beta values for the Time and Time x Time Out of the Water effects exceeded the critical beta value based on the 80% power, the observed beta value for the ADHD effect did not exceed the critical beta value based on the 80% power. These results suggest that this analysis was underpowered due to the relatively small sample of individuals with co-occurring ADHD in the present sample (


Non Fungible Tokens (NFTs) are digital assets that represent objects like art, collectible, and in-game items. They are traded online, often with cryptocurrency, and are generally encoded within smart contracts on a blockchain. Public attention towards NFTs has exploded in 2021, when their market has experienced record sales, but little is known about the overall structure and evolution of its market. Here, we analyse data concerning 6.1 million trades of 4.7 million NFTs between June 23, 2017 and April 27, 2021, obtained primarily from Ethereum and WAX blockchains. First, we characterize statistical properties of the market. Second, we build the network of interactions, show that traders typically specialize on NFTs associated with similar objects and form tight clusters with other traders that exchange the same kind of objects. Third, we cluster objects associated to NFTs according to their visual features and show that collections contain visually homogeneous objects. Finally, we investigate the predictability of NFT sales using simple machine learning algorithms and find that sale history and, secondarily, visual features are good predictors for price. We anticipate that these findings will stimulate further research on NFT production, adoption, and trading in different contexts. The Art Newspaper podcast1, signalling both the impact of Non Fungible Tokens (NFTs) on the art world and the novelty they represent for most of the general public. The revolution is not confined to the art market. While NFT adoption in gaming has already reached a certain maturity, for example concerning the trade of in-game objects, different other industries, especially those involved with the production of digital content such as music or video, are experimenting with the technology. Overall, in the first four months of 2021, the NFT volume has exceeded 2 billion USD, ten times more than the entire NFT trading volume in 20202.“WTF are NFTs? Why crypto is dominating the art market” is the title of the February 21, 2021 episode of 3. More broadly, an NFT allows to establish the “provenance” of the assigned digital object, offering indisputable answers to such questions as who owns, previously owned, and created the NFT, as well as which of the many copies is the original. Several types of digital objects can be associated to an NFT including photos, videos, and audio. NFTs are now being used to commodify digital objects in different contexts, such as art, gaming, and sports collectibles. Originally NFTs were part of the Ethereum blockchain but increasingly more blockchains have implemented their own versions of NFTs4.So, what’s an NFT? An NFT is a unit of data stored on a blockchain that certifies a digital asset to be unique and therefore not interchangeable, while offering a unique digital certificate of ownership for the NFT5. In December 2017, CryptoKitties congested the Ethereum network6. By many considered a chief example of the irrationality driving the cryptocurrency market in 20177, CryptoKitties remained the only popular example of NFTs for almost 2 years. In July 2020, the NFT market started to grow2 and attracted a huge attention in March 2021, when the artist known as Beeple sold an NFT of his work for $69.3 million at Christie’s8. The purchase resulted in the third-highest auction price achieved for a living artist, after Jeff Koons and David Hockney9. Several other record sales followed11: three Cryptopunks—a collection of 10,000 unique automatically generated digital characters—were sold at $11.8, $7.6, and $7.6 million dollars, respectively; the first tweet was sold at $2.9 million dollars; and the Auction Winner Picks Name, an NFT with music video and dance track, sold at $1.33 million dollars. The profitability of NFTs has motivated celebrities to create their own NFTs, with collectibles of NBA and famous football players getting sold for hundreds of thousands dollars12.The first popular example of NFTs is CryptoKitties, a collection of artistic images representing virtual cats that are used in a game on Ethereum that allows players to purchase, collect, breed, and sell them on Ethereum3; components, protocols, standards, and desired properties13; new blockchain-based protocols to trace physical goods14; and the implications that NFTs have on the art world16, in particular as they allow to share secondary sale royalties with the artist. Empirical studies aiming at characterizing properties of the market have focused on a limited number of NFT collections, such as, CryptoKitties18, Cryptopunks, and Axie19, or on a single NFT market, such as, Decentraland20 or SuperRare22. These analyses revealed that the digital abundance of NFTs in digital games has led to a substantial decrease of their value17, and that, even if NFT prices are driven by the prices of cryptocurrencies19, the NFT market could be prone to speculation20. Further, it was shown that NFTs valued by experts are more successful21, and that, based on 16,000 NFTs sold on the SuperRare market, the structure of the the NFT co-ownership network is highly centralized, and small-world-like23.Research on NFTs is still limited, and focuses mostly on technical aspects, such as copyright regulationsIn this paper, we provide a first comprehensive quantitative overview of the NFT market. To this end, we analyse a large dataset including 6.1 million trades of 4.7 million NFTs in 160 cryptocurrencies, primarly Ethereum and WAX, and covering the period between June 23, 2017 and April 27, 2021. We start by analysing the overall statistical properties of the NFT market and its evolution over time. Then, we study the network of interactions between NFT traders, and the network of NFT assets. NFTs are further clustered based on their visual features. Finally, we present the results of regression and classification models predicting the occurrence of NFT secondary sales and their price.24, a specialized company that track NFTs sales, and OpenSea25, one of the largest NFT marketplace. However, the exact classification of different categories in which NFTs are used is outside of the scope of the present paper. For example, Art objects can be in some cases classified as Collectibles, while some Game objects may present sophisticated aesthetic and cultural properties that may qualify them as Art.We break down our analysis by NFT categories, which are classified by manual inspection, with references to the classification proposed by NonFungible Corporationcollections, sets of NFTs that, in most cases, share some common features. Collections can be widely different in nature, from sets of collectible cards, to selections of art masterpieces, to virtual spaces in online games. Most collections can be categorised in six categories: Art, Collectible, Games, Metaverse, Other, and Utility  for each category  purchases at least one NFT from the latter (the seller). Each link has a weight corresponding to the total number of items that the buyer bought from the seller.How do traders interact with each other? Are there central actors? We approach these questions adopting a network science approachs, defined as the total number of purchases and sales made by each trader, is distributed as a power law d, with First, we study the behaviour of individual NFT traders by focusing on properties of the nodes. We find that traders activity is highly heterogeneous: the strength of traders (nodes) see Fig. a, such tsee Fig. b. This rsee Fig. c. A specsee Fig. ”.Figure r28, that measures the correlation between the sum of the weights of all outgoing links (the outgoing strength) of a given node with the average sum of the weights of incoming links (the incoming strength) of its neighbours. We find that the assortativity, which takes value Secondly, we turn to properties of the network links, describing interactions between pairs of traders. We find that the distribution of link weights is well characterized by a power law distribution, with the top 10% of buyer–seller pairs contributing to the total number of transactions as much as the remaining 90% see “”. An int29 of the network under this partition of nodes. The modularity is a metric bounded between Q of the collections partition is Finally, we focus on the network structure. Building upon the result that traders are specialized, we assign each trader to their top collection, and we study the modularity001 see “”. It revnetwork of NFTs, where nodes are NFTs and a directed link exists between two NFTs that are purchased “in sequence”, e.g. a link is created from an NFT to another when a buyer purchases the former and then the latter, with no purchases between the two  into NFT collections. We find that the modularity Q of the collections partition is 30, defined as groups of nodes such that, starting from a given NFTs, it is possible to reach any other NFTs in the SCC following directed links. The largest SCC include NFTs traded in the WAX blockchain, consisting of 35% of all NFTs, while the second largest includes NFTs traded in the Ethereum blockchain, consisting of 20% of all NFTs  between them, a value that goes from zero  to one (for highly different images). We measured such distance between pictures within the same collection and across collections.NFTs are linked to digital assets of different types, including videos, text, animated GIFs, and audio. Currently, the most popular NFTs are imagesThe average CD calculated between items which belong to the same collection is significantly lower  to the AlexNet vectors. PCA uses linear combinations of the 4096-dimensional vectors to project them into vectors with an arbitrarily lower number of dimensions and such that the variance of datapoints in the projected space is maximized. Considering the whole sample, which consists of about 1.25 million graphical objects, the first five principal components explain together about the 38.3% of the total variance, progressively distributed from PC1 to PC5 as follow: 20.3%, 7.3%, 4.0%, 3.8% and 2.7%. The PC1 to PC5 scores are used to test the capacity of visual features for predicting sales (see next subsection), while PC1, PC2, and PC3 for visually representing the data through a 3D scatter plot and showing intra-categories homogeneity . Last, the prediction is most accurate when trying to predict the occurrence of a secondary sale over longer periods of time  we analyzed the main properties of the market, (2) we built and studied the traders and NFTs networks and found that most traders are specialised, (3) we showed that NFT collections tend to be visually homogeneous, and (4) we explored the predictability of NFT prices revealing that, while past history is as expected the best predictor, also NFT specific properties, such as the visual features of the associated digital object, help increase predictability.33 and their clustering in a lower-dimensional space34, machine learning for price prediction35, or market modelling36. We also did not consider collective attention as measured e.g. from social media or Wikipedia, which can be a further source of information about market behaviour39. Fourth, we considered mostly the Ethereum and WAX blockchains, but several other platforms offer smart contracts and NFTs. Finally, our price prediction exercise did not include information about the creator of the  object associated to the NFTs. While this is due mainly to the dataset, and in many cases the identity of the creator is not available or does not exist , it is likely that in certain contexts, and specifically for art, this can be an important aspect to consider.It is important to highlight the main limitations of our study, which represent also directions for future work. First, we gathered data from a variety of online NFT marketplaces and not directly from the Ethereum or WAX blockchains, so that we have likely missed a number of “independent” NFT producers. Second, we mostly adopted an accepted categorisation for the NFTs, which includes a number of arbitrary decisions and could however be further refined (as every categorization). Third, since our primary goal was to provide a general overview of the market, we did not extensively explore all the available methods e.g., for the features extraction from imagesOverall, NFTs are a new tool that satisfies some of the needs of creators, users, and collectors of a large class of digital and non-digital objects. As such, they are probably here to stay or, at least, they represent a first step towards new tools to deal with property and provenance of such assets. We anticipate that our study will help accelerate new research on NFT in a broad array of disciplines, including economics, law, cultural evolution, art history, computational social science, and computer science. The results will also help practitioners make sense of a rapidly evolving landscape and inform the design of more efficient marketplaces as well as the associated regulation.We summarize our data collection below and provide a detailed description of our data manipulations in “24, a company that tracks historical NFT sales data to build NFT valuations. Other Ethereum blockchain data were downloaded from four open-source APIs: CryptoKitties sales40, Gods-Unchained41, Decentraland42, and OpenSea43. With OpenSea that allows trading in multiple cryptocurrencies. We also monitored the WAX blockchain, through tracking transactions in the Atomic API44.Our dataset includes only transactions representing purchases of NFTs, whose ownership change following that transaction. We exclude from our analysis any transactions representing the minting of NFTs or bids during an auction. We track different cryptocurrencies. Etherum blockchain data for the collections SuperRare, Makersplace, Knownorigin, Cryptopunks, and Asyncart were shared by NonFungible CorporationArt consisting of digital artworks such as images, videos, or GIFs; Collectible representing items of interest to collectors; Games including digital object used in competitive games; Metaverse consisting of pieces of virtual worlds; Utility representing items having a specific function; and Other including the remaining collections. More details on the NFT categorization are explained in “We group NFTs into six categories: 45 implementation of AlexNet46, a deep convolutional neural network architecture designed for image classification. We initialize AlexNet with weights pre-trained on ImageNet47, a widely-used reference dataset of labeled images. Given an image in input, AlexNet passes it through multiple layers of transformation. The second to last layer  is a vector consisting of 4096 values that constitute a dense representation of the input image into a high-dimensional space. These vectors can be used for a variety of tasks such as similarity ranking, clustering, or classification. To reduce the dimensionality of AlexNet vectors, we extracted their principal components using Principal Component Analysis (PCA)48, and selected the five most relevant ones. PCA projects each point of the high-dimensional space into a space with a desired number of dimensions, while preserving the data variation as much as possible.For each NFT in our dataset (except for less than 3000 exceptions) we managed to collect at least one URL that points to a copy of the NFT’s digital object. We focused only on objects with image file formats  and GIFs, for a total of about 1.2 million unique graphical objects associated with 4.7 million unique NFTs. Note that a single digital object can be related to multiple NFTs; this happens for example for identical playing cards that are minted in multiple copies, each associated with a different NFT. Since our algorithm for visual feature extraction works with static images, we converted the animated GIFs to PNGs by extracting central frame of each GIF. In order to succinctly represent the visual features that characterize an image, we encode it into a latent space using a neural network. Specifically, we pick the PyTorchSupplementary Information.


Introduction: Job satisfaction is vital for the optimal functioning of medical practitioners. Herein, we report our experience of restructuring the internship program by identifying the gaps, developing, implementing strategies to overcome gaps and sharing the results of the pre-implementation and post-implementation audit, as an example for establishing a system for improving intern’s work-based learning and satisfaction in a university hospital setting.Methods: Using Kern’s six-step instructional model, a prospective mixed-method study was conducted at Aga Khan University Hospital. In phase 1 (2013) gaps were identified by evaluating various aspects of the internship program. Strategies were developed and implemented to overcome the identified gaps. In phase 2 (2014-2016) the impact of these developmental strategies was assessed.Results: A total of 65 interns, 30 residents, and 22 faculty members participated in phase I, while 71 interns participated in phase II. The reformation of orientation sessions, including practical exposure and content of sessions, opportunities to enhance hands-on experience and supervision in inpatient areas, operating rooms, supervision by fellows, supervision for hands-on procedures, career counseling, and mentorship, led to significant improvement in satisfaction. It was identified that the lack of hands-on opportunities can be overcome by surgical skills-based workshops. These reforms led to an overall rise in intern satisfaction .Conclusion: Periodic restructuring of an existing program helps to improve the work-based learning experience and overall satisfaction among interns. This not only maximizes learning but also eases interns into their postgraduate life and workload subsequently enabling them to become more competent and well-rounded health practitioners. Job satisfaction among hospital employees has been a topic of extensive research . Lower jStudy design and settingThis was a prospective mixed-method study conducted at AKUH in two phases during 2013-2016. The study was designed to identify the gaps in the internship program at AKUH and to develop and implement the strategies to overcome the identified gaps (phase I), and to evaluate the impact of these strategies in improving the quality of the internship program after one year of implementation (phase II).The study was approved by the hospital Ethics Review Committee of AKUH, approval 4327-MED-ERC-16. The study participants were the interns, residents working in various subspecialties, program directors, and coordinators of various disciplines. The study was conducted in two phases. In phase I, to identify the gaps, interns who had graduated from the internship program in the preceding two years (2011-2012), as well as the interns who were enrolled during the year 2013, residents working in various subspecialties, program directors, and coordinators of various disciplines were surveyed using a cross-sectional study design through a standardized questionnaire. At the end of phase I, based upon feedback from study participants, strategies were designed and implemented. During phase II the impact of implemented strategies on the quality of the internship program was measured by using a “single-arm cohort” study design. This was done by approaching the interns subsequently enrolled in the internship program for the next three years (2014-2016) using the same questionnaire filled out by the interns in phase I. All these questionnaires were standardized for objective measurement of responses and were pre-tested before administration. The internal consistency of each questionnaire was evaluated by calculating Cronbach’s alpha.Data collection and enrollmentTo evaluate and identify the gaps in various aspects of our internship program and to evaluate the impact of strategies, the Kern six-step model  was usedDetails of internal and external reviews conducted in 2012 were reviewed. During phase I (2013) focus group discussions were carried out with small groups of interns, residents, and faculty members. Separate questionnaires were designed for interns, residents and faculty members, focusing on three broad categories: 1) orientation of the internship program 2) intern’s sub-specialty rotations 3) on job facilities and overall quality of the program. These questionnaires had both open-ended and closed-ended questions and utilized a 7-point Likert scale for ratings. The questionnaires were pre-tested and their internal consistency calculated through Cronbach’s alpha was 0.832, 0.971 and 0.981 for interns’, residents’ and faculty’s questionnaires respectively.Gaps were identified and potential areas for improvement were prioritized. Strategies were developed to overcome the gaps and after necessary approvals, these strategies were subsequently implemented. The program was re-evaluated at the end of the year for the next three years, i.e. 2014-2016, using the same questionnaire.Data was entered and analyzed by using SPSS version 22 . Frequencies (%) were calculated for categorical variables and mean ± SD for quantitative variables. The responses were measured using a Likert scale on an ordinal rating scale value from 1 to 7 and were further converted into two nominal categories of ≤4 and >4, which were represented in terms of frequencies and proportions. A cut-off value ≤4 was considered as “not satisfied” and >4 as “satisfied”.To measure the difference in the quality of the program after implementing the strategies, the ordinal response data of phase I and phase II interns were compared by the Mann-Whitney U test. A p-value of <0.05 was considered significant. Qualitative assessment was done for open-ended questions and all responses to open-ended questions were transcribed for qualitative analysis. After reviewing all the responses, a coding frame was constructed. Topics in these responses were identified and categorized into themes and analyzed as such by calculating responses in percentages.A total of 65 interns, 30 residents, and 22 faculty members participated in phase I (pre-implementation) and 71 interns participated in phase II (post-implementation). Among interns, 64% were graduates of private-sector medical colleges. The mean age of interns was 24.8 ± 1.1 years and 68.4% were female. Most of the interns aimed to join the AKUH internship program due to the high impact of the institution with training facilities matching international standards, structured programs with a secure workplace, and good salary structure. The responses of participants in phase 1 are described in Table Orientation sessionsBefore the commencement of the internship, the PGME arranges a three-day mandatory orientation session for interns, for which they are paid as well. The utility of these sessions was assessed Tables , 2, 3. TRotations in various subspecialtiesThe PMDC requires each intern to rotate in subspecialties related to Medicine and Surgery for the duration of six months each. Institutions can adjust rotations in a given domain as per availability. The AKUH offered a fixed rotation schedule with several combinations of sub-specialties. However, such preset combinations were not offered in medicine. In phase 1 approximately 63% were satisfied with the available combinations of rotations and almost one-third of interns found rotations in compliance with objectives. Hence, restructured 12 months of rotational plans were implemented consists of various combinations of rotations that interns can choose during orientation based on their admission test merit. The learning objectives of all rotations were developed and shared beforehand. Although the difference was not statistically significant, the mean rank score was higher for compliance with objectives in phase II.The satisfaction level was low for learning opportunities in phase I. Hence, in each rotation opportunities were created to enhance hands-on experience and supervision. A statistically significant improvement in satisfaction was noticed for inpatient areas , operating rooms , supervision by fellows  and supervision for hands-on procedures  . Though the mean rank score for the revised content of mandatory sessions was higher in phase II (67.4 vs 57.0), the difference was not statistically significant. A similar observation was found in the case of facilities at the workplace and compliance with the AKU policy of 80 hours/week averaged over three months.Surgical skills workshopConsidering gaps in teaching basic surgical skills, simulation-based surgical skills workshops were conducted biannually providing an opportunity to learn and improve their basic surgical skills. These skills taught during workshops were basic instrument handling skills, suturing techniques, hand tying, urethral catheterization, back slab application, chest tube insertion, nasogastric tube insertion, gowning and gloving, the method of using a hand-held doppler, wound care and the method of performing a sterile dressing in the simulation lab. In the end, the overall satisfaction level for the internship program was assessed. The proportion of interns satisfied with the program was significantly higher in phase 2  and the mean rank for overall satisfaction was also higher in phase 2 .Maintaining high job satisfaction while facilitating work-based training for a diverse group of young medical graduates is a huge challenge ,14. LikeAnnually 76 interns are enrolled after a merit-based, competitive evaluation process of written exam and OSCE with approximately more than 500 applicants from all over the country having diverse academic backgrounds. Based on an elaborate external review conducted in 2012, the Internship Committee at AKUH revamped its program, adopting the six-step approach described by Kern as an instructional design process ,15. RateOur results supported the fact that introducing fundamental changes to an existing program can improve overall satisfaction for a work-based training program. The role of regulating working hours and introducing mentoring opportunities has been shown to improve job satisfaction, and our results further endorse this notion ,10. The Restructuring of orientation program with exposure in clinical areas under supervision provided an opportunity to familiarize interns with the existing systems, policies operating mechanisms, safe handovers, and daily working mechanisms. With the implementation of the 12-month rotational plan, interns were able to select rotations of their choice. Sharing of learning objectives of all rotations beforehand helped the interns to focus on what they can learn during a specific rotation. The restructuring of mandatory teaching sessions and topics covered found useful. These results were consistent with existing evidence. For instance, sharing a checklist of expected competencies and desired objectives at the start of surgical rotation was found to be associated with better performance and satisfaction levels . HoweverIn a busy tertiary care hospital with multiple learners including fellows, residents, interns, students creating equal opportunities for all and supervision by faculties at the workplace is difficult and this is what interns have reported in our study as well . HoweverSurgical skills workshop, in particular, not only allowed an opportunity for interns to learn the basics of surgical procedures but also a close interaction with surgical faculty, which helped to improve their work environments. Following the same strategy, periodic workshops to enhance other competencies including communication skills are expected to improve the desired outcome.The internship year is unique, bridging between two immensely established, structured, curriculum-based programs i.e. undergraduate and postgraduate medical education. While providing opportunities to groom in challenging patient-centered workplace, rotations in various subspecialties help them to choose a further course of their careers ,18. MostOur study was thus able to conclude that while the internship year is extremely stressful and demanding, several interventions can be put into place to improve work-based learning and satisfaction among interns thus improving their experience and performance. In this two-phase study, in light of the responses received in the initial round, we were able to effectively restructure different aspects of the internship for the second phase, thereby significantly increasing intern satisfaction and learning. Thus the regular audits, feedback from all stakeholders, and adopting strategies to improve an academic program will help to achieve the desired outcomes. These attempts subsequently enable the interns to become more competent and well-rounded health practitioners.


Language diversity is increasing in the United States. This growth has implications for language preference, cost, quality, and client outcomes in health services settings. However, language diversity among medical and allied health professionals is lacking. Education pipeline programs are a mechanism to prepare bi- and multi-lingual diverse students to enter health careers. The Community of Bilingual English-Spanish Speakers Exploring Issues in Science and Health (CBESS) is one such program. Through peer mentorship from Leadership Trainees (LT), and a multicomponent 17-month education curriculum, CBESS was designed to increase interest in STEM careers among English-Spanish bilingual high school youth. In 2020, the COVID-19 pandemic interrupted high school students’ education and forced programs to innovate. CBESS was no exception. The most significant modifications were to a) expectations of SRs for a successful Summer Virtual Research Program (SVRP), b) LT roles, and c) scope and delivery of summer science content. A preliminary evaluation was conducted from data collected through pre-post surveys, process data, and focus groups. Among the outcomes were a significant increase in science knowledge among SVRP youth participants as well as no significant differences between cohort 1 and 2 suggesting that changes did not impede program goals. LTs highlighted skills needed and role of mentors. Adaptations were successful and will continue with the 2021 cohort. Racial, ethnic, and language diversity continues to grow in the U.S., yet ethnic and linguistic diversity within health careers is not keeping up with current demographics. In the United States, language diversity is at an all-time high with one in five households speaking a language other than English at home . FurtherCost, quality, and health outcomes are influenced, in part, by language barriers. Service seekers who encounter language barriers have unequal access to healthcare services and more adverse health outcomes compared with patients who do not experience language barriers . For exaDespite the growing number of students overall, college-enrolled Latino students still are not representative of the general population. During 2016-2017, 19% of undergraduate students and approximately 11% of graduate students across US institutions identified as Latino . It is wThe CBESS program is a cohort-based, multicomponent Science, Technology Engineering, and Math plus health  education program. CBESS is co-led by two principal investigators (authors 6 and 7), coordinated by the second author, and housed within the Raggio Research Center for STEM Education which is directed by the fourth author at the University of Nevada Reno (UNR). CBESS leadership collaborate with inter-collegiate, local and state partners and stakeholders, along with pipeline components of the Nevada IDeA Network of Biomedical Research Excellence (INBRE). CBESS was developed to increase interest in health and STEM careers among English-Spanish bilingual youth in northern Nevada. Three components of the CBESS program are: a) Leadership Trainees (LTs) or near-peer mentors; b) Student Researchers (SRs) who participate in the 17-month multicomponent STEM + health education program; and c) a Community of Practice (CoP) , which sThis section describes the participants, program structure, and programmatic adaptations that were made to the CBESS program because of the COVID-19 pandemic as well as the preliminary evaluation approach.CBESS began in fall 2017 with funding from the Science Education Partnership Award (SEPA) of the National Institutes of Health. This multi-component education research program utilizes a well-matched intervention-comparison evaluation design. Annually, cohort recruitment begins during the fall months for undergraduate Leadership Trainees (LTs), high school Student Researchers  and the comparison group. Each year five LTs are recruited from across the UNR campus, including the Nevada INBRE biomedical student pipeline program. Eligible LTs are bilingual upper-level undergraduate students interested in healthcare and/or STEM careers. The LT opportunity is posted to CBESS social media platforms, the Latino Research Center bi-weekly newsletter, the CoP network, teacher preparation programs and paper flyers hung in various UNR buildings that house STEM fields . Applicant selection is based on interviews, a review of the applicant transcripts and experience, and expressed interest in STEM + health fields. The application review process and participant selection is determined by a committee comprised of CoP members and CBESS leadership team. Promising applicants are invited to an interview where they are asked questions in Spanish. This is an opportunity to gauge bilingual level. Over the 17-month program and under the supervision of the CBESS program coordinator (author 2), the LTs receive extensive professional development training to serve as near-peer mentors SRs and assist the CBESS program coordinator during the 17-month program. LTs are provided a $4,000 incentive for their participation.Each year, a cohort of up to 32 student researchers (SRs) and 64 comparison students are recruited from various high schools in northern Nevada, predominantly Washoe County school district, and neighboring areas. For example, cohort 1 (C1) enrolled 32 participants while in 2020,28 SRs were enrolled. For each enrolled SR, two well-matched comparison students, based on demographics, are enrolled for a 1:2 intervention:comparison ratio for program evaluation. CBESS applicants who were not selected and students who initiated an application but did not complete it were invited into the comparison group. Students in the comparison group do not participate in any CBESS activities, completed evaluation materials at the same time as intervention students and received a $150 gift card incentive at the end of the 17-month program.The SRs are predominantly first-generation college goers, and bilingual Spanish-English high school students who have an interest in STEM and/or health careers. Recruitment occurs using multiple strategies including CBESS social media platforms, the CoP network, interaction with high school counselors, and presentations to high school health classes. A promising strategy has been to include a CBESS SR alumni to talk to interested students about CBESS programming and expectations. The program coordinator provides contact information in case an interested student has additional questions. A holistic application process is utilized. Applicants submit demographic and academic information, two letters of recommendation , three short essays that respond to questions about career interest and experience, and one required essay written in Spanish to gauge bilingual level). Like LTs, the application review process and participant selection are determined by a CoP member and leadership team committee. SRs do not receive an incentive nor are the LTs or SRs required to pay any out-of-pocket fees for their participation in CBESS. An inquiry-based professional development component positions the LTs as STEM insiders and prepares them to engage SRs in STEM-healthcare content as mentors . STEM-heThe 17-month program for SRs begins in early Spring with 4 monthly career exploration events  reflective of the inputs, activities, or outputs, and short-, medium-, and long-term impacts. Several methods were used to evaluate the CBESS program and inform necessary changes. First, a summative evaluation of the CBESS program was completed using a pre-post survey for the summer research program. Student researchers and the comparison group were provided the pre-survey to complete at the time that they agreed to participate in the program. The post test was completed immediately after the summer program and at the end of the 17-month program by both the SRs and the comparison group. To assess the comparative impact of the in-person model and the virtual model, the results of the post-assessment survey, administered immediately after the three-week summer research program, were compared.The survey questions consisted of closed- and open-ended questions, administered electronically through the UNR-licensed Qualtrics system. Closed-ended questions were adapted from the National Science Foundation Scientific Work Experience Programs for Teachers . The AcaTo understand the experiences of the LTs, a semi-structured focus group with four of the five cohort 2 LTs (80%) was conducted to get their perceptions on being a mentor and having a larger role delivering the summer content. The semi-structured focus group guide contained the following questions: Describe a way in which CBESS helped you gain skills for your current academic or work status; How were you able to assess student’s knowledge and skills to mentor them throughout this program? and; Would you recommend this LT experience to a friend? Why or why not?This focus group was conducted using Zoom Video Communication in April 2021. The focus group was audio and video recorded and lasted 60 minutes. The focus group was facilitated by a CBESS intern and MPH student at the University of Nevada. The fifth author and external evaluator observed the focus groups and asked several questions of the participants to further clarify any questions. Having two facilitators who were external to the CBESS program was ethically important to ensure that LTs did not feel coerced to participate or did not censure responses. The focus group was transcribed verbatim and thematically analyzed .Before implementing the SVRP, CBESS faculty conducted a brief survey to ascertain what concerns SRs had about a virtual curriculum, if SRs were interested in learning more about COVID-19, and how many hours a day they were willing to engage in synchronous activities during the 3-week program. The primary concerns participants expressed were related to access to remote learning technology. For instance, several students either did not have access to a computer or shared a device with other family members. Unstable Wi-Fi signal was also mentioned as a primary concern. In addition, one student was concerned about not having and needing to purchase materials that would be required for at-home experiments. Fortunately, CBESS faculty had previously purchased laptop computers for use by the first cohort and these were readily available to loan to SRs for the duration of the SVRP.In the first CBESS cohort, the University’s online instruction management system was minimally utilized . In prepRegarding the number of hours per day SRs were willing to engage in synchronous activities, the average response was 3.4 hours, with responses ranging from one to six hours. As most SRs self-identified as Latino, and given that COVID-19 was impacting Latino/Hispanic families in greater numbers see  - both iCBESS faculty made all expectations very clear prior to beginning summer programming during “Kick-Off” meetings with each LT / SR group, which helped set up a standard that was maintained throughout the three-week program. This included small details, such as expecting all SRs to log on 10 minutes ahead of start time to ensure that all technology was working properly. If an SR was not logged on by five minutes to start time, their LT would contact them via text or phone call. This ensured that all students were ready to go, and that no time was wasted at the beginning of each daily session. CBESS continues to implement this approach with the current cohort, during CEEs, and will do so again in the next virtual summer program in the 2021/22 cycle. To keep SRs engaged during the SVRP, CBESS team incentivized participation by having weekly raffles for students with university swag as prizes. For instance, students were entered into the raffle each time they asked a question or offered an answer or meaningful comment during synchronous programming. This strategy was not used during the in-person program.SRs were expected to maintain professional and respectful etiquette throughout the Zoom activities. Most of these expectations were specific to on-line etiquette which is different than in-person etiquette because of the ability of participants to do other things simultaneously without being detected . Because of the physical distance, corrections took longer . CBESS was also aware that all students would not have a designated desk space, but during orientations, personnel discussed potential strategies with students on how to create a space where they would be able to participate with minimal distractions.The LT training occurred as normal see , albeit As a result, in 2020, formal mentoring began prior to the summer program, whereas in the first cohort mentoring began after the completion of the summer program see . For theAdditionally, LTs were required to take on several distinct roles during the SVRP, including monitoring and tracking daily attendance and participation of the SRs. In cohort 1, the SRs were living on the UNR campus and attendance was immediately known because the groups lived and traveled together. Whereas cohort 2 SRs were able to log in and out of the Zoom sessions potentially without anyone noticing. Thus, LT involvement in tracking student participation became necessary. The LTs were tasked with privately messaging SRs on the Zoom chat box or via email if SRs were exhibiting any inappropriate or unprofessional behavior such as lying down in bed during the sessions. During the in-person programming the CBESS session faculty  were expected to address inappropriate behavior. Furthermore, LTs facilitated daily debrief sessions with their respective SR groups. These sessions were scheduled at the group level, sometimes occurring immediately after synchronous programming, whereas some groups were later in the evenings; each group was able to accommodate work and online course class schedules for both LTs and SRs. These debriefs gave SRs an opportunity to connect with their LT and group in a more informal setting without program staff present, allowing for more open discussion and creating a space where students could express their honest concerns or feedback. During in-person programming, LTs debriefed at the end with the SR groups.For cohort 2, each LT implemented creative ways to engage their students and groups to develop rapport. However, cohort 2 LTs reported that the daily debrief sessions were the most important opportunity for team building and to develop relationships with their SRs. Several LTs had SRs reflect on the day with a “Rose, Thorn, Bud” activity where each SR would identify one highlight or success (rose), one challenge or frustration (thorn), and something they were looking forward to (bud). LTs acknowledged that choosing a different SR to start this activity daily helped each student feel more comfortable and confident in participating. Another strategy that LTs implemented was to play a quick game during debrief sessions that was completely removed from topics related to science or CBESS programming. Examples of games included categories, scavenger hunts and guess the movie. Later in the evening, the CBESS program coordinators would have a separate debrief with all the LTs. Finally, during the SVRP science content LTs were subject to a new training that prepared them to facilitate science modules as described in the next section below.The science content provided during the in-person summer program had previously focused on general aspects of learning science, including biology paired with science and engineering processes (Scientific Methods and Engineering Design Process), or the Nature of Science. Faculty aimed to review some basic concepts and processes in biology , physics and engineering design with students and to demonstrate what these foundational concepts look like in action by providing SRs various hands-on laboratory research experiences in-person within the university environment and within the community. For the most part, while COVID-19 did not change the general idea of the science content objectives, most of the didactic content was adapted to make it relevant to the current pandemic. Content included basic concepts in microbiology . As such, SRs were able to make connections about how a better understanding of the biology related to COVID-19 could benefit CBESS participants, families, and communities see .When creating the learning COVID-19 modules for the CBESS program, authors 3 and 4 decided to approach the concepts from the perspective of a student that had very little to no background in the life sciences. While the students within the summer program may have taken a life science or Biology course prior to acceptance, this approach would allow everyone to be on the same level moving forward with other concepts/modules.Another consideration was to decide if the modules would be asynchronous or synchronous in delivery. The initial idea was to build asynchronous modules, but after speaking with the students and hearing their opinions about online learning, we decided to build both an asynchronous portion that was followed up with multiple synchronous sessions led by LTs. CBESS collaborated with Educational Innovations, Inc. to build custom science kits with all required materials and supplies. These hands-on materials were shipped in advance to the students directly by Educational Innovations, Inc. so that they could begin the investigations, but then complete them in a synchronous setting.The COVID-19 trajectory of content learning began with an exploration of the ideas of living (biotic) and non-living (abiotic) elements within our biosphere, moving into DNA as the code of life, expanding into bacteria as the simplest form of life, and finishing with basic viral design and function. This final module led to a more in-depth examination of the coronavirus  subgroup. All of these lesson plans utilized the 5E inquiry-based teaching model  that incThe activities started with an exploration of nature in their neighborhoods to identify biotic/abiotic factors which enabled SRs to think critically about the characteristics of living things. The exploration activity was followed up with another activity that involved the use of jellybeans to illustrate differences in physical characteristics of living and non-living things in classification. The students then learned that within living things, DNA was one of the major criteria of the definition of life. The next module focused on DNA where students created a DNA strand from plastic straw pieces and briefly transcribed it to RNA.The following module explored bacteria, simply because many people confuse the difference between bacteria and viruses when they become ill (example of getting antibiotics for the flu). The students constructed a Winogradsky column to observe bacteria colonies found in their neighborhoods. To connect bacteria, and eventually COVID-19, to personal health and hygiene, the students then were exposed to the role of handwashing in personal hygiene and as it related to the current emphasis on handwashing due to COVID-19. By using Glogerm™, the students learned proper hand-washing procedures where a hand-held blacklight showed the students where they missed washing their hands thoroughly.Finally, now that all the basic background was in place, the students learned about viruses in general, how/why they are considered non-living, but contain everything they need to be living, how viruses are classified using the Baltimore Classification System, understanding the role of RNA and mimicking the behaviors of enveloped viruses to reinforce the adaptability and perseverance of COVID-19.Since the modules were to be facilitated by the LTs, the LTs needed to be trained on both the content and activities so that they could facilitate the synchronous sessions with the SRs. As a reminder, the LTs were upper-level undergraduate students who may not have had any formal or informal educator training. A day was set aside when the creators of the learning modules went through each of the modules with all the LTs and then one LT chose one of the modules to take the lead in presenting to the SRs during the SVRP. Each LT delivered one module during one SVRP day for approximately 20-25 minutes, so this training involved describing the underlying concept of each activity afforded by each kit, performing the activity with SRs (e.g. building the model of the DNA helix), having the LTs reiterate certain points, ask questions to SRs, rehearse occasionally, and clear up any misunderstandings. It also allowed feedback to be given to the authors of the kits if/when there were issues or omissions the LTs discovered. This training was conducted remotely due to COVID-19, but also modeled what some of the learning would look like for the SRs. Every LT seemed comfortable with the material, and their presentation improved with the training.These modules were presented by CBESS LTs, who setup the topics while providing explanations and the other LTs not instructing provided backup by monitoring the Zoom chat box and responding to questions, while the SRs performed the activities and conducted the write-ups on their own and remotely. The total time was approximately 90 minutes. To further enhance the learning experience afforded by each module, reinforce some basic concepts in microbiology, and relate those concepts to the pandemic current, SRs virtually attended up to 4 different interactive presentations. These presentations were provided by renowned scientists conducting research in basic or applied sciences at the university or state health lab. For example, for cohort 2, in week 1, following the CBESS session on bacteria, a research lecture was given by the Director of Nevada Health State Lab to reinforce lessons learned and to place concepts into practice. During these virtual sessions, the SRs had an opportunity to ask questions to each of the four speakers interactively or on the Zoom chat box. Following each presentation, all unanswered questions from SRs were collected from the chat box and answered by the speaker on a spreadsheet which was provided to SRs the following day. While the subjects broached were far-ranging and deep, these modules were developed in a way to afford accessibility not only with the remote learning delivery but also with the amount and type of information given to the SRs to facilitate conceptual understanding and specialized language that accompanied the content in Spanish and English. Because all the LTs, both program coordinators, and a PI (sixth author) were bilingual they were able to provide explanations, expand on a concept, and/or answer questions in Spanish if that was preferable to students.The evaluation results presented here reflect the virtual delivery format (cohort 2) between the pre and immediate post, which equals 6 months. In addition, to assess the impact of the virtual model, the results of the post-assessment survey, administered immediately after the three-week summer research program, were compared between cohort 1 (in-person) and cohort 2 .Remarkably, CBESS faculty were able to successfully retain 28 SRs throughout the summer virtual research program during the pandemic. Unfortunately, during the second phase of virtual programmatic content, CBESS experienced modest attrition with two students withdrawing during their senior year and the outreach portion of the program; in addition, two SRs completed the post-assessment, but not the pre-assessment. This resulted in the analysis of data from 24 SRs.An in-depth evaluation pertaining to the effectiveness of the CBESS program on participants and in relation to comparison groups is still ongoing. However, in brief, the summative evaluation data show the summer virtual residential program (SVRP) was successful as suggested by the evaluation data. For instance, summative evaluation data demonstrated that SRs improved with respect to “Self Confidence in Knowledge of Science”  while no significant differences or positive trends regarding bilingualism as an asset and CBESS-related factors were observed. In addition, (>60%) of SRs strongly agreed that CBESS provided the necessary knowledge and tools to apply to college and for pursuing STEM and health careers (data not shown). Interestingly, the comparison group (N=37), which did not receive treatment, performed significantly worse in some areas, including bilingualism, ethnicity, and attitude towards college, suggesting that students that did not receive the educational content worsened in terms of their attitude and insight towards being bilingualism as an asset and ethnicity, presumably due to the anxiety, uncertainty and mental stress caused by the COVID19 pandemic during the summer and fall of 2020. In contrast, the SRs benefited from substantial near peer and adult mentoring to engage and follow up on their well-being.the expectations for the program were clear,” 85% strongly agreed and 15% agreed with the statement. When asked to rate their agreement with the statement, “I knew who to go to with questions and how to find her/him” 85% strongly agreed, 11% agreed, and 4% selected neutral. When asked to rate their agreement with the statement, “I understood what I was supposed to present and how,” 78% strongly agreed, 15% agreed, and 7% were neutral. Furthermore, when asked about skills learned during the program, SR participants responded “Research, collaborative, and communication skills” as well as “Researching in different databases and writing in APA format.”From the process evaluation, there was overwhelming agreement among the 24 SRs that the program was well planned and implemented. When asked to rate their agreement with the statement, “Do you have any general-thoughts or comments about the CBESS Summer Virtual Research Program or suggestions to improve future events? Six SRs expounded on how great the summer program was, with no reference to the format. Responses included suggestions for ice breakers, more labs and experiments, workshops on making graphs and charts. One mentioned the “amazing” mentors because “they were what truly made my day” while another stated that it was the “best experience” of the summer. Six SRs reported that it was a great experience, but references were made to the program being virtual. A typical response in this group was, “Although the summer program was all virtual, I enjoyed the entire three weeks, with the science kits and meeting new organizations, it was definitely an experience.” However, it must be noted that two SRs specifically expressed a desire to get to know other SRs in person and three SRs were more emphatic about their desire to have the program in-person. One stated, “I think the virtual part lost the personal connections or relationships that could have happened.” Perhaps typical of suggestions for improvement, four indicated a desire for more time. Two SRs mentioned the research project and other assignments; another wanted longer breaks; another wanted more discussion about college options.In addition to being asked to rate the program, SRs were asked the open-ended question: t-tests were conducted on each set of scales. No significance differences were found between the two groups  Mentorship skills, b) Mentorship role. These themes are discussed next and selected quotes are presented as supporting evidence.Leadership trainees reported that they learned how to work with a diverse group of students. Specifically, the LTs gained skill needed for mentoring, such as active listening. For example, one participant stated, “I think really one of my number one skills is having patience and really learning how to listen, because I think it’s easy to put yourself in a role, you know you’re in college these kids are looking up to you.” Additionally, the expanded role of the LTs in cohort 2 required time management, a skill that was reinforced during the summer program. As another LT stated,The students learn that but I learned that as well, and time management and all those things but not just school but in everything else that we had to do during CBESS and that was a great skill to learn as well and It just helped me realize how important everyone in the Community is and how important research is.LTs expressed that being in a mentorship role came with the need to set boundaries and regular communication. The leadership trainees reported having good relationships with their students that allowed them give information and advice for the work they were doing, but setting boundaries was necessary. For example, one LT stated,Being able to kind of have that bond with your students but also being able to set boundaries has really kind of highlighted the importance of being able to specify certain professional positions in your life right, so I think that the CBESS program has definitely highlighted how to be able to interact with other people.To build a solid foundation for a mentor-mentee relationship, leadership trainees regularly interacted with SRs through monthly check-in meetings. Over the course of the program, career paths became a common discussion point and the more the SRs learned, their interests diversified, “Having those monthly meetings and we talked with them about what careers they’re thinking of now, like the evolution of it [career selection] and was nice to kind of see their growth.” All in all, the LTs expressed the personal benefit of mentorship; the pride that comes from being a part of and witnessing student growth. For example,I think that I’ve seen a huge change within our students and with that being said, I feel some type of pride almost being able to realize that we’ve come so far within the program and realizing that not only has there been a personal impact within us as mentors, but also within our mentees because they’ve done a lot.As mentioned in the Despite the current pandemic disrupting social life for SRs and in-person instruction, our summative evaluation data showed that the second cohort of SRs recruited by CBESS had an overall positive experience during the virtual SVRP. Indeed, our evaluation data showed significant improvements in attaining basic concepts in biology (microbiology and cell biology), enhanced interest in STEM careers, and on the importance of leveraging their bilingualism as an asset by which bilingual speakers can enhance their learning of STEM content through a different didactic lens. In the next phase of CBESS, a hybrid program is planned to represent strengths and qualities of in-person and remote-learning modes. Decision on mode of interactions will be built based on qualitative and quantitative summative evaluation data (forthcoming).It is worth noting that student learning and engagement were not only retained for cohort 2 but some significant gains in student development  were enhanced similar to cohort 1 in 2019 suggesting that CBESS was able to successfully provide an overall positive experience during the pandemic. Importantly, our evaluation data showed that CBESS in-person programming is adaptable to long distance learning, synchronous/dynamic activities can be maintained by shipping science kits to student, can be scaled and scaffolded with science learning modules, and is resilient in response to social/environmental crisis such as the current pandemic. Through LT focus group participation, it became apparent that listening and time management skills are essential for being a mentor. However, while many aspects of student learning, CBESS programming, and mentorship were retained for cohort 2, we recognize that parents of SRs were not engaged significantly as in cohort 1 which predominantly occurred during the CEEs.CBESS faculty realized that this was a flaw in the program design. As part of ongoing efforts to sustain CBESS, we are looking to engage high school counselors and administrators from both urban and rural high schools in northern Nevada to support a learning community to provide STEM/ health bilingual resources and parent engagement. Secondly, we will have LTs meet with parents or caregivers in addition to SRs during program check-ins. Going forward, CBESS will not only strive to provide instruction and mentorship remotely for cohort 3 and beyond but will add parent engagement activities that invite parents to Community of Practice events, parent-instructor focus groups, CEEs and youth-led dissemination events.Evaluation data reported in this manuscript have focused on only the changes to programming that were necessary because of the current pandemic. With the third cohort completing the program in 2022, a comprehensive evaluation is currently underway. Future evaluation reports will detail program outcomes for three cohorts of intervention and comparison students. Qualitative and quantitative, pre and post program data, received from three cohorts of leadership trainees is also under review. However, we have noted that the data obtained from focus groups are more meaningful and relevant, especially given the small number of LTs for each cohort (4-5 people). A 5-year longitudinal assessment of career interests and paths chosen by SRs and LTs from cohorts 1-3 is also on-going. These data are preliminary and will be included in follow-up reports as CBESS cohort 1 students, in 2021, will be third-year program alumni.Appendix A