Applications of Supervised Machine Learning in Autism Spectrum Disorder Research: a Review <s> Alternating Decision Tree <s> We gratefully acknowledge the families and individuals who have contributed their biomaterials, time, and financial resources to AGRE, especially Marianne Toedtman, AGRE family recruiter; Ed Berry, phlebotomist; Andrew Smith, M.D., pediatric neurologist; Paul Law, M.D., M.P.H., for database development; and Nancy Jones, AGRE Web master. We specifically thank Sallie and Tom Bernard, for their generous financial support of AGRE, and the Schering-Plough Research Institute and Pfizer Inc., for their contributions to AGRE. We also thank Maricela Alarcon, Ph.D., for her error checking and advice; Jianjun Liu, Ph.D., for updating of the genotyping data; and scientists who have started to utilize AGRE, for their helpful comments and criticism. The members of the AGRE Steering Committee are: W. Ted Brown, New York State Institute for Basic Research in Developmental Disabilities, Staten Island; Maya Bucan, University of Pennsylvania, Philadelphia; Joseph Buxbaum, Mt. Sinai School of Medicine, New York; T. Conrad Gilliam, Columbia University Genome Center, New York; David A. Greenberg, Mt. Sinai School of Medicine, New York; David H. Ledbetter, University of Chicago, Chicago; Bruce L. Miller, University of California, San Francisco; Stanley F. Nelson, UCLA School of Medicine, Los Angeles; Jonathan Pevsner, Kennedy Krieger Institute, Baltimore; Jerome I. Rotter, Cedars-Sinai Medical Center, Los Angeles; Carol Samango-Sprouse, Children’s National Medical Center, Baltimore; Gerard D. Schellenberg, University of Washington and Veterans Affairs Medical Center, Seattle; Rudolph E. Tanzi, Massachusetts General Hospital, Boston; and Kirk C. Wilhelmsen, University of California, San Francisco. <s> BIB001 </s> Applications of Supervised Machine Learning in Autism Spectrum Disorder Research: a Review <s> Alternating Decision Tree <s> Autism Diagnostic Observation Schedule (ADOS) Modules 1–3 item and domain total distributions were reviewed for 1,630 assessments of children aged 14 months to 16 years with an autism spectrum disorder (ASD) or with heterogeneous non-spectrum disorders. Children were divided by language level and age to yield more homogeneous cells. Items were chosen that best differentiated between diagnoses and were arranged into domains on the basis of multi-factor item-response analysis. Reflecting recent research, the revised algorithm now consists of two new domains, Social Affect and Restricted, Repetitive Behaviors (RRB), combined to one score to which thresholds are applied, resulting in generally improved predictive value. <s> BIB002 </s> Applications of Supervised Machine Learning in Autism Spectrum Disorder Research: a Review <s> Alternating Decision Tree <s> Introduction.- Getting Started.- Working with Data.- Loading Data.- Exploring Data.- Interactive Graphics.- Transforming Data.- Descriptive and Predictive Analytics.- Cluster Analysis.- Association Analysis.- Decision Trees.- Random Forests.- Boosting.- Support Vector Machines.- Model Performance Evaluation.- Deployment. <s> BIB003 </s> Applications of Supervised Machine Learning in Autism Spectrum Disorder Research: a Review <s> Alternating Decision Tree <s> The Autism Diagnostic Observation Schedule-Generic (ADOS) is one of the most widely used instruments for behavioral evaluation of autism spectrum disorders. It is composed of four modules, each tailored for a specific group of individuals based on their language and developmental level. On average, a module takes between 30 and 60min to deliver. We used a series of machine-learning algorithms to study the complete set of scores from Module 1 of the ADOS available at the Autism Genetic Resource Exchange (AGRE) for 612 individuals with a classification of autism and 15 non-spectrum individuals from both AGRE and the Boston Autism Consortium (AC). Our analysis indicated that 8 of the 29 items contained in Module 1 of the ADOS were sufficient to classify autism with 100% accuracy. We further validated the accuracy of this eight-item classifier against complete sets of scores from two independent sources, a collection of 110 individuals with autism from AC and a collection of 336 individuals with autism from the Simons Foundation. In both cases, our classifier performed with nearly 100% sensitivity, correctly classifying all but two of the individuals from these two resources with a diagnosis of autism, and with 94% specificity on a collection of observed and simulated non-spectrum controls. The classifier contained several elements found in the ADOS algorithm, demonstrating high test validity, and also resulted in a quantitative score that measures classification confidence and extremeness of the phenotype. With incidence rates rising, the ability to classify autism effectively and quickly requires careful design of assessment and diagnostic tools. Given the brevity, accuracy and quantitative nature of the classifier, results from this study may prove valuable in the development of mobile tools for preliminary evaluation and clinical prioritization—in particular those focused on assessment of short home videos of children—that speed the pace of initial evaluation and broaden the reach to a significantly larger percentage of the population at risk. Translational Psychiatry (2012) 2, e100; doi:10.1038/tp.2012.10; published online 10 April 2012 <s> BIB004 </s> Applications of Supervised Machine Learning in Autism Spectrum Disorder Research: a Review <s> Alternating Decision Tree <s> Current approaches for diagnosing autism have high diagnostic validity but are time consuming and can contribute to delays in arriving at an official diagnosis. In a pilot study, we used machine learning to derive a classifier that represented a 72% reduction in length from the gold-standard Autism Diagnostic Observation Schedule-Generic (ADOS-G), while retaining >97% statistical accuracy. The pilot study focused on a relatively small sample of children with and without autism. The present study sought to further test the accuracy of the classifier (termed the observation-based classifier (OBC)) on an independent sample of 2616 children scored using ADOS from five data repositories and including both spectrum (n=2333) and non-spectrum (n=283) individuals. We tested OBC outcomes against the outcomes provided by the original and current ADOS algorithms, the best estimate clinical diagnosis, and the comparison score severity metric associated with ADOS-2. The OBC was significantly correlated with the ADOS-G (r=−0.814) and ADOS-2 (r=−0.779) and exhibited >97% sensitivity and >77% specificity in comparison to both ADOS algorithm scores. The correspondence to the best estimate clinical diagnosis was also high (accuracy=96.8%), with sensitivity of 97.1% and specificity of 83.3%. The correlation between the OBC score and the comparison score was significant (r=−0.628), suggesting that the OBC provides both a classification as well as a measure of severity of the phenotype. These results further demonstrate the accuracy of the OBC and suggest that reductions in the process of detecting and monitoring autism are possible. <s> BIB005 </s> Applications of Supervised Machine Learning in Autism Spectrum Disorder Research: a Review <s> Alternating Decision Tree <s> Machine learning has immense potential to enhance diagnostic and intervention research in the behavioral sciences, and may be especially useful in investigations involving the highly prevalent and heterogeneous syndrome of autism spectrum disorder. However, use of machine learning in the absence of clinical domain expertise can be tenuous and lead to misinformed conclusions. To illustrate this concern, the current paper critically evaluates and attempts to reproduce results from two studies (Wall et al. in Transl Psychiatry 2(4):e100, 2012a; PloS One 7(8), 2012b) that claim to drastically reduce time to diagnose autism using machine learning. Our failure to generate comparable findings to those reported by Wall and colleagues using larger and more balanced data underscores several conceptual and methodological problems associated with these studies. We conclude with proposed best-practices when using machine learning in autism research, and highlight some especially promising areas for collaborative work at the intersection of computational and behavioral science. <s> BIB006 </s> Applications of Supervised Machine Learning in Autism Spectrum Disorder Research: a Review <s> Alternating Decision Tree <s> Abstract ::: The Mobile Autism Risk Assessment (MARA) is a new, electronically administered, 7-question autism spectrum disorder (ASD) screen to triage those at highest risk for ASD. Children 16 months–17 years (N = 222) were screened during their first visit in a developmental-behavioral pediatric clinic. MARA scores were compared to diagnosis from the clinical encounter. Participant median age was 5.8 years, 76.1 % were male, and most participants had an intelligence/developmental quotient score >85; 69 of the participants (31 %) received a clinical diagnosis of ASD. The sensitivity of the MARA in detecting ASD was 89.9 % [95 % CI = 82.7–97]; the specificity was 79.7 % [95 % CI = 73.4–86.1]. In a high-risk clinical setting, the MARA shows promise as a screen to distinguish ASD from other developmental/behavioral disorders. <s> BIB007
ADTree models, also known as acyclic directed graphical models, involve combining many one-level decision trees (i.e., decision stumps) to obtain a representation in which each stump consists of a decision node and two prediction nodes BIB003 . One main benefit of leveraging a decision tree is being able to interpret and understand the impact of each variable in the tree. While other methods, such as an RF, may lead to better performance from a predictability standpoint, they may lack the transparency that a decision tree gives. Decision trees can also be leveraged to identify unique segments in a population where multiple models can be applied rather than only a single model for the entire population. ADTree models have been used in the reviewed ASD research in attempts to enhance diagnostic and screening practices. Behavioral and Neuroimaging Data As mentioned previously, ASD is currently diagnosed via standardized behavioral assessments, which can be lengthy and time consuming to administer. In an attempt to accelerate the diagnostic process,  set out to identify a subset of ADI-R items that could be used to accurately classify ASD. A dataset of 891 individuals with ASD and 75 individuals without ASD who completed the ADI-R was initially tested. Using seven of the 93 ADI-R items, an ADTree classifier was found to perform best with an accuracy of 99.9%. The seven-item classifier was further examined in two additional samples of individuals with ASD (n = 1654; n = 322). Across both samples, the classifier was found to have an accuracy of nearly 100%. Specificity was evaluated using two small samples of individuals without ASD (n = 5; n = 12) as well as 1000 artificially generated score sheets used as control data, with specificity ranging from 93.8 to 99%, for both real and artificial data. In a similar study, BIB004 set out to identify a subset of items included in the ADOS that could be used to accurately detect ASD. An initial sample including 612 individuals with ASD and 15 individuals without ASD who had completed module one of the ADOS were tested. Using eight of the 29 items measured in module one of the ADOS, an ADTree classifier again demonstrated a superior performance with a classification accuracy of 100%. The eight-item classifier was further validated using two additional samples of individuals with ASD (n = 110; n = 366) and 1000 artificial controls. Across both ASD samples, the eight-item classifier was found to have a sensitivity of 100%. Using the simulated control data, a specificity of 94% was revealed. The findings of these two studies by Wall and colleagues show promise in the use of machine learning techniques to sizably reduce the number of behavioral assessment items required to accurately classify ASD. In a replication study, BIB006 discussed the methodological issues that they felt raised concern in the studies conducted by Wall and colleagues  BIB004 ). The primary concern was the unbalanced datasets used to test and validate the classifiers, which contained little to no negative cases (i.e., participants without ASD). The authors conducted experiments to replicate the results of Wall and colleagues' ADTree procedures using larger, though not publicly available, datasets with more balanced samples of positive and negative cases. Both experiments failed to replicate the level of accuracy achieved by Wall and colleagues for both the reduced-item ADI-R and ADOS classifiers. Furthermore, the authors did not find that clinician administration time decreased using the reduced-item classifiers. These classifiers have undergone additional evaluation for use as preclinical screening tools to detect risk of ASD. BIB005 set out to further validate the eight-item ADOS classifier, referred to as the observation-based classifier (OBC). The study dataset included a more balanced sample of 2333 children with ASD and 283 children without ASD, who had completed the ADOS. The OBC, which uses an ADTree procedure to classify ASD, demonstrated significant correlations with both original ADOS and ADOS-2 BIB002 ) scoring procedures, revealing a highest reported accuracy of 95.8%, sensitivity of 97.7%, and specificity of 83.5%. OBC outcomes were also compared to best estimate clinical diagnoses, showing an accuracy of 96.8%, sensitivity of 97.1%, and a specificity of 83.3%. While the results are not as robust as those initially reported by BIB004 , the OBC may be a viable screening tool to detect risk of ASD. The seven-item ADI-R classifier has also undergone further evaluation by BIB007 . In this study, a mobile application, the Mobile Autism Risk Assessment (MARA), was used to administer the ADI-R questionnaire. Participants included 222 children and adolescents (69 of which were diagnosed with ASD), between the ages of 16 months and 17 years old, who were visiting a developmental-behavioral pediatric clinic for the first time. Participant's MARA screening outcomes were compared to clinical diagnoses made during the visit. MARA utilizes an ADTree trained on the answer sheets of 891 subjects with ASD and 75 subjects without ASD from the Autism Genetic Resource Exchange (AGRE; BIB001 database to generate scores. The model demonstrated a sensitivity of 89.9% and a specificity of 79.7%.