Learning Analytics in Medical Education
JMEF Forum 
July 14, 2021
Rodrigo B. Cavalcanti, MD MSc
Associate professor Department of Medicine, Faculty of Medicine, University of TorontoDirector, HoPingKong Centre, University Heath NetworkClinician Educator, Royal College of Physicians and Surgeons of Canada
Objectives
Define learning analytics and related concepts
Review 3 examples of learning analytics
Outline benefits of learning analytics
Propose future directions for development
Definitions
Learning Analytics
Institutional Research
Programmatic Assessment
Learning Analytics
What:measurement, collection, analysis and 
reporting of data  about learners
Learning Analytics
What:measurement, collection, analysis and 
reporting of data  about learners
Why: to understand and optimize learningand the environments where it occurs
Society for Learning Analytics Research (SoLAR) 
solaresearch.org
Learning Analytics
Why now:
Information age and 
digitalization
Greater ability to collect 
educational data
Greater ability to collate 
and analyze data
Institutional Research
My understanding, in Japanese context 
Institutional Research ~ Learning Analytics
Institutional Research
Support effective decision-making and educationmanagement, through development and operation of a system for data collection, analysis and provision, in relation to education and learning activities
Programmatic Assessment
Traditional Assessment: 
Development of instruments 
for assessing individuals
New Framework:
Individual assessment fits into 
program of assessment 
Programmatic Assessment
Assessment of learner performance helps build programs
Dimensions of a program of assessment include:
Goals of ProgramProgram in Action Support for ProgramDocumenting the ProgramImproving the ProgramAccountability
Dijkstra, van der Vleuten, Schuwirth AHSE. 2010
Programmatic Assessment
Defining Goals - What is assessed = what is learned
Documenting - Demonstrate learner competence
Supporting - Faculty Development and Psychometrics
Program Evaluation and Quality Improvement
Accountability to Stakeholders
Dijkstra, van der Vleuten, Schuwirth AHSE. 2010
Examples of Learning Analytics
Examples of Institutional Research
Examples
Individual task: ankle radiographs in ER
Martin Pusic – NYU (now Harvard)
Resident performance in Internal Medicine
Eric Warm – University of Cincinnati
National data: US neurosurgery programs 
Stan Hamstra - ACGME
Learning Analytics for skills: Radiograph Interpretation
Martin Pusic MD PhD
NY and Harvard
Ankle Radiograph 
Interpretation
Important skill for paediatric emergency physicians
Requires experience
Skill acquisition dependent on clinical exposure
But clinical experience variable
Item Bank
. . . 
Case 1      Case2       Case3       Case4     Case 5                        Case xx
Performance: Ankle X Rays
Measured through Likelihood Ratios
Comparing Resident and Radiologist readings
To account for Correct and Incorrect readings
Positive Likelihood Ratio
LR+ = Sn/1-Sp
Negative Likelihood Ratio
LR- = 1-Sn/Sp
Resident Reading
Radiologist Reading
17131925313743495561677379859197 103 109 115 121 127 133 139 145 151 157 163 169 175 181 187 193 199 205 211 217 223 229
LR Estimate
Case Number
LR+ and LR-
LR--LR+
Resident 1
1713 19 25 31 37 43 49 55 61 67 73 79 85 91 97 103109115121127133139145151157163169175181187193199205211217223229
LR Estimate
Case Number
LR+ and LR-
LR--LR+
Resident 2
Time vs. Competency Based Training 
Pusic M et al Acad Med. 2011
Time vs. Competency Based
Pusic M et al Acad Med. 2011
For Teachers
Are we providing trainees with enough clinical experience?
How to move from ‘time-based’ models to competencies?
Resident Evaluation:
Cincinnati Internal Medicine Dashboard
Eric J. Warm M.D., F.A.C.P.
Program Director, Internal Medicine
Richard W. and Sue P. Vilter Professor of Medicine
University of Cincinnati College of Medicine
Cincinnati Int Med Dashboard
Three year training Program
Workplace based Assessments:
FacultyPeersAllied HealthSelf
Organized by:
ACGME competenciesContent knowledge: e.g. Diabetes
From assessments to analytics
ACGME Competencies
Practice-Based Learning and Improvement.
Patient Care and Procedural Skills.
Systems-Based Practice.
Medical Knowledge.
Interpersonal and Communication Skills.
Professionalism.
Graduating 3rd Year Resident
Warm EJ, Held JD, Hellmann M, Kelleher M, Kinnear B, Lee C, O'Toole JK, Mathis B, Mueller C, Sall D, Tolentino J, Schauer DP. Entrusting Observable Practice Activities and Milestones Over the 36 Months of an Internal Medicine Residency. Acad Med. 2016 Jun 28. [Epub ahead of print] PubMed PMID: 27355780.
Graduating 3rd Year Resident
Warm EJ, Held JD, Hellmann M, Kelleher M, Kinnear B, Lee C, O'Toole JK, Mathis B, Mueller C, Sall D, Tolentino J, Schauer DP. Entrusting Observable Practice Activities and Milestones Over the 36 Months of an Internal Medicine Residency. Acad Med. 2016 Jun 28. [Epub ahead of print] PubMed PMID: 27355780.
n = 329524
Warm EJ, Held JD, Hellmann M, Kelleher M, Kinnear B, Lee C, O'Toole JK, Mathis B, Mueller C, Sall D, Tolentino J, Schauer DP. Entrusting Observable Practice Activities and Milestones Over the 36 Months of an Internal Medicine Residency. Acad Med. 2016 Jun 28. [Epub ahead of print] PubMed PMID: 27355780.
Warm EJ, Held JD, Hellmann M, Kelleher M, Kinnear B, Lee C, O'Toole JK, Mathis B, Mueller C, Sall D, Tolentino J, Schauer DP. Entrusting Observable Practice Activities and Milestones Over the 36 Months of an Internal Medicine Residency. Acad Med. 2016 Jun 28. [Epub ahead of print] PubMed PMID: 27355780.
r2 = 0.7642
n = 3508
Single 
Resident
Warm EJ, Held JD, Hellmann M, Kelleher M, Kinnear B, Lee C, O'Toole JK, Mathis B, Mueller C, Sall D, Tolentino J, Schauer DP. Entrusting Observable Practice Activities and Milestones Over the 36 Months of an Internal Medicine Residency. Acad Med. 2016 Jun 28. [Epub ahead of print] PubMed PMID: 27355780.
For Teachers
Assessments beyond faculty can be very informative
Learning curves vary!  A LOT!
Learning curves require multiple assessments (>1000) 
Country Wide Program Evaluation
Stanley Hamstra, PhD
University of Toronto
2.8M data points
•~22 sub-compete
>130,000 residents
>10,000 programs
One example
Neurosurgery Training Programs across USA
Residents assessment on multiple domains of practice:
Traumatic Brain InjuryCritical CareBrain TumoursSpinal SurgeryPediatric NeurosurgeryVascular NeurosurgeryPain and Peripheral NervesSurgical Treatment of Epilepsy and Movement Disorders
ACGME Milestones
Collating assessment data at a program level can help assess the program itself
For Teachers
Examples
Individual task: ankle radiographs in ER
Martin Pusic – NYU (now Harvard)
Resident performance in Internal Medicine
Eric Warm – University of Cincinnati
National data: US neurosurgery programs 
Stan Hamstra - ACGME
Benefits of Learning Analytics
Uses of learning analytics
Improve individual student performance
Early attention and feedback
Improve teaching and learning
Design individual learning experiences
Improve curriculum 
Design, support, systems
Learners
Improved feedback on skill acquisition
Improved learning experiences
Variable time for skill acquisition
Teachers
Feedback from data on assessments
What are they assessing?
How do they compare to peers?
What can they do better?
Institutions
Programmatic Assessment -Institutional Research
Are goals appropriate?
Are goals being met?
What programs need 
development?
Future Directions
Future Directions 
Learner: Better task specific analytics
Program: Systematic analysis of goals
Institution: Strategy and development, dashboards
Learner Level
Define competencies 
Structure systematic learning experiences
Data collection to understand skill acquisition 
Training Program Level
Define Competencies 
Collect assessment data that allows:
Feedback on skill acquisition
Design better learning experiences
Understand learning curves
Training Program Level
Define Competencies 
Collect assessment data that allows:
Feedback on skill acquisition
Design better learning experiences
Understand learning curves
At Institutional Level
Plan Programmatic Assessment 
Define goals of training
Develop data collection, analysis and reporting
Two key concepts in analytics
SamplingData Analysis
Sampling
Data analysis
Decisions regarding types of analyses
Data visualization
Dashboard Development
Competency Based Dashboard
Dr. Brent Thoma  
Emergency Medicine University of Saskatchewan
Dashboard to display all assessments for residents 
over 5 year competency based program
https://cbme.usask.ca/#/
Dashboard Development
Objectives
Define learning analytics and related concepts
Review 3 examples of learning analytics
Outline benefits of learning analytics
Propose future directions for development
Thank You
rodrigo.cavalcanti@uhn.ca