Modélisation, élaboration et évaluation de rapports à visée diagnostique des données du PIRLS 2011
This study uses the cognitive diagnostic approach (CDA) to provide teachers with precise and detailed feedback on the cognitive strengths and weaknesses of students through comprehensible diagnostic reports. It was conducted with the aim of addressing teachers’ pressing need for diagnostic information based on large-scale tests, as well as the lack of empirical research on diagnostic report development and evaluation. More specifically, our study aims to: (1) model data from 4,762 Canadian students who took the 2011 PIRLS test and (2) develop and evaluate diagnostic reports for teachers, using modeling results. In order to attain the first objective, necessary underlying skills to solve the 35 items in PIRLS 2011 were identified based on the PIRLS 2011 development framework and with a panel of three experts. Two templates, Q1 and Q2, were then developed, representing the connexions between these skills and items. With these two “Q” templates, we modeled the data with the DINA and G-DINA models, which enabled us to assess how the models fit the data and to estimate the diagnostic quality of the items as well as the skill mastery profiles of the students.  As for the second objective, a panel of five experts first rephrased the identified skills descriptions in accessible terms for teachers in the classroom. Next, among the skill mastery profiles obtained from the modeling, we chose the profile that was most representative of the students. Then, three report formats were developed based on this standard profile, using the design protocol derived from our literature review. Finally, 98 elementary school teachers, educational consultants and remedial teachers evaluated these reports by means of a survey focusing on report preference, report quality evaluation and report comprehension. The results point to a preferred report format among the participants. This preference is explained by the familiarity of the type of chart presented, and it is constant regardless of contextual information. The participants evaluated the quality of the reports very positively, with no variation according to contextual variables. However, there is a significant difference between report quality evaluation and report preference. Participants who evaluate their preferred format show more positive perceptions. Regarding report comprehension, participants have a very good understanding of all three formats. The group that best understands the reports is the one that made the most positive quality evaluation. The results of the study point to the possibility of receiving, through accessible, reliable, and comprehensible diagnostic reports, more precise and detailed feedback on the strengths and weaknesses shown by students in large-scale tests. Our study is the first to apply a report development and evaluation framework based on a cognitive diagnostic approach, thus helping improve support for reading acquisition in elementary schools.
DINA G-DINA PIRLS 2011 Cognitive diagnostic approach Diagnostic classification model Reading Diagnostic report development Diagnostic report evaluation