Graduation Year

2018

Document Type

Dissertation

Degree

Ph.D.

Degree Name

Doctor of Philosophy (Ph.D.)

Degree Granting Department

Educational Measurement and Research

Major Professor

Robert Dedrick, Ph.D.

Co-Major Professor

Yi-Hsin Chen, Ph.D.

Committee Member

Robert Dedrick, Ph.D.

Committee Member

Yi-Hsin Chen, Ph.D.

Committee Member

John Ferron, Ph.D.

Committee Member

Sarah van Ingen, Ph.D.

Keywords

Bayesian Estimation, Measurement, Middle School Mathematics, Retrofitted Model, University of Chicago School Mathematics Project

Abstract

This study retrofitted a Diagnostic Classification Model (DCM) known as the Fusion model onto non-diagnostic test data from of the University of Chicago School Mathematics Project (UCSMP) Algebra and Geometry Readiness test post-test used with Transition Mathematics (Third Edition, Field-Trial Version). The test contained 24 multiple-choice middle school math items, and was originally given to 95 advanced 6th grade and 293 7th grade students. The use of these test answers for this study was an attempt to show that by using cognitive diagnostic analysis techniques on test items not constructed for that purpose, highly predictable multidimensional cognitive attribute profiles for each test taker could be obtained. These profiles delineated whether a given test taker was a master or non-master for each attribute measured by the test, thus allowing detailed diagnostic feedback to be disseminated to both the test takers and their teachers.

The full version of the non-compensatory Fusion model, specifically, along with the Arpeggio software package, was used to estimate test taker profiles on each of the four cognitive attributes found to be intrinsic to the items on this test, because it handled both slips and guesses by test takers and accounted for residual skills not defined by the four attributes and twenty-four items in the Q-matrix. The attributes, one or more of which was needed to correctly answer an item, were defined as: Skills— those procedures that students should master with fluency; e.g., multiplying positive and negative numbers; Properties—which deal with the principles underlying the mathematics concepts being studied, such as being able to recognize and use the Repeated-Addition Property of Multiplication; Uses—which deal with applications of mathematics in real situations ranging from routine "word problems" to the development and use of mathematical models, like finding unknowns in real situations involving multiplication; and, Representations—which deal with pictures, graphs, or objects that illustrate concepts.

Ultimately, a Q-matrix was developed from the rating of four content experts, with the attributes needed to answer each item clearly delineated. A validation of this Q-matrix was obtained from the Fusion model Arpeggio application to the data as test taker profiles showed which attributes were mastered by each test taker and which weren’t. Masters of the attributes needed to be acquired to successfully answer a test item had a proportion-correct difference from non-masters of .44, on average. Regression analysis produced an R-squared of .89 for the prediction of total scores on the test items by the attribute mastery probabilities obtained from the Fusion model with the final Q-matrix. Limitations of the study are discussed, along with reasons for the significance of the study.

Download

Share

COinS