Adaptive Testing with Multidimensional Pairwise Preference Items: Improving the Efficiency of Personality and Other Noncognitive Assessments

Document Type

Article

Publication Date

7-2012

Keywords

item response theory, IRT, computerized adaptive testing, CAT, ideal point, pairwise preference, multidimensional forced choice

Digital Object Identifier (DOI)

https://doi.org/10.1177/1094428112444611

Abstract

Assessment of noncognitive constructs in organizational research and practice is challenging because of response biases that can distort test scores. Researchers must also deal with time constraints and the ensuing trade-offs between test length and the number of constructs measured. This article describes a novel way of improving the efficiency of noncognitive assessments using computer adaptive testing (CAT) with multidimensional pairwise preference (MDPP) items. Tests composed of MDPP items are part of a broader family of forced choice measures that ask respondents to choose between two or more equally desirable statements in an effort to combat response distortion. The authors conducted four computer simulations to explore the influences of test design, dimensionality, and the advantages of adaptive item selection for trait score and error estimation with tests involving as many as 25 dimensions. Overall, adaptive MDPP testing produced gains in accuracy over nonadaptive MDPP tests comparable to those observed with traditional unidimensional CATs. In addition, an empirical illustration involving a 15-dimension MDPP CAT administered in a field setting showed patterns of correlations that were consistent with expectations, thus showing construct validity.

Was this content written or created while at USF?

Yes

Citation / Publisher Attribution

Organizational Research Methods, v. 15, issue 3, p. 463-487

Link to Full Text

Share

COinS