An Examination of the Comparative Reliability, Validity, and Accuracy of Performance Ratings Made Using Computerized Adaptive Rating Scales

Document Type


Publication Date



iterative paired comparison, computerized adaptive rating scales, job performance rating, method comparisons, error of measurement, validity, reliability

Digital Object Identifier (DOI)



This laboratory research compared the reliability, validity, and accuracy of a computerized adaptive rating scale (CARS) format and 2 relatively common and representative rating formats. The CARS is a paired-comparison rating task that uses adaptive testing principles to present pairs of scaled behavioral statements to the rater to iteratively estimate a ratee's effectiveness on 3 dimensions of contextual performance. Videotaped vignettes of 6 office workers were prepared, depicting prescripted levels of contextual performance, and 112 subjects rated these vignettes using the CARS format and one or the other competing format. Results showed 23%–37% lower standard errors of measurement for the CARS format. In addition, validity was significantly higher for the CARS format (d = .18), and Cronbach's accuracy coefficients showed significantly higher accuracy, with a median effect size of .08. The discussion focuses on possible reasons for the results.

Was this content written or created while at USF?


Citation / Publisher Attribution

Journal of Applied Psychology, v. 86, issue 5, p. 965-973