Graduation Year

2016

Document Type

Thesis

Degree

Ph.D.

Degree Name

Doctor of Philosophy (Ph.D.)

Degree Granting Department

Educational Measurement and Research

Major Professor

Robert Dedrick, Ph.D.

Committee Member

John Ferron, Ph.D.

Committee Member

Jenni Wolgemuth, Ph.D.

Committee Member

Waynne James, Ed.D.

Keywords

testing environment, human factors, computer-based testing

Abstract

Despite the increasing number of individuals taking computer-based tests, little is known about how examinees perceive computer-based testing environments and the extent to which these testing environments are perceived to affect test performance. The purpose of the present study was to assess the testing environment as perceived by individuals taking the National Physical Therapy Examination (NPTE), a high-stakes licensure examination. Perceptions of the testing environments were assessed using an examinee self-report questionnaire. The questionnaire included items that measured individuals’ preference and perception of specific characteristics of the environment, along with demographic information and one open-ended item. Questionnaires were distributed by email to the 210 accredited physical therapy programs at the time, encouraging programs to forward the instrument by email to the most recent class of physical therapy graduates. Two hundred and sixteen respondents completed the study, representing 101 testing centers in 31 states.

Data from these 216 examinees were used to answer four research questions. The first research question focused on the examinees’ environmental preferences for the NPTE testing environment and the relation between these preferences and examinees’ background characteristics (e.g., sex, program GPA, age, online experience, online testing experience, comfort level with online testing, and preferred testing time). A clear preference toward one end of the scale was observed for preferring a quiet room and a desktop area that had a great deal of adjustability. Examinees’ preferences and their demographic characteristics were not strongly related with the seven demographic variables accounting for < 7% of the variability in examinees’ environmental preferences.

The second research question used the data from multiple examinees nested within the same testing center to examine the within- and between-center variability in examinees’ perceptions of the testing environment and their satisfaction with the environment. Results indicated that the majority of the variance in these variables was within testing centers with average between-center variability equal to .032 for the perception ratings and .078 for the satisfaction ratings. Research questions (RQ) three and four explored whether examinees’ background characteristics (RQ 3) and center characteristics (RQ 4) were significantly related to the 12 environmental perception ratings, 12 satisfaction ratings, and two items representing examinees’ perceptions of the effect of the testing environment on their performance and the likelihood they would choose the same center again. In terms of examinee characteristics, age, online testing experience, and comfort with online testing were the most consistent predictors of the various examinee ratings. The most consistent predictors for the satisfaction ratings were examinees’ online test comfort, online test experience, and age. For center characteristics, the newness of the center and the room density of the center were the most consistent predictors of examinee ratings. For satisfaction ratings, the most consistent predictor was the newness of the center. Center newness was significantly related to the outcome variables related to the size, lighting and sound of the center which may reflect changes in building standards and materials.

The results of the study suggest the need for further exploration of the environmental and human factors that may impact individuals taking high stakes examinations in testing centers. Although there may not be an effect on all examinees, there may be subsets of individuals who are more sensitive to the effects of the testing environment on performance. Further exploration of the uniformity of testing environments is also needed to minimize error and maximize potential threats to test security.

Download

Share

COinS