The Influence of Examiner Fidelity on First Graders’ Oral and Written Measure
Mentor Information
Trina Spencer (College of Behavioral & Community Sciences)
Description
The extent to which examiners adhered to assessment protocols can influence the outcomes of a study. Yet, assessment fidelity data are rarely reported in evaluation studies (Reed et al., 2014). Little is known about the extent to which examiner adherence to standardization procedures influence student outcomes on oral and written language measures. Using extant data from a randomized group study investigating the effects of an academic language intervention on 155 first grade students’ oral and written language skills, we examined assessment fidelity. Expository oral retell (ERO) and narrative writing (NGW) language samples were collected from students at three time points: pretest, posttest and follow up. Research assistants, who were blind to student group assignments, independently transcribed and scored samples. About 30% of samples were randomly selected for fidelity examination, which involved another research assistant listening to audio recordings of the data collection session. Fidelity and student outcomes for ERO and NGW measures were analyzed using a two-level hierarchical linear model (HLM) with participant data (Level 1) nested within schools (Level 2) across three time points (pretest, posttest, and follow-up). We hypothesized that low examiner NGW and ERO fidelity to assessment standardization will negatively affect student outcomes.
The Influence of Examiner Fidelity on First Graders’ Oral and Written Measure
The extent to which examiners adhered to assessment protocols can influence the outcomes of a study. Yet, assessment fidelity data are rarely reported in evaluation studies (Reed et al., 2014). Little is known about the extent to which examiner adherence to standardization procedures influence student outcomes on oral and written language measures. Using extant data from a randomized group study investigating the effects of an academic language intervention on 155 first grade students’ oral and written language skills, we examined assessment fidelity. Expository oral retell (ERO) and narrative writing (NGW) language samples were collected from students at three time points: pretest, posttest and follow up. Research assistants, who were blind to student group assignments, independently transcribed and scored samples. About 30% of samples were randomly selected for fidelity examination, which involved another research assistant listening to audio recordings of the data collection session. Fidelity and student outcomes for ERO and NGW measures were analyzed using a two-level hierarchical linear model (HLM) with participant data (Level 1) nested within schools (Level 2) across three time points (pretest, posttest, and follow-up). We hypothesized that low examiner NGW and ERO fidelity to assessment standardization will negatively affect student outcomes.
