Graduation Year

2024

Document Type

Dissertation

Degree

Ph.D.

Degree Name

Doctor of Philosophy (Ph.D.)

Degree Granting Department

Educational Measurement and Research

Major Professor

Robert Dedrick, Ph.D.

Committee Member

Eunsook Kim, Ph.D.

Committee Member

John Ferron, Ph.D.

Committee Member

Sanghoon Park, Ph.D.

Keywords

Behavioral, Cognitive, Emotional, ESM, Longitudinal, Online Engagement

Abstract

This dissertation focuses on the development and initial validation of the In-The-Moment Asynchronous Student Engagement Questionnaire (ITMASEQ), designed to measure undergraduate student engagement—emotional, cognitive, and behavioral—in asynchronous online learning environments. The study responds to the growing need for dynamic, in-the-moment assessments of student engagement, a construct critical to understanding academic success and retention, particularly in online education.

The ITMASEQ included a total of 25 questions: Five contextual questions (study day, study place, company during the study, primary online activity, type of primary online activity), eight emotional engagement questions (enjoyment, feeling calm, anxious, focused, distracted, etc.), seven cognitive engagement questions (interest, challenge, mental effort, etc.) and five behavioral questions (persistency, following directions, etc.). The Experience Sampling Method (ESM) was employed to capture real-time engagement data across multiple interactions: student-content, student-instructor, student-student, and student-AI. The data were collected for 8 weeks, once a week from 83 students who took asynchronous online courses from the College of Education for the main study.

The rigorous item development and review process using multiple expert panels and student feedback about the clarity and relevance of the items strengthened the content validity of the ITMASEQ. Cronbach’s alphas for the three engagement subscales (emotional, cognitive, and behavioral) across the eight time periods were generally low with emotional engagement showing moderate reliabilities (> .70) for five of the eight time periods and cognitive engagement showing moderate reliability for one time point. Reliability was below .70 for behavioral engagement for each time point. While the study offers some initial and promising insights into the structure of the ITMASEQ, the scale is not yet ready for practical application. Further refinement is needed to enhance its robustness and improve its practicality in effectively capturing the construct. Given that the current version of the ITMASEQ includes more items than most Experience Sampling Method (ESM) studies, its practicality warrants reconsideration. While the comprehensive nature of the tool initially seemed advantageous for capturing a wide range of data, a shorter version could offer substantial benefits, particularly in terms of usability and reducing respondent burden.

Download

Share

COinS