Graduation Year

2014

Document Type

Thesis

Degree

Ph.D.

Degree Granting Department

Educational Measurement and Research

Major Professor

John M. Ferron

Keywords

Hierarchical, Multilevel, N=1, Single-subject, Synthesis

Abstract

Single-case interventions allow for the repeated measurement of a case or participant across multiple time points, to assess the treatment¡͞s effect on one specific case or participant. The basic interrupted time series design includes two phases: baseline and treatment. Raudenbush and Byrk (2002) demonstrated that a meta-analysis of large group designs can be seen as a special case of multi-level analysis with participants (level-one) nested within studies (level-two). Raw data from a set of single case design studies have a similar structure. Van den Noortgate and Onghena (2003) illustrated the use of a two-level model to analyze data in primary single-case studies. In 2008, Van den Noortgate and Onghena later proposed that if raw data from several single case designs are used in a meta-analysis, scores can be varied at each of the three levels: over occasions (level-one), across participants from the same study (level-two), and across studies (level-three).

The multi-level approach allows for a large degree of flexibility in modeling the data (Goldstein & Yang, 2000; Hox & de Leeuw, 1997). Researchers can make various methodological decisions when specifying the model to approximate the data. Those decisions are critical since parameters can be biased if the statistical model is not correctly specified. The first of these decisions is how to model the level-one error structure--is it correlated or uncorrelated? Recently, the investigation of the Van den Noortgate and Onghena¡͞s (2008) three-level meta-analytic model has increased and shown promising results (Owens & Ferron, 2011; Ugille, Moeyaert, Beretvas, Ferron, & Van den Noortgate, 2012 ). These studies have shown the fixed effects tend to be unbiased and the variance components have been problematic across a range of conditions. Based on a thorough literature review, no one has looked at the model in relation to the use of fit indices or log likelihood tests to select an appropriate level-one error structure.

The purpose of the study was two-fold: 1) to determine the extent to which the various fit indices can correctly identify the level-one covariance structure; and 2) to investigate the effect of various forms of misspecification of the level-one error structure when using a three-level meta-analytic single-case model. This study used Monte Carlo simulation methods to address the aforementioned research questions. Multiple design, data, and analysis factors were manipulated in this study. The study used a 2x2x2x2x2x5x7 factorial design. Seven experimental variables were manipulated in this study: 1) The number of primary studies per meta-analysis (10 and 30); 2) The number of participants per primary study (4 and 8); 3)The series length per participant (10 and 20); 4)Variances of the error terms (most of the variance at level-one: [¦Ò2=1;¡¼ ¦²¡½_u = 0.5, 0.05, 0.5, 0.05; ¡¼ ¦²¡½_v = 0.5, 0.05, 0.5, 0.05] and most of the variance at the upper levels: [¦Ò2=1;¡¼ ¦²¡½_u = 2, 0.2, 2, 0.2; ¡¼ ¦²¡½_v = 2, 0.2, 2, 0.2]); 5) The levels for the fixed effects (0, 2 [corresponding to the shift in level]; and 0, 0.2[corresponding to the shift in slope]) 6)Various types of covariance structures were used for data generation (ID, AR(1), and ARMA (1,1); and 7) The form of model specification [i.e. ID, AR(1), ARMA (1,1)], and error structure selected by AIC, AICC, BIC, and the LRT.

The results of this study found that the fixed effects tend to mostly be unbiased, however, the variance components were extremely biased with particular design factors. The study also concluded that the use of fit indices to select the correct level-1 structure was appropriate for certain error structures. The accuracy of the fit indices tend to increase for the simpler level-one error structures. There were multiple implications for the applied single-case researcher, for the meta-analyst, and for the methodologist. Future research included investigating different estimation methods, such as Bayesian approach, to improve the estimates of the variance components and coupling multiple violations of the error structures, such as non-normality at levels two and three.

Download

Share

COinS