Graduation Year

2014

Document Type

Dissertation

Degree

Ph.D.

Degree Name

Doctor of Philosophy (Ph.D.)

Department

Curriculum & Instruction

Degree Granting Department

Educational and Psychological Studies

Major Professor

Liliana Rodríguez-Campos, Ph.D.

Committee Member

John Ferron, Ph.D.

Committee Member

Robert Dedrick, Ph.D.

Committee Member

Roger Boothroyd, Ph.D.

Keywords

evaluation capacity building, instrument development, internal evaluation, mnemonic device, nonprofit organization

Abstract

Evaluation capacity development (ECD) has been acknowledged as a system of processes to help organizations achieve sustainable evaluation practice. Examining the existing evaluation capacity of an organization before starting an ECD process is necessary and will increase the possibilities of success, determined by the establishment or strengthening of an evaluation system into the organization. In response to this need, this study involved the designing of the Organizational Readiness for Evaluation Capacity Development (ORECD) checklist and its initial validation, using a mixed method research design. The study was conducted in four phases, including: (a) the design of the ORECD checklist based on a review of the literature; (b) a review of the ORECD checklist by five experts to obtain face and content validity evidences, with emphasis on relevance and clarity of the items and how well the items fit the corresponding component; (c) a pretesting about the appropriateness of the wording of the items and format of the ORECD checklist by a sample of doctoral graduate students with formal training in evaluation and professional evaluators; and (d) a field study with 32 nonprofit organizations to determine the utility and benefits of using the ORECD checklist and potential improvements to the instrument. This phase generated information about the psychometric properties as well as consequential validity evidence. Findings indicated that the ORECD checklist has great potential to determine the readiness of an organization to develop evaluation capacity, as demonstrated by the feedback received from various groups of participants, establishing face, content, and consequential validity. Results from the psychometric analysis showed correlations that, for the most part, suggested that the components are measuring aspects of the same construct. In addition, the alpha for most of the components supported the reliability of the ORECD checklist. The two components with alphas close to but below .70 required modifications in order to improve their reliability. Also, it was necessary to modify or reword some of the items. Ongoing efforts should provide information about how the changes made to the ORECD checklist are working and additional validity evidences as the one that can be obtained through factor analysis. This will allow the exploration of the underlying structure of the ORECD checklist and its components. It is expected that the ORECD checklist can be a contribution to the body of literature about ECD helping to address organizational readiness in order to support and sustain the development of evaluation capacity within organizations.

Download

Share

COinS