%AHowley, Caitlin [ICF, Fairfax, VA, USA]%ACampbell, Johnavae [ICF, Fairfax, VA, USA]%ACowley, Kimberly [ICF, Fairfax, VA, USA]%ACook, Kimberly [ICF, Fairfax, VA, USA]%BJournal Name: American Journal of Evaluation; Journal Volume: 43; Journal Issue: 4; Related Information: CHORUS Timestamp: 2022-12-07 08:18:46 %D2022%ISAGE Publications %JJournal Name: American Journal of Evaluation; Journal Volume: 43; Journal Issue: 4; Related Information: CHORUS Timestamp: 2022-12-07 08:18:46 %K %MOSTI ID: 10361234 %PMedium: X %TPathways and Structures: Evaluating Systems Changes in an NSF INCLUDES Alliance %X

In this article, we reflect on our experience applying a framework for evaluating systems change to an evaluation of a statewide West Virginia alliance funded by the National Science Foundation (NSF) to improve the early persistence of rural, first-generation, and other underrepresented minority science, technology, engineering, and mathematics (STEM) students in their programs of study. We begin with a description of the project and then discuss the two pillars around which we have built our evaluation of this project. Next, we present the challenge we confronted (despite the utility of our two pillars) in identifying and analyzing systems change, as well as the literature we consulted as we considered how to address this difficulty. Finally, we describe the framework we applied and examine how it helped us and where we still faced quandaries. Ultimately, this reflection serves two key purposes: (1) to consider a few of the challenges of measuring changes in systems, and (2) to discuss our experience applying one framework to address these issues.

%0Journal Article