skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.


Title: Pathways and Structures: Evaluating Systems Changes in an NSF INCLUDES Alliance
In this article, we reflect on our experience applying a framework for evaluating systems change to an evaluation of a statewide West Virginia alliance funded by the National Science Foundation (NSF) to improve the early persistence of rural, first-generation, and other underrepresented minority science, technology, engineering, and mathematics (STEM) students in their programs of study. We begin with a description of the project and then discuss the two pillars around which we have built our evaluation of this project. Next, we present the challenge we confronted (despite the utility of our two pillars) in identifying and analyzing systems change, as well as the literature we consulted as we considered how to address this difficulty. Finally, we describe the framework we applied and examine how it helped us and where we still faced quandaries. Ultimately, this reflection serves two key purposes: (1) to consider a few of the challenges of measuring changes in systems, and (2) to discuss our experience applying one framework to address these issues.  more » « less
Award ID(s):
1834569 1834586 1834575 1834595 1834601
PAR ID:
10361234
Author(s) / Creator(s):
 ;  ;  ;  
Publisher / Repository:
SAGE Publications
Date Published:
Journal Name:
American Journal of Evaluation
Volume:
43
Issue:
4
ISSN:
1098-2140
Format(s):
Medium: X Size: p. 632-646
Size(s):
p. 632-646
Sponsoring Org:
National Science Foundation
More Like this
  1. From out-competing grandmasters in chess to informing high-stakes healthcare decisions, emerging methods from artificial intelligence are increasingly capable of making complex and strategic decisions in diverse, high-dimensional and uncertain situations. But can these methods help us devise robust strategies for managing environmental systems under great uncertainty? Here we explore how reinforcement learning (RL), a subfield of artificial intelligence, approaches decision problems through a lens similar to adaptive environmental management: learning through experience to gradually improve decisions with updated knowledge. We review where RL holds promise for improving evidence-informed adaptive management decisions even when classical optimization methods are intractable and discuss technical and social issues that arise when applying RL to adaptive management problems in the environmental domain. Our synthesis suggests that environmental management and computer science can learn from one another about the practices, promises and perils of experience-based decision-making. This article is part of the theme issue ‘Detecting and attributing the causes of biodiversity change: needs, gaps and solutions’. 
    more » « less
  2. Millions of people across the world live off-grid not by choice but because they live in rural areas, have low income, and have no political clout. Delivering sustainable energy solutions to such a substantial amount of the world’s population requires more than a technological fix; it requires leveraging the knowledge of underserved populations working together with a transdisciplinary team to find holistically derived solutions. Our original research has resulted in an innovative Convergence Framework integrating the fields of engineering, social sciences, and communication, and is based on working together with communities and other stakeholders to address the challenges posed by delivering clean energy solutions. In this paper, we discuss the evolution of this Framework and illustrate how this Framework is being operationalized in our on-going research project, cocreating hybrid renewable energy systems for off-grid communities in the Brazilian Amazon. The research shows how this Framework can address clean energy transitions, strengthen emerging industries at local level, and foster Global North–South scholarly collaborations. We do so by the integration of social science and engineering and by focusing on community engagement, energy justice, and governance for underserved communities. Further, this solution-driven Framework leads to the emergence of unique approaches that advance scientific knowledge, while at the same time addressing community needs. 
    more » « less
  3. Carnegie Mellon University, Johns Hopkins University, and New York University created the Project Equity-focused Launch to Empower and Value AGEP Faculty to Thrive in Engineering (ELEVATE) Alliance (National Science Foundation Awards #2149995, #2149798 #2149899 from the Division of Equity for Excellence in STEM in the Directorate for STEM Education) to develop a model to promote the equitable advancement of early career tenure-track engineering faculty from populations of interest to the Alliances for Graduate Education and the Professoriate (AGEP) program. The goal of this AGEP Faculty Career Pathways Alliance Model (FCPAM) is to develop, implement, self-study, and institutionalize a career pathway model that can be adapted for use at other similar institutions for advancing early career engineering faculty who are: African Americans, Hispanic Americans, American Indians, Alaska Natives, Native Hawaiians, and Native Pacific Islanders. This NSF AGEP FCPAM will provide a framework for institutional change at private, highly selective research institutions that will enable all faculty to be members of a collaborative community. Improving the experience of these faculty can lead to increased diversity in the engineering faculty and ultimately result in graduating more engineering students from diverse populations and increasing diversity in the engineering workforce. The Alliance interventions will focus on three major areas, 1) equity-focused institutional change designed to make structural changes that support the advancement of AGEP faculty, 2) identity-affirming mentorship that acknowledges and provides professional support to AGEP faculty holistically, recognizing all parts of their identity and 3) inclusive professional development that equips all engineering faculty and institutional leaders with skills to implement inclusive practices and equips AGEP faculty for career advancement. In this paper, we will discuss the process of creating a leadership team to address these focus areas and assess the processes and procedures that currently exist at the three institutions as we begin to institutionalize these change efforts. We provide an overview of the project and efforts to date. We will also present our process for engaging in our initial self-study evaluation and next steps. 
    more » « less
  4. Using the framework of servingness as conceptualized in research of Hispanic Serving Institutions (HSI), measures of student’s success can be characterized using academic and non-academic outcomes (i.e. liberatory outcomes). Academic outcomes are commonly institutional quantitative measures such as GPA, time to graduation, retention, etc. However, within the framework of servingness of an institution, there are additional indicators, including identifying the experiences of students and their non-academic outcomes. A self-advocacy professional development program has been developed that focuses on non-academic outcomes of PhD graduate students in science, technology, engineering and math (STEM) programs at an Urban R1 HSI. Self-advocacy originates from the American Counseling Association (ACA) and the Learning Disabilities (LD) communities for effective counseling that promotes academic success and is based on a social justice framework. The pillars of the self-advocacy program are centered on (i) Empowerment, (ii) Promoting self-awareness and (iii) Social Justice and the programming is aligned to these three pillars. An overview of the project evaluation and focus group findings for the program are presented. 
    more » « less
  5. null (Ed.)
    Our work with teams funded through the National Science Foundation REvolutionizing Engineering and Computer Science Departments (RED) program began in 2015. Our project—funded first by a NSF EAGER grant, and then by a NSF RFE grant—focuses on understanding how the RED teams make change on their campuses and how this information about change can be captured and communicated to other STEM programs that seek to make change happen. Because our RED Participatory Action Research (REDPAR) Project is a collaboration between researchers (Center for Evaluation & Research for STEM Equity at the University of Washington) and practitioners (Making Academic Change Happen Workshop at Rose-Hulman Institute of Technology), we have challenged ourselves to develop means of communication that allow for both aspects of the work—both research and practice—to be treated equitably. As a result, we have created a new dissemination channel—the RED Participatory Action Project Tipsheet. The tipsheet format accomplishes several important goals. First, the content is drawn from both the research conducted with the RED teams and the practitioners’ work with the teams. Each tipsheet takes up a single theme and grounds the theme in the research literature while offering practical tips for applying the information. Second, the format is accessible to a wide spectrum of potential users, remaining free of jargon and applicable to multiple program and departmental contexts. Third, by publishing the tipsheets ourselves, rather than submitting them to an engineering education research journal, we make the information timely and freely available. We can make a tipsheet as soon as a theme emerges from the intersection of research data and observations of practice. During the poster session at ASEE 2019, we will share the three REDPAR Tipsheets that have been produced thus far: Creating Strategic Partnerships, Communicating Change, and Shared Vision. We will also work with attendees to demonstrate how the tipsheet content is adaptable to the attendees’ specific academic context. Our goal for the poster session is to provide attendees with tipsheet resources that are useful to their specific change project. 
    more » « less