skip to main content


Title: Relationships Between Quantitative Measures of Evaluation Plan and Program Model Quality and a Qualitative Measure of Participant Perceptions of an Evaluation Capacity Building Approach
Award ID(s):
0814364
NSF-PAR ID:
10012542
Author(s) / Creator(s):
; ; ;
Date Published:
Journal Name:
Journal of Mixed Methods Research
Volume:
9
Issue:
2
ISSN:
1558-6898
Page Range / eLocation ID:
154 to 177
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. To reshape energy systems towards renewable energy resources, decision makers need to decide today on how to make the transition. Energy scenarios are widely used to guide decision making in this context. While considerable effort has been put into developing energy scenarios, researchers have pointed out three requirements for energy scenarios that are not fulfilled satisfactorily yet: The development and evaluation of energy scenarios should (1) incorporate the concept of sustainability, (2) provide decision support in a transparent way and (3) be replicable for other researchers. To meet these requirements, we combine different methodological approaches: story-and-simulation (SAS) scenarios, multi-criteria decision-making (MCDM), information modeling and co-simulation. We show in this paper how the combination of these methods can lead to an integrated approach for sustainability evaluation of energy scenarios with automated information exchange. Our approach consists of a sustainability evaluation process (SEP) and an information model for modeling dependencies. The objectives are to guide decisions towards sustainable development of the energy sector and to make the scenario and decision support processes more transparent for both decision makers and researchers. 
    more » « less
  2. Bauerle, Cynthia (Ed.)
    Most science, technology, engineering, and mathematics (STEM) departments inadequately evaluate teaching, which means they are not equipped to recognize or reward effective teaching. As part of a project at one institution, we observed that departmental chairs needed help recognizing the decisions they would need to make to improve teaching evaluation practices. To meet this need, we developed the Guides to Advance Teaching Evaluation (GATEs), using an iterative development process. The GATEs are designed to be a planning tool that outlines concrete goals to guide reform in teaching evaluation practices in STEM departments at research-intensive institutions. The GATEs are grounded in the available scholarly literature and guided by existing reform efforts and have been vetted with STEM departmental chairs. The GATEs steer departments to draw on three voices to evaluate teaching: trained peers, students, and the instructor. This research-based resource includes three components for each voice: 1) a list of departmental target practices to serve as goals; 2) a characterization of common starting places to prompt reflection; and 3) ideas for getting started. We provide anecdotal examples of potential uses of the GATEs for reform efforts in STEM departments and as a research tool to document departmental practices at different time points. 
    more » « less
  3. Program leaders put a tremendous amount of thought into how they recruit students for engineering summer camps. Recruitment methods can include information sessions, established partnerships with school districts, and teacher or school counselor nominations of students. This study seeks to assess if the methods used to recruit students broaden participation or have any impact on students’ perceptions of engineering. Two identical week-long summer camps were hosted by the University of Texas at Austin (UT Austin) in the summer of 2022. Camps were entirely free for all campers. A specific goal of the camp was to promote engineering as a career pathway for students from groups that have been historically excluded from STEM majors. Campers were rising 8th and 9th grade students in two cities near UT Austin; this age was intentionally identified as students who have sufficient STEM backgrounds to engage in meaningful engineering design challenges, and who are also at a critical inflection point with respect to decisions that put them on a trajectory to study engineering in college. Summer camp topics ranged from additive manufacturing to the chemical properties of water proofing, and students did activities such as constructing a prosthetic limb from recovered materials or designing an electronic dance game pad. In one camp session, students primarily found out about the camp by being nominated by counselors at their schools, with an intentional focus on recruiting students who might not otherwise be exposed to engineering. In the other camp session, parents signed up campers after hearing about the camp via information sent through the schools. All students who applied were accepted to the camps. Identical pre- and post-camp surveys asked campers questions about their knowledge of what engineers do, their interest in math and science, and what factors are important to them when choosing a career. Survey analysis showed that there were statistically significant differences in answers to questions between the groups in the pre-camp surveys, but post-camp surveys show that these differences disappeared after participating in the summer camp. Students whose parents directly enrolled them in the camp had higher pre-camp interest in science and technology; thus, counselor nominations may be a method to recruit students who might not have been interested in engineering had they not attended the camp. Additionally, prior to participating, campers recruited via counselor nominations had a narrower view of what engineers do than the parent-enrolled campers, but after camp the two groups had similar perceptions of what engineers do. The results of this study confirm literature findings regarding the importance of exposing young learners to engineering as a profession and broaden their views of opportunities in this field. The recruitment methods used for these camps show that nomination-based recruitment methods have the potential for greater impact on changing students’ engineering trajectories. 
    more » « less