Laboratory experiments play a critical role in the professional work of chemical engineers [1,2]. Experiments are used in many facets of engineering (Figure 1). Hands-on laboratory experiences at the junior and senior levels typically reinforce concepts learned in course work and offer opportunities to practice technical communication skills that will benefit students in their future careers. Thus, regardless of whether they are headed to industry or graduate school, experimental design and communication are critical skills in the professional formation of chemical engineers. Despite increasing calls for modernizing chemical engineering curricula in line with professional concerns [1], many laboratory experiments have remained relatively unchanged. One reason for this is the high cost of laboratory equipment that prevents most departments from purchasing new equipment unless necessary. As much of the laboratory equipment is fixable by replacing parts, it is not uncommon for teaching laboratories to have the same equipment for decades. For example, in one of our study sites, the valve used for an experiment on level control on a water tank is being replaced, which is not an uncommon occurrence. However, that particular valve was manufactured in 1947 and has been used on that exact experiment since 1952 (Figure 2). Likewise, a set of packed-bed columns have been used by students in the same experiment since 1978 (Figure 3). Our purpose here is not to critique the enduring (and sustainable) use of equipment, but to recognize how it shapes both faculty and students' expectations about experimental objectives. Given that the equipment is unchanging, it is not surprising to find that the experiments also endure. To preface the main purpose of this work, we present an example of how enduring laboratory equipment can be used in new ways-through course-based undergraduate research experiences (CUREs). This example illustrates what is possible in terms of students having consequential agency, but it is also an example of instruction that can be challenging to implement. Using instrumental case study of a senior-level chemical engineering laboratory course, in our prior research, we investigated students' perceptions of their agency in a CURE [3] and here we draw from that study's analysis, which focused on how students negotiated uncertainty because of the ambiguity of their choices in their CURE. Students were tasked with selecting a catalyst and several experimental conditions, and while they did use published research to inform their choices, they quickly realized that their results might differ from their expectations. Initially, some students worried that if they chose poorly, they could end up with a low grade. The instructors reassured students (through meetings and rubrics) that if they could explain their choices, they would not lose points because their catalyst did not perform as expected. For instance, an instructor explained "We're giving you a chance to practice," and if your experiment doesn't work as expected, "Fine, what can we learn from this data?" In this process, most students came to recognize that failure is endemic to the research process and even a learning opportunity. For instance, one student explained a failed experiment, "That's just experiments. That's how science works." Some students jointly expressed frustration about the openness of the experiment while recognizing its value for their learning, "So for me, honestly, I like-like prompts and straightforward. Like, this is what I want, is structured. It's easier. But [the CURE is] more challenging and in the end, like more rewarding. I thought it was cool that we got to choose our own catalyst and it gives the people that are actually super interested in ChemE and like doing research and stuff, um, the opportunity to pursue some of those things. So, I think it is really cool. It's hard." Others primarily focused on how having agency to make choices was motivating for them, "I liked having the freedom, to kind of decide what it was that we wanted to do. And kind of, you feel like you're running your own experiment." While these results are supportive of CUREs, implementing a CURE comes with substantial instructional challenges. With this understanding that students can be supported to navigate such high-agency laboratory experiments, we seek to investigate how to incorporate instructionally feasible opportunities for student agency into more typical engineering laboratory experiments. As a first step, the current study focuses on development of a new survey that will allow researchers and instructors to examine how more feasible changes impact students' perceptions of their agency. Few studies have examined ways students are impacted by agency in laboratory experiments. We argue that students need opportunities to develop the capacity to make informed and consequential decisions, as these opportunities help them develop professional capacities.
The purpose of this study was therefore to investigate student perceptions of their agency across a variety of experiments. To guide our work, we addressed the following research question:
Experiments survey?
To investigate the question, we developed and implemented a new survey at two institutions and across varied laboratory experiments. While our broader aim is to investigate four domains-(1) experimental design; (2) data collection; (3) data analysis and interpretation; and (4) communication of the experiment-for the purpose of survey development, we restricted our focus to two of the domains-(1) experimental design and (4) communication. This focus allowed us to test a larger suite of possible items while reducing the potential of survey fatigue and will lead to future studies using a refined survey to measure agency across all four domains.
In traditional laboratory courses, experiments are "cookbook" in that they are highly prescriptive and students make few or no consequential decisions [4]. Typically, faculty focus on complexity-a term that references the number of variables and the relations between variables [5]-and this provides students with opportunities to manipulate variables and hopefully understand concepts. However, professional chemical engineers also make decisions about what questions to investigate, how to design experiments to investigate their questions, and how to analyze their data [6]. These are ill-structured problems-that is, problems in which there are multiple possible answers [5]. Though ill-structured problems are sometimes brought into the classroom through design and course-based undergraduate research experiences [7], such approaches are fraught with feasibility issues in laboratory courses that prevent wide-scale adoption [8,9].
To theorize the kinds of agency that students need opportunities to develop, we extend the notion of framing agency. Set in ill-structured design courses, framing agency is defined as the capacity to make decisions that are consequential to how design problems are framed and reframed, and thus, how and what is learned in the process of proposing a solution [10][11][12][13].
To characterize whether a learning experience offers students opportunities to make consequential decisions, we borrow a term from sociology-opportunity structure [14,15], a term used to explain how the organization of society influences decision making, by shaping perceptions of what is possible and promising. In this way, we can understand how students' prior experiences with learning experiences, and especially chemistry labs, shapes their expectations about their chemical engineering laboratory courses. Given a preponderance of well-structured prior experiences, it would be unsurprising to find that students also expect their chemical engineering experiments should offer few chances to make decisions, culminating in a known correct solution. Indeed, in our past related work, we found that students tended to perceive all of the laboratory experiments, even those that emphasized discovery and included some unknown outcomes, as inauthentic and offering few opportunities to make consequential decisions [16].
Making experiments fully authentic-meaning, based in problems that exist externally [17] is unrealistic in terms of time, cost, and capacity, including the acknowledgment that ill-structured problems are difficult to propose, implement, support, and evaluate, evidenced by the limited uptake of such approaches. More authentic tasks can also limit participation, as students may report low agency if they perceive the requisite decisions as beyond their reach [18]. Thus, there is a need for investigations into how to enhance agency in ways that are impactful for student learning and development yet feasible for faculty to manage.
Limited research has hinted at feasible ways to enhance agency. For instance, research on graduate students suggests students appreciate simulation tools that offer them greater agency in supplemental study [19]. Studies on undergraduates suggest students may struggle with openended experiments, but ultimately appreciate their salience for professional practice [20]. In other settings, such as in physics [21,22] and environmental engineering [23], higher-agency laboratory experiments were appreciated by students. While this suggests students may be open to changes, these studies shed little light on the relationships between agency, learning, and identity. Recent research on an undergraduate course sequence suggests students can benefit from scaffolded instruction followed by more self-directed laboratory experiments [24]. However, this approach shares challenges with fully authentic research experiences in that it can be challenging to support students, even in teams, who propose varied experimental designs. This limits both scalability and adoption, despite its value. One approach to counter this is providing constraints that limit the ill-structuredness of the problem [25], but this approach can over-constrain opportunities. These studies highlight the importance of cumulative opportunities to develop agency in experimental design practices, as well as the need for more nuanced investigations into how different domains of agency impact students' professional learning and identity formation.
Collectively, this literature highlights that agency matters in learning, but our current state of knowledge is not nuanced enough. As the highest agency approaches have not been broadly adopted, a nuanced understanding of how agency in each domain contributes to students' professional learning and identity development can shed light on feasible, scalable approaches.
We used a previously-developed survey [16] that had been adapted from a measure of student agency in design [10]. To bring the survey from the design context into the laboratory experiment context, we identified the decisions students might make in experimental design and communication, such as choosing variable or duration of an experiment and making choices about which figures to present in a technical report. We adapted items following research-based guidance [26].
Students at two universities (N=259) evaluated a recently completed experiment in their chemical engineering laboratory course. Because of the varied schedules and courses, this resulted in data for eight experiments, which provided an opportunity to evaluate a survey instrument and the validity of the data for understanding students' agency perceptions. The experiments included bomb calorimetry of sucrose, friction and fluid flow (one version at each university), batch distillation of ethanol and water, optimization of the selective catalytic hydrogenation of acetylene to ethylene, heat exchanger optimization, continuous stirred tank reactor, and reaction kinetics.
We conducted exploratory factor analysis (EFA), with an aim of moving toward a measure of each of the four domains, using fewer survey questions to avoid fatigue. EFA is a method used in validation studies to determine how responses to survey questions group together [27]; when related items group together well, it provides evidence that those questions are measuring the same underlying construct. EFA includes several metrics for determining when to remove items that do not group with others [27].
We followed standard techniques in EFA [27], including using principal axis factoring as our extraction method to account for a non-normal distribution of data [28], as is expected with survey data. We chose an oblique rotation method (promax), as this is appropriate in educational and social science surveys in which some correlation between factors is both anticipated and useful [29,30]. The Kaiser-Meyer-Olkin (KMO) [31] measure of sampling adequacy was 0.78, which met the recommendation of ≥ .70 [32] and Bartlett's test of sphericity was significant, p < .001 [33]. These tests indicate the data were appropriate for EFA. We retained items that were not cross-loaded, that had loadings above 0.4 or below -0.4 [34,35], and that belonged to factors with a Cronbach's alpha ≥ .70 [28,30,36,37].
We examined the items removed, which tended to be those that asked about team decision making. In discussing with the full research team, we realized some courses did not use teamwork, and this resulted in students being unsure about how to answer such questions, which likely resulting in the cross-loading observed. We therefore omitted all questions that referenced teams. As is common, we re-ran the EFA after removing cross loaded items.
We sought to evaluate whether the survey measured the intended constructs. In contrast to commonplace usage of surveys in educational settings, educational researchers do not assume that any single survey question can provide a measure of a construct on its own. This is not true for variables that can be measured more directly. As an example, one's height is a variable that can be measured directly and reported in a single question, whereas one's sense of tallness or shortness is a construct that would require more than one question to provide an adequate measure.
We used EFA to assess whether the questions grouped together, suggesting they measured the same underlying constructs, which are referred to as "latent factors" in typical EFA practice. We found support for five latent factors (Table 1):
• Agency as responsibility. Students' perceptions about their responsibility for making consequential decisions in the experiment overall; • Authenticity. Students' perceptions about whether their experiment has utility outside the classroom; • Agency in the communication domain. Students' perceptions about their responsibility for making consequential decisions related to communicating their experiment; • Agency in the experimental design domain. Students' perceptions about their responsibility for making consequential decisions related to experimental design; and • Opportunity structure. Students' perceptions about whether the experiment permitted them to make consequential decisions.
Thus, the questions grouped into conceptually clear and expected latent factors. This is evidence that the survey can provide valid data for informing curricular decisions and for additional research studies involving these constructs. The final column of Table 1, α if deleted, indicates that some items may be removed in future versions of the survey; specifically, the overall factor α will not be negatively impacted by removing items such as "How responsible or not responsible have you felt for the preparation of the presentation?" and "Considering the experiment, have you had many or few opportunities to make decisions personally related to your experimental design and interpretation of results?" Table 1. Exploratory Factor Analysis (EFA) results, presented following standards [27], shows five latent factors recovered. The highlighted cells show questions that load at a level that meets standards (> .40 or < -.40); this loading indicates which questions group together as a latent factor. These factors are retained if they have a Cronbach's alpha ≥ .70; the factors can then be named by researchers to reflect the latent construct measured. The mean and standard deviation are for the responses to the item (not the mean of the loadings). The α if deleted indicates the impact of removing the item on that factor's α; this may only be calculated when there are three or more items.
Incorporating opportunities for consequential agency into more traditional laboratory experiments is important and worthwhile, however, it is not well understood how and what choices would have the most impact on the student's perception of agency. Therefore, we conducted exploratory factor analysis to evaluate a survey as a means to measure different facets of agency. We found strong support for items measuring agency as responsibility, authenticity, agency in the communication domain, agency in the experimental design domain, and opportunity structure. In our ongoing work, we plan to extend the survey to measure agency in all four domains: (1) experimental design, (2) experimental oversight & data collection, (3) data analysis & interpretation, and (4) communication. Specifically, we plan to investigate how relatively minor changes to the opportunity structure of each domain contributes to students' perceptions that they can make consequential decisions (Table 2).
Experimental design, oversight & data collection, and analysis & interpretation may vary in their consequentiality. One way to understand this issue is to consider the consequences of unmet aims under low versus high agency conditions. When students propose their own experimental objective that is under-constrained or unanswerable and craft a protocol that tests a different hypothesis, they are likely to miss intended learning objectives related to the concepts. However, they may instead learn about experimental design. In contrast, when students simply carry out the protocol to meet the objective set by the instructor, they might not notice a missed step in a long set of procedures, and when the outcome differs from the expectation, they may miss learning about both the concept and experimental design. Our ongoing work investigates the most impactful, feasible changes that enhance consequential agency, and in turn, support professional learning and engineering identity development.