This paper focuses on qualitative findings from an initial pilot study from a larger, ongoing project at a small, Mid-Atlantic private college. This exploratory study includes the perceptions of two engineering faculty members and one educational support staff using mastery-based teaching and assessment in a project-based engineering program. A semi-structured interview with multiple open-ended questions were used to prompt participants to share their experiences with assessment in relation to their self-efficacy around teaching and their perceptions of assessment in relation to their students' failure mindset, metacognition (awareness of learning processes), and agency (ownership of learning). Directed content and thematic analysis were used to identify codes and develop themes in relation to how participants described certain features of assessment in their engineering program. Preliminary results will illustrate features of mastery-based learning that faculty highlighted as particularly challenging or successful and related lessons learned. The initial themes and patterns identified in this preliminary pilot study will be used to set up a more focused secondary full data collection phase in the larger study. Additionally, this poster serves as an opportunity to initiate important dialogue around the implementation of mastery-based assessment and project-based learning in engineering programs and to better support engineering faculty in incorporating elements of mastery-based teaching and assessment.

Elizabethtown College, in partnership with Greenway Institute, has founded the Greenway Center for Equity and Sustainability in Engineering which re-designs and re-centers engineering education around a mission-driven focus on sustainability and the core equity practices that students from underrepresented groups identify as drivers of their success. The Center reimagines engineering education from the ground up at a new and separate location, implementing an integrated package of best practices in a way that existing infrastructure and institutions cannot. It provides a supportive, inclusive community where students learn engineering by working in teams on hands-on multidisciplinary engineering challenges and every student can develop the competence, confidence, and connections they need to thrive in engineering.

In the existing higher education models, systemic inequities in preparation, barriers to entry, societal stereotypes, and harsh program cultures amplify underrepresentation in engineering of these historically excluded groups [1]. The high-stakes, competitive, individualistic nature of engineering programs makes students from underrepresented groups feel isolated and overwhelmed [2], [3]. Engineering programs tend to have intense curricula with little flexibility and rigid prerequisite structure, preventing students from shaping their learning, pursuing what they're passionate about, and connecting their learning to societal problems [4], [5]. As of 2020, women earned only 21% of the nation's engineering degrees and made up only 15% of the engineering workforce [6]. Black and Hispanic students earned 7% and 12% of STEM degrees, lower than their representations in undergraduate college of 10% and 15% respectively.

Rural students are also a population of interest, as they are amongst the most underrepresented in engineering. Many lack access to the role models and educational opportunities that would inspire them to pursue engineering. The smaller size of many rural schools prevents them from offering advanced STEM or AP courses; rural students enter college with comparatively weak math backgrounds and thus are less equipped for degrees in engineering on entry [7]. In dissertation research, De Urquidi found that 8% of all college students, but only 4% of engineering students, come from rural remote communities [8]. Furthermore, while the world is in dire need of sustainability, the engineering field is in dire need of equity [9]. It is widely recognized that the two are inextricable [10], [11], yet traditional engineering programs embody biases and practices that perpetuate inequity. For example, most institutions of higher education employ grading schemes that rank students, have competitive and isolating cultures, emphasize passivating lectures [12], and are isolated from real-world engineering impacts. Too many "equity" programs are focused on helping underrepresented students survive in programs that are structurally unsupportive. We are creating a program that is equitable and sustainability-centered by design in a resource-constrained context representative of real-world conditions and thus more likely to yield scalable, replicable results.

All this suggests that simply recruiting more students from underrepresented groups will not suffice. The Center's approach involves recognizing that equity and sustainability need each other; that the drastic changes needed to meet the needs of underrepresented students calls for a new learning enterprise dedicated to equity from its inception, not just a support system bolted on to a traditional institution; that continuous self-improvement is necessary; and that incremental strategies have been ineffective and we must substantially reorganize the core educational approach to overcome systemic and institutional barriers.

To answer the call for substantially reorganizing the core educational approach, The Center integrates a suite of best practices based on current research. Educational literature indicates that (1) developing a strong sense of purpose or mission, (2) organizing learning around hands-on work on real engineering challenges, (3) emphasizing mastery learning and assessment, and (4) strong mentoring are all effective tools for supporting success in underrepresented students in engineering [1], [4], [13], [14]. When well implemented, these principles have demonstrated effectiveness at student engagement, improving academic outcomes, bolstering student selfefficacy, and fostering a sense of belonging. All the Center's programs are organized around these practices, thus providing all students with access to relevant, supportive instruction that gives them the opportunity to develop the competence, confidence, and connections they need to thrive in engineering.

This work particularly focuses on the Center's innovative use of close faculty mentoring paired with mastery assessment in an entirely hands-on, multidisciplinary, problem-based curriculum. Through our findings, we seek to better understand faculty perspectives of teaching with mastery assessment to support a user-oriented perspective that can help other engineering faculty navigate the challenges of using evidence-based teaching practices in their own classrooms.

RQ1: How do faculty perceive the experience of teaching with a mastery-based approach in a project-based curriculum? RQ2: How does the role of faculty change in a mastery-assessed, project-based curriculum? RQ3: Are there strategies that faculty can employ to enrich positive impacts of mastery-based learning on their students?

This paper reports primary, qualitative findings from an ongoing NSF-funded initiative (NSF Grant #) exploring faculty perceptions in in an entirely hands-on, multidisciplinary, problembased curriculum. This work is based on a pilot version of this program in which a small group of students engage in a semester consisting of a series of interdisciplinary, hands-on project modules, through which they are coached by faculty and academic support staff to achieve and demonstrate mastery of a prescribed set of skills mapped to standard engineering science courses including Statics, Circuits, Calculus III, and Physics II. An exploratory approach was taken in this pilot phase to guide future research directions. This study utilized semi-structed interviews with two faculty and one educational support staff participants to begin to understand the experiences and perceptions of teaching in a mastery-assessed, project-based curriculum. For conciseness, we are referring to all three participants as 'faculty' due to the team-teaching nature of the program and the high level of involvement from the educational support staff in developing and delivering the curriculum.

Due to the small size of the study population, identifying demographic information has been redacted and pseudonyms (Professor Alpha, Professor Beta, and Professor Gamma) were assigned by the researchers. It is important to consider both the small size of this study and the unique context of the program as one considers the applicability of this study's finding to their own engineering courses and programs. While the larger goal of this work is to scale this program, this is a pilot study whose findings are predominantly intended to initiate discussion and influence future work and direction.

To explore participant's perceptions of teaching in this innovative environment, this work used semi-structured interviews. This interview approach is particularly suited for exploratory work as it balances pre-set questions to keep the data collection focused with spontaneous questions to explore, deepen understanding, and clarify answers to earlier questions [15]. Interviews were conducted by the third author during the latter half of the fall semester and were audio recorded before being transcribed by Otter.ai (Otter.ai Inc, 2023) and edited for clarity by the second author.

Interview questions were derived from theory and prompted participants to reflect on their experiences with mastery-based learning, features of the program, individual and community efficacy as educators, as well as their perceptions of the student's failure mindset, attitudes toward assessment, performance/ competence, metacognition (thinking about learning process), agency (ownership of learning), and engineering identity (Table 1). The semi-structured nature of these interviews allowed for additional follow-up questions that permitted a better understanding of how these concepts were or were not supported by programmatic features.

The six steps of thematic analysis outlined by Braun and Clarke [16] were leveraged to identify major meaningful patterns in participants mindsets and interpretation in their engineering programs. First, researchers familiarized themselves with the data through re-reading transcripts, listening to the audio file, and making initial notes on the data [16], [17]. Second, transcripts underwent a round of exploratory coding in which structural codes and open codes were simultaneously applied. Structural codes were derived from the theory guiding the larger study and were used to index the data and provide a theoretical overview [16], [18]. Open coding was included to capture nuances of the experiences that were not adequately captured by the predetermined structural codes or to highlight elements that emerged through participant responses.

In this work, coding was used as a heuristic tool that goes beyond labeling data but to link or connect data to data, data to ideas, and ideas to ideas [18].

Third, researchers searched for initial themes in the data by focusing on similarity and overlap between codes. A pattern coding pass was applied to collapse and cluster codes together [16], [18]. Fourth, researchers reviewed the various themes that emerged and discussed whether they were themes, codes, or actually nuances of the same thing and should be combined. Fifth, these codes were defined and a named before finally being written and described as the findings of this study [16]. Three major themes were identified that focused on how faculty perceived the relationship of mastery-based learning to their role and to the student learning experience. These preliminary themes are detailed in the following section.

A first theme that emerged consistently amongst the participants focused on the relationship of mastery-based learning to productive failure, risk-taking, and persistence of the students.

In particular, in the context of their mastery approach, all three participants described "failure" as an opportunity to learn:

looks like… there are multiple ways you could do mastery-based learning. One is not showing up. Another is not trying, not putting yourself out there. Then I suppose there's the possibility of just getting the material that we have. There are numerous opportunities and structures to try… and both our team and the students ourselves will help you get there. I think failure would generally be not trying. (Professor. Alpha) What does failure look like? It is a learning opportunity. It's like a normal step of the process. It's celebrated as an opportunity to learn a little bit more. .. It's not like you're a failure. It's that you need a little bit more practice here. (Professor Beta) I don't think the students ever really see themselves as failing is the thing, I think it's either like they have mastery of something or they don't have mastery of something yet. (Professor Gamma) The faculty further explained their perception that mastery-based learning supported risk taking and persistence in learning by giving students multiple opportunities to iterate and how that impacted their perception of student attitude and anxiety: It's much more lighthearted. I don't totally know how to say..I mean, that's probably more than that too. Yeah, to contrast this from the environment that I was teaching in before I see so many students just getting buried under stress about workload stress about passing X, Y and Z classes. And kind of doing things to check them off. I think that the approach the students here are taking they just seem like mentally well, in a way that is not typical and it not is not the environment that I'm used to. It's a bit more of a relaxed environment. The stakes of each individual assignment don't feel as high. And also, it's a much more collaborative, there's not a lot of competition because they can each go as far as they want to any subject without feeling like there's going to be a curve. (Professor Gamma) They may have been disappointed that they were wrong, but they've never been, like, offended or visibly upset about it. And they, at least verbally, very clearly state that that's the thing that they like, the ability to iterate. (Professor Alpha) Students feel less fragile. It's not confidence exactly, but they're related somehow. I think that when students kind of feel like they have infinite pathways to success, they aren't mired down when they don't get something the first time or even the fifth time. And I'm seeing students trying something for the seventh and eighth time and expecting to succeed at it. So I think that it improves confidence. (Professor Gamma) A second theme centered on the faculty's use of mastery-based teaching strategies that fostered student metacognition, or awareness of their own thought processes, including frequent feedback (both formal and informal), cognitive coaching and talking out loud about problems and strategies. I think it came from the way that our faculty talk with our students about assessment and their learning. And I hear them turn it on, the students, really often. So you know, like, saying things like "you can pursue this outcome but you don't have to." Or "you know, if you don't pass this this quiz this time, let's just touch base and figure out what you need to work on next…. I think you still hear them internalize like speaking because they internalized these processes. You hear them catching themselves or hear them talking through and catching each other on pieces of feedback that we initially started to give. (Professor Beta) It puts us in more of a coaching role, where I think it's more clear to students that I am on my students side, but I think other kinds of assessment can make it feel like you're pitted against each other, faculty against student… I mean there's a power stuff there too, but I think it can really feel like that to the students that it's pitting us against each other.. And with mastery-based, I think that the coaching vibe comes across much more clearly. And my sense is that the students really feel like we're on their side, and that we want them to succeed and you know, we're committed to doing our part to seeing them succeed, whatever that looks like… So a lot of it I do think is me and [the other faculty's] early messaging about … when students are like, taking a quiz and not getting it the first time, then our message to them is "not yet." the next step is let's talk together about the next steps." So like, the recourse from that is a conversation and an action plan. (Professor Gamma) …I think feedback is a big one that I've been thinking about and I think I'm surprised at how good some industry folks are at giving good feedback. And how sometimes we as teachers can be really hesitant to give harsh feedback or maybe honest feedback is more a word for it. Because we don't want to hurt their feelings or make them feel like they're not on the right path. But yeah, then we would bring the students out to the real world and, like, industry folks just have so much to say. They're coming from some real-world contexts, and they just can share that really clearly with the students. And I think it's, I think the students see the value in that and so even if it's harsh, like your feedback, they appreciate it. [When the students] have gotten very direct feedback in a number of ways and they like are open to it. They are thoughtful about it. And they are like incredibly willing to not be defensive and change course. (Professor Beta)

Notably, these practices were focused on cultivating not specific knowledge, but rather expert practice; e.g. thinking like expert about the disciplines they were studying: … it's not about the specific skills but about kind of the collective ability to work problems, to approach new things, to learn how to learn, it's like higher level stuff… which is to say that it's far more important to know when to use calculus and to know how to use it and so like identity but that is like really expert level thinking that's really hard to get students to, to get to you have to really know a discipline pretty well to get to that point. And so I guess Yeah, I would say… Knowing when to apply some calculus theorem is even more important than knowing the theorem itself. (Professor Gamma)

Finally, a third theme expounded that faculty perceive their teaching practices and coaching fostered a change in mindset toward more agency, or more ownership of their learning.

One of the gifts of mastery-based learning is really encouraging students to be able to explore a topic in a semi-independent way. In a sort of coached but independent way. And the students had a lot of autonomy over whether or not they did that. So I had students who really grabbed the bull by the horns and did that.

Those were different students in different subjects, which was cool to see and I think is a real strength. (Professor Gamma) It was a change in mindset of from being a passive kind of receiver of curriculum, to being an active learner and needing to figure out like, start from the end. Okay, this is the outcome they want to get was seen and how we get there and then figuring out the steps to get there. (Professor Beta) On the wind turbine project (Student name) was doing experiments in a pretty good scientific method. Multiple measurements and different conditions. But he was measuring in a way that made about half the data useless. So we kind of threw together a couple of different options on what he could do. I'd actually kind of want him to do something slightly different than what he ended up doing. But by just having this conversation and doing those things he came up with a new strategy and it was a superior strategy and a good one. An ideal outcome is when you not only redirect to something more efficient, but it's something more efficient that is their creation versus something you just feed to them. (Professor Alpha) I think that's been a huge thing that I've noticed is they started off like talking about A's and being really nervous about like, what do I have to do to get this grade? And now you hear them talk about how deeply they learned something and what interests them and feeling like they, again, like the greatest, so much less relevant because what they are really engaged in is something that can't be quantified by numbers. It's about their own personal growth. (Professor Beta) A related exploratory study involving semi-structured interviews of the pilot cohort students, being presented as a Work in Progress paper at ASEE 2024, confirms that the faculty perceptions were consistent with the students' experiences. Reframing failure as an opportunity to learn: Before starting at Greenway, my definition of failure was that I'm dumb or I can't achieve something. But here my definition of failure is I'm not there yet. And the key word is yet. It kind of allows me to see that progress is like steps. It's not like a dead end road. Just because you made it halfway up the steps you still don't see the top doesn't mean that you should stop. And so it's definitely changed my view to where I see failure as progress. (Student #1) Failure is …if there is a word that has kind of changed meaning, especially since this program and in this context, it does not mean that I have done something bad or wrong. It just meant that I didn't get the expected outcome that I wanted. And it's a way to learn. you don't learn from succeeding on the first try… Failing means that you have an opportunity to grow, and then being able to take that and continue on with it through the mastery-based learning turns a failure into a step toward success. (Student #4) Mastery-based learning's impact on anxiety: Because of the mastery-based assessment, I'm a lot more comfortable taking risks and trying new things. Because I know if it takes me a while, I can keep trying until eventually I get it. And I'm not going to be punished for taking a while to learn something. So I can try new things. And if I don't know how easy they're going to be I'm not too worried about that. Whereas in a traditional classroom environment, if I wanted to maybe take risks in that class or try something, I would be a bit more hesitant because I know I've got to practice certain things for the test. And if I don't know them by the test, I'm going to get a bad grade and it's going to hurt my GPA overall. Whereas at --where I can take risks, I can choose what I apply to be something that I know will be harder and I'll get more out of because I can take the time I need to take and I won't be punished for it. (Student #2) I think it's boosted my confidence with myself. That I know that I can take my time to achieve what I want to achieve. And I know that I am allowed to make mistakes… that no one's perfect, and that's to be expected. And that's normal, and that's not an issue. (Student #4) On metacognition: The instructors help in that way, because they sort of purposefully don't give you….They don't just lay out the strategy to do a problem or give you… and they don't even give you every tool to solve it,...they make sure that you have to dig a little bit, which sort of makes it stick in your brain better. So that kind of in itself makes the students have to take risks. And then personally, I guess I don't mind making something hard for myself to make sure that I try to do it more… I am very confident that I understand the engineering that I want to understand. I'm not 100% Confident in statics … If I wanted to go farther in statics, I'd probably have to learn more. (Student #3) That changes my experience of learning, because then I'm number one, I'm not afraid to fail. Number two, I push myself to actually understand the concept because I'm not just trying to memorize the definition out of a book, because the book is right in front of me. And I could read that definition, I have to make sure I actually understand what it means though, which is great to be able to apply it. And overall, it's pushes you to a different type of learning. (Student #1) On agency: Yeah, so like with the mastery-based and with a lot of how open ended a lot of the outcomes were here, I was able to pick specific things that I wanted to hit. And this is why with transferable (skills), I can swap in and out specific zones that I'm less interested in or feel don't hold as much weight as others, and I can focus in on things that I know are going to stick with me, so that I can both focus on how I'm going to learn these things, and also, what things I'm going to focus on. (Student #4) I choose to learn more about certain topics that interest me as well. Back in a traditional classroom environment, even if there's a certain topic that interests me, if I want to learn more about it, that's all on my own time. Because all I need to know and all that I'm graded on and all that it really matters that I know is what's going to be on the test. Whereas here because I can choose my learning outcomes, I can say, "hey, this is a topic that really interests me, I'd like to learn more about it. I'd like to relate this to engineering." And then I can go out and I can do that and instead of being…not punished but choosing to learn more about topics that interest me being, at best, is something that doesn't really matter to the course. Now here it's something that's actively rewarded. (Student #2)

The findings of this exploratory study suggest that faculty perceive several advantages and positive outcomes for students when teaching with a mastery-based approach in a project-based curriculum. A related exploratory student of the pilot cohort of this program corroborates these faculty perceptions. In particular, faculty noted a positive reframing of failure which also enhanced student agency and reduced academic anxiety. The findings suggest that more work should explore the power of mastery-based learning in reframing learning as an iterative process, in ways that redefine risk and failure and motivate students to challenge themselves and take ownership of their learning in a way that may reduce student anxiety. Related studies by researchers at the first author's institution are being proposed to ASEE 2024 that suggest that mastery-based learning employed in a traditional engineering program enhances a productive failure mindset, decreases test anxiety, and increases students' sense of belonging.

Secondly, additional work should better understand the different role for faculty in this teaching context, and how they can be most effective in that role. For example, elucidating the faculty coaching behaviors and practices that build student metacognition, which helps create their sense of agency and responsibility for their own learning. By using strategies that explicitly give students language and encourage introspection about their own learning, faculty may be able to better operationalize potential positive impacts of mastery-based and problembased learning. It may also be that explicitly layering metacognition around problem solving and assessment is valuable in helping students learn certain skills and knowledge.

While these results are preliminary and exploratory, they suggest that other programs might consider expanding their use of mastery-based learning while positioning instructors as coaches who explicitly use the language of metacognition (learning to learn) and who, through formal course structures or informal coaching practices, encourage students to reflect on their own learning process.

In the larger landscape of engineering education, expanding this work beyond the exploratory study could speak to the gap in the literature around how the well-established constructs of metacognition, student agency, problem-based learning, and mastery-based learning come together and influence one another.

We must update engineering curriculum to serve today's diverse students and address our most pressing global issues such as climate change. To do this, we must reimagine engineering education to include more hands-on, multidisciplinary, problem-based learning as well as a new role for faculty in guiding, coaching, and mediating the overwhelming wealth of textbook information students' have at their fingertips. In addition, based on our experiences offering training on mastery-based learning, faculty are already increasingly interested in moving toward a more formative and individualized type of assessment.

These substantial changes can be frustrating for faculty and administrators as they require time and effort. Therefore, we need programs like The ---Center, Iron Range, Olin College, and others to push the boundaries of innovation while learning what works in various contexts and how to make these changes efficiently and effectively. From this work, we have learned that faculty engaging in mastery-based assessment and project-based learning can enhance the positive impacts from these practices for their students if they also explicitly teach and discuss metacognition (learning how to learn).