An official website of the United States government
Background. While educational change often involves bold talk about disruptive ideas that eventually need to be institutionalized, a critical but often less visible element of sustaining change is work such as maintaining a shared vision, onboarding new people, negotiating small issues in light of department culture, and coordinating big changes with existing efforts. While knowledge about these forms of invisible work exist in other disciplines, these issues seem understudied in engineering education. This work approaches this issue of invisible knowledge with a design orientation, and specifically draws on the field of design-based research. Increasingly, design is recognized as a knowledge producing activity, resulting in insights into generative ways of defining problems, frameworks for generating solutions to problems, examples of what it looks like to connect theory to specific problems. Purpose: As a design effort, this work asks: How might a specific department create a sustainable practice to support the invisible work of coordinating and sustaining change? As a scholarly effort, this instance of design can result in a culminating problem definition, a solution framework, and examples of theory use that represent knowledge contributions. Approach: A mechanical engineering department in a small, private educational institution worked for four months to develop a sustainable practice to support invisible work of coordinating and sustaining change. Following an initial commitment of 60 minutes once every three weeks and 3-hour retreat to explore possibilities, the department then iteratively designed and then carried out sample conversations. Each iteration involved specifying the goals of the conversation, how to have the conversation (the design) and the rationale for connecting the design to the goals. Traces from the process represent the data for this work. Results. Over time, the conversations came to be designed along four dimensions: topic, time allocation, turn-taking, and traces. We have learned that topics that are of immediate relevance to everyone are particularly powerful (initial topics included "being back on campus" and "navigating in-person"). We are currently leveraging a time allocation that devotes the most time to hearing from each participant on the topic, then time for the group to cautiously explore synthesis, and finally time for the group to weigh in on future conversation topics. Approaches to turn-taking have involved decentralization (e.g., each current speaker invites the next speaker) and respect (speakers have a chance to "pass" and then choose the next speaker). Finally, we are experimenting with how to balance the creation of traces as a natural part of the process, such as through real-time transcription in the chat feature of zoom. Undergirding each of these dimensions are connections to the intended goals, connections to relevant theory, and connections to the long-term goal of sustainability. In presenting these ideas, we will focus on how the information being offered connects to the current body of knowledge in engineering education. Conclusion. It is promising to treat the work of department culture as a design problem. The ideas in this framework may serve as inspiration to others seeking to create their own sustainable mechanisms but with different conditions. During the winter and spring of 2022, the approach will be additionally tested via six deployments, and insights will be shared in subsequent publications.
more »
« less
Work in progress: Designing a sustainable mechanism for discursively navigating change
Background. While educational change often involves bold talk about disruptive ideas that eventually need to be institutionalized, a critical but often less visible element of sustaining change is work such as maintaining a shared vision, onboarding new people, negotiating small issues in light of department culture, and coordinating big changes with existing efforts. While knowledge about these forms of invisible work exist in other disciplines, these issues seem understudied in engineering education. This work approaches this issue of invisible knowledge with a design orientation, and specifically draws on the field of design-based research. Increasingly, design is recognized as a knowledge producing activity, resulting in insights into generative ways of defining problems, frameworks for generating solutions to problems, examples of what it looks like to connect theory to specific problems. Purpose: As a design effort, this work asks: How might a specific department create a sustainable practice to support the invisible work of coordinating and sustaining change? As a scholarly effort, this instance of design can result in a culminating problem definition, a solution framework, and examples of theory use that represent knowledge contributions. Approach: A mechanical engineering department in a small, private educational institution worked for four months to develop a sustainable practice to support invisible work of coordinating and sustaining change. Following an initial commitment of 60 minutes once every three weeks and 3-hour retreat to explore possibilities, the department then iteratively designed and then carried out sample conversations. Each iteration involved specifying the goals of the conversation, how to have the conversation (the design) and the rationale for connecting the design to the goals. Traces from the process represent the data for this work. Results. Over time, the conversations came to be designed along four dimensions: topic, time allocation, turn-taking, and traces. We have learned that topics that are of immediate relevance to everyone are particularly powerful (initial topics included "being back on campus" and "navigating in-person"). We are currently leveraging a time allocation that devotes the most time to hearing from each participant on the topic, then time for the group to cautiously explore synthesis, and finally time for the group to weigh in on future conversation topics. Approaches to turn-taking have involved decentralization (e.g., each current speaker invites the next speaker) and respect (speakers have a chance to "pass" and then choose the next speaker). Finally, we are experimenting with how to balance the creation of traces as a natural part of the process, such as through real-time transcription in the chat feature of zoom. Undergirding each of these dimensions are connections to the intended goals, connections to relevant theory, and connections to the long-term goal of sustainability. In presenting these ideas, we will focus on how the information being offered connects to the current body of knowledge in engineering education. Conclusion. It is promising to treat the work of department culture as a design problem. The ideas in this framework may serve as inspiration to others seeking to create their own sustainable mechanisms but with different conditions. During the winter and spring of 2022, the approach will be additionally tested via six deployments, and insights will be shared in subsequent publications.
more »
« less
- Award ID(s):
- 1730354
- PAR ID:
- 10427039
- Date Published:
- Journal Name:
- Proceedings ASEE annual conference
- ISSN:
- 0190-1052
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
The current paper discusses conversation-based assessments (CBAs) created with prompt engineering for LLMs based on Evidence-Centered Design (ECD). Conversation-based assessments provide students the opportunity to discuss a given topic with artificial agent(s). These conversations elicit evidence of students’ knowledge, skills and abilities that may not be uncovered by traditional tests. We discuss our previous method of creating such conversations with regular expressions and latent semantic analysis in an expensive methodology requiring time and various expertise. Thus, in this novel work, we created a prompt-engineered version of CBAs based on evidence-centered design that remains on the domain topic throughout the conversation as well as provides evidence of the student knowledge in a less expensive way. We present the methodology for creating these prompts, compare responses to various student speech acts between the previous version and the prompt engineered version, and discuss the evidence gleaned from the conversation and based on the prompt. Finally, limitations, conclusions and implications of this work are discussed.more » « less
-
The Language, Culture, and Knowledge-building through Science (LaCuKnoS) project tests and refines a model of science teaching and learning that brings together current research on the role of language in science communication, the role of cultural and community connections in science engagement, and the ways people apply science knowledge to their daily decision making. One key component of the model brings families together as co-learners and co-teachers through family learning experiences. We describe our work to promote more robust family conversations about science in our lives within an existing research practice partnership, using a two-tiered qualitative conversational analysis to compare the family conversations that result from three family engagement models: (a) family science festivals; (b) family science workshops; and (c) family science home learning. More specifically, this paper addresses the question: How do families describe and evaluate science in their lives and communities during family conversations that occur during each of these three engagement models? Discourse analysis using the appraisal dimension of systemic functional linguistics highlights the affective components of families evaluating science in their lives, as well as how each model provided unique affordances for different communicative goals. These findings are used to propose a set of design principles to guide the continued exploration of community-sustaining and family-centric models of family engagement as a key strategy for broadening science participation.more » « less
-
There is a growing body of work to characterize elementary engineering classroom talk and its influence on students’ learning. One form of classroom talk is the whole-class conversation, which can be an important site for growth in students’ ideas and ways of thinking about engineering design problems and solutions. With intentional teacher facilitation, whole-class conversations can help students refine their engineering reasoning, consider new ideas, and make new connections between different ways of defining or solving a problem. Participating in these conversations can also help students expand their engineering thinking to include perspectives of care and socio-ethical deliberations. In a multi-year collaboration of classroom teachers and university researchers, we have been enacting and studying five different genres of whole-class engineering Design Talks in first-grade through sixth-grade classrooms: problem scoping talks, idea generation talks, design-in-progress talks, design synthesis talks, and impact talks. As a teacher-researcher community of practice, we have video recorded these “Design Talks” in teachers’ classrooms. These classroom video clips have helped us explore a range of questions about how to structure Design Talks. This paper reports on a qualitative study focused on teacher perceptions of their experiences with Design Talks in their classrooms. Specifically, we ask: How do elementary teachers perceive the benefits and challenges of intentionally facilitated whole-class conversations during engineering design units? Study participants were the six classroom teachers in our Design Talks community of practice. Data sources include field notes from teacher-researcher meetings over three years and teachers’ written responses to open-ended reflection questions. We applied thematic analysis techniques (Braun & Clarke, 2006), including initial coding followed by thematic mapping. We found four themes that characterize how teachers perceive the benefits and challenges of whole-class engineering design conversations. Teachers find that these conversations help them employ asset-based pedagogies while at the same time helping their students synthesize designs and their underlying concepts, take a perspective of care in engineering design, and learn to listen, empathize, and communicate.more » « less
-
Benjamin, Paaßen; Carrie, Demmans Epp (Ed.)This paper explores the differences between two types of natural language conversations between a student and pedagogical agent(s). Both types of conversations were created for formative assessment purposes. The first type is conversation-based assessment created via knowledge engineering which requires a large amount of human effort. The second type, which is less costly to produce, uses prompt engineering for LLMs based on Evidence-Centered design to create these conversations and glean evidence about students¿½f knowledge, skills and abilities. The current work compares linguistic features of the artificial agent(s) discourse moves in natural language conversations created by the two methodologies. Results indicate that more complex conversations are created by the prompt engineering method which may be more adaptive than the knowledge engineering approach. However, the affordances of prompt engi-neered, LLM generated conversation-based assessment may create more challenges for scoring than the original knowledge engineered conversations. Limitations and implications are dis-cussed.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
