More Like this

1. Design and testing of a quantitative instrument to evaluate engineering research center participation

   Melo de Lyra, Marcus V. ; Carberry, Adam R. ; Larson, Jean S. ; Zhao, Zhen ; Barr, Christopher ( January 2023 , Design and Testing of a Quantitative Instrument to Evaluate Engineering Research Center Participation)

   This research paper reports the in-progress validation of a quantitative instrument designed to assess the perceived impact of participating in a National Science Foundation (NSF)-funded Engineering Research Center (ERC). A multi-institutional consortium composed of ERC education directors, researchers, and evaluators from six NSF-funded ERCs designed easily accessible evaluation instruments and tools that specifically help measure anticipated outcomes for ERC participants for all ERCs. The total effort underway by the consortium includes creating a suite of qualitative and quantitative instruments, an evaluator toolkit, and a user-friendly online platform to host the inventory materials. This paper focuses on the quantitative instrument created to evaluate the experiences of those who engage with a center. It consists of Likert-type questions assessing the impact of the ERC on participants' self-reported: 1) understanding of the ERC, 2) research and communication skills, 3) climate of inclusion, 4) mentorship experiences, and 5) program satisfaction. The instrument also included additional demographic questions and questions to capture STEM-related future plans. The instrument was designed using multiple rounds of design iterations and pilot tests. Separate surveys used by individual ERCs were compiled and categorized to ensure all requirements from the National Science Foundation were met. The web-based survey was administered to six ERCs during the Summer of 2021, Fall of 2021, and Spring of 2022. A total of 549 responses were collected; 535 were used following data cleaning procedures. Sample sizes for each component of the survey varied because some ERCs chose to only use some parts of the new instrument. Exploratory Factor Analyses (EFA) were performed to identify latent factors and items that needed further revision. The following factors emerged from our analyses: 1) ERC general understanding; 2) development of research skills; 3) development of professional skills; 4) experience in the ERC; 5) feelings toward the ERC; 6) Beliefs about the ERC, 7) mentors performance; and 8) mentorship experience. The results provide preliminary evidence that the survey can be used across ERCs. This effort is the first that has been undertaken to develop a shared ERC instrument. The data collected was used to continue in-progress validation. The collaborative nature of this effort can provide ways for ERCs to benchmark impacts of their efforts and share effective practices across ERCs and other similarly structured STEM centers going forward. 

   more » « less
2. Initial Steps in Developing Classroom Observation Rubrics Designed Around Instructional Practices that Support Equity and Access in Classrooms with Potential for “Success”

   https://doi.org/10.1177/01614681221140963  

   Wilson, Jonee ( November 2022 , Teachers College Record: The Voice of Scholarship in Education)

   Background: The field of mathematics education has made progress toward generating a set of instructional practices that could support improvements in the learning opportunities made available to groups of students who historically have been underserved and marginalized. Studies that contribute to this growing body of work are often conducted in learning environments that are framed as “successful.” As a researcher who is concerned with issues of equity and who acknowledges the importance of closely attending to the quality of the mathematical activity in which students are being asked to engage, my stance on “successful learning environments” pulls from both Gutiérrez’s descriptions of what characterizes classrooms as aiming for equity and the emphasis on the importance of conceptually oriented goals for student learning that is outlined in documents like the Standards. Though as a field we are growing in our knowledge of practices that support these successful learning environments, this knowledge has not yet been reflected in many of the observational tools, rubrics, and protocols used to study these environments. In addition, there is a growing need to develop empirically grounded ways of attending to the extent to which the practices that are being outlined in research literature actually contribute to the “success” of these learning environments. Purpose: The purpose of this article is to explore one way of meeting this growing need by describing the complex work of developing a set of classroom observation rubrics (the Equity and Access Rubrics for Mathematics Instruction, EAR-MI) designed to support efforts in identifying and observing critical features of classrooms characterized as having potential for “success.” In developing the rubrics, I took as my starting place findings from an analysis that compared a set of classrooms that were characterized as demonstrating aspects of successful learning environments and a set of classrooms that were not characterized as successful. This paper not only describes the process of developing the rubrics, but also outlines some of the qualitative differences that distinguished more and less effective examples of the practices the rubrics are designed to capture. Research Design: In designing the rubrics, I engaged in multiple cycles of qualitative analyses of video data collected from a large-scale study. Specifically, I iteratively designed, tested, and revised the developing rubrics while consistently collaborating with, consulting with, and receiving feedback from different experts in the field of education. Conclusions: Although I fully acknowledge and recognize that there are several tensions and limitations of this work, I argue that developing rubrics like the EAR-MI is still worthwhile. One reason that I give for continuing these types of efforts is that it contributes to the work of breaking down forms of practice into components and identifying key aspects of specific practices that are critical for supporting student learning in ways that make potentially productive routines of action visible to and learnable by others, which may ultimately contribute to the development of more successful learning environments. I also argue that rubrics like the EAR-MI have the potential to support researchers in developing stronger evidence of the effectiveness of practices that prior research has identified as critical for marginalized students and in more accurately and concretely identifying and describing learning environments as having potential for “success.” 

   more » « less
3. The Engineering Design Process Portfolio Scoring Rubric (EDPPSR): Initial Validity and Reliability

   Klein-Gardner, S. ( June 2022 , American Society for Engineering Education (ASEE) Conference & Exposition)

   Research prior to 2005 found that no single framework existed that could capture the engineering design process fully or well and benchmark each element of the process to a commonly accepted set of referenced artifacts. Compounding the construction of a stepwise, artifact driven framework is that engineering design is typically practiced over time as a complex and iterative process. For both novice and advanced students, learning and applying the design process is often cumulative, with many informal and formal programmatic opportunities to practice essential elements. The Engineering Design Process Portfolio Scoring Rubric (EDPPSR) was designed to apply to any portfolio that is intended to document an individual or team driven process leading to an original attempt to design a product, process, or method to provide the best and most optimal solution to a genuine and meaningful problem. In essence, the portfolio should be a detailed account or “biography” of a project and the thought processes that inform that project. Besides narrative and explanatory text, entries may include (but need not be limited to) drawings, schematics, photographs, notebook and journal entries, transcripts or summaries of conversations and interviews, and audio/video recordings. Such entries are likely to be necessary in order to convey accurately and completely the complex thought processes behind the planning, implementation, and self-evaluation of the project. The rubric is comprised of four main components, each in turn comprised of three elements. Each element has its own holistic rubric. The process by which the EDPPSR was created gives evidence of the relevance and representativeness of the rubric and helps to establish validity. The EDPPSR model as originally rendered has a strong theoretical foundation as it has been developed by reference to the literature on the steps of the design process through focus groups and through expert review by teachers, faculty and researchers in performance based, portfolio rubrics and assessments. Using the unified construct validity framework, the EDDPSR’s validity was further established through expert reviewers (experts in engineering design) providing evidence supporting the content relevance and representativeness of the EDPPSR in representing the basic process of engineering design. This manuscript offers empirical evidence that supports the use of the EDPPSR model to evaluate student design-based projects in a reliable and valid manner. Intra-class correlation coefficients (ICC) were calculated to determine the inter-rater reliability (IRR) of the rubric. Given the small sample size we also examined confidence intervals (95%) to provide a range of values in which the estimate of inter-reliability is likely contained. 

   more » « less
4. Activating social change together: A qualitative synthesis of collaborative change research, evaluation and design literature

   https://doi.org/http:dx.doi.org/10.5130/ijcre.v12i1.6693  

   Busch, Melida D ; Jean-Baptiste, Elizabeth ; Person, Pamela F ; Vaughn, Lisa M. ( December 2019 , Gateways)

   Researchers, evaluators and designers from an array of academic disciplines and industry sectors are turning to participatory approaches as they seek to understand and address complex social problems. We refer to participatory approaches that collaboratively engage/ partner with stakeholders in knowledge creation/problem solving for action/social change outcomes as collaborative change research, evaluation and design (CCRED). We further frame CCRED practitioners by their desire to move beyond knowledge creation for its own sake to implementation of new knowledge as a tool for social change. In March and May of 2018, we conducted a literature search of multiple discipline-specific databases seeking collaborative, change-oriented scholarly publications. The search was limited to include peerreviewed journal articles, with English language abstracts available, published in the last five years. The search resulted in 526 citations, 236 of which met inclusion criteria. Though the search was limited to English abstracts, all major geographic regions (North America, Europe, Latin America/Caribbean, APAC, Africa and the Middle East) were represented within the results, although many articles did not state a specific region. Of those identified, most studies were located in North America, with the Middle East having only one identified study. We followed a qualitative thematic synthesis process to examine the abstracts of peer-reviewed articles to identify practices that transcend individual disciplines, sectors and contexts to achieve collaborative change. We surveyed the terminology used to describe CCRED, setting, content/topic of study, type of collaboration, and related benefits/outcomes in order to discern the words used to designate collaboration, the frameworks, tools and methods employed, and the presence of action, evaluation or outcomes. Forty-three percent of the reviewed articles fell broadly within the social sciences, followed by 26 percent in education and 25 percent in health/medicine. In terms of participants and/ or collaborators in the articles reviewed, the vast majority of the 236 articles (86%) described participants, that is, those who the research was about or from whom data was collected. In contrast to participants, partners/collaborators (n=32; 14%) were individuals or groups who participated in the design or implementation of the collaborative change effort described. In terms of the goal for collaboration and/or for doing the work, the most frequently used terminology related to some aspect of engagement and empowerment. Common descriptors for the work itself were ‘social change’ (n=74; 31%), ‘action’ (n=33; 14%), ‘collaborative or participatory research/practice’ (n=13; 6%), ‘transformation’ (n=13; 6%) and ‘community engagement’ (n=10; 4%). Of the 236 articles that mentioned a specific framework or approach, the three most common were some variation of Participatory Action Research (n=30; 50%), Action Research (n=40; 16.9%) or Community-Based Participatory Research (n=17; 7.2%). Approximately a third of the 236 articles did not mention a specific method or tool in the abstract. The most commonly cited method/tool (n=30; 12.7%) was some variation of an arts-based method followed by interviews (n=18; 7.6%), case study (n=16; 6.7%), or an ethnographic-related method (n=14; 5.9%). While some articles implied action or change, only 14 of the 236 articles (6%) stated a specific action or outcome. Most often, the changes described were: the creation or modification of a model, method, process, framework or protocol (n=9; 4%), quality improvement, policy change and social change (n=8; 3%), or modifications to education/training methods and materials (n=5; 2%). The infrequent use of collaboration as a descriptor of partner engagement, coupled with few reported findings of measurable change, raises questions about the nature of CCRED. It appears that conducting CCRED is as complex an undertaking as the problems that the work is attempting to address. 

   more » « less
5. Developing a Model of Disciplinary Literacy Instruction for K-12 Engineering Education: Comparing the Literacy Practices of Electrical and Mechanical Engineers (Fundamental)

   Green, T. ; Wilson-Lopez, A. ; Minichiello, A. ( July 2019 , ASEE Annual Conference proceedings)

   Despite efforts to diversify the engineering workforce, the field remains dominated by White, male engineers. Research shows that underrepresented groups, including women and minorities, are less likely to identify and engage with scientific texts and literacy practices. Often, children of minority groups and/or working-class families do not receive the same kinds of exposure to science, technology, engineering, and mathematics (STEM) knowledge and practices as those from majority groups. Consequently, these children are less likely to engage in school subjects that provide pathways to engineering careers. Therefore, to mitigate the lack of diversity in engineering, new approaches able to broadly support engineering literacy are needed. One promising approach is disciplinary literacy instruction (DLI). DLI is a method for teaching students how advanced practitioners in a given field generate, interpret, and evaluate discipline-specific texts. DLI helps teachers provide access to to high quality, discipline-specific content to all students, regardless of race, ethnicity, gender, or socio-economic status, Therefore, DLI has potential to reduce literacy-based barriers that discourage underrepresented students from pursuing engineering careers. While models of DLI have been developed and implemented in history, science, and mathematics, little is known about DLI in engineering. The purpose of this research is to identify the authentic texts, practices, and evaluative frameworks employed by professional engineers to inform a model of DLI in engineering. While critiques of this approach may suggest that a DLI model will reflect the literacy practices of majority engineering groups, (i.e., White male engineers), we argue that a DLI model can directly empower diverse K-16 students to become engineers by instructing them in the normed knowledge and practices of engineering. This paper presents a comparative case study conducted to investigate the literacy practices of electrical and mechanical engineers. We scaffolded our research using situated learning theory and rhetorical genre studies and considered the engineering profession as a community of practice. We generated multiple types of data with four participants (i.e., two electrical and two mechanical engineers). Specifically, we generated qualitative data, including written field notes of engineer observations, interview transcripts, think-aloud protocols, and engineer logs of literacy practices. We used constant comparative analysis (CCA) coding techniques to examine how electrical and mechanical engineers read, wrote, and evaluated texts to identify the frameworks that guide their literacy practices. We then conducted within-group and cross-group constant comparative analyses (CCA) to compare and contrast the literacy practices specific to each sub-discipline Findings suggest that there are two types of engineering literacy practices: those that resonate across both mechanical and electrical engineering disciplines and those that are specific to each discipline. For example, both electrical and mechanical engineers used test procedures to review and assess steps taken to evaluate electrical or mechanical system performance. In contrast, engineers from the two sub-disciplines used different forms of representation when depicting components and arrangements of engineering systems. While practices that are common across sub-disciplines will inform a model of DLI in engineering for K-12 settings, discipline-specific practices can be used to develop and/or improve undergraduate engineering curricula. 

   more » « less