skip to main content
Explore Scholarly Publications and Datasets in the NSF-PAR

Title: I Played a Song with the Help of a Magic Banana: Assessing Short-term Making Events
Purpose Designers of learning experiences are concerned with how people learn across a range of timescales from a semester to a single moment in time. And just as designing experiences at different timescales requires unique goals, tools, and processes, measuring what people learn from their interactions is also timescale-specific. The aim of our work is twofold: 1) To understand how learners describe their experiences with short-term, introductory maker experiences and; 2) To test a method for assessing learners’ experiences authentic to short-term learning. Design We collected written responses from participants at a two-day event, STEM Center Learning Days. Through an analysis of 707 unique instances of learner responses to participation in drop-in maker activities, we examined how participants describe their short-term learning experiences. Findings We found that although some activities appear to onlookers to create passive experiences for learners, these seemingly passive moments have a significant impact on learners. In addition, some learners described themselves as working in tandem with tools to make something work and other learners viewed the tools as working autonomously. We found that our assessment method allowed us to gain an understanding of how learners describe their experiences offering important implications for understanding short-term learning events. Originality/ Implications Our findings provide researchers studying short-term learning in more » its natural setting a new method to understand how learners make sense of their individual experience. Further, designers of short-term learning experiences may gain insights into their unique activities and indications of where additional guidance and scaffolds will improve small learning moments. Keywords Makerspaces, STEM, assessment « less
Authors:
Award ID(s):
1639915
Publication Date:
NSF-PAR ID:
10219804
Journal Name:
Information and learning sciences
ISSN:
2398-5348
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ( , Conference on Research in Equity and Sustained Participation in Engineering, Computing, and Technology (RESPECT))
    To make computer science (CS) more equitable, many educational efforts are shifting foci from access and content understanding to include identification, agency, and social change. As part of these efforts, we look at how learners perceive themselves in relation to what they believe CS is and what it means to participate in CS. Informed by three design lenses, unblackboxing, culturally responsive computing, and creative production, we designed a physical computing kit and activities. Drawing from qualitative analysis of interviews, artifacts, and observation of six young people in a weeklong summer workshop, we report on the experiences of two young Black women designers. We found that using these materials young people were able to: leverage personal goals and prior experiences in computing work; feel as if they were figuring out computing systems; and recognize computational technologies as created by people for particular purposes. We observed that while the mix of materials and activities created some frustration for participants, it also prompted processes of community building and inquiry. We discuss implications for design of computational tools in equity-centered CS education and pose seamfulness as an emergent heuristic when designing for learning that engages young people with the social, not just material, systemsmore »of computing.« less
  2. ; ( , National Science Foundation -  Hispanic-Serving Institutions (HSI) Program Principal Investigator (PI) Meeting, November 2019.)
    Need/Motivation (e.g., goals, gaps in knowledge) The ESTEEM implemented a STEM building capacity project through students’ early access to a sustainable and innovative STEM Stepping Stones, called Micro-Internships (MI). The goal is to reap key benefits of a full-length internship and undergraduate research experiences in an abbreviated format, including access, success, degree completion, transfer, and recruiting and retaining more Latinx and underrepresented students into the STEM workforce. The MIs are designed with the goals to provide opportunities for students at a community college and HSI, with authentic STEM research and applied learning experiences (ALE), support for appropriate STEM pathway/career, preparation and confidence to succeed in STEM and engage in summer long REUs, and with improved outcomes. The MI projects are accessible early to more students and build momentum to better overcome critical obstacles to success. The MIs are shorter, flexibly scheduled throughout the year, easily accessible, and participation in multiple MI is encouraged. ESTEEM also establishes a sustainable and collaborative model, working with partners from BSCS Science Education, for MI’s mentor, training, compliance, and building capacity, with shared values and practices to maximize the improvement of student outcomes. New Knowledge (e.g., hypothesis, research questions) Research indicates that REU/internship experiences canmore »be particularly powerful for students from Latinx and underrepresented groups in STEM. However, those experiences are difficult to access for many HSI-community college students (85% of our students hold off-campus jobs), and lack of confidence is a barrier for a majority of our students. The gap between those who can and those who cannot is the “internship access gap.” This project is at a central California Community College (CCC) and HSI, the only affordable post-secondary option in a region serving a historically underrepresented population in STEM, including 75% Hispanic, and 87% have not completed college. MI is designed to reduce inequalities inherent in the internship paradigm by providing access to professional and research skills for those underserved students. The MI has been designed to reduce barriers by offering: shorter duration (25 contact hours); flexible timing (one week to once a week over many weeks); open access/large group; and proximal location (on-campus). MI mentors participate in week-long summer workshops and ongoing monthly community of practice with the goal of co-constructing a shared vision, engaging in conversations about pedagogy and learning, and sustaining the MI program going forward. Approach (e.g., objectives/specific aims, research methodologies, and analysis) Research Question and Methodology: We want to know: How does participation in a micro-internship affect students’ interest and confidence to pursue STEM? We used a mixed-methods design triangulating quantitative Likert-style survey data with interpretive coding of open-responses to reveal themes in students’ motivations, attitudes toward STEM, and confidence. Participants: The study sampled students enrolled either part-time or full-time at the community college. Although each MI was classified within STEM, they were open to any interested student in any major. Demographically, participants self-identified as 70% Hispanic/Latinx, 13% Mixed-Race, and 42 female. Instrument: Student surveys were developed from two previously validated instruments that examine the impact of the MI intervention on student interest in STEM careers and pursuing internships/REUs. Also, the pre- and post (every e months to assess longitudinal outcomes) -surveys included relevant open response prompts. The surveys collected students’ demographics; interest, confidence, and motivation in pursuing a career in STEM; perceived obstacles; and past experiences with internships and MIs. 171 students responded to the pre-survey at the time of submission. Outcomes (e.g., preliminary findings, accomplishments to date) Because we just finished year 1, we lack at this time longitudinal data to reveal if student confidence is maintained over time and whether or not students are more likely to (i) enroll in more internships, (ii) transfer to a four-year university, or (iii) shorten the time it takes for degree attainment. For short term outcomes, students significantly Increased their confidence to continue pursuing opportunities to develop within the STEM pipeline, including full-length internships, completing STEM degrees, and applying for jobs in STEM. For example, using a 2-tailed t-test we compared means before and after the MI experience. 15 out of 16 questions that showed improvement in scores were related to student confidence to pursue STEM or perceived enjoyment of a STEM career. Finding from the free-response questions, showed that the majority of students reported enrolling in the MI to gain knowledge and experience. After the MI, 66% of students reported having gained valuable knowledge and experience, and 35% of students spoke about gaining confidence and/or momentum to pursue STEM as a career. Broader Impacts (e.g., the participation of underrepresented minorities in STEM; development of a diverse STEM workforce, enhanced infrastructure for research and education) The ESTEEM project has the potential for a transformational impact on STEM undergraduate education’s access and success for underrepresented and Latinx community college students, as well as for STEM capacity building at Hartnell College, a CCC and HSI, for students, faculty, professionals, and processes that foster research in STEM and education. Through sharing and transfer abilities of the ESTEEM model to similar institutions, the project has the potential to change the way students are served at an early and critical stage of their higher education experience at CCC, where one in every five community college student in the nation attends a CCC, over 67% of CCC students identify themselves with ethnic backgrounds that are not White, and 40 to 50% of University of California and California State University graduates in STEM started at a CCC, thus making it a key leverage point for recruiting and retaining a more diverse STEM workforce.« less
  3. ; ; ( , International Conference on Program Comprehension (ICPC))
    Developers and computing students are usually expected to master multiple programming languages. To learn a new language, developers often turn to online search to find information and code examples. However, insights on how learners perform code search when working with an unfamiliar language are lacking. Understanding how learners search and the challenges they encounter when using an unfamiliar language can motivate future tools and techniques to better support subsequent language learners. Research on code search behavior typically involves monitoring developers during search activities through logs or in situ surveys. We conducted a study on how computing students search for code in an unfamiliar programming language with 18 graduate students working on VBA tasks in a lab environment. Our surveys explicitly asked about search success and query reformulation to gather reliable data on those metrics. By analyzing the combination of search logs and survey responses, we found that students typically search to explore APIs or find example code. Approximately 50% of queries that precede clicks on documentation or tutorials successfully solved the problem. Students frequently borrowed terms from languages with which they are familiar when searching for examples in an unfamiliar language, but term borrowing did not impede search success. Editmore »distances between reformulated queries and non-reformulated queries were nearly the same. These results have implications for code search research, especially on reformulation, and for research on supporting programmers when learning a new language.« less
  4. ; ; ( , 2019 ASEE Annual Conference & Exposition)
    When asked about how they deal with unforeseen problems, novice learners often describe a process of “trial and error.” This process might fairly be described as iteration, a critical step in the design process, but falls short of the practices that engineering education needs to develop. In the face of novel and multifaceted problems, future engineers must be comfortable and competent not just trying again, but identifying failure points, troubleshooting, and running systematic tests with relevant data. To examine the abilities of novice designers to test and effectively refine ideas and prototypes, we conducted qualitative analysis of structured interviews, audio, video, and designs of 11 girls, ages 9 -11, working on computational papercrafts as part of a museum-based STEAM summer camp. The projects involved design and construction of expressive paper and cardboard sculptures with gears and linkages powered by servomotors. Over the course of one day, the girls generated designs inspired by a camp theme, then had to work with mechanics, electronics and craft to create working versions that would be displayed as part of a public exhibit. Computational papercraft was selected because it lowers cost and intimidation. Our design conjecture was that by making materials familiar and abundant, learnersmore »would have more relevant knowledge, could easily modify and replicate components, and would therefore be better able to recognize potential faults and more likely to engage in testing and refinement. We also supported design and troubleshooting with a customized circuit board and an online gear simulator. In the first stage of this study, we looked at what engineering practices emerged, given these conditions. We asked: What opportunities for testing and refinement did computational papercrafts open up? What resources and tools do young learners employ when testing and refining designs? Analysis showed that technical supports for testing and refinement were successful in supporting valued testing and refinement practices as youth pursued personal goals. Use of the simulator and customized microcontroller allowed for consideration of multiple alternatives and for “trial before error.” Learners were able to conduct focused tests on subsystems of their paper machines, and to make “small bets,” keeping initial ideas and designs fluid. Inexpensive materials also allowed them to test and refine at late project stages, without feeling that they were wasting time or materials. The analysis sheds light on young students practices of testing and refinement, and how to best support young people as they begin learning trajectories in engineering. The approach is especially relevant within making-oriented engineering education and other settings working to broaden participation in engineering.« less
  5. ( , HCI International)
    Today’s classrooms are remarkably different from those of yesteryear. In place of individual students responding to the teacher from neat rows of desks, one more typically finds students working in groups on projects, with a teacher circulating among groups. AI applications in learning have been slow to catch up, with most available technologies focusing on personalizing or adapting instruction to learners as isolated individuals. Meanwhile, an established science of Computer Supported Collaborative Learning has come to prominence, with clear implications for how collaborative learning could best be supported. In this contribution, I will consider how intelligence augmentation could evolve to support collaborative learning as well as three signature challenges of this work that could drive AI forward. In conceptualizing collaborative learning, Kirschner and Erkens (2013) provide a useful 3x3 framework in which there are three aspects of learning (cognitive, social and motivational), three levels (community, group/team, and individual) and three kinds of pedagogical supports (discourse-oriented, representation-oriented, and process-oriented). As they engage in this multiply complex space, teachers and learners are both learning to collaborate and collaborating to learn. Further, questions of equity arise as we consider who is able to participate and in which ways. Overall, this analysis helps usmore »see the complexity of today’s classrooms and within this complexity, the opportunities for augmentation or “assistance to become important and even essential. An overarching design concept has emerged in the past 5 years in response to this complexity, the idea of intelligent augmentation for “orchestrating” classrooms (Dillenbourg, et al, 2013). As a metaphor, orchestration can suggest the need for a coordinated performance among many agents who are each playing different roles or voicing different ideas. Practically speaking, orchestration suggests that “intelligence augmentation” could help many smaller things go well, and in doing so, could enable the overall intention of the learning experience to succeed. Those smaller things could include helping the teacher stay aware of students or groups who need attention, supporting formation of groups or transitions from one activity to the next, facilitating productive social interactions in groups, suggesting learning resources that would support teamwork, and more. A recent panel of AI experts identified orchestration as an overarching concept that is an important focus for near-term research and development for intelligence augmentation (Roschelle, Lester & Fusco, 2020). Tackling this challenging area of collaborative learning could also be beneficial for advancing AI technologies overall. Building AI agents that better understand the social context of human activities has broad importance, as does designing AI agents that can appropriately interact within teamwork. Collaborative learning has trajectory over time, and designing AI systems that support teams not just with a short term recommendation or suggestion but in long-term developmental processes is important. Further, classrooms that are engaged in collaborative learning could become very interesting hybrid environments, with multiple human and AI agents present at once and addressing dual outcome goals of learning to collaborate and collaborating to learn; addressing a hybrid environment like this could lead to developing AI systems that more robustly help many types of realistic human activity. In conclusion, the opportunity to make a societal impact by attending to collaborative learning, the availability of growing science of computer-supported collaborative learning and the need to push new boundaries in AI together suggest collaborative learning as a challenge worth tackling in coming years.« less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email