Research based assessments have a productive and storied history in PER. While useful for conducting research on student learning, their utility is limited for instructors interested in improving their own courses. We have developed a new assessment design process that leverages three-dimensional learning, evidence-centered design, and self-regulated learning to deliver actionable feedback to instructors about supporting their students' learning. We are using this approach to design the Thermal and Statistical Physics Assessment (TaSPA), which also allows instructors to choose learning goals that align with their teaching. Perhaps more importantly, this system will be completely automated when it is completed, making the assessment scalable with minimal burden on instructors and researchers. This work represents an advancement in how we assess physics learning at a large scale and how the PER community can better support physics instructors and students.
more »
« less
Patterns in Elementary-Age Student Responses to Personalized & Generic Code Comprehension Questions
The CS community has struggled to assess student learning at the K-8 level, with techniques ranging from one-on-one interviews to written assessments. While scalable, automated techniques exist for analyzing student code, a scalable method for assessing student comprehension of their own code has remained elusive. This study is a first step in bridging the gap between the knowledge gained from interviews and the time efficiency and scalability of written assessments and automated analysis. The goal of this study is to understand how student answers on various types of questions differ depending on whether they are being asked about their own code or generic code. We find that while there were no statistically-significant differences in overall scores, questions about generic and personalized code of comparable complexity are far from equivalent. Our qualitative analyses revealed interesting patterns in student responses, inviting further research into this assessment technique. In particular, students answered differently from students with generic code when presented with individual blocks from their code taken out of context and placed into different code snippets, and students answered in a way that demonstrates a functional, instead of structural, understanding on Explain in Plain English (EiPE) questions.
more »
« less
- Award ID(s):
- 1660871
- PAR ID:
- 10158898
- Date Published:
- Journal Name:
- SIGCSE '20: Proceedings of the 51st ACM Technical Symposium on Computer Science Education
- Page Range / eLocation ID:
- 514 to 520
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
more » « less
Abstract We investigated the difficulties that physics students in upper-level undergraduate quantum mechanics and graduate students after quantum and statistical mechanics core courses have with the Fermi energy, the Fermi–Dirac distribution and total electronic energy of a free electron gas after they had learned relevant concepts in their respective courses. These difficulties were probed by administering written conceptual and quantitative questions to undergraduate students and asking some undergraduate and graduate students to answer those questions while thinking aloud in one-on-one individual interviews. We find that advanced students had many common difficulties with these concepts after traditional lecture-based instruction. Engaging with a sequence of clicker questions improved student performance, but there remains room for improvement in their understanding of these challenging concepts.
-
Background and context. “Explain in Plain English” (EiPE) questions ask students to explain the high-level purpose of code, requiring them to understand the macrostructure of the program’s intent. A lot is known about techniques that experts use to comprehend code, but less is known about how we should teach novices to develop this capability. Objective. Identify techniques that can be taught to students to assist them in developing their ability to comprehend code and contribute to the body of knowledge of how novices develop their code comprehension skills. Method. We developed interventions that could be taught to novices motivated by previous research about how experts comprehend code: prompting students to identify beacons, identify the role of variables, tracing, and abstract tracing. We conducted think-aloud interviews of introductory programming students solving EiPE questions, varying which interventions each student was taught. Some participants were interviewed multiple times throughout the semester to observe any changes in behavior over time. Findings. Identifying beacons and the name of variable roles were rarely helpful, as they did not encourage students to integrate their understanding of that piece in relation to other lines of code. However, prompting students to explain each variable’s purpose helped them focus on useful subsets of the code, which helped manage cognitive load. Tracing was helpful when students incorrectly recognized common programming patterns or made mistakes comprehending syntax (text-surface). Prompting students to pick inputs that potentially contradicted their current understanding of the code was found to be a simple approach to them effectively selecting inputs to trace. Abstract tracing helped students see high-level, functional relationships between variables. In addition, we observed student spontaneously sketching algorithmic visualizations that similarly helped them see relationships between variables. Implications. Because students can get stuck at many points in the process of code comprehension, there seems to be no silver bullet technique that helps in every circumstance. Instead, effective instruction for code comprehension will likely involve teaching a collection of techniques. In addition to these techniques, meta-knowledge about when to apply each technique will need to be learned, but that is left for future research. At present, we recommend teaching a bottom-up, concrete-to-abstract approach.more » « less
-
As computing skills become necessary for 21st-century students, infused computational thinking (CT) lessons must be created for core courses to truly provide computing education for all. This will bring challenges as students will have widely varying experience and programming ability. Additionally, STEM teachers might have little experience teaching CT and instructing using unfamiliar technology might create discomfort. We present a design pattern for infused CT assignments that scaffold students and teachers into block-based programming environments. Beginning with existing code, students and teachers work together 'Using' and comprehending code before 'Modifying' it together to fix their programs. The activity ends with students 'Choosing' their own extensions from a pre-set list. We present a comparison of two implementations of a simulation activity, one ending with student choosing how to extend their models and one having all students create the same option. Through triangulating data from classroom observations, student feedback, teacher interviews, and programming interaction logs, we present support for student and teacher preference of the 'Student-Choice' model. We end with recommended strategies for developing curricula that follow our design model.more » « less
-
We evaluated the impact of the current COVID-19 pandemic and systemic racism on Underrepresented Minority (URM) students pursuing higher education in the STEM fields. Given the ongoing pandemic and the wave of protests in response to a series of police brutalities and systemic racism, URM students were thrown into uncharted territory. We reached out to a group of Black and Latino students who were already engaged in STEM. We began surveys and interviews by asking participants how they were and how their family and communities were doing. Next, participants answered questions about academic progress, challenges, and what support would be helpful. Our framework was based on a mixed-methods approach that draws on the work of Michael Patton (Qualitative Research & Evaluation Methods: Integrating Theory and Practice, 2014) and Veronica Thomas (American Journal of Evaluation, 2016, 38 (1), 7–28). Qualitative data from interviews were collected to capture perceptions, experiences, and recommendations of the study participants. Survey data were collected to reach as many students as possible and to provide numerical self-assessments of student experience, progression, and obstacles. All qualitative data were coded thematically using Atlas. ti, with the goal of illuminating emerging themes, and quantitative data were reviewed using descriptive statistics. Themes emerging from both data sets were compared, contrasted, and integrated in order to develop consistent findings that would enhance URM student perseverance and persistence in the face of confounding adversities. This study shows that ILSAMP COVID-19 Study participants maintained a commitment to pursuing a career in STEM. The findings of this study also indicate that the participants are stressed by their immediate circumstances and by the ongoing racism of U.S. society. These students ask for additional financial, academic, and networking support during the disruptions caused by the pandemic. More specifically, students request continued advising and connection with STEM professionals who can help them envision and enact a pathway to their own careers in STEM during this tumultuous period. The study validates the importance of key elements of the national LSAMP model as reported by Clewell et al. (Revitalizing the Nation’s Talent Pool in STEM, 2006). These are: academic integration, social integration, and professional integration. In addition, it identifies several other factors that are key to student success, including interventions that directly address racial trauma and economic hardship.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1145/3328778.3366833
