An official website of the United States government
Involving students in scientific modeling practice is one of the most effective approaches to achieving the next generation science education learning goals. Given the complexity and multirepresentational features of scientific models, scoring student-developed models is time- and cost-intensive, remaining one of the most challenging assessment practices for science education. More importantly, teachers who rely on timely feedback to plan and adjust instruction are reluctant to use modeling tasks because they could not provide timely feedback to learners. This study utilized machine learn- ing (ML), the most advanced artificial intelligence (AI), to develop an approach to automatically score student- drawn models and their written descriptions of those models. We developed six modeling assessment tasks for middle school students that integrate disciplinary core ideas and crosscutting concepts with the modeling practice. For each task, we asked students to draw a model and write a description of that model, which gave students with diverse backgrounds an opportunity to represent their understanding in multiple ways. We then collected student responses to the six tasks and had human experts score a subset of those responses. We used the human-scored student responses to develop ML algorithmic models (AMs) and to train the computer. Validation using new data suggests that the machine-assigned scores achieved robust agreements with human consent scores. Qualitative analysis of student-drawn models further revealed five characteristics that might impact machine scoring accuracy: Alternative expression, confusing label, inconsistent size, inconsistent position, and redundant information. We argue that these five characteristics should be considered when developing machine-scorable modeling tasks.
more »
« less
Employing automatic analysis tools aligned to learning progressions to assess knowledge application and support learning in STEM
Abstract We discuss transforming STEM education using three aspects: learning progressions (LPs), constructed response performance assessments, and artificial intelligence (AI). Using LPs to inform instruction, curriculum, and assessment design helps foster students’ ability to apply content and practices to explain phenomena, which reflects deeper science understanding. To measure the progress along these LPs, performance assessments combining elements of disciplinary ideas, crosscutting concepts and practices are needed. However, these tasks are time-consuming and expensive to score and provide feedback for. Artificial intelligence (AI) allows to validate the LPs and evaluate performance assessments for many students quickly and efficiently. The evaluation provides a report describing student progress along LP and the supports needed to attain a higher LP level. We suggest using unsupervised, semi-supervised ML and generative AI (GAI) at early LP validation stages to identify relevant proficiency patterns and start building an LP. We further suggest employing supervised ML and GAI for developing targeted LP-aligned performance assessment for more accurate performance diagnosis at advanced LP validation stages. Finally, we discuss employing AI for designing automatic feedback systems for providing personalized feedback to students and helping teachers implement LP-based learning. We discuss the challenges of realizing these tasks and propose future research avenues.
more »
« less
- Award ID(s):
- 2200757
- PAR ID:
- 10554278
- Publisher / Repository:
- Springer Science + Business Media
- Date Published:
- Journal Name:
- International Journal of STEM Education
- Volume:
- 11
- Issue:
- 1
- ISSN:
- 2196-7822
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
This work-in-progress paper explores university students’ perspectives on Generative Artificial Intelligence (GAI) tools, such as ChatGPT, an increasingly prominent topic in the academic community. There is ongoing debate about whether faculty should teach students how to use GAI tools, restrict their usage to maintain academic integrity, or establish regulatory guidelines for sustained integration into higher education. Unfortunately, limited research exists beyond surface-level policies and educator opinions regarding GAI, and its full impact on student learning remains largely unknown. Therefore, understanding students' perceptions and how they use GAI is crucial to ensuring its effective and ethical integration into higher education. As GAI continues to disrupt traditional educational paradigms, this study seeks to explore how students perceive its influence on their learning and problem-solving. As part of a larger mixed-methods study, this work-in-progress paper presents preliminary findings from the qualitative portion using a phenomenological approach that answers the research question: How do university students perceive disruptive technologies like ChatGPT affecting their education and learning? By exploring the implications of Artificial Intelligence (AI) tools on student learning, academic integrity, individual beliefs, and community norms, this study contributes to the broader discourse on the role of emerging technologies in shaping the future of teaching and learning in education.more » « less
-
Generative Artificial Intelligence (GAI) has emerged in recent years as an innovative tool with promising potential for enhancing student learning across a broad spectrum of academic disciplines. GAI not only offers students personalized and adaptive learning experiences, but it is also playing an increasingly important role in various industries. As technologies evolve and society adapts to the growing AI revolution, it becomes necessary to train students of all disciplines to become proficient in using GAI. This work builds on studies that have established the effectiveness of intelligent tutoring systems, adaptive learning environments, and the use of virtual reality in education. This work-in-progress paper presents preliminary findings related to the relationship between university students’ area of study and the frequency at which they utilize GAI to aid their learning. Data for this study were collected using a survey distributed to students from eight different colleges at a large Western university as part of a larger ongoing project geared towards gaining insight into student perceptions and use of GAI in higher education. The goal of the overall project is to establish a foundational understanding of how disruptive technologies, like GAI, can promote learner agency. By exploring why and how students choose to engage with these technologies, the project seeks to find proactive approaches to integrate GAI technology into education, ultimately enhancing teaching and learning practices across various disciplines.more » « less
-
Abstract Argumentation, a key scientific practice presented in theFramework for K-12 Science Education, requires students to construct and critique arguments, but timely evaluation of arguments in large-scale classrooms is challenging. Recent work has shown the potential of automated scoring systems for open response assessments, leveraging machine learning (ML) and artificial intelligence (AI) to aid the scoring of written arguments in complex assessments. Moreover, research has amplified that the features (i.e., complexity, diversity, and structure) of assessment construct are critical to ML scoring accuracy, yet how the assessment construct may be associated with machine scoring accuracy remains unknown. This study investigated how the features associated with the assessment construct of a scientific argumentation assessment item affected machine scoring performance. Specifically, we conceptualized the construct in three dimensions: complexity, diversity, and structure. We employed human experts to code characteristics of the assessment tasks and score middle school student responses to 17 argumentation tasks aligned to three levels of a validated learning progression of scientific argumentation. We randomly selected 361 responses to use as training sets to build machine-learning scoring models for each item. The scoring models yielded a range of agreements with human consensus scores, measured by Cohen’s kappa (mean = 0.60; range 0.38 − 0.89), indicating good to almost perfect performance. We found that higher levels ofComplexityandDiversity of the assessment task were associated with decreased model performance, similarly the relationship between levels ofStructureand model performance showed a somewhat negative linear trend. These findings highlight the importance of considering these construct characteristics when developing ML models for scoring assessments, particularly for higher complexity items and multidimensional assessments.more » « less
-
Pandey, Sumali (Ed.)This original research article focuses on the investigation of the use of generative artificial intelligence (GAI) use among students in communication-intensive STEM courses and how this engagement shapes their scientific communication practices, competencies, confidence, and science identity. Using a mixed-methods approach, patterns were identified in how students perceived their current science identity and use of incorporating artificial intelligence (AI) into writing, oral, and technical tasks. Thematic analysis reveals that students use AI for a range of STEM communication endeavors such as structuring lab reports, brainstorming presentation ideas, and verifying code. While many minoritized students explicitly describe AI as a confidence-boosting, timesaving, and competence-enhancing tool, others—particularly those from privileged backgrounds—downplay its influence, despite evidence of its significant role in their science identity. These results suggest the reframing of science identity as being shaped by technological usage and social contingency. This research illuminates both the potential and pitfalls of AI-use in shaping the next generation of scientists.more » « less
