An official website of the United States government
This study proposes and demonstrates how computer‐aided methods can be used to extend qualitative data analysis by quantifying qualitative data, and then through exploration, categorization, grouping, and validation. Computer‐aided approaches to inquiry have gained important ground in educational research, mostly through data analytics and large data set processing. We argue that qualitative data analysis methods can also be supported and extended by computer‐aided methods. In particular, we posit that computing capacities rationally applied can expand the innate human ability to recognize patterns and group qualitative information based on similarities. We propose a principled approach to using machine learning in qualitative education research based on the three interrelated elements of the assessment triangle: cognition, observation, and interpretation. Through the lens of the assessment triangle, the study presents three examples of qualitative studies in engineering education that have used computer‐aided methods for visualization and grouping. The first study focuses on characterizing students' written explanations of programming code, using tile plots and hierarchical clustering with binary distances to identify the different approaches that students used to self‐explain. The second study looks into students' modeling and simulation process and elicits the types of knowledge that they used in each step through a think‐aloud protocol. For this purpose, we used a bubble plot and a k‐means clustering algorithm. The third and final study explores engineering faculty's conceptions of teaching, using data from semi‐structured interviews. We grouped these conceptions based on coding similarities, using Jaccard's similarity coefficient, and visualized them using a treemap. We conclude this manuscript by discussing some implications for engineering education qualitative research.
more »
« less
Work-in-Progress Paper: Creating and Validating the Conceptual Assessment for Sedimentology courses
Contribution: In this work-in-progress paper we describe the process of creating and validating a conceptual assessment in the field of sedimentology for undergraduate geoscience courses. The mechanism can aid future geoscience educators and researchers in the process of academic assessment development aligned with learning objectives in these courses. Background: Prior literature review supports the benefits of using active learning tools in STEM (Science, Technology, Engineering, and Mathematics) courses. This paper is part of a larger project to develop and incorporate research-based active learning software in sedimentology and other geoscience courses to improve grade point average (GPA) and time to graduation for Hispanic students at Texas A&M University. To evaluate the novel tool, we designed and validated the conceptual assessment instrument presented in this work. Research Question: What is the process to develop and validate a conceptual assessment for sedimentology? Methodology: This paper follows quantitative analysis and the assessment triangle approach and focuses on cognition, observation, and interpretation to design and evaluate the conceptual assessment. In the cognition element of the triangle, we explain the mechanism for creating the assessment instrument using students' learning objectives. The observation element explains the mechanism of data collection and the instrument revision. The interpretation element explains the results of the validation process using item response theory and reliability measures. We collected the conceptual assessment data from 17 participants enrolled in two courses where sedimentology topics are taught. Participants were geology majors in one of the courses and engineering majors in the other. Findings: The team developed a conceptual assessment that included eight multiple-choice (MCQ) and four open-ended response questions. The results of the design process described the conceptualization of questions and their validation. Also, the validity of created rubrics was established using inter-rater reliability measures, which showed good agreement between raters. Additionally, the results of the validation process indicated that the conceptual assessment was designed for students with average abilities.
more »
« less
- Award ID(s):
- 2448851
- PAR ID:
- 10614737
- Publisher / Repository:
- IEEE
- Date Published:
- ISSN:
- 2377-634X
- ISBN:
- 979-8-3503-5150-7
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Chinn, C.; Tan, E.; Chan, C.; Kali, Y. (Ed.)Collaboration is an important learning process. During collaborative learning, students engage in group activities where they converge on goals, solve problems and make joint decisions. To understand the process of collaboration, we focused on how behavior and interaction patterns contribute to the social-relational space of collaboration. We have designed a multilayered conceptual model for the collaboration process and an observation rubric that identifies behaviors and interactions during collaboration that serves as the foundation for machine learning models that can provide behavioral insight into the process of collaboration. This study reports results on several validation studies performed to establish a validation argument for our collaboration conceptual model and collaboration rubric. Through disconfirming evidence, interrater reliability testing, expert reviews, and focus group interviews, we found that our stratified architecture of collaboration and rubric provide valid accounts and descriptions of human behavior and interactions that can be used to substantiate the collaboration process.more » « less
-
The association between student motivation and learning, and changes in motivation across a course, were evaluated for students enrolled in one-semester foundation-level inorganic chemistry courses at multiple postsecondary institutions across the United States. The Academic Motivation Scale for Chemistry (AMS-Chemistry) and the Foundations of Inorganic Chemistry American Chemical Society Exam (i.e., a content knowledge measure) were used in this study. Evidence of validity, reliability, and longitudinal measurement invariance for data obtained from the AMS-Chemistry instrument with this population were found using methodologies appropriate for ordinal, non-parametric data. Positive and significant associations between intrinsic motivation measures and academic performance corroborate theoretical and empirical investigations; however, a lack of pre/post changes in motivation suggest that motivation may be less malleable in courses primarily populated by chemistry majors. Implications for inorganic chemistry instructors include paths for incorporating engaging pedagogies known to promote intrinsic motivation and methods for incorporating affect measures into assessment practices. Implications for researchers include a need for more work that disaggregates chemistry majors when evaluating relationships between affect and learning, and when making pre/post comparisons. Additionally, this work provides an example of how to implement more appropriate methods for treating data in studies using Likert-type responses and nested data.more » « less
-
The association between student motivation and learning, and changes in motivation across a course, were evaluated for students enrolled in one-semester foundation-level inorganic chemistry courses at multiple postsecondary institutions across the United States. The Academic Motivation Scale for Chemistry (AMS-Chemistry) and the Foundations of Inorganic Chemistry American Chemical Society Exam (i.e., a content knowledge measure) were used in this study. Evidence of validity, reliability, and longitudinal measurement invariance for data obtained from the AMS-Chemistry instrument with this population were found using methodologies appropriate for ordinal, non-parametric data. Positive and significant associations between intrinsic motivation measures and academic performance corroborate theoretical and empirical investigations; however, a lack of pre/post changes in motivation suggest that motivation may be less malleable in courses primarily populated by chemistry majors. Implications for inorganic chemistry instructors include paths for incorporating engaging pedagogies known to promote intrinsic motivation and methods for incorporating affect measures into assessment practices. Implications for researchers include a need for more work that disaggregates chemistry majors when evaluating relationships between affect and learning, and when making pre/post comparisons. Additionally, this work provides an example of how to implement more appropriate methods for treating data in studies using Likert-type responses and nested data.more » « less
-
Abstract This paper presents the development and validation of the 17-item mathematics Graduate Student Instructor Observation Protocol (GSIOP) at two universities. The development of this instrument attended to some unique needs of novice undergraduate mathematics instructors while building on an existing instrument that focused on classroom interactions particularly relevant for students’ development of conceptual understanding, called the Mathematical Classroom Observation Protocol for Practices (MCOP2). Instrument validation involved content input from mathematics education researchers and upper-level mathematics graduate student instructors at two universities, internal consistency analysis, interrater reliability analysis, and structure analyses via scree plot analysis and exploratory factor analysis. A Cronbach-Alpha level of 0.868 illustrated a viable level for internal consistency. Crosstabulation and correlations illustrate high level of interrater reliability for all but one item, and high levels across all subsections. Collaborating a scree plot with the exploratory factor analysis illustrated three critical groupings aligning with the factors from the MCOP2(student engagement and teacher facilitation) while adding a third factor, lesson design practices. Taken collectively, these results indicate that the GSIOP measures the degree to which instructors’ and students’ actions in undergraduate mathematics classrooms align with practices recommended by the Mathematical Association of America (MAA) using a three-factor structure of teacher facilitation, student engagement, and design practices.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
