skip to main content

Explore Research Products in the NSF-PAR

Title: A Multivariate Probit Model for Learning Trajectories: A Fine-Grained Evaluation of an Educational Intervention
Advances in educational technology provide teachers and schools with a wealth of information about student performance. A critical direction for educational research is to harvest the available longitudinal data to provide teachers with real-time diagnoses about students’ skill mastery. Cognitive diagnosis models (CDMs) offer educational researchers, policy makers, and practitioners a psychometric framework for designing instructionally relevant assessments and diagnoses about students’ skill profiles. In this article, the authors contribute to the literature on the development of longitudinal CDMs, by proposing a multivariate latent growth curve model to describe student learning trajectories over time. The model offers several advantages. First, the learning trajectory space is high-dimensional and previously developed models may not be applicable to educational studies that have a modest sample size. In contrast, the method offers a lower dimensional approximation and is more applicable for typical educational studies. Second, practitioners and researchers are interested in identifying factors that cause or relate to student skill acquisition. The framework can easily incorporate covariates to assess theoretical questions about factors that promote learning. The authors demonstrate the utility of their approach with an application to a pre- or post-test educational intervention study and show how the longitudinal CDM framework can provide fine-grained assessment of experimental effects.  more » « less
Award ID(s):
1758688
NSF-PAR ID:
10302520
Author(s) / Creator(s):
 ;  
Date Published:
Journal Name:
Applied Psychological Measurement
Volume:
44
Issue:
7-8
ISSN:
0146-6216
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ( , British Journal of Educational Technology)
    Abstract  
    more » « less
  2. ; ( , American Society for Engineering Education Annual Conference and Exhibition)
    Cyclical models are often used to describe how students learn and develop. These models usually focus on the cognitive domain and describe how knowledge and skills are learned within a course or classroom. By providing insights into how students learn and thus how an instructor can support learning, these models and the schemas drawn from them also influence beliefs about learning and thus how educational programs are designed and developed. In this paper the authors present an alternative cyclical model of learning that is drawn from a philosophy of enactivism rather than rational dualism. In comparison with the dualism inherent in viewpoints derived from Descartes where learners construct internal mental representation from inputs received from the external world, in enactivism development occurs through continual dynamic interactions between an agent and their environment. Enactivism thus emphasizes the role environments play in learning and development. The model developed in this paper hypothesizes that the environment in which learning typically occurs can be represented by three elements: the learner’s identity and culture which informs personally significant goals and values; the affordances a degree program offers in areas of knowledge, identity, and context which informs the capabilities of the environment; and the implicit and explicit goals of education as they are negotiated and understood by learners and teachers. These three elements are strongly coupled and together define the ever-changing learning environment. The paper explores how changing technologies and cultures affect each of these three elements in regards to students’ ability to become technologically literate. While rational or dualist views of education see such environmental changes as peripheral to developing accurate representations of truth, enactivism posits that environment significantly affects the process of education. Because each student or faculty member is a participant in a learning organization changes within the organization—whether externally or internally driven—change the learning process. If education is deemed successful when students can transfer learning to new contexts, dualist models assume transfer is weakly coupled to educational environments while the enactivist viewpoint posits that environments strongly affect transfer. The enactivist model can inform efforts to encourage technological literacy. Like many areas in STEM, education technological literacy has sought to identify and support learning outcomes that specify effective teaching or content interventions which enable learners to become more technologically literate. From the enactivist perspective, however, technological literacy is achieved by placing individuals into an environment in which they must navigate technology-induced challenges, with success defined as learning processes that allow learners to manage tensions inherent in their environment. Because most students already live in such environments teaching definable or enumerable outcomes makes less sense than helping student to be metacognitive and reflective how they manage and relate with technology. 
    more » « less
  3. ; ( , 2022 CoNECD (Collaborative Network for Engineering & Computing Diversity))
    There are significant disparities between the conferring of science, technology, engineering, and mathematics (STEM) bachelor’s degrees to minoritized groups and the number of STEM faculty that represent minoritized groups at four-year predominantly White institutions (PWIs). Studies show that as of 2019, African American faculty at PWIs have increased by only 2.3% in the last 20 years. This study explores the ways in which this imbalance affects minoritized students in engineering majors. Our research objective is to describe the ways in which African American students navigate their way to success in an engineering program at a PWI where the minoritized faculty representation is less than 10%. In this study, we define success as completion of an undergraduate degree and matriculation into a Ph.D. program. Research shows that African American students struggle with feeling like the “outsider within” in graduate programs and that the engineering culture can permeate from undergraduate to graduate programs. We address our research objective by conducting interviews using navigational capital as our theoretical framework, which can be defined as resilience, academic invulnerability, and skills. These three concepts come together to denote the journey of an individual as they achieve success in an environment not created with them in mind. Navigational capital has been applied in education contexts to study minoritized groups, and specifically in engineering education to study the persistence of students of color. Research on navigational capital often focuses on how participants acquire resources from others. There is a limited focus on the experience of the student as the individual agent exercising their own navigational capital. Drawing from and adapting the framework of navigational capital, this study provides rich descriptions of the lived experiences of African American students in an engineering program at a PWI as they navigated their way to academic success in a system that was not designed with them in mind. This pilot study took place at a research-intensive, land grant PWI in the southeastern United States. We recruited two students who identify as African American and are in the first year of their Ph.D. program in an engineering major. Our interview protocol was adapted from a related study about student motivation, identity, and sense of belonging in engineering. After transcribing interviews with these participants, we began our qualitative analysis with a priori coding, drawing from the framework of navigational capital, to identify the experiences, connections, involvement, and resources the participants tapped into as they maneuvered their way to success in an undergraduate engineering program at a PWI. To identify other aspects of the participants’ experiences that were not reflected in that framework, we also used open coding. The results showed that the participants tapped into their navigational capital when they used experiences, connections, involvement, and resources to be resilient, academically invulnerable, and skillful. They learned from experiences (theirs or others’), capitalized on their connections, positioned themselves through involvement, and used their resources to achieve success in their engineering program. The participants identified their experiences, connections, and involvement. For example, one participant who came from a blended family (African American and White) drew from the experiences she had with her blended family. Her experiences helped her to understand the cultures of Black and White people. She was able to turn that into a skill to connect with others at her PWI. The point at which she took her familial experiences to use as a skill to maneuver her way to success at a PWI was an example of her navigational capital. Another participant capitalized on his connections to develop academic invulnerability. He was able to build his connections by making meaningful relationships with his classmates. He knew the importance of having reliable people to be there for him when he encountered a topic he did not understand. He cultivated an environment through relationships with classmates that set him up to achieve academic invulnerability in his classes. The participants spoke least about how they used their resources. The few mentions of resources were not distinct enough to make any substantial connection to the factors that denote navigational capital. The participants spoke explicitly about the PWI culture in their engineering department. From open coding, we identified the theme that participants did not expect to have role models in their major that looked like them and went into their undergraduate experience with the understanding that they will be the distinct minority in their classes. They did not make notable mention of how a lack of minority faculty affected their success. Upon acceptance, they took on the challenge of being a racial minority in exchange for a well-recognized degree they felt would have more value compared to engineering programs at other universities. They identified ways they maneuvered around their expectation that they would not have representative role models through their use of navigational capital. Integrating knowledge from the framework of navigational capital and its existing applications in engineering and education allows us the opportunity to learn from African American students that have succeeded in engineering programs with low minority faculty representation. The future directions of this work are to outline strategies that could enhance the path of minoritized engineering students towards success and to lay a foundation for understanding the use of navigational capital by minoritized students in engineering at PWIs. Students at PWIs can benefit from understanding their own navigational capital to help them identify ways to successfully navigate educational institutions. Students’ awareness of their capacity to maintain high levels of achievement, their connections to networks that facilitate navigation, and their ability to draw from experiences to enhance resilience provide them with the agency to unleash the invisible factors of their potential to be innovators in their collegiate and work environments. 
    more » « less
  4. ; ; ; ; ; ; ; ; ; ( , Review of Education)
    Abstract Why the new findings matter

    The process of teaching and learning is complex, multifaceted and dynamic. This paper contributes a seminal resource to highlight the digitisation of the educational sciences by demonstrating how new machine learning methods can be effectively and reliably used in research, education and practical application.

    Implications for educational researchers and policy makers

    The progressing digitisation of societies around the globe and the impact of the SARS‐COV‐2 pandemic have highlighted the vulnerabilities and shortcomings of educational systems. These developments have shown the necessity to provide effective educational processes that can support sometimes overwhelmed teachers to digitally impart knowledge on the plan of many governments and policy makers. Educational scientists, corporate partners and stakeholders can make use of machine learning techniques to develop advanced, scalable educational processes that account for individual needs of learners and that can complement and support existing learning infrastructure. The proper use of machine learning methods can contribute essential applications to the educational sciences, such as (semi‐)automated assessments, algorithmic‐grading, personalised feedback and adaptive learning approaches. However, these promises are strongly tied to an at least basic understanding of the concepts of machine learning and a degree of data literacy, which has to become the standard in education and the educational sciences.

    Demonstrating both the promises and the challenges that are inherent to the collection and the analysis of large educational data with machine learning, this paper covers the essential topics that their application requires and provides easy‐to‐follow resources and code to facilitate the process of adoption.

     
    more » « less
  5. ; ( , 2023 ASEE Annual Conference & Exposition)
    his work-in-progress paper expands on a collaboration between engineering education researchers and machine learning researchers to automate the analysis of written responses to conceptually challenging questions in statics and dynamics courses (Authors, 2022). Using the Concept Warehouse (Koretsky et al., 2014), written justifications of ConcepTests (CTs) were gathered from statics and dynamics courses in a diverse set of two- and four-year institutions. Written justifications for CTs have been used to support active learning pedagogies which makes them important to investigate how students put together their problem-solving narratives of understanding. However, despite the large benefit that analysis of student written responses may provide to instructors and researchers, manual review of responses is cumbersome, limits analysis, and can be prone to human bias. In efforts to improve the analysis of student written responses, machine learning has been used in various educational contexts to analyze short and long texts (Burstein et al., 2020; Burstein et al., 2021). Natural Language Processing (NLP) uses transformer-based machine learning models (Brown et al., 2020; Raffel et al., 2019) which can be used through fine-tuning or in-context learning methods. NLP can be used to train algorithms that can automate the coding of written responses. Only a few studies for educational applications have leveraged transformer-based machine learning models further prompting an investigation into its use in STEM education. However, work in NLP has been criticized for heightening the possibility to perpetuate and even amplify harmful stereotypes and implicit biases (Chang et al., 2019; Mayfield et al., 2019). In this study, we detail the aim to use NLP for linguistic justice. Using methods like text summary, topic modeling, and text classification, we identify key aspects of student narratives of understanding in written responses to mechanics and statics CTs. Through this process, we seek to use machine learning to identify different ways students talk about a problem and their understanding at any point in their narrative formation process. Thus, we hope to help reduce human bias in the classroom and through technology by giving instructors and researchers a diverse set of narratives that include insight into their students’ histories, identities, and understanding. These can then be used towards connecting technological knowledge to students’ everyday lives. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email