An official website of the United States government
Abstract Complex interactive test items are becoming more widely used in assessments. Being computer-administered, assessments using interactive items allow logging time-stamped action sequences. These sequences pose a rich source of information that may facilitate investigating how examinees approach an item and arrive at their given response. There is a rich body of research leveraging action sequence data for investigating examinees’ behavior. However, the associated timing data have been considered mainly on the item-level, if at all. Considering timing data on the action-level in addition to action sequences, however, has vast potential to support a more fine-grained assessment of examinees’ behavior. We provide an approach that jointly considers action sequences and action-level times for identifying common response processes. In doing so, we integrate tools from clickstream analyses and graph-modeled data clustering with psychometrics. In our approach, we (a) provide similarity measures that are based on both actions and the associated action-level timing data and (b) subsequently employ cluster edge deletion for identifying homogeneous, interpretable, well-separated groups of action patterns, each describing a common response process. Guidelines on how to apply the approach are provided. The approach and its utility are illustrated on a complex problem-solving item from PIAAC 2012.
more »
« less
Exploring latent states of problem‐solving competence using hidden Markov model on process data
Abstract The response process of problem‐solving items contains rich information about respondents' behaviours and cognitive process in the digital tasks, while the information extraction is a big challenge. The aim of the study is to use a data‐driven approach to explore the latent states and state transitions underlying problem‐solving process to reflect test‐takers' behavioural patterns, and to investigate how these states and state transitions could be associated with test‐takers' performance. We employed the Hidden Markov Modelling approach to identify test takers' hidden states during the problem‐solving process and compared the frequency of states and/or state transitions between different performance groups. We conducted comparable studies in two problem‐solving items with a focus on the US sample that was collected in PIAAC 2012, and examined the correlation between those frequencies from two items. Latent states and transitions between them underlying the problem‐solving process were identified and found significantly different by performance groups. The groups with correct responses in both items were found more engaged in tasks and more often to use efficient tools to solve problems, while the group with incorrect responses was found more likely to use shorter action sequences and exhibit hesitative behaviours. Consistent behavioural patterns were identified across items. This study demonstrates the value of data‐driven based HMM approach to better understand respondents' behavioural patterns and cognitive transmissions underneath the observable action sequences in complex problem‐solving tasks.
more »
« less
- Award ID(s):
- 1633353
- PAR ID:
- 10449512
- Publisher / Repository:
- Wiley-Blackwell
- Date Published:
- Journal Name:
- Journal of Computer Assisted Learning
- Volume:
- 37
- Issue:
- 5
- ISSN:
- 0266-4909
- Page Range / eLocation ID:
- p. 1232-1247
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Computerized assessments and interactive simulation tasks are increasingly popular and afford the collection of process data, i.e., an examinee’s sequence of actions (e.g., clickstreams, keystrokes) that arises from interactions with each task. Action sequence data contain rich information on the problem-solving process but are in a nonstandard, variable-length discrete sequence format. Two methods that directly extract features from the raw action sequences, namely multidimensional scaling and sequence-to-sequence autoencoders, produce multidimensional numerical features that summarize original sequence information. This study explores the utility of action sequence features in understanding how problem-solving behavior relates to cognitive proficiencies and demographic characteristics. This is empirically illustrated with the process data from the 2012 PIAAC PSTRE digital assessment. Regularized regression results showed that action sequence features are more predictive of examinees’ demographic and cognitive characteristics compared to final outcomes. Partial least squares analysis further aided the identification of behavioral patterns systematically associated with demographic/cognitive characteristics.more » « less
-
Recent studies show increasing interest in using process data (e.g., response time, response actions) to enhance measurement accuracy for respondents’ latent traits. Yet, few have explored the possibility of incorporating process information into cognitive diagnostic models (CDMs). This study proposes a novel CDM approach that utilizes a four-component joint modeling approach with response action sequences (i.e., similarity and efficiency), response time, and item responses. We employed the Markov Chain Monte Carlo method for parameter estimation and evaluated the performance of the proposed model using both an empirical study and two simulation studies. The results suggest that the process data can improve respondents’ classification accuracy under varied conditions and support the interpretation of the association between process and response data.more » « less
-
The impact of randomness on model training is poorly understood. How do differences in data order and initialization actually manifest in the model, such that some training runs outperform others or converge faster? Furthermore, how can we interpret the resulting training dynamics and the phase transitions that characterize different trajectories? To understand the effect of randomness on the dynamics and outcomes of neural network training, we train models multiple times with different random seeds and compute a variety of metrics throughout training, such as the norm, mean, and variance of the neural network's weights. We then fit a hidden Markov model (HMM) over the resulting sequences of metrics. The HMM represents training as a stochastic process of transitions between latent states, providing an intuitive overview of significant changes during training. Using our method, we produce a low-dimensional, discrete representation of training dynamics on grokking tasks, image classification, and masked language modeling. We use the HMM representation to study phase transitions and identify latent "detour" states that slow down convergence.more » « less
-
Abstract Recent evidence indicates that individual behavioural variation in animals, defined as consistent individual differences in behaviour across contexts and time, influence ecological and evolutionary processes, and a growing number of studies demonstrate that individual behavioural variation can play a large role in shaping grouping dynamics among social animals. We studied the common degu,Octodon degus, a social rodent, to evaluate whether individual behavioural variation underlies social organization and the reproductive success of individuals within groups. We examined social groups in a population in central-north Chile during one breeding season, tested 67 adults in an open field test (i.e., the propensity to explore an unfamiliar environment) and 62 adults in a poke test (i.e., the propensity to charge an object) to quantify individual behavioural variation, determined assortment based on individual behavioural differences across 19 social groups, and performed genetic analyses to assess reproductive success. We found that the response to the poke test was repeatable, while none of the behaviours from an open field test were. The repeatable behaviour during the poke test was not associated to components of social organization (group composition), or to reproductive success. These findings imply that individual behavioural variation did not affect grouping patterns or direct fitness in this degu population.more » « less
