skip to main content


Search for: All records

Award ID contains: 2100320

Note: When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external site maintained by the publisher. Some full text articles may not yet be available without a charge during the embargo (administrative interval).
What is a DOI Number?

Some links on this page may take you to non-federal websites. Their policies may differ from this site.

  1. Barany, A. ; Damsa, C. (Ed.)
    Regular expression (regex) based automated qualitative coding helps reduce researchers’ effort in manually coding text data, without sacrificing transparency of the coding process. However, researchers using regex based approaches struggle with low recall or high false negative rate during classifier development. Advanced natural language processing techniques, such as topic modeling, latent semantic analysis and neural network classification models help solve this problem in various ways. The latest advance in this direction is the discovery of the so called “negative reversion set (NRS)”, in which false negative items appear more frequently than in the negative set. This helps regex classifier developers more quickly identify missing items and thus improve classification recall. This paper simulates the use of NRS in real coding scenarios and compares the required manual coding items between NRS sampling and random sampling in the process of classifier refinement. The result using one data set with 50,818 items and six associated qualitative codes shows that, on average, using NRS sampling, the required manual coding size could be reduced by 50% to 63%, comparing with random sampling. 
    more » « less
  2. Barany,  A. ; Damsa, C. (Ed.)
    Analysis of policy ecosystems can be challenging due to the volume of documentary and ethnographic data and the complexity of the interactions that define the ecology of such a system. This paper uses climate change adaptation policy as a case study with which to explore the potential for QE methods to model policy ecosystems. Specifically, it analyzes policies and draft policies constructed by three different categories of governmental entity—nations, state and local governments, and tribal governments or Indigenous communities—as well as guidance for policy makers produced by the United Nations Intergovernmental Panel on Climate Change and other international agencies, as a first step toward mapping the ecology of climate change adaptation policy. This case study is then used to reflect on the strengths of QE methods for analyzing policy ecosystems and areas of opportunity for further theoretical and methodological development. 
    more » « less
  3. The present conceptual literature review analyzes 50 studies that systematically examined the effects of authentic learning settings on cognitive or motivational learning outcomes. The analysis focuses on describing the context of the studies, the design elements of authentic learning settings, and the pursued intentions of authenticity. The review further describes the effects of authentically designed learning settings on cognitive outcomes, motivational outcomes, and learners’ perceived authenticity revealed by previous research. Building on these findings, we conducted Epistemic Network Analysis (ENA) of contrasting cases to identify design elements and intentions of authenticity characterizing studies that show high effectiveness for cognitive and motivational outcomes versus those with low effectiveness. The ENA results suggest, for instance, that providing authentic materials (as a design element of authentic learning settings) to resemble real-life experiences (as an intention of authenticity) could be a double-edged sword, as they feature both authentically designed learning settings with low effects on cognitive outcomes and settings with high effects on motivational outcomes. Overall, the results of the present literature review point to critical limitations of previous research, such as a lack of clear definitions and operationalizations of authentic learning. Consequently, we draw specific conclusions about how future research could improve our understanding of how to create and implement powerful methods of authentic learning. 
    more » « less
  4. Barany, A. ; Damsa, C. (Ed.)
    In quantitative ethnography (QE) studies which often involve large da-tasets that cannot be entirely hand-coded by human raters, researchers have used supervised machine learning approaches to develop automated classi-fiers. However, QE researchers are rightly concerned with the amount of human coding that may be required to develop classifiers that achieve the high levels of accuracy that QE studies typically require. In this study, we compare a neural network, a powerful traditional supervised learning ap-proach, with nCoder, an active learning technique commonly used in QE studies, to determine which technique requires the least human coding to produce a sufficiently accurate classifier. To do this, we constructed multi-ple training sets from a large dataset used in prior QE studies and designed a Monte Carlo simulation to test the performance of the two techniques sys-tematically. Our results show that nCoder can achieve high predictive accu-racy with significantly less human-coded data than a neural network. 
    more » « less
  5. Mitrovic, A. ; Bosch, N. (Ed.)
    Regular expression (regex) coding has advantages for text analysis. Humans are often able to quickly construct intelligible coding rules with high precision. That is, researchers can identify words and word patterns that correctly classify examples of a particular concept. And, it is often easy to identify false positives and improve the regex classifier so that the positive items are accurately captured. However, ensuring that a regex list is complete is a bigger challenge, because the concepts to be identified in data are often sparsely distributed, which makes it difficult to identify examples of \textit{false negatives}. For this reason, regex-based classifiers suffer by having low recall. That is, it often misses items that should be classified as positive. In this paper, we provide a neural network solution to this problem by identifying a \textit{negative reversion set}, in which false negative items occur much more frequently than in the data set as a whole. Thus, the regex classifier can be more quickly improved by adding missing regexes based on the false negatives found from the negative reversion set. This study used an existing data set collected from a simulation-based learning environment for which researchers had previously defined six codes and developed classifiers with validated regex lists. We randomly constructed incomplete (partial) regex lists and used neural network models to identify negative reversion sets in which the frequency of false negatives increased from a range of 3\\%-8\\% in the full data set to a range of 12\\%-52\\% in the negative reversion set. Based on this finding, we propose an interactive coding mechanism in which human-developed regex classifiers provide input for training machine learning algorithms and machine learning algorithms ``smartly" select highly suspected false negative items for human to more quickly develop regex classifiers. 
    more » « less
  6. Learning analytics uses large amounts of data about learner interactions in digital learning environments to understand and enhance learning. Although measurement is a central dimension of learning analytics, there has thus far been little research that examines links between learning analytics and assessment. This special issue of Computers in Human Behavior highlights 11 studies that explore how links between learning analytics and assessment can be strengthened. The contributions of these studies can be broadly grouped into three categories: analytics for assessment (learning analytic approaches as forms of assessment); analytics of assessment (applications of learning analytics to answer questions about assessment practices); and validity of measurement (conceptualization of and practical approaches to assuring validity in measurement in learning analytics). The findings of these studies highlight pressing scientific and practical challenges and opportunities in the connections between learning analytics and assessment that will require interdisciplinary teams to address: task design, analysis of learning progressions, trustworthiness, and fairness – to unlock the full potential of the links between learning analytics and assessment. 
    more » « less