More Like this

1. LSTM Neural Network Assisted Regex Development for Qualitative Coding

   Cai, Z.; Eagan, B.; Marquart C.; Shaffer, D (January 2022, Advances in Quantitative Ethnography: Fourth International Conference, International Conference on Quantitative Ethnography 2022)

   Barany, A.; Damsa, C. (Ed.)

   Regular expression (regex) based automated qualitative coding helps reduce researchers’ effort in manually coding text data, without sacrificing transparency of the coding process. However, researchers using regex based approaches struggle with low recall or high false negative rate during classifier development. Advanced natural language processing techniques, such as topic modeling, latent semantic analysis and neural network classification models help solve this problem in various ways. The latest advance in this direction is the discovery of the so called “negative reversion set (NRS)”, in which false negative items appear more frequently than in the negative set. This helps regex classifier developers more quickly identify missing items and thus improve classification recall. This paper simulates the use of NRS in real coding scenarios and compares the required manual coding items between NRS sampling and random sampling in the process of classifier refinement. The result using one data set with 50,818 items and six associated qualitative codes shows that, on average, using NRS sampling, the required manual coding size could be reduced by 50% to 63%, comparing with random sampling. 

   more » « less
2. Does Active Learning Reduce Human Coding?: A Systematic Comparison of a Neural Network with nCoder

   Choi, J.; Ruis, A. R.; Cai, Z.; Eagan, B. R.; Shaffer, D. W. (January 2022, Advances in Quantitative Ethnography: Fourth International Conference, International Conference on Quantitative Ethnography 2022)

   Barany, A.; Damsa, C. (Ed.)

   In quantitative ethnography (QE) studies which often involve large da-tasets that cannot be entirely hand-coded by human raters, researchers have used supervised machine learning approaches to develop automated classi-fiers. However, QE researchers are rightly concerned with the amount of human coding that may be required to develop classifiers that achieve the high levels of accuracy that QE studies typically require. In this study, we compare a neural network, a powerful traditional supervised learning ap-proach, with nCoder, an active learning technique commonly used in QE studies, to determine which technique requires the least human coding to produce a sufficiently accurate classifier. To do this, we constructed multi-ple training sets from a large dataset used in prior QE studies and designed a Monte Carlo simulation to test the performance of the two techniques sys-tematically. Our results show that nCoder can achieve high predictive accu-racy with significantly less human-coded data than a neural network. 

   more » « less
3. Estimating Instance-dependent Label-noise Transition Matrix using a Deep Neural Network

   Yang, Shuo; Yang, Erkun; Han, Bo; Liu, Yang; Xu, Min; Niu, Gang; Liu, Tongliang (January 2022, International Conference on Machine Learning)

   In label-noise learning, estimating the transition matrix is a hot topic as the matrix plays an important role in building statistically consistent classifiers. Traditionally, the transition from clean labels to noisy labels (i.e., clean-label transition matrix (CLTM)) has been widely exploited to learn a clean label classifier by employing the noisy data. Motivated by that classifiers mostly output Bayes optimal labels for prediction, in this paper, we study to directly model the transition from Bayes optimal labels to noisy labels (i.e., Bayes-label transition matrix (BLTM)) and learn a classifier to predict Bayes optimal labels. Note that given only noisy data, it is ill-posed to estimate either the CLTM or the BLTM. But favorably, Bayes optimal labels have less uncertainty compared with the clean labels, i.e., the class posteriors of Bayes optimal labels are one-hot vectors while those of clean labels are not. This enables two advantages to estimate the BLTM, i.e., (a) a set of examples with theoretically guaranteed Bayes optimal labels can be collected out of noisy data; (b) the feasible solution space is much smaller. By exploiting the advantages, we estimate the BLTM parametrically by employing a deep neural network, leading to better generalization and superior classification performance. 

   more » « less
4. No position-specific interference from prior lists in cued recognition: A challenge for position coding (and other) theories of serial memory

   https://doi.org/10.1016/j.cogpsych.2024.101641  

   Logan, Gordon D; Cox, Gregory E; Lilburn, Simon D; Ulrich, Jana E (March 2024, Cognitive Psychology)

   Rotello, C (Ed.)

   Position-specific intrusions of items from prior lists are rare but important phenomena that distinguish broad classes of theory in serial memory. They are uniquely predicted by position coding theories, which assume items on all lists are associated with the same set of codes representing their positions. Activating a position code activates items associated with it in current and prior lists in proportion to their distance from the activated position. Thus, prior list intrusions are most likely to come from the coded position. Alternative “item dependent” theories based on associations between items and contexts built from items have difficulty accounting for the position specificity of prior list intrusions. We tested the position coding account with a position-cued recognition task designed to produce prior list interference. Cuing a position should activate a position code, which should activate items in nearby positions in the current and prior lists. We presented lures from the prior list to test for position-specific activation in response time and error rate; lures from nearby positions should interfere more. We found no evidence for such interference in 10 experiments, falsifying the position coding prediction. We ran two serial recall experiments with the same materials and found position-specific prior list intrusions. These results challenge all theories of serial memory: Position coding theories can explain the prior list intrusions in serial recall and but not the absence of prior list interference in cued recognition. Item dependent theories can explain the absence of prior list interference in cued recognition but cannot explain the occurrence of prior list intrusions in serial recall. 

   more » « less
5. Difficulty-based Sampling for Debiased Contrastive Representation Learning doi

   Jang, Taeuk; Wang, Xiaoqian (June 2023, IEEE Conference on Computer Vision and Pattern Recognition)

   Contrastive learning is a self-supervised representation learning method that achieves milestone performance in various classification tasks. However, due to its unsupervised fashion, it suffers from the false negative sample problem: randomly drawn negative samples that are assumed to have a different label but actually have the same label as the anchor. This deteriorates the performance of contrastive learning as it contradicts the motivation of contrasting semantically similar and dissimilar pairs. This raised the attention and the importance of finding legitimate negative samples, which should be addressed by distinguishing between 1) true vs. false negatives; 2) easy vs. hard negatives. However, previous works were limited to the statistical approach to handle false negative and hard negative samples with hyperparameters tuning. In this paper, we go beyond the statistical approach and explore the connection between hard negative samples and data bias. We introduce a novel debiased contrastive learning method to explore hard negatives by relative difficulty referencing the bias-amplifying counterpart. We propose triplet loss for training a biased encoder that focuses more on easy negative samples. We theoretically show that the triplet loss amplifies the bias in self-supervised representation learning. Finally, we empirically show the proposed method improves downstream classification performance. 

   more » « less