More Like this

1. What Can an Accent Identifier Learn? Probing Phonetic and Prosodic Information in a Wav2vec2-based Accent Identification Model

   https://doi.org/10.21437/Interspeech.2023-2254  

   Yang, Mu; Shekar, Ram C.; Kang, Okim; Hansen, John H. (August 2023, ISCA INTERSPEECH-2023)

   N/A (Ed.)

   This study is focused on understanding and quantifying the change in phoneme and prosody information encoded in the Self-Supervised Learning (SSL) model, brought by an accent identification (AID) fine-tuning task. This problem is addressed based on model probing. Specifically, we conduct a systematic layer-wise analysis of the representations of the Transformer layers on a phoneme correlation task, and a novel word-level prosody prediction task. We compare the probing performance of the pre-trained and fine-tuned SSL models. Results show that the AID fine-tuning task steers the top 2 layers to learn richer phoneme and prosody representation. These changes share some similarities with the effects of fine-tuning with an Automatic Speech Recognition task. In addition, we observe strong accent-specific phoneme representations in layer 9. To sum up, this study provides insights into the understanding of SSL features and their interactions with fine-tuning tasks. 

   more » « less
2. Unsupervised Parsing with S-DIORA: Single Tree Encoding for Deep Inside-Outside Recursive Autoencoders

   https://doi.org/10.18653/v1/2020.emnlp-main.392  

   Drozdov, Andrew; Rongali, Subendhu; Chen, Yi-Pei; O’Gorman, Tim; Iyyer, Mohit; McCallum, Andrew (November 2020, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP))

   null (Ed.)

   The deep inside-outside recursive autoencoder (DIORA; Drozdov et al. 2019) is a self-supervised neural model that learns to induce syntactic tree structures for input sentences *without access to labeled training data*. In this paper, we discover that while DIORA exhaustively encodes all possible binary trees of a sentence with a soft dynamic program, its vector averaging approach is locally greedy and cannot recover from errors when computing the highest scoring parse tree in bottom-up chart parsing. To fix this issue, we introduce S-DIORA, an improved variant of DIORA that encodes a single tree rather than a softly-weighted mixture of trees by employing a hard argmax operation and a beam at each cell in the chart. Our experiments show that through *fine-tuning* a pre-trained DIORA with our new algorithm, we improve the state of the art in *unsupervised* constituency parsing on the English WSJ Penn Treebank by 2.2-6% F1, depending on the data used for fine-tuning. 

   more » « less
3. STREAMER: Streaming Representation Learning and Event Segmentation in a Hierarchical Manner

   Mounir, R; Vijayaraghavan, S; Sarkar, S (December 2023, NeurIPS Proceedings)

   We present a novel self-supervised approach for hierarchical representation learning and segmentation of perceptual inputs in a streaming fashion. Our research addresses how to semantically group streaming inputs into chunks at various levels of a hierarchy while simultaneously learning, for each chunk, robust global representations throughout the domain. To achieve this, we propose STREAMER, an architecture that is trained layer-by-layer, adapting to the complexity of the input domain. In our approach, each layer is trained with two primary objectives: making accurate predictions into the future and providing necessary information to other levels for achieving the same objective. The event hierarchy is constructed by detecting prediction error peaks at different levels, where a detected boundary triggers a bottom-up information flow. At an event boundary, the encoded representation of inputs at one layer becomes the input to a higher-level layer. Additionally, we design a communication module that facilitates top-down and bottom-up exchange of information during the prediction process. Notably, our model is fully self-supervised and trained in a streaming manner, enabling a single pass on the training data. This means that the model encounters each input only once and does not store the data. We evaluate the performance of our model on the egocentric EPIC-KITCHENS dataset, specifically focusing on temporal event segmentation. Furthermore, we conduct event retrieval experiments using the learned representations to demonstrate the high quality of our video event representations. 

   more » « less
4. Using Developer Eye Movements to Externalize the Mental Model Used in Code Summarization Tasks

   Abid, N.; Maletic, J.I.; Sharif, B. (June 2019, 11th ACM Symposium on Eye tracking Research and Applications (ETRA))

   cognition model (i.e., bottom-up or top-down) applied during program comprehension tasks. The cognition models examine how programmers understand source code by describing the temporary information structures in the programmer’s short term memory. The two types of models that we are interested in are top-down and bottom-up. The top-down model is normally applied as-needed (i.e., the domain of the system is familiar). The bottom-up model is typically applied when a developer is not familiar with the domain or the source code. An eye-tracking study of 18 developers reading and summarizing Java methods is used as our dataset for analyzing the mental cognition model. The developers provide a written summary for methods assigned to them. In total, 63 methods are used from five different systems. The results indicate that on average, experts and novices read the methods more closely (using the bottom-up mental model) than bouncing around (using top-down). However, on average novices spend longer gaze time performing bottom-up (66s.) compared to experts (43s.) 

   more » « less
5. Transient top‐down and bottom‐up effects of resources pulsed to multiple trophic levels

   https://doi.org/10.1002/ecy.3197  

   McCary, Matthew A.; Phillips, Joseph S.; Ramiadantsoa, Tanjona; Nell, Lucas A.; McCormick, Amanda R.; Botsch, Jamieson C. (November 2020, Ecology)

   Abstract Pulsed fluxes of organisms across ecosystem boundaries can exert top‐down and bottom‐up effects in recipient food webs, through both direct effects on the subsidized trophic levels and indirect effects on other components of the system. While previous theoretical and empirical studies demonstrate the influence of allochthonous subsidies on bottom‐up and top‐down processes, understanding how these forces act in conjunction is still limited, particularly when an allochthonous resource can simultaneously subsidize multiple trophic levels. Using the Lake Mývatn region in Iceland as an example system of allochthony and its potential effects on multiple trophic levels, we analyzed a mathematical model to evaluate how pulsed subsidies of aquatic insects affect the dynamics of a soil–plant–arthropod food web. We found that the relative balance of top‐down and bottom‐up effects on a given food web compartment was determined by trophic position, subsidy magnitude, and top predators’ ability to exploit the subsidy. For intermediate trophic levels (e.g., detritivores and herbivores), we found that the subsidy could either alleviate or intensify top‐down pressure from the predator. For some parameter combinations, alleviation and intensification occurred sequentially during and after the resource pulse. The total effect of the subsidy on detritivores and herbivores, including top‐down and bottom‐up processes, was determined by the rate at which predator consumption saturated with increasing size of the allochthonous subsidy, with greater saturation leading to increased bottom‐up effects. Our findings illustrate how resource pulses to multiple trophic levels can influence food web dynamics by changing the relative strength of bottom‐up and top‐down effects, with bottom‐up predominating top‐down effects in most scenarios in this subarctic system. 

   more » « less