More Like this

1. STREAMER: Streaming Representation Learning and Event Segmentation in a Hierarchical Manner

   Mounir, R; Vijayaraghavan, S; Sarkar, S (December 2023, NeurIPS Proceedings)

   We present a novel self-supervised approach for hierarchical representation learning and segmentation of perceptual inputs in a streaming fashion. Our research addresses how to semantically group streaming inputs into chunks at various levels of a hierarchy while simultaneously learning, for each chunk, robust global representations throughout the domain. To achieve this, we propose STREAMER, an architecture that is trained layer-by-layer, adapting to the complexity of the input domain. In our approach, each layer is trained with two primary objectives: making accurate predictions into the future and providing necessary information to other levels for achieving the same objective. The event hierarchy is constructed by detecting prediction error peaks at different levels, where a detected boundary triggers a bottom-up information flow. At an event boundary, the encoded representation of inputs at one layer becomes the input to a higher-level layer. Additionally, we design a communication module that facilitates top-down and bottom-up exchange of information during the prediction process. Notably, our model is fully self-supervised and trained in a streaming manner, enabling a single pass on the training data. This means that the model encounters each input only once and does not store the data. We evaluate the performance of our model on the egocentric EPIC-KITCHENS dataset, specifically focusing on temporal event segmentation. Furthermore, we conduct event retrieval experiments using the learned representations to demonstrate the high quality of our video event representations. 

   more » « less
2. Feedforward attentional selection in sensory cortex

   https://doi.org/10.1038/s41467-023-41745-1  

   Westerberg, Jacob A; Schall, Jeffrey D; Woodman, Geoffrey F; Maier, Alexander (December 2023, Nature Communications)

   Abstract Salient objects grab attention because they stand out from their surroundings. Whether this phenomenon is accomplished by bottom-up sensory processing or requires top-down guidance is debated. We tested these alternative hypotheses by measuring how early and in which cortical layer(s) neural spiking distinguished a target from a distractor. We measured synaptic and spiking activity across cortical columns in mid-level area V4 of male macaque monkeys performing visual search for a color singleton. A neural signature of attentional capture was observed in the earliest response in the input layer 4. The magnitude of this response predicted response time and accuracy. Errant behavior followed errant selection. Because this response preceded top-down influences and arose in the cortical layer not targeted by top-down connections, these findings demonstrate that feedforward activation of sensory cortex can underlie attentional priority. 

   more » « less
3. Runtime Monitoring of Deep Neural Networks Using Top-Down Context Models Inspired by Predictive Processing and Dual Process Theory

   Roy, A.; Cobb, A.; Bastian, Nathaniel; Jalaian, Brian; Jha, Susmit (April 2022, AAAI Spring Symposium 2022)

   Deep neural networks (DNNs) have achieved near-human level accuracy on many datasets across different domains. But they are known to produce incorrect predictions with high confidence on inputs far from the training distribution. This challenge of lack of calibration of DNNs has limited the adoption of deep learning models in high-assurance systems such as autonomous driving, air traffic management, cybersecurity, and medical diagnosis. The problem of detecting when an input is outside the training distribution of a machine learning model, and hence, its prediction on this input cannot be trusted, has received significant attention recently. Several techniques based on statistical, geometric, topological, or relational signatures have been developed to detect the out-of-distribution (OOD) or novel inputs. In this paper, we present a runtime monitor based on predictive processing and dual process theory. We posit that the bottom-up deep neural networks can be monitored using top-down context models comprising two layers. The first layer is a feature density model that learns the joint distribution of the original DNN’s inputs, outputs, and the model’s explanation for its decisions. The second layer is a graph Markov neural network that captures an even broader context. We demonstrate the efficacy of our monitoring architecture in recognizing out-of-distribution and out-of-context inputs on the image classification and object detection tasks. 

   more » « less
4. Permafrost Region Greenhouse Gas Budgets Suggest a Weak CO ₂ Sink and CH ₄ and N ₂ O Sources, But Magnitudes Differ Between Top‐Down and Bottom‐Up Methods

   https://doi.org/10.1029/2023GB007969  

   Hugelius, G; Ramage, J; Burke, E; Chatterjee, A; Smallman, T L; Aalto, T; Bastos, A; Biasi, C; Canadell, J G; Chandra, N; et al (October 2024, Global Biogeochemical Cycles)

   Abstract Large stocks of soil carbon (C) and nitrogen (N) in northern permafrost soils are vulnerable to remobilization under climate change. However, there are large uncertainties in present‐day greenhouse gas (GHG) budgets. We compare bottom‐up (data‐driven upscaling and process‐based models) and top‐down (atmospheric inversion models) budgets of carbon dioxide (CO₂), methane (CH₄) and nitrous oxide (N₂O) as well as lateral fluxes of C and N across the region over 2000–2020. Bottom‐up approaches estimate higher land‐to‐atmosphere fluxes for all GHGs. Both bottom‐up and top‐down approaches show a sink of CO₂in natural ecosystems (bottom‐up: −29 (−709, 455), top‐down: −587 (−862, −312) Tg CO₂‐C yr⁻¹) and sources of CH₄(bottom‐up: 38 (22, 53), top‐down: 15 (11, 18) Tg CH₄‐C yr⁻¹) and N₂O (bottom‐up: 0.7 (0.1, 1.3), top‐down: 0.09 (−0.19, 0.37) Tg N₂O‐N yr⁻¹). The combined global warming potential of all three gases (GWP‐100) cannot be distinguished from neutral. Over shorter timescales (GWP‐20), the region is a net GHG source because CH₄dominates the total forcing. The net CO₂sink in Boreal forests and wetlands is largely offset by fires and inland water CO₂emissions as well as CH₄emissions from wetlands and inland waters, with a smaller contribution from N₂O emissions. Priorities for future research include the representation of inland waters in process‐based models and the compilation of process‐model ensembles for CH₄and N₂O. Discrepancies between bottom‐up and top‐down methods call for analyses of how prior flux ensembles impact inversion budgets, more and well‐distributed in situ GHG measurements and improved resolution in upscaling techniques. 

   more » « less
5. Juvenile predation overwhelms nutritional effects on female ungulate fat reserves in a high‐predation system

   https://doi.org/10.1111/1365-2435.70343  

   Bealer, Nicole P; Hebblewhite, Mark; Williams, Sara H; Berg, Jodi E; Martin, Hans W; Merrill, Evelyn H (April 2026, Functional Ecology)

   Abstract Ungulate body fat reserves reflect the nutritional environment, often serving as a useful indicator of bottom‐up resource availability. However, body fat reserves also integrate energetic costs associated with avoiding predation risk and reproductive effort, and it is unknown how bottom‐up and top‐down factors integrate to affect body fat reserves.We used generalized mixed‐effects models to evaluate how bottom‐up, top‐down and intrinsic factors explained variation in winter ingesta‐free body fat (IFBF) of female elk (Cervus canadensis) near Banff National Park, Alberta, Canada from 2015 to 2021.We found two top models were competitive, with calf survival during the first 3–4 months of life and female age included in both models. There was more support for bottom‐up forage effects than indirect predation risk effects. Based on model predictions, females with a calf surviving through the previous summer had 3.6 percentage points (95% CI: 2.4%, 4.9%) lower IFBF the subsequent winter than females without a surviving calf.Because calf survival during the first 3–4 months of life is largely driven by predation in this system, we suggest top‐down effects of predation through calf mortality influenced body fat around nine times more than bottom‐up factors in our study. Under high predation, variation in body fat levels of female ungulates across a population may not reflect only bottom‐up influences on body condition, but also differences in predation. Thus, both bottom‐up and top‐down factors must be considered when assessing the nutritional environment. Read the freePlain Language Summaryfor this article on the Journal blog. 

   more » « less