More Like this

1. STREAMER: Streaming Representation Learning and Event Segmentation in a Hierarchical Manner

   Mounir, R; Vijayaraghavan, S; Sarkar, S (December 2023, NeurIPS Proceedings)

   We present a novel self-supervised approach for hierarchical representation learning and segmentation of perceptual inputs in a streaming fashion. Our research addresses how to semantically group streaming inputs into chunks at various levels of a hierarchy while simultaneously learning, for each chunk, robust global representations throughout the domain. To achieve this, we propose STREAMER, an architecture that is trained layer-by-layer, adapting to the complexity of the input domain. In our approach, each layer is trained with two primary objectives: making accurate predictions into the future and providing necessary information to other levels for achieving the same objective. The event hierarchy is constructed by detecting prediction error peaks at different levels, where a detected boundary triggers a bottom-up information flow. At an event boundary, the encoded representation of inputs at one layer becomes the input to a higher-level layer. Additionally, we design a communication module that facilitates top-down and bottom-up exchange of information during the prediction process. Notably, our model is fully self-supervised and trained in a streaming manner, enabling a single pass on the training data. This means that the model encounters each input only once and does not store the data. We evaluate the performance of our model on the egocentric EPIC-KITCHENS dataset, specifically focusing on temporal event segmentation. Furthermore, we conduct event retrieval experiments using the learned representations to demonstrate the high quality of our video event representations. 

   more » « less
2. Runtime Monitoring of Deep Neural Networks Using Top-Down Context Models Inspired by Predictive Processing and Dual Process Theory

   Roy, A.; Cobb, A.; Bastian, Nathaniel; Jalaian, Brian; Jha, Susmit (April 2022, AAAI Spring Symposium 2022)

   Deep neural networks (DNNs) have achieved near-human level accuracy on many datasets across different domains. But they are known to produce incorrect predictions with high confidence on inputs far from the training distribution. This challenge of lack of calibration of DNNs has limited the adoption of deep learning models in high-assurance systems such as autonomous driving, air traffic management, cybersecurity, and medical diagnosis. The problem of detecting when an input is outside the training distribution of a machine learning model, and hence, its prediction on this input cannot be trusted, has received significant attention recently. Several techniques based on statistical, geometric, topological, or relational signatures have been developed to detect the out-of-distribution (OOD) or novel inputs. In this paper, we present a runtime monitor based on predictive processing and dual process theory. We posit that the bottom-up deep neural networks can be monitored using top-down context models comprising two layers. The first layer is a feature density model that learns the joint distribution of the original DNN’s inputs, outputs, and the model’s explanation for its decisions. The second layer is a graph Markov neural network that captures an even broader context. We demonstrate the efficacy of our monitoring architecture in recognizing out-of-distribution and out-of-context inputs on the image classification and object detection tasks. 

   more » « less
3. Feedforward attentional selection in sensory cortex

   https://doi.org/10.1038/s41467-023-41745-1  

   Westerberg, Jacob A; Schall, Jeffrey D; Woodman, Geoffrey F; Maier, Alexander (December 2023, Nature Communications)

   Abstract Salient objects grab attention because they stand out from their surroundings. Whether this phenomenon is accomplished by bottom-up sensory processing or requires top-down guidance is debated. We tested these alternative hypotheses by measuring how early and in which cortical layer(s) neural spiking distinguished a target from a distractor. We measured synaptic and spiking activity across cortical columns in mid-level area V4 of male macaque monkeys performing visual search for a color singleton. A neural signature of attentional capture was observed in the earliest response in the input layer 4. The magnitude of this response predicted response time and accuracy. Errant behavior followed errant selection. Because this response preceded top-down influences and arose in the cortical layer not targeted by top-down connections, these findings demonstrate that feedforward activation of sensory cortex can underlie attentional priority. 

   more » « less
4. Permafrost Region Greenhouse Gas Budgets Suggest a Weak CO ₂ Sink and CH ₄ and N ₂ O Sources, But Magnitudes Differ Between Top‐Down and Bottom‐Up Methods

   https://doi.org/10.1029/2023GB007969  

   Hugelius, G; Ramage, J; Burke, E; Chatterjee, A; Smallman, T L; Aalto, T; Bastos, A; Biasi, C; Canadell, J G; Chandra, N; et al (October 2024, Global Biogeochemical Cycles)

   Abstract Large stocks of soil carbon (C) and nitrogen (N) in northern permafrost soils are vulnerable to remobilization under climate change. However, there are large uncertainties in present‐day greenhouse gas (GHG) budgets. We compare bottom‐up (data‐driven upscaling and process‐based models) and top‐down (atmospheric inversion models) budgets of carbon dioxide (CO₂), methane (CH₄) and nitrous oxide (N₂O) as well as lateral fluxes of C and N across the region over 2000–2020. Bottom‐up approaches estimate higher land‐to‐atmosphere fluxes for all GHGs. Both bottom‐up and top‐down approaches show a sink of CO₂in natural ecosystems (bottom‐up: −29 (−709, 455), top‐down: −587 (−862, −312) Tg CO₂‐C yr⁻¹) and sources of CH₄(bottom‐up: 38 (22, 53), top‐down: 15 (11, 18) Tg CH₄‐C yr⁻¹) and N₂O (bottom‐up: 0.7 (0.1, 1.3), top‐down: 0.09 (−0.19, 0.37) Tg N₂O‐N yr⁻¹). The combined global warming potential of all three gases (GWP‐100) cannot be distinguished from neutral. Over shorter timescales (GWP‐20), the region is a net GHG source because CH₄dominates the total forcing. The net CO₂sink in Boreal forests and wetlands is largely offset by fires and inland water CO₂emissions as well as CH₄emissions from wetlands and inland waters, with a smaller contribution from N₂O emissions. Priorities for future research include the representation of inland waters in process‐based models and the compilation of process‐model ensembles for CH₄and N₂O. Discrepancies between bottom‐up and top‐down methods call for analyses of how prior flux ensembles impact inversion budgets, more and well‐distributed in situ GHG measurements and improved resolution in upscaling techniques. 

   more » « less
5. Single crystal neutron and magnetic measurements of Rb ₂ Mn ₃ (VO ₄ ) ₂ CO ₃ and K ₂ Co ₃ (VO ₄ ) ₂ CO ₃ with mixed honeycomb and triangular magnetic lattices

   https://doi.org/10.1039/C9DT03389K  

   Smith Pellizzeri, Tiffany M.; Sanjeewa, Liurukara D.; Pellizzeri, Steven; McMillen, Colin D.; Garlea, V. Ovidiu; Ye, Feng; Sefat, Athena S.; Kolis, Joseph W. (April 2020, Dalton Transactions)

   null (Ed.)

   Two new alkali vanadate carbonates with divalent transition metals have been synthesized as large single crystals via a high-temperature (600 °C) hydrothermal technique. Compound I , Rb 2 Mn 3 (VO 4 ) 2 CO 3 , crystallizes in the trigonal crystal system in the space group P 3̄1 c , and compound II , K 2 Co 3 (VO 4 ) 2 CO 3 , crystallizes in the hexagonal space group P 6 3 / m . Both structures contain honeycomb layers and triangular lattices made from edge-sharing MO 6 octahedra and MO 5 trigonal bipyramids, respectively. The honeycomb and triangular layers are connected along the c -axis through tetrahedral [VO 4 ] groups. The MO 5 units are connected with each other by carbonate groups in the ab -plane by forming a triangular magnetic lattice. The difference in space groups between I and II was also investigated with Density Functional Theory (DFT) calculations. Single crystal magnetic characterization of I indicates three magnetic transitions at 77 K, 2.3 K, and 1.5 K. The corresponding magnetic structures for each magnetic transition of I were determined using single crystal neutron diffraction. At 77 K the compound orders in the MnO 6 -honeycomb layer in a Néel-type antiferromagnetic orientation while the MnO 5 triangular lattice ordered below 2.3 K in a colinear ‘up–up–down’ fashion, followed by a planar ‘Y’ type magnetic structure. K 2 Co 3 (VO 4 ) 2 CO 3 ( II ) exhibits a canted antiferromagnetic ordering below T N = 8 K. The Curie–Weiss fit (200–350 K) gives a Curie–Weiss temperature of −42 K suggesting a dominant antiferromagnetic coupling in the Co 2+ magnetic sublattices. 

   more » « less