An official website of the United States government
Abstract Hyperdimensional computing (HDC) addresses massively parallel implementations of symbolic computations that are both more transparent than ANNs and LLMs and more suitable for in-memory computing on highly energy-efficient analog hardware. It captures an essential aspects of brain computations: objects, concepts, and their attributes are encoded by very sparse distributed representations. But currently known methods for binding these tokens together entail deficits in flexible information retrieval.We show that a mechanism which the brain employs for binding, Behavioral Time Scale Synaptic Plasticity (BTSP), overcomes these deficiencies by adding attractor features to high-dimensional representations. They drastically improve the capability to recover from composed representations the tokens which have been bound together in them. One arrives in this way at a functionally more powerful HDC paradigm that provides new perspectives both for understanding how brains carry out symbolic computations, and for implementing them in novel energy-efficient and massively parallel neuromorphic hardware.
more »
« less
Behavioral Time Scale Synaptic Plasticity (BTSP) endows binding of distributed representations with flexible retrieval options
Abstract Human reasoning depends on reusing pieces of information by binding them together in new ways, thereby “making infinite uses of finite means” (Alexander von Humboldt). Needed for that is a binding mechanism that enables fast composition and decomposition of tokens of information. Binding can easily be implemented in symbolic computations through parentheses and ordering of symbols. But it is a highly nontrivial operation for distributed representations, where the tokens are encoded by activity patterns in large neural networks or large language models, or more abstractly, by a high dimensional vector. Vector Symbolic Architectures (VSAs) provide partial solutions, but are lacking the flexibility of the brain in information retrieval, e.g. retrieving the tokens from a composed representation or retrieval of a composed representation by just providing some tokens as a cue. We show that a mechanism which the brain employs for binding distributed representations, Behavioral Time Scale Synaptic Plasticity (BTSP), overcomes these deficiencies. In particular, it combines binding with attractor features that make information retrieval substantially more flexible and robust. We evaluate its performance on various applications, including encoding and decoding complex visual scenes and hierarchical binding. We also show that it enhances models for natural language processing and abstract brain computation. BTSP-based binding only requires binary synaptic weights and simple local synaptic plasticity, and can therefore easily be implemented through in-memory computing or other innovative designs for energy-efficient AI implementations.
more »
« less
- Award ID(s):
- 2318152
- PAR ID:
- 10711775
- Publisher / Repository:
- bioRxiv
- Date Published:
- Format(s):
- Medium: X
- Institution:
- bioRxiv
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
null (Ed.)Variable binding is a cornerstone of symbolic reasoning and cognition. But how binding can be implemented in connectionist models has puzzled neuroscientists, cognitive psychologists, and neural network researchers for many decades. One type of connectionist model that naturally includes a binding operation is vector symbolic architectures (VSAs). In contrast to other proposals for variable binding, the binding operation in VSAs is dimensionality-preserving, which enables representing complex hierarchical data structures, such as trees, while avoiding a combinatoric expansion of dimensionality. Classical VSAs encode symbols by dense randomized vectors, in which information is distributed throughout the entire neuron population. By contrast, in the brain, features are encoded more locally, by the activity of single neurons or small groups of neurons, often forming sparse vectors of neural activation. Following Laiho et al. (2015), we explore symbolic reasoning with a special case of sparse distributed representations. Using techniques from compressed sensing, we first show that variable binding in classical VSAs is mathematically equivalent to tensor product binding between sparse feature vectors, another well-known binding operation which increases dimensionality. This theoretical result motivates us to study two dimensionality-preserving binding methods that include a reduction of the tensor matrix into a single sparse vector. One binding method for general sparse vectors uses random projections, the other, block-local circular convolution, is defined for sparse vectors with block structure, sparse block-codes. Our experiments reveal that block-local circular convolution binding has ideal properties, whereas random projection based binding also works, but is lossy. We demonstrate in example applications that a VSA with block-local circular convolution and sparse block-codes reaches similar performance as classical VSAs. Finally, we discuss our results in the context of neuroscience and neural networks.more » « less
-
We investigate the task of retrieving information from compositional distributed representations formed by hyperdimensional computing/vector symbolic architectures and present novel techniques that achieve new information rate bounds. First, we provide an overview of the decoding techniques that can be used to approach the retrieval task. The techniques are categorized into four groups. We then evaluate the considered techniques in several settings that involve, for example, inclusion of external noise and storage elements with reduced precision. In particular, we find that the decoding techniques from the sparse coding and compressed sensing literature (rarely used for hyperdimensional computing/vector symbolic architectures) are also well suited for decoding information from the compositional distributed representations.Combining these decoding techniqueswith interference cancellation ideas from communications improves previously reported bounds (Hersche et al., 2021) of the information rate of the distributed representations from 1.20 to 1.40 bits per dimension for smaller codebooks and from 0.60 to 1.26 bits per dimension for larger codebooks.more » « less
-
Abstract Variation in the strength of synapses can be quantified by measuring the anatomical properties of synapses. Quantifying precision of synaptic plasticity is fundamental to understanding information storage and retrieval in neural circuits. Synapses from the same axon onto the same dendrite have a common history of coactivation, making them ideal candidates for determining the precision of synaptic plasticity based on the similarity of their physical dimensions. Here, the precision and amount of information stored in synapse dimensions were quantified with Shannon information theory, expanding prior analysis that used signal detection theory (Bartol et al., 2015). The two methods were compared using dendritic spine head volumes in the middle of the stratum radiatum of hippocampal area CA1 as well-defined measures of synaptic strength. Information theory delineated the number of distinguishable synaptic strengths based on nonoverlapping bins of dendritic spine head volumes. Shannon entropy was applied to measure synaptic information storage capacity (SISC) and resulted in a lower bound of 4.1 bits and upper bound of 4.59 bits of information based on 24 distinguishable sizes. We further compared the distribution of distinguishable sizes and a uniform distribution using Kullback-Leibler divergence and discovered that there was a nearly uniform distribution of spine head volumes across the sizes, suggesting optimal use of the distinguishable values. Thus, SISC provides a new analytical measure that can be generalized to probe synaptic strengths and capacity for plasticity in different brain regions of different species and among animals raised in different conditions or during learning. How brain diseases and disorders affect the precision of synaptic plasticity can also be probed.more » « less
-
null (Ed.)Classical information retrieval systems such asBM25 rely on exact lexical match and carryout search efficiently with inverted list index. Recent neural IR models shifts towards soft semantic matching all query document terms,but they lose the computation efficiency of exact match systems.This paper presents COIL, a contextualized exact match retrieval architecture that brings semantic lexical matching. COIL scoring is based on overlapping query document tokens’ contextualized representations. The new architecture stores contextualized token representations in inverted lists, bringing together the efficiency of exact match and the representation power of deep language models. Our experimental results show COIL outperforms classical lexical retrievers and state-of-the-art deep LM retrievers with similar or smaller latency.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript
- Posted Content:
- https://doi.org/10.1101/2025.05.15.654220
