Search for: All records

In supervised continual learning, a deep neural network (DNN) is updated with an ever-growing data stream. Unlike the offline setting where data is shuffled, we cannot make any distributional assumptions about the data stream. Ideally, only one pass through the dataset is needed for computational efficiency. However, existing methods are inadequate and make many assumptions that cannot be made for real-world applications, while simultaneously failing to improve computational efficiency. In this paper, we propose a novel continual learning method, SIESTA based on wake/sleep framework for training, which is well aligned to the needs of on-device learning. The major goal of SIESTA is to advance compute efficient continual learning so that DNNs can be updated efficiently using far less time and energy. The principal innovations of SIESTA are: 1) rapid online updates using a rehearsal-free, backpropagation-free, and data-driven network update rule during its wake phase, and 2) expedited memory consolidation using a compute-restricted rehearsal policy during its sleep phase. For memory efficiency, SIESTA adapts latent rehearsal using memory indexing from REMIND. Compared to REMIND and prior arts, SIESTA is far more computationally efficient, enabling continual learning on ImageNet-1K in under 2 hours on a single GPU; moreover, in the augmentation-free setting it matches the performance of the offline learner, a milestone critical to driving adoption of continual learning in real-world applications. 

Abstract Replay is the reactivation of one or more neural patterns that are similar to the activation patterns experienced during past waking experiences. Replay was first observed in biological neural networks during sleep, and it is now thought to play a critical role in memory formation, retrieval, and consolidation. Replay-like mechanisms have been incorporated in deep artificial neural networks that learn over time to avoid catastrophic forgetting of previous knowledge. Replay algorithms have been successfully used in a wide range of deep learning methods within supervised, unsupervised, and reinforcement learning paradigms. In this letter, we provide the first comprehensive comparison between replay in the mammalian brain and replay in artificial neural networks. We identify multiple aspects of biological replay that are missing in deep learning systems and hypothesize how they could be used to improve artificial neural networks. 

Abstract Lamellar phases of alkyldiacetylenes in which the alkyl chains lie parallel to the substrate represent a straightforward means for scalable 1‐nm‐resolution interfacial patterning. This capability has the potential for substantial impacts in nanoscale electronics, energy conversion, and biomaterials design. Polymerization is required to set the 1‐nm functional patterns embedded in the monolayer, making it important to understand structure–function relationships for these on‐surface reactions. Polymerization can be observed for certain monomers at the single‐polymer scale using scanning probe microscopy. However, substantial restrictions on the systems that can be effectively characterized have limited utility. Here, using a new multi‐scale approach, we identify a large, previously unreported difference in polymerization efficiency between the two most widely used commercial diynoic acids. We further identify a core design principle for maximizing polymerization efficiency in these on‐surface reactions, generating a new monomer that also exhibits enhanced polymerization efficiency.