Search for: All records

Pruning the parameters of deep neural networks has generated intense interest due to potential savings in time, memory and energy both during training and at test time. Recent works have identified, through an expensive sequence of training and pruning cycles, the existence of winning lottery tickets or sparse trainable subnetworks at initialization. This raises a foundational question: can we identify highly sparse trainable subnetworks at initialization, without ever training, or indeed without ever looking at the data? We provide an affirmative answer to this question through theory driven algorithm design. We first mathematically formulate and experimentally verify a conservation law that explains why existing gradient-based pruning algorithms at initialization suffer from layer-collapse, the premature pruning of an entire layer rendering a network untrainable. This theory also elucidates how layer-collapse can be entirely avoided, motivating a novel pruning algorithm Iterative Synaptic Flow Pruning (SynFlow). This algorithm can be interpreted as preserving the total flow of synaptic strengths through the network at initialization subject to a sparsity constraint. Notably, this algorithm makes no reference to the training data and consistently competes with or outperforms existing state-of-the-art pruning algorithms at initialization over a range of models (VGG and ResNet), datasets (CIFAR-10/100 and Tiny ImageNet), and sparsity constraints (up to 99.99 percent). Thus our data-agnostic pruning algorithm challenges the existing paradigm that, at initialization, data must be used to quantify which synapses are important. 

Recently, deep feedforward neural networks have achieved considerable success in modeling biological sensory processing, in terms of reproducing the input-output map of sensory neurons. However, such models raise profound questions about the very nature of explanation in neuroscience. Are we simply replacing one complex system (a biological circuit) with another (a deep network), without understanding either? Moreover, beyond neural representations, are the deep network's computational mechanisms for generating neural responses the same as those in the brain? Without a systematic approach to extracting and understanding computational mechanisms from deep neural network models, it can be difficult both to assess the degree of utility of deep learning approaches in neuroscience, and to extract experimentally testable hypotheses from deep networks. We develop such a systematic approach by combining dimensionality reduction and modern attribution methods for determining the relative importance of interneurons for specific visual computations. We apply this approach to deep network models of the retina, revealing a conceptual understanding of how the retina acts as a predictive feature extractor that signals deviations from expectations for diverse spatiotemporal stimuli. For each stimulus, our extracted computational mechanisms are consistent with prior scientific literature, and in one case yields a new mechanistic hypothesis. Thus overall, this work not only yields insights into the computational mechanisms underlying the striking predictive capabilities of the retina, but also places the framework of deep networks as neuroscientific models on firmer theoretical foundations, by providing a new roadmap to go beyond comparing neural representations to extracting and understand computational mechanisms. 

Abstract KAGRA, the underground and cryogenic gravitational-wave detector, was operated for its solo observation from February 25 to March 10, 2020, and its first joint observation with the GEO 600 detector from April 7 to April 21, 2020 (O3GK). This study presents an overview of the input optics systems of the KAGRA detector, which consist of various optical systems, such as a laser source, its intensity and frequency stabilization systems, modulators, a Faraday isolator, mode-matching telescopes, and a high-power beam dump. These optics were successfully delivered to the KAGRA interferometer and operated stably during the observations. The laser frequency noise was observed to limit the detector sensitivity above a few kilohertz, whereas the laser intensity did not significantly limit the detector sensitivity. 

We search for gravitational-wave (GW) transients associated with fast radio bursts (FRBs) detected by the Canadian Hydrogen Intensity Mapping Experiment Fast Radio Burst Project, during the first part of the third observing run of Advanced LIGO and Advanced Virgo (2019 April 1 15:00 UTC–2019 October 1 15:00 UTC). Triggers from 22 FRBs were analyzed with a search that targets both binary neutron star (BNS) and neutron star–black hole (NSBH) mergers. A targeted search for generic GW transients was conducted on 40 FRBs. We find no significant evidence for a GW association in either search. Given the large uncertainties in the distances of our FRB sample, we are unable to exclude the possibility of a GW association. Assessing the volumetric event rates of both FRB and binary mergers, an association is limited to 15% of the FRB population for BNS mergers or 1% for NSBH mergers. We report 90% confidence lower bounds on the distance to each FRB for a range of GW progenitor models and set upper limits on the energy emitted through GWs for a range of emission scenarios. We find values of order 10⁵¹–10⁵⁷erg for models with central GW frequencies in the range 70–3560 Hz. At the sensitivity of this search, we find these limits to be above the predicted GW emissions for the models considered. We also find no significant coincident detection of GWs with the repeater, FRB 20200120E, which is the closest known extragalactic FRB.