More Like this

1. Using convolutional neural networks to develop starting models for near-surface 2-D full waveform inversion

   https://doi.org/10.1093/gji/ggac179  

   Vantassel, Joseph P. ; Kumar, Krishna ; Cox, Brady R. ( May 2022 , Geophysical Journal International)

   Non-invasive subsurface imaging using full waveform inversion (FWI) has the potential to fundamentally change near-surface (<30 m) site characterization by enabling the recovery of high-resolution (metre-scale) 2-D/3-D maps of subsurface elastic material properties. Yet, FWI results are quite sensitive to their starting model due to their dependence on local-search optimization techniques and inversion non-uniqueness. Starting model dependence is particularly problematic for near-surface FWI due to the complexity of the recorded seismic wavefield (e.g. dominant surface waves intermixed with body waves) and the potential for significant spatial variability over short distances. In response, convolutional neural networks (CNNs) are investigated as a potential tool for developing starting models for near-surface 2-D elastic FWI. Specifically, 100 000 subsurface models were generated to be representative of a classic near-surface geophysics problem; namely, imaging a two-layer, undulating, soil-over-bedrock interface. A CNN has been developed from these synthetic models that is capable of transforming an experimental wavefield acquired using a seismic source located at the centre of a linear array of 24 closely spaced surface sensors directly into a robust starting model for FWI. The CNN approach was able to produce 2-D starting models with seismic image misfits that were significantly less than the misfits from other common starting model approaches, and in many cases even less than the misfits obtained by FWI with inferior starting models. The ability of the CNN to generalize outside its two-layered training set was assessed using a more complex, three-layered, soil-over-bedrock formation. While the predictive ability of the CNN was slightly reduced for this more complex case, it was still able to achieve seismic image and waveform misfits that were comparable to other commonly used starting models, despite not being trained on any three-layered models. As such, CNNs show great potential as tools for rapidly developing robust, site-specific starting models for near-surface elastic FWI.

    

   more » « less
2. Coresets for Data-efficient Training of Machine Learning Models

   Mirzasoleiman, Baharan ; Bilmes, Jeff ; Leskovec, Jure. ( January 2020 , International Conference on Machine Learning (ICML))

   null (Ed.)

   Incremental gradient (IG) methods, such as stochastic gradient descent and its variants are commonly used for large scale optimization in machine learning. Despite the sustained effort to make IG methods more data-efficient, it remains an open question how to select a training data subset that can theoretically and practically perform on par with the full dataset. Here we develop CRAIG, a method to select a weighted subset (or coreset) of training data that closely estimates the full gradient by maximizing a submodular function. We prove that applying IG to this subset is guaranteed to converge to the (near)optimal solution with the same convergence rate as that of IG for convex optimization. As a result, CRAIG achieves a speedup that is inversely proportional to the size of the subset. To our knowledge, this is the first rigorous method for data-efficient training of general machine learning models. Our extensive set of experiments show that CRAIG, while achieving practically the same solution, speeds up various IG methods by up to 6x for logistic regression and 3x for training deep neural networks. 

   more » « less
3. Coresets for Data-efficient Training of Machine Learning Models

   Mirzasoleiman, R. ; Bilmes, J. ; Leskovec, J. ( April 2020 , International Conference on Machine Learning (ICML))

   Incremental gradient (IG) methods, such as stochastic gradient descent and its variants are commonly used for large scale optimization in machine learning. Despite the sustained effort to make IG methods more data-efficient, it remains an open question how to select a training data subset that can theoretically and practically perform on par with the full dataset. Here we develop CRAIG, a method to select a weighted subset (or coreset) of training data that closely estimates the full gradient by maximizing a submodular function. We prove that applying IG to this subset is guaranteed to converge to the (near)optimal solution with the same convergence rate as that of IG for convex optimization. As a result, CRAIG achieves a speedup that is inversely proportional to the size of the subset. To our knowledge, this is the first rigorous method for data-efficient training of general machine learning models. Our extensive set of experiments show that CRAIG, while achieving practically the same solution, speeds up various IG methods by up to 6x for logistic regression and 3x for training deep neural networks. 

   more » « less
4. Internal models in control, bioengineering, and neuroscience

   Bin.M. ; Huang, J. ; Isidori, A. ; Marconi, L. ; Mischiati, M. ; E. D. Sontag, E.D. ( October 2021 , Annual review of control robotics and autonomous systems)

   Cells respond to biochemical and physical internal as well as external signals. These signals can be broadly classified into two categories: (a) ``actionable'' or ``reference'' inputs that should elicit appropriate biological or physical responses such as gene expression or motility, and (b) ``disturbances'' or ``perturbations'' that should be ignored or actively filtered-out. These disturbances might be exogenous, such as binding of nonspecific ligands, or endogenous, such as variations in enzyme concentrations or gene copy numbers. In this context, the term robustness describes the capability to produce appropriate responses to reference inputs while at the same time being insensitive to disturbances. These two objectives often conflict with each other and require delicate design trade-offs. Indeed, natural biological systems use complicated and still poorly understood control strategies in order to finely balance the goals of responsiveness and robustness. A better understanding of such natural strategies remains an important scientific goal in itself and will play a role in the construction of synthetic circuits for therapeutic and biosensing applications. A prototype problem in robustly responding to inputs is that of ``robust tracking'', defined by the requirement that some designated internal quantity (for example, the level of expression of a reporter protein) should faithfully follow an input signal while being insensitive to an appropriate class of perturbations. Control theory predicts that a certain type of motif, called integral feedback, will help achieve this goal, and this motif is, in fact, a necessary feature of any system that exhibits robust tracking. Indeed, integral feedback has always been a key component of electrical and mechanical control systems, at least since the 18th century when James Watt employed the centrifugal governor to regulate steam engines. Motivated by this knowledge, biological engineers have proposed various designs for biomolecular integral feedback control mechanisms. However, practical and quantitatively predictable implementations have proved challenging, in part due to the difficulty in obtaining accurate models of transcription, translation, and resource competition in living cells, and the stochasticity inherent in cellular reactions. These challenges prevent first-principles rational design and parameter optimization. In this work, we exploit the versatility of an Escherichia coli cell-free transcription-translation (TXTL) to accurately design, model and then build, a synthetic biomolecular integral controller that precisely controls the expression of a target gene. To our knowledge, this is the first design of a functioning gene network that achieves the goal of making gene expression track an externally imposed reference level, achieves this goal even in the presence of disturbances, and whose performance quantitatively agrees with mathematical predictions. 

   more » « less
5. Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold

   Yaras, Can ; Peng Wang ; Zhihui Zhu ; Laura Balzano ; Qing Qu ( January 2022 , Advances in neural information processing systems)

   When training overparameterized deep networks for classification tasks, it has been widely observed that the learned features exhibit a so-called “neural collapse” phenomenon. More specifically, for the output features of the penultimate layer, for each class the within-class features converge to their means, and the means of different classes exhibit a certain tight frame structure, which is also aligned with the last layer’s classifier. As feature normalization in the last layer becomes a common practice in modern representation learning, in this work we theoretically justify the neural collapse phenomenon under normalized features. Based on an un-constrained feature model, we simplify the empirical loss function in a multi-class classification task into a nonconvex optimization problem over the Riemannian manifold by constraining all features and classifiers over the sphere. In this context, we analyze the nonconvex landscape of the Riemannian optimization problem over the product of spheres, showing a benign global landscape in the sense that the only global minimizers are the neural collapse solutions while all other critical points are strict saddle points with negative curvature. Experimental results on practical deep networks corroborate our theory and demonstrate that better representations can be learned faster via feature normalization. Code for our experiments can be found at https://github.com/cjyaras/normalized-neural-collapse. 

   more » « less