An official website of the United States government
Understanding the generalization of deep neural networks is one of the most important tasks in deep learning. Although much progress has been made, theoretical error bounds still often behave disparately from empirical observations. In this work, we develop margin-based generalization bounds, where the margins are normalized with optimal transport costs between independent random subsets sampled from the training distribution. In particular, the optimal transport cost can be interpreted as a generalization of variance which captures the structural properties of the learned feature space. Our bounds robustly predict the generalization error, given training data and network parameters, on large scale datasets. Theoretically, we demonstrate that the concentration and separation of features play crucial roles in generalization, supporting empirical results in the literature. The code is available at https://github.com/chingyaoc/kV-Margin.
more »
« less
IMMIGRATE: A Margin-Based Feature Selection Method with Interaction Terms
Traditional hypothesis-margin researches focus on obtaining large margins and feature selection. In this work, we show that the robustness of margins is also critical and can be measured using entropy. In addition, our approach provides clear mathematical formulations and explanations to uncover feature interactions, which is often lack in large hypothesis-margin based approaches. We design an algorithm, termed IMMIGRATE (Iterative max-min entropy margin-maximization with interaction terms), for training the weights associated with the interaction terms. IMMIGRATE simultaneously utilizes both local and global information and can be used as a base learner in Boosting. We evaluate IMMIGRATE in a wide range of tasks, in which it demonstrates exceptional robustness and achieves the state-of-the-art results with high interpretability.
more »
« less
- PAR ID:
- 10310391
- Date Published:
- Journal Name:
- Entropy
- Volume:
- 22
- Issue:
- 3
- ISSN:
- 1099-4300
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract Offshore meteoric groundwater (OMG) has long been hypothesized to be a driver of seafloor geomorphic processes in continental margins worldwide. Testing this hypothesis has been challenging because of our limited understanding of the distribution and rate of OMG flow and seepage, and their efficacy as erosive/destabilizing agents. Here we carry out numerical simulations of groundwater flow and slope stability using conceptual models and evolving stratigraphy—for passive siliciclastic and carbonate margin cases—to assess whether OMG and its evolution during a late Quaternary glacial cycle can generate the pore pressures required to trigger mechanical instabilities on the seafloor. Conceptual model results show that mechanical instabilities using OMG flow are most likely to occur in the outer shelf to upper slope, at or shortly before the Last Glacial Maximum sea‐level lowstand. Models with evolving stratigraphy show that OMG flow is a key driver of pore pressure development and instability in the carbonate margin case. In the siliciclastic margin case, OMG flow plays a secondary role in preconditioning the slope to failure. The higher degree of spatial/stratigraphic heterogeneity of carbonate margins, lower shear strengths of their sediments, and limited generation of overpressures by sediment loading may explain the higher susceptibility of carbonate margins, in comparison to siliciclastic margins, to mechanical instability by OMG flow. OMG likely played a more significant role in carbonate margin geomorphology (e.g., Bahamas, Maldives) than currently thought.more » « less
-
null (Ed.)Several recent results provide theoretical insights into the phenomena of adversarial examples. Existing results, however, are often limited due to a gap between the simplicity of the models studied and the complexity of those deployed in practice. In this work, we strike a better balance by considering a model that involves learning a representation while at the same time giving a precise generalization bound and a robustness certificate. We focus on the hypothesis class obtained by combining a sparsity-promoting encoder coupled with a linear classifier, and show an interesting interplay between the expressivity and stability of the (supervised) representation map and a notion of margin in the feature space. We bound the robust risk (to $$\ell_2$$-bounded perturbations) of hypotheses parameterized by dictionaries that achieve a mild encoder gap on training data. Furthermore, we provide a robustness certificate for end-to-end classification. We demonstrate the applicability of our analysis by computing certified accuracy on real data, and compare with other alternatives for certified robustness.more » « less
-
At convergent margins, plates collide producing a subduction process. When an oceanic plate collides with a continental plate, the denser (i.e., oceanic) plate subducts beneath the less dense (continental) plate. This process results in the transportation of carbon and other volatiles into Earth’s deep interior and is counterbalanced by volcanic outgassing. Sampling deeply-sourced seeps and fumaroles throughout a convergent margin allows us to assess the processes that control the inventory of volatiles and their interaction with the deep subsurface microbial communities. The Andean Convergent Margin is volcanically active in four distinct zones: the Northern Volcanic Zone, the Central Volcanic Zone, the Southern Volcanic Zone and the Austral Volcanic Zone, which are each characterised by significantly different subduction parameters like crustal thickness, age of subduction and subduction angle. These differences can change subduction dynamics along the convergent margin, possibly influencing the recycling efficiency of carbon and volatiles and its interaction with the subsurface microbial communities. We carried out a scientific expedition, sampling along a ~800 km convergent margin segment of the Andean Convergent Margin in the Central Volcanic Zone of northern Chile, between 17 °S and 24 °S, sampling fluids, gases and sediments, in an effort to understand interactions between microbiology, deeply-sourced fluids, the crust, and tectonic parameters. We collected samples from 38 different sites, representing a wide diversity of seep types in different geologic contexts. Here we report the field protocols and the descriptions of the sites and samples collected.more » « less
-
According to the efficient coding hypothesis, neural populations encode information optimally when representations are high-dimensional and uncorrelated. However, such codes may carry a cost in terms of generalization and robustness. Past empirical studies of early visual cortex (V1) in rodents have suggested that this tradeoff indeed constrains sensory representations. However, it remains unclear whether these insights generalize across the hierarchy of the human visual system, and particularly to object representations in high-level occipitotemporal cortex (OTC). To gain new empirical clarity, here we develop a family of object recognition models with parametrically varying dropout proportion , which induces systematically varying dimensionality of internal responses (while controlling all other inductive biases). We find that increasing dropout produces an increasingly smooth, low-dimensional representational space. Optimal robustness to lesioning is observed at around 70% dropout, after which both accuracy and robustness decline. Representational comparison to large-scale 7T fMRI data from occipitotemporal cortex in the Natural Scenes Dataset reveals that this optimal degree of dropout is also associated with maximal emergent neural predictivity. Finally, using new techniques for achieving denoised estimates of the eigenspectrum of human fMRI responses, we compare the rate of eigenspectrum decay between model and brain feature spaces. We observe that the match between model and brain representations is associated with a common balance between efficiency and robustness in the representational space. These results suggest that varying dropout may reveal an optimal point of balance between the efficiency of high-dimensional codes and the robustness of low dimensional codes in hierarchical vision systems.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.3390/e22030291
