Search for: All records

The process of training deep learning models produces a huge amount of meta-data, including but not limited to losses, hidden feature embeddings, and gradients. Model diagnosis tools have been developed to analyze losses and feature embeddings with the aim to improve the performance of these models. However, gradients, despite carrying rich information that is potentially relevant for model interpretation and data debugging, have yet to be fully explored due to their size and complexity. Each single gradient has a size as large as the number of parameters of the neural net - often measured in the tens of millions. This makes it extremely challenging to efficiently collect, store, and analyze large numbers of gradients in these models. In this work, we develop MetaStore to fill this gap. MetaStore leverages our observation that storing certain compact intermediate results produced in the back propagation process, namely, the prefix and suffix gradients, is sufficient for the exact restoration of the original gradient. These prefix and suffix gradients are much more compact than the original gradients, thus allowing us to address the gradient collection and storage challenges. Furthermore, MetaStore features a rich set of analytics operators that allow the users to analyze the gradients for data debugging or model interpretation. Rather than first having to restore the original gradients and then run analytics on top of this decompressed view, MetaStore directly executes these operators on the compact prefix and suffix structures, making gradient-based analytics efficient and scalable. Our experiments on popular deep learning models such as VGG, BERT, and ResNet and benchmark image and text datasets demonstrate that MetaStore outperforms strong baseline methods from 4 to 678x in storage costs and from 2 to 1000x in running time. 

Abstract Eastern boundary upwelling systems (EBUSs) host equatorward wind-driven near-surface currents overlying poleward subsurface undercurrents. Various previous theories for these undercurrents have emphasized the role of poleward alongshore pressure gradient forces (APFs). Energetic mesoscale variability may also serve to accelerate undercurrents via mesoscale stirring of the potential vorticity gradient imposed by the continental slope. However, it remains unclear whether this eddy rectification mechanism contributes substantially to driving poleward undercurrents in EBUS. This study isolates the influence of eddy rectification on undercurrents via a suite of idealized simulations forced either by alongshore winds, with or without an APF, or by randomly generated mesoscale eddies. It is found that the simulations develop undercurrents with strengths comparable to those found in nature in both wind-forced and randomly forced experiments. Analysis of the momentum budget reveals that the along-isobath undercurrent flow is accelerated by isopycnal advective eddy momentum fluxes and the APF and retarded by frictional drag. The undercurrent acceleration may manifest as eddy momentum fluxes or as topographic form stress depending on the coordinate system used to compute the momentum budget, which reconciles these findings with previous work that linked eddy acceleration of the undercurrent to topographic form stress. The leading-order momentum balance motivates a scaling for the strength of the undercurrent that explains most of the variance across the simulations. These findings indicate that eddy rectification is of comparable importance to the APF in driving poleward undercurrents in EBUSs and motivate further work to diagnose this effect in high-resolution models and observations and to parameterize it in coarse-resolution ocean/climate models.