Search for: All records

Batch normalization (BN) is a popular and ubiquitous method in deep learning that has been shown to decrease training time and improve generalization performance of neural networks. Despite its success, BN is not theoretically well understood. It is not suitable for use with very small mini-batch sizes or online learning. In this paper, we propose a new method called Batch Normalization Preconditioning (BNP). Instead of applying normalization explicitly through a batch normalization layer as is done in BN, BNP applies normalization by conditioning the parameter gradients directly during training. This is designed to improve the Hessian matrix of the loss function and hence convergence during training. One benefit is that BNP is not constrained on the mini-batch size and works in the online learning setting. Furthermore, its connection to BN provides theoretical insights on how BN improves training and how BN is applied to special architectures such as convolutional neural networks. For a theoretical foundation, we also present a novel Hessian condition number based convergence theory for a locally convex but not strong-convex loss, which is applicable to networks with a scale-invariant property. 

People experience the consequences of a rapidly changing Arctic as the combined effects of physical conditions; responses of biological resources; impacts on infrastructure; decisions influencing adaptive capacities; and both environmental and international influences on economics and well-being. Living and innovating in Arctic environments over millennia, Indigenous Peoples have evolved holistic knowledge providing resilience and sustainability. Indigenous expertise is augmented by scientific abilities to reconstruct past environments and to model and predict future changes. Applying the combined understanding of Indigenous and scientific experts will be important if decision makers (from communities to governments) are to help mitigate and adapt to a rapidly changing Arctic. Considerable discussion among diverse collaborators suggests that addressing unprecedented Arctic environmental changes requires hearing one another, aligning values, and collaborating across knowledge systems, disciplines, and sectors of society.