More Like this

1. Neural Representations Observed

   https://doi.org/10.1007/s11023-018-9459-4  

   Thomson, Eric; Piccinini, Gualtiero (March 2018, Minds and Machines)
2. Cognitive flexibility as the shifting of brain network flows by flexible neural representations

   https://doi.org/10.1016/j.cobeha.2024.101384  

   Cole, Michael W (June 2024, Current Opinion in Behavioral Sciences)

   Our ability to overcome habitual responses in favor of goal-driven novel responses depends on frontoparietal cognitive control networks (CCNs). Recent and ongoing work is revealing the brain network and information processes that allow CCNs to generate cognitive flexibility. First, working memory processes necessary for flexible maintenance and manipulation of goal-relevant representations were recently found to depend on short-term network plasticity (in contrast to persistent activity) within CCN regions. Second, compositional (i.e. abstract and reusable) rule representations maintained within CCNs have been found to reroute network activity flows from stimulus to response, enabling flexible behavior. Together, these findings suggest cognitive flexibility is enhanced by CCN-coordinated network mechanisms, utilizing compositional reuse of neural representations and network flows to flexibly accomplish task goals. 

   more » « less
3. WIRE: Wavelet Implicit Neural Representations

   Saragadam, V.; LeJeune, D.; Tan, J.; Balakrishnan, G.; Veeraraghavan, A.; Baraniuk, R.G. (January 2023, Proceedings IEEE Computer Society Conference on Computer Vision and Pattern Recognition)

   Implicit neural representations (INRs) have recently advanced numerous vision-related areas. INR performance depends strongly on the choice of activation function employed in its MLP network. A wide range of nonlinearities have been explored, but, unfortunately, current INRs designed to have high accuracy also suffer from poor robustness (to signal noise, parameter variation, etc.). Inspired by harmonic analysis, we develop a new, highly accurate and robust INR that does not exhibit this tradeoff. Our Wavelet Implicit neural REpresentation (WIRE) uses as its activation function the complex Gabor wavelet that is well-known to be optimally concentrated in space–frequency and to have excellent biases for representing images. A wide range of experiments (image denoising, image inpainting, super-resolution, computed tomography reconstruction, image overfitting, and novel view synthesis with neural radiance fields) demonstrate that WIRE defines the new state of the art in INR accuracy, training time, and robustness. 

   more » « less
4. WIRE: Wavelet Implicit Neural Representations

   Saragadam, V.; LeJeune, D.; Tan, J.; Balakrishnan, G.; Veeraraghavan, A.; Baraniuk, R.G. (January 2023, Proceedings IEEE Computer Society Conference on Computer Vision and Pattern Recognition)

   Implicit neural representations (INRs) have recently advanced numerous vision-related areas. INR performance depends strongly on the choice of activation function employed in its MLP network. A wide range of nonlinearities have been explored, but, unfortunately, current INRs designed to have high accuracy also suffer from poor robustness (to signal noise, parameter variation, etc.). Inspired by harmonic analysis, we develop a new, highly accurate and robust INR that does not exhibit this tradeoff. Our Wavelet Implicit neural REpresentation (WIRE) uses as its activation function the complex Gabor wavelet that is well-known to be optimally concentrated in space–frequency and to have excellent biases for representing images. A wide range of experiments (image denoising, image inpainting, super-resolution, computed tomography reconstruction, image overfitting, and novel view synthesis with neural radiance fields) demonstrate that WIRE defines the new state of the art in INR accuracy, training time, and robustness. 

   more » « less
5. Formation of Representations in Neural Networks

   Ziyin, Liu; Chuang, Isaac; Galanti, Tomer; Poggio, Tomaso (October 2024, Center for Brains, Minds and Machines (CBMM))

   Understanding neural representations will help open the black box of neural networks and advance our scientific understanding of modern AI systems. However, how complex, structured, and transferable representations emerge in modern neural networks has remained a mystery. Building on previous results, we propose the Canonical Representation Hypothesis (CRH), which posits a set of six alignment relations to universally govern the formation of representations in most hidden layers of a neural network. Under the CRH, the latent representations (R), weights (W), and neuron gradients (G) become mutually aligned during training. This alignment implies that neural networks naturally learn compact representations, where neurons and weights are invariant to task-irrelevant transformations. We then show that the breaking of CRH leads to the emergence of reciprocal power-law relations between R, W, and G, which we refer to as the Polynomial Alignment Hypothesis (PAH). We present a minimal-assumption theory demonstrating that the balance between gradient noise and regularization is crucial for the emergence the canonical representation. The CRH and PAH lead to an exciting possibility of unifying major key deep learning phenomena, including neural collapse and the neural feature ansatz, in a single framework. 

   more » « less