Over the last several decades, computational neuroscience researchers have proposed a variety of "biologically plausible" models of synaptic plasticity that seek to provide normative accounts of a variety of learning processes in the brain-these models aim to explain how modifications of cellular properties such as excitability and synaptic strengths can improve performance on a variety of important tasks for an organism, using only the information "locally" available to a real neuron in the brain. However, there is no consensus on what "biologically plausible" really means. This is due to a lack of any precise definition of locality, with individual studies adopting different, architecturespecific heuristics that are often not explicitly articulated; further, at an experimental level, the precise quantities available to neurons for computing synaptic updates are still up for debate [1,2], and likely vary across cell types [3,4,5] and brain regions [6,7,8,9,10].
As an example, some studies postulate that global neuromodulatory signals containing reward information could be available at individual synapses, in addition to the more traditional pre-and post-synaptic information typically required for Hebbian plasticity [11,12,1]. Within this broad class, individual studies differ in their assumptions about this reward information (e.g. is it completely global [13] or potentially distinct for different layers or regions [14]) and in the precise details about the pre-and post-synaptic signals available (e.g. is it spike-timing [15] that is available or spike-rate [16]?). Furthermore, there are many more signals that could theoretically be available at synapses: for example, other studies propose that individual pyramidal neurons could receive detailed error (or target) feedback information at their apical dendrites [17,18,19,2] which can be used for learning. Previous work has taxonomized different normative plasticity models according to these feedback signals [20]. Within this space of normative plasticity models, it can be difficult to identify which features of a given model constitute necessary, testable predictions that can be used for experimental verification-i.e., if this information were not available at a synapse, learning with the given algorithm would be impossible-and which features are due to arbitrary choices that may vary with more or less realistic neural network architectures.
As a consequence of these heterogeneous conceptions of locality and given the advancements in the field, it is arguably necessary to develop a clear and precise way to identify what variables synapses should have access to in a learning algorithm, whenever claims of locality are made. In this paper, we focus on formalizing this process. Our central contributions are as follows:
1. We develop an architecture-independent formal framework for locality that we term Splocality, which requires one to precisely specify the set of variables that synapses are assumed to have access to within a model: different choices of allowed variables for the set S produce different classes of locality. Importantly, this definition can be used for any learning algorithm applied to any stochastic neural network. 2. We use our framework to group existing plasticity models into different locality classes.
Intriguingly, we find that many different algorithms derive their locality from similar principles, even though they belong to different locality classes. 3. We show that our different locality classes make different experimental predictions, which makes it possible to identify which algorithms can be cleanly distinguished experimentally and which cannot.
Our ultimate goal is a formal definition of locality that makes it easy to identify which variables are locally available to synapses in a learning algorithm within any specific model architecture. However, this can be challenging. For example, when we consider a single-compartment neuron model it seems reasonable to assume that the postsynaptic voltage is locally available to synapses. In contrast, in a multi-compartment model, voltages can be different in each compartment, and synapses likely only have access to voltages within nearby compartments. Ideally, we would avoid a definition that simply requires us to list all of the variables that must be made available to the synapse, as that would carry little conceptual weight. However, we will begin by taking this approach, as it will help us to build the groundwork for a more conceptually useful definition in what follows.
Notation. We have X as a random vector in R N X denoting network variables (e.g. voltage, spikerate, inputs, etc.), and Θ as a random vector in R N Θ denoting network parameters (e.g. synaptic weights, spike threshold, etc.). We combine these as Z = [X, Θ]-the concatenation of the two vectors. We will use Z ∼ p(Z) to denote a random vector with support Ω ⊆ R N Θ +N X , but will also overload this notation to indicate a vector subject to either universal or existential quantification (e.g. ∃Z s.t. df (Z) dZ = 0) or a dummy variable in an expectation (e.g. E p(Z) [Z] = µ). We will use the subscript "̸ = i" to denote removal of the i th element of a vector; for example Z ̸ =i = [Z 1 , . . . , Z i-1 , Z i+1 , . . . , Z N X +N Θ ] ⊤ , and will occasionally use the more general notation Z ̸ =ν , where ν is a set of indices, to denote the vector with the associated set of elements removed.
We begin with a definition of locality that formalizes the concept of a set of variables, S k ⊆ Z, to which the k th synapse can have access in order to compute synaptic updates (e.g. some neural voltages, reward signals, etc.). We then consider the set of these sets across all synapses, S = {S 1 , . . . , S N Θ }. This inspires the following definition: Definition 2.1. Given parameters Θ ∈ R N Θ , random variables X ∼ p(X|Θ) ∈ R N X , and function f (•) : R N X +N Θ → R N Θ , the update function f (Z) is S-local with respect to some set S = {S k : S k ⊆ Z} if and only if ∀k, i:
In short, S-locality requires that if parameter update f k (Z) has a direct dependence on Z i , then Z i must be included in the set S k of "allowed" variables for that parameter. For example, in the classic approach to locality in a neural network, for any synapse W ij ∈ Θ, we might set S ij = {r i , r j } ⊆ Z, allowing any S-local parameter update under this definition to use pre-(r j ) and post-synaptic (r i ) firing rate information, and nothing else. If we wanted to allow a third factor such as a global neuromodulatory signal, R, to project to diffusely to every synapse in the network, as in rewardmodulated synaptic plasticity rules, we might take S ij = {r i , r j , R}. In this manner, for any model architecture, we can manually construct a set S ij that defines our notion of locality for each synapse.
While our definition is intuitive and flexible, it is cumbersome: for every parameter we have to explicitly define which variables are local. This means that S-locality is architecture-specific, requiring an evaluation of which S k is an acceptable set of variables for each synapse's update rule.
For example, if we switched from a single-to a multi-compartment model, we would now have to modify S ij to accept only the voltage of the compartment where the synapse is located. Ideally, our definition would make it obvious whether a learning rule is local in both models without having to redefine the list of which variables are local. To do that, we will use the structure of the distribution p(X, Θ) to construct a definition of locality that captures intuitions across network architectures.
Despite its ability to concretely define locality for any model, S-locality is architecture-specific and cumbersome: in this section, we will develop an alternative operationalized notion of locality, called p-locality (where p stands for "probability"). Our idea is to formulate locality in terms of adjacency in the computational graph underlying network dynamics. To do this we will leverage conditional dependencies in the joint distribution p(X, Θ) to generate an architecture-general concept of locality 3 . This is considerably more useful as both a conceptual tool and organizing principle for normative synaptic plasticity models. However, we will still call on S-locality later to formulate our final locality definition: Sp-locality.
Further, it is important to note that for this definition, as with S-locality, we do not restrict the functional form of parameter updates f k (•)-we only restrict which variables they are allowed to depend on. This is in contrast to previous, complementary work which has focused on defining allowable local computations and memory complexity constraints for 'biologically plausible' learning algorithms [21]. Our decision not to restrict f k (•) enables us to abstract away the details of 'allowed' subcellular computations, and will consequently enable us to make much more general statements about the locality properties of normative plasticity algorithms as a whole, without having to focus on the specifics of particular models of neural dynamics.
Definition 2.2. p-locality: Consider a probability distribution p(Z) over network activity variables and parameters, and an update function f : R N X +N Θ → R N Θ , mapping variables and parameters to parameter updates. Assume that ∂p(Zi|Z ̸ =i ) ∂Θ k and ∂f k (Z) ∂Zi exist for all Z ∈ Ω and i, k indices. The update function f (Z) is local with respect to p (p-local) if and only if ∀k, i:
To unpack this definition and render it more intuitive, as with S-locality, ∂f k (Z) ∂Zi indicates a test to see whether the kth index of f has a direct dependence on a particular variable Z i -we assume a b c ...
Figure 1: Common neural network graphical models and the permissible parameter updates for a synapse W ij under p-locality. a. Feedforward neural networks can be described as DAGs, so variables that are not children or coparents of W ij are excluded from parameter updates under p-locality (by Property 2.1). b. For fixed weights in a dynamic neural network like a recurrent neural network (RNN), p-locality excludes variables that are not children and coparents of W ij for all time points (also by Property 2.1). c. For an undirected graphical model like a Boltzmann machine, variables that are not neighbors of W ij are excluded under p-locality (by Property 2.2).
that f k (•) is differentiable primarily for simplicity, and differentiable approximations can always be substituted in practice if f k (•) happens to not be differentiable. If there is a dependency, then we want that associated random variable Z i to be considered local to Θ k . To measure this, we have selected the Fisher Information of p(Z i |Z ̸ =i ) with respect to Θ k . This measurement quantifies direct influence. For example, if Z i is a neuron downstream of neuron Z j , and Θ k is a synapse onto Z j , then by conditioning on Z j , Θ k is no longer able to have any effect on the conditional probability distribution of Z i through its influence on Z j . In other words, the statistical influence of Θ k on Z i is mediated through the influence of Z j on Z i , and thus it is indirect. By conditioning on Z j , we seal off one pathway of indirect influence. By conditioning on all Z ̸ =i , we seal off all pathways of indirect influence, which makes
It is worth stressing that p(Z) = p(X|Θ)p(Θ) is a joint distribution over both X and Θ, even though most algorithms operate only on the conditional distribution p(X|Θ). As a consequence, the marginal probability that we place on p(Θ) can be a free choice, which is useful for clarifying the assumptions in the model. For example, we will typically assume that synapses do not have any marginalized dependence on one another, i.e. p(Θ) = k p(Θ k ).
Having defined p-locality, we can now talk about its interesting properties, which will give intuitions for how it functions and which will figure prominently in our proofs of algorithms' locality. While any p-local update could also technically be reformulated as a S-local rule, by taking the set S to be the set allowed under p-locality, the conciseness of p-locality and its convenient mathematical properties will prove to be its principal benefits over S-locality. For proofs of these properties, see Appendices A-C. To see the power and generality of p-locality as a definition, we first provide two properties that show how one can immediately make statements about the locality of a learning algorithm by simply inspecting graphs associated with the model. Assume all quantities are defined as in Definition 2.2 and that, for certain properties, the joint density satisfies mild regularity constraints (see Appendix C.1). For the first two properties we further assume that p(Z) is strictly positive (see Appendix B).
Then we have the following:
By these properties, as long as the conditional dependencies of p(Z) can be summarized by an UG or DAG (two classes which subsume many modern neural network architectures, see Figure 1 for examples), we can identify many of the variables disallowed by p-locality. Therefore, for such a network, we can decide whether p-locality conforms to intuitions about biological plausibility as easily as we can for a hand-crafted set of allowed variables under S-locality; for most practical neural network architectures, we will see below that p-locality does behave intuitively. Thus, with p-locality we have a definition of locality that does not require an exhaustive list of the variables available to a synapse, but rather, which relies on the implicit computational structure of the model (i.e. p(Z)). Property 2.3. For any function b : R N
This demonstrates that p-local functions can be arbitrarily combined without the combination losing the p-local property. It also shows that p-locality places no restrictions on the functional form of parameter updates, so long as they are exclusively functions of variables allowed under p-locality. Property 2.4. For any function f (•) : R N X +N Θ → R N Θ , there exists a probability distribution p(Z) such that the random variable
This is intended as a cautionary note: unless p defines a neural network-like probability distribution, p-locality does not necessarily conform to intuitions about biological locality (see Appendix F for examples). That f (Z) is p-local for some probability distribution says nothing about a function f (Z); p-locality only becomes informative when we narrow our focus to probability distributions with biological relevance.
The following properties will turn out to be extremely important in Section 3: all algorithms that we survey below with provable p-locality properties will involve the score function or the derivative of the unnormalized energy function.
Property 2.5. The derivative of the log joint distribution ∂ log p(X,Θ) ∂Θ is p-local.
Property 2.6. For a probability distribution given by p(Z) = 1 Z exp (-E(Z)), where Z is a normalizing constant, the expression ∂ ∂Θ E(Z) is p-local. Property 2.7. If the parameter marginal distribution factorizes as p(Θ) = k p(Θ k ), i.e. the parameters are independent from one another, then the score function ∂ log p(X|Θ) ∂Θ is p-local.
Property 2.8. For a mixture distribution p 12 (Z, γ) = p 1 (Z) γ p 2 (Z) 1-γ p(γ) for some binary variable γ ∈ {0, 1} with nonzero probabilities, if f (Z) is p 1 -local (or equivalently p 2 -local), then f (Z) is p 12 -local.
Intuitively, this feature holds because mixture distributions introduce more dependencies between parameters Θ and random variables X. Therefore, p-locality under a mixture of probability distributions is more permissive than p-locality under any single one of its constituent probability distributions.
To make our definition of p-locality more concrete, we will now explore how it functions for a common network architecture. Consider a simplified linear-nonlinear feedforward neuron model with L layers of neurons r (l) (X = r (1) , ..., r (L) ) connected by synaptic weights W (l) (Θ = W 1 , ..., W (L) ), with additive Gaussian noise at each layer. Conditioned on our feedforward weight matrices and stimuli s, the probability distribution of neural firing rates is given by:
p(r
where for notational simplicity we have taken r (0) = s, h(•) is a pointwise nonlinearity, and W (l) i:
corresponds to the ith row of W (l) . As discussed above, most algorithms operate on the conditional distribution, p(X|Θ), so we only define a prior over parameters for the purposes of assessing plocality. To ensure that our joint distribution is a directed graphical model, and to ensure that we have no dependencies between parameters, we will assume that the parameters for the network are all independently distributed, i.e. p(W) = i,j,l p(W
where W (l) ij is the synapse between a b c Figure 2: Different versions of Sp-locality. a. Rp-locality functions as p-locality, but also allows synapses to use rewards R for parameter updates, corresponding to an additional diffuse neuromodulatory signal. b. γp md -locality allows synapses to use p m -local variables and p d -local variables, as well as the global gating variable γ. For the Wake-Sleep algorithm, a γp md -local probability distribution could correspond to a network of pyramidal neurons receiving p m -related synapses in their apical compartments (orange) and p d -related synapses in their basal compartments. The variable γ controls whether p d or p m synapses affect cellular dynamics. c. Backpropagation and its approximations are e i p-local, which functions like p-locality, except it additionally permits a neuron-specific error signal to be used for parameter updates. Typically, e is constructed by sequentially propagating error signals backwards through a feedforward network.
postsynaptic neuron i and presynaptic neuron j in layer l. To make our choice of conditional probability distribution more concrete (see Figure 1a), we can see that neural firing rates for this model can be sampled by:
where η (l) ∼ N (0, 1). This corresponds to an ordinary multilayer perceptron neural network with noise added at every layer. We can use this model to get intuition for how this choice of probability distribution p constrains the variables allowed for an update for a particular parameter W (l) ij . Now, our probability distribution corresponds to a directed graphical model with dependencies given by Eq. 3, so by Property 2.1, we know that variables that are not parents, coparents, or children of W (l) ij violate p-locality if included in the parameter update for W (l) ij . Figure 1 summarizes the variables allowed under p-locality for such a DAG. W
i , whose coparents are given by W (l) ik s.t. k ̸ = j (all other synapses onto r (l) i ), and r (l-1) (all presynaptic neurons). This shows that updates for synapses in our simple feedforward network cannot depend on neurons that are in layers > l or < l -1 while maintaining p-locality. We also found that dependencies on neurons within the postsynaptic layer l for indices k ̸ = i were also not permissible. As a concrete example, by Property 2.7, we found that the score function of p(X|Θ) is p-local if the parameters are assumed to be independently distributed. Our example satisfies this criterion. Its score function is given by: ∂ log p(r|s,W) 1) . By inspection, this function only requires information about the post-and presynaptic neurons, as well as the summed input to the postsynaptic neuron, V i . This simple example clearly outlines why p-locality aligns with expected notions of locality. More importantly, the use of p-locality also has the benefit of generalizing cleanly to alternative network architectures. For example, if we were to instead inspect a multilayer recurrent neural network (Figure 1b), we would see that for W (l) ij the firing rates r (l) i (t) and all presynaptic neural firing rates would be permissible for all time steps, because the activities at each timestep are children or coparents of W (l) ij . For another classical example of a biologically plausible probability distribution corresponding to an UG (as in Figure 1c.), see Appendix D for a discussion of a linear continuous Boltzmann machine.
In almost all situations, a given class of plasticity algorithms will produce updates that are almost p-local, in that synapses are required to have access to some small number of global state variables that a biological system could in principle project diffusely throughout a network, but which would not be allowed under generic p-locality (e.g. reward, which is usually produced far downstream of any given synapse in a neural network). For this eventuality, we introduce a notion called Sp-locality, which is best considered a hybrid of S-locality and p-locality. Definition 2.3. Given parameters Θ ∈ R N Θ and random variables X ∈ R N X , concatenated as Z = [X, Θ] ⊤ , and a function f (Z) : R N X +N Θ → R N Θ , the update function f (Z) is Sp-local with respect to probability distribution p(Z) and some set S = {S k : S k ⊆ Z} if and only if ∀k
, where h k is an arbitrary function and
The value of this definition is that it negotiates a compromise between the architecture-generality of p-locality, and the flexibility of S-locality. Obviously if we take S k = Z ∀k, all functions are S-local, and likewise, if we define p sufficiently generally, all functions are p-local (Property 2.4). However, if some class of optimization algorithms operating on a predefined probability distribution p are provably guaranteed to produce parameter updates that are Sp-local for some small set S, then we will have obtained a very concise description of the types of information required by that algorithm.
3 Applying Sp-locality to normative plasticity models Given a probability distribution over a neural network states and parameters, p(X, Θ), which could be experimentally motivated or even observed, it is natural to ask which learning algorithms respect locality principles in the way parameter updates are made during learning. Alternatively, if we denote an algorithm which operates on a probability distribution p(X, Θ) to output an update f (Z) by A(p(X, Θ)) = f (Z), where f (Z) : R N X +N Θ → R N Θ , we can ask which probability distributions p(X, Θ) and variable collections S leave f (Z) Sp-local. This is one way Sp-locality could potentially facilitate experimentally testing biologically plausible learning models.
To demonstrate the utility of our approach, in this section we characterize the Sp-locality properties of a wide variety of algorithms that have historically been used to produce normative models of synaptic plasticity, as summarized in Table 1. Though our framework covers many algorithms, here we expand on three prototypical algorithms in detail, focusing on features that are relevant for our discussion of the algorithms' locality properties. More algorithmic details as well as theorem proofs are provided in Appendix E. Schematics depicting biological interpretations for the types of Sp-locality discussed below are provided in Figure 2.
REINFORCE [13], also known as policy gradient learning, produces reward-modulated Hebbian parameter updates for neural networks [1] similar to the one discussed in Section 2.4. Here we will show that REINFORCE is Rp-local, meaning that it is Sp-local where S = {S k = R ∀k}, which assumes each synapse has access to a global scalar reward signal R that needs to be delivered diffusely to all synapses in a network.
As an example, consider the network defined by Eq. 3. The REINFORCE update for this network, for a single sample of the network state r and reward R, is given by:
where again 1) . This is just the scalar multiplication of the reward signal R with the score function; because the score function is p-local, the full update is Rp-local (see Appendix E for more detail).
Whereas REINFORCE has been used to model reinforcement learning in neural networks, the Wake-Sleep algorithm has been used to model unsupervised learning [22]. It assumes that a neural network has two modes of operation, controlled by a scalar variable γ that determines whether the network is in inference mode ('wake', γ = 1) or in generative mode ('sleep,' γ = 0). In all, neural activity samples are drawn from the mixture distribution given by p md (r, γ|W, M) = p d (r|W) γ p m (r|M) (1-γ) p(γ), where p(γ) defines the probability of sampling from the wake phase (p d ; d corresponds to 'data') or the sleep phase (p m ; m corresponds to 'model'), and W corresponds to the feedforward weights while M correspond to feedback weights. For conciseness, in all subsequent sections we will denote the mixture distribution p 1 (Z) γ p 2 (Z) 1-γ p(γ) by M ix (p 1 (Z), p 2 (Z)). As an example, building on the feedforward network defined by Eq. 3 by adding a generative feedback pathway, we have neural dynamics given by:
where h w (•) corresponds to the 'wake' nonlinearity, and h s (•) corresponds to the 'sleep' nonlinearity. Under this formulation, γ gates, for all neurons, whether activity is driven by the feedforward or feedback pathways; there are several hypotheses for how this could be implemented at a neuronal level, including global neuromodulatory or inhibitory gating of the apical and basal dendrites of pyramidal neurons in the cortex [23] (Fig 2b).
The Wake-Sleep algorithm updates M to fit p m as a generative model of the network's input data, and updates W to fit p d to perform approximate inference with respect to that generative model, in a manner similar to variational autoencoders (VAE) [24,25]; unlike VAE training with backpropagation, however, the Wake-Sleep algorithm is γp md -local, meaning
As a concrete example, the Wake-Sleep parameter update for W (l) ij for a single sample of the state r from the network above is given by:
where
i: r (l-1) ; a similar update holds for M. Similar to REINFORCE, this parameter update is only the combination of a scalar variable γ and the score function, and is consequently γp mdlocal (see Appendix E for the proof). Interestingly, Wake-Sleep is not the only algorithm to obey this form of locality: several other normative plasticity models, including Boltzmann machine learning [26], equilibrium propagation [27], and impression learning [23] have essentially the same p-locality properties. However, some of these have additional requirements on the probability distribution that make them less biologically plausible. For example, Boltzmann machine learning is γp md -local only for distributions that can be captured with energy-based models (which typically require weight symmetry).
Here we provide a characterization of the locality of the backpropagation algorithm [28], which as we will show using our definitions, is only a local algorithm with biologically implausible assumptions [20]. This demonstration is primarily important because the Sp-locality properties of several biologically plausible backpropagation approximations, including feedback alignment [29], weight mirror [30], and Burstprop [31] satisfy a similar notion of Sp-locality, but using more biologically realistic assumptions about S (see Table 1 and Appendix E). Derivations for the backpropagation algorithm and its approximations require more stringent network assumptions (here we assume a feedforward multilayer perceptron network as in Section 2.4), because the algorithm itself is not well-defined for arbitrary probabilistic network architectures. i is given by their respective gradient approximations.
i p-locality, where S = {S
i ∀i, j, l}, includes more restrictive biological assumptions than the one provided by Rp-locality and γp md -locality, because it relies heavily on the neuron-specific error ê(l) i , which must be then accounted for using physiological mechanisms. For our network architecture given by Eq. 3, the functional form of all backpropagation approximations listed in Table 1 is given by:
where êi is an algorithm-specific approximation of dL dri . Similar to γp md -local algorithms, recent work has suggested that this error signal could be propagated through networks by a top-down error signal relayed to the apical dendrites of pyramidal neurons [31] (see Fig. 2c). Thus, these algorithms make specific biological predictions, driven by the need for ê(l) i to be accessible to each synapse.
We have constructed a formal definition of locality, Sp-locality, that combines set-based and probabilistic graphical model-based formalisms (S-locality and p-locality, respectively). Our goal was standardizing and operationalizing notions of locality, and we also demonstrated the usefulness of these definitions in characterizing normative plasticity models. Notably, this approach enabled the identification of distinct classes of Sp-locality (Table 1) which subsume learning algorithms spanning more than thirty years of research, from Boltzmann machines [26] to modern incarnations like equilibrium propagation [27]. Importantly, our framework can be applied to any neural network architecture, incorporating networks that are recurrent [16,32,23,27,26,33], spiking [34,12], and multilayer [29,22,30] into a single framework. This is thanks, in large part, to the architecture-generality of p-locality, which abstracts away specific details of network models, defining locality instead in terms of statistical dependencies-this makes our framework much more powerful than simpler linguistic descriptions of locality. As we discuss in Appendix F, this abstraction comes at the cost of requiring researchers to specify a distribution, p, that is grounded in biological constraints. We note that for non-biological probability distributions the variables allowed to be "local" under p-locality can violate standard intuitions; however, for standard neural networks like those discussed in Section 2.4 and Appendix D, p-locality behaves as one would expect, with "local" learning rules being those that use pre-and postsynaptic activity.
Table 1 organizes existing normative plasticity models into distinct classes. The first locality class, Rp-locality, requires individual synapses to have access to reward information; this class encompasses many reward-modulated Hebbian plasticity models [1]. The second class, p (m) -locality, encompasses predictive coding and its variations [33,35], and aligns with more traditionally Hebbian plasticity updates (often at the cost of unrealistic network architectures; see Table 1). The third class, γp mdlocality, requires neural networks to have two distinct modes of operation (e.g. the 'wake' and 'sleep' modes in the Wake-Sleep algorithm) gated by a scalar global variable γ. Thus, γp md -locality requires synapses to have access to the value of γ to compute updates, and subsumes many algorithms that require generative feedback pathways for learning [22,23]. The final class of algorithms are ê(l) i plocal for feedforward neural networks, where ê(l) i is a neuron-specific approximate error signal, and includes feedback alignment [29], weight mirror [30], and Burstprop [31], as well as temporal variants of these algorithms [36,37]. These four categories-Rp-locality, p (m) -locality, γp md -locality, and ê(l)
i -locality-subsume, to our knowledge, almost all existing normative plasticity models. Because these plasticity models are sometimes only constructed for specific network architectures (which may violate known facts about the brain), we also specify the broadest class of network architectures for which we have successfully derived a model's Sp-locality properties (proven in Appendix E). As noted in Appendix F, 'biological plausibility' requires both network architectures and parameter updates to be in line with existing experimental evidence, while Sp-locality provides a formal framework for assessing only the latter of these two factors. We therefore stress that Sp-locality properties are only relevant for neuroscience if they can be determined for biologically plausible network architectures. As a consequence, classes of network architectures that encompass a broad range of biologically plausible network models (such as DAGs) are desirable, whereas algorithms that are only viable for implausible architectures (e.g. those that imply weight symmetry or weight transport) are undesirable.
Interestingly, the distinct classes of locality also delineate distinct sets of experimentally testable predictions for plasticity. Critically, Sp-locality abstracts away details that are not important for testing predictions, and helps identify important features of learning algorithms. For example, the REINFORCE (Eq. 6) and Wake-Sleep (Eq. 8) algorithms are equivalent with respect to the 'p' in Sp-locality, which tells us that the set of allowed local variables under p-locality itself (e.g. pre-and post-synaptic information) are not the best targets for experimental testing between these algorithms, and instead the focus must be on the variables included in 'S' (a reward signal R versus a clamping variable γ that switches the network between different modes of synaptic plasticity and neural activity). Thus, by identifying whether plasticity is modulated by reward, neuron-specific error teaching signals, transitions in network dynamics, or none of the above, one may narrow the space of possible candidate plasticity models down to Rp-, ê(l) i -, γp md -, or p (m) -locality, respectively. These distinct phenomena are necessary features of the learning algorithms in question for any neural network architecture used and are consequently good targets for experimental testing. In contrast, other features of parameter updates are more about specific architectural choices, e.g. switching from a rate-based network [16] to spiking [15], from feedforward [22] to recurrent [32], or singleto multi-compartment [38]. Thus, beyond clarifying and categorizing the locality properties of normative plasticity models, our hope is that our framework will also support experimental efforts to differentiate between such models, helping the field to focus in on the most important predictions to test experimentally.
To gain intuition for how p-locality functions, we will introduce another notion of locality, called Markov locality, which will use the language of Markov blankets. We will prove that under relatively relaxed conditions p-locality and Markov locality are equivalent. This will allow us to relate the notion of locality to various graph structures commonly used to represent probability distributions, and will be a key step in proving Properties 2.1 and 2.2.
We start by defining the Markov boundary, M(X, S), of a random variable X contained in a set of random variables S, as a minimal set such that p(X|S) = p(X|M(X, S)). The Markov boundary defines a minimal set of variables such that, conditioned on these variables, conditioning on no additional random variables in S changes the probability of X [39]. Similarly, we define the Markov blanket, M (X, S) for X in S as any set of variables such that conditioning on M (X, S), makes X conditionally independent from all other variables [39]. In this way, the Markov boundary is a Markov blanket but not all blankets are boundaries. Definition A.1. Markov locality: Given probability distribution p(Z) and function f : R N X +N Θ → R N Θ , the update function f (Z) is Markov-local with respect to the distribution p over Z if and only if ∀ k:
Markov locality requires that the set of variables used in the parameter update f k (Z) is a subset of the Markov boundary of the parameter itself. A Markov boundary can be thought of as the set of variables that 'locally' communicate with the parameter Θ k , thus providing a natural measure of locality.
Importantly, for Markov-locality to be of use, we would like the Markov boundaries of random variables in the model of interest to be unique. Without this requirement there will be ambiguity, for a given p, in terms of which updates are considered local and which are not. To guarantee this, we ask that the conditional independence relationships implied by p satisfy four properties, commonly referred to as graphoid properties [39,40]. A sufficient condition for these to hold is that the distribution have a strictly positive density (see Appendix for more details B). With this, and some mild regularity assumptions, we can prove the following equivalence between Markov locality and p-locality: Theorem A.1. Assume all quantities are as in A.1, that the conditional independence relationships implied by p(Z) satisfy the four graphoid properties given in Section B, and that mild regularity assumptions are satisfied by the joint distribution (see Section C.1). Then Equation 2 holds if and only if Equation A.1 also holds.
Proof. This proof relies on Lemma A.1, proved below.
We wish to prove Eq. 2 ⇐⇒ Eq. A.1. It suffices to show the following:
Using the contrapositive for the left and right implications separately shows that Equation A.2 is equivalent to
which means that it suffices to prove Equation A.3 for the proof. Observe that
which follows from the regularity assumptions. From here, the proof follows by Lemma A.1.
Lemma A.1. Let X, Θ, Z, k and i, and p be defined as in Theorem A.1. Then
Proof. First, observe that
by the chain rule. By applying the fundamental theorem of calculus to this derivative, which we can do by the assumption of differentiability on R, we find that p(
where we have also used that a probability distribution integrates to 1, and that p(Z k |Z ̸ ={i,k} ) will be equal to zero outside Ω. Because Z ∈ Ω is arbitrary, from the above Z i is independent of Z k given the other random variables in Z. Using Lemma B.1 (which we can do by assumption of the graphoid properties), we see that if
by Eq. A.6 -A.8, Z i / ∈ M(Z k , Z). Conversely, if we start with the assumption that Z i / ∈ M(Z k , Z), we immediately get Z k / ∈ M(Z i , Z), by Lemma B.1, and see that p(Z i |Z ̸ =i ) must not be a function of Z k for all Z; thus, the derivative w.r.t. Z k is equal to zero for Z ∈ Ω. Applying Equation A.6 completes the proof.
In this section we compile several properties, definitions, and results on Markov boundaries and Probabilistic Graphical Models (PGMs) that underlie Theorem A.1, and Properties 2.1 and 2.2. We begin by setting up notation. Let us assume that we have a joint probability distribution, P , over a set of random variables S, and that W , X, Y , Z ⊂ S, and U , V , T ∈ S. We use X ⊥ ⊥ Y |Z to mean that the set of random variables X is independent of the set Y given set Z, and assume that the reader is familiar with the notion of a directed graph, an undirected graph, and graph separation. If a set X contains only a single random variable U then we abuse notation and write U in place of X.
The following four properties-known as the graphoid properties (or axioms-see e.g. [39,40])-are useful for getting well-behaved Markov boundaries and in assigning graphical representation to probability distributions: Definition B.1. Pseudo-graphoid properties:
Importantly, it is known that these properties are satisfied when we have a density p that is strictly positive w.r.t. to its base product measure [40]. Here, measure is used in a measure theoretic sense; e.g. we have assumed throughout the paper that the base measure is simply a product of a multidimensional Lebesgue measure over R k1 , for some k 1 , and a counting measure over N k2 or some subset of N k2 , for some k 2 . Roughly speaking, this positivity property means that there are no purely deterministic relationships between variables.
The key result that we use to guarantee that the Markov boundaries we discuss in the paper are well-defined is given in [39]. We state a paraphrased and shortened version below for completeness: Theorem B.1. Theorem 4, Chapter 3 in [39]: every P with conditional independence relations satisfying the four pseudo-graphoid properties has a unique Markov boundary for each X.
We now add two more simple results on Markov boundaries, used in the proof of Theorem A.1: Lemma B.1. If P has conditional independence relations satisfying the four pseudo-graphoid properties we have:
• for every U ∈ S, M(U, S) is contained in every Markov blanket of U .
These follow simply from the graphoid properties so we omit the proof.
Lastly, we make specific what we mean when we say that a graph is an undirected or directed graphical model for a distribution. Definition B.2. Let G be an undirected graph where each node corresponds to a random variable in S. We say that G is an Undirected Graph (UG) for P if whenever X and Y are separated by Z in G, X ⊥ ⊥ Y |Z is true under P . Note that this corresponds to the notion of I-map in [39].
Definition B.3. Let G d be a directed graph with each vertex corresponding to a random variable in S. We say that G d is a Directed Graph for P if the variable under P corresponding to any node in the graph is conditionally independent of all variables corresponding to nodes that are non-descendants given the variables corresponding to parents. This is equivalent to G d satisfying the Markov condition described in Definition 1.9 of [41].
For the first two properties we assume the requirements of Theorem A.1 are satisfied. For all properties except 2.3, 2.4, and 2.8 we assume p satisfies mild regularity constraints (see Section C.1).
Proof. By Theorem A.1 we get that ∂f k (Z) ∂Zi can only be non-zero on the unique Markov boundary of Z i if it is p-local. By the definition of a DAG, the parents, co-parents, and children of Z i form a Markov blanket for it (see e.g. [41] Th. 2.13). By Lemma B.1 the boundary is included in all Markov blankets so ∂f k (Z) ∂Zi can only be non-zero on some subset of the parents, co-parents, and children of Z i . Property 2.2 Assume that G defines an Undirected Graph (UG) for p.
Proof. As above, by Theorem A.1 we get that ∂f k (Z) ∂Zi can only be non-zero on the unique Markov boundary of Z i if it is p-local. A UG for a distribution is an I-map for it, and conditioning on the neighbours in an I-map renders a node independent from the other nodes in the graph by definitionthus the neighbours form a Markov blanket. By Lemma B.1 the Markov boundary is included in every blanket so ∂f k (Z) ∂Zi can only be non-zero on some subset of the neighbours in the UG.
, where f and g are p-local and h k is differentiable, b(Z) is also p-local.
Proof. Suppose that ∂b k (Z) ∂Xi ̸ = 0. We need to show that
∂Xi ̸ = 0, we have:
This implies that either ∂f k (Z) ∂Zi ̸ = 0 or ∂g k (Z) ∂Zi ̸ = 0 (or both). No matter which is true, by virtue of the p-locality of f and g, we have the consequence:
which concludes our proof. This demonstrates that p-local functions can be more or less arbitrarily combined without the combination losing the p-local property.
Property 2.4 For any function f (•) : R N X +N Θ → R N Θ , there exists a probability distribution p(Z) such that the random variable f (Z) with Z ∼ p(Z) is p-local.
Proof. We can prove this property by construction. Take p({X i : ∂f (X) ∂Xi ̸ = 0}|Θ) = N (Θ T Θ, I), i.e. the distribution of every variable contained within f has mean parameter dependence on all Θ variables. The probability distributions for all other variables Z are otherwise unconstrained. Then for all i such that ∂f (X) ∂Xi ̸ = 0, we have:
Property 2.5 The derivative of the log joint distribution ∂ log p(X,Θ) ∂Θ is p-local.
Here, it's more useful to work with the equivalent (contrapositive) requirement for p-locality, i.e., we need to show ∀k, i:
Proof. First, we see that:
where the first implication follows from the fact that the Fisher Information integral is effectively a weighted sum of elements, each of which is ≥ 0. If the function on the right were nonzero for some Z, then the Fisher Information would also be nonzero. Assuming that log p has differentiable partial derivatives, we can interchange the order of differentiation, giving:
which concludes the proof.
Property 2.6 For a probability distribution given by p(Z) = 1 Z exp (-E(Z)), the expression
Proof. The proof is almost identical to the proof for Property 2.5. From Property 2.5, we have that:
Using our definition of p, we have:
which concludes the proof.
Property 2.7 If the marginal parameter distribution factorizes as p(Θ) = k p(Θ k ), i.e. the parameters are independent from one another, then the score function ∂ log p(X|Θ) ∂Θ is p-local.
Proof. Again, we make heavy use of Property 2.5, which states:
It is important to note that the left-hand equation only holds true if Z i ̸ = Θ k : under p-locality, an update equation for parameter Θ k can always include its own value. So for the remainder of the proof we will assume that Z i ̸ = Θ k . Now, log(p(X|Θ)) = log p(Z) -log p(Θ). We also have:
where for the last equality we have used the assumption that Z i ̸ = Θ k . These two equations collectively imply:
which concludes the proof.
Property 2.8 For a mixture distribution p 12 (Z, γ) = p 1 (Z) γ p 2 (Z)
1-γ p(γ) for some binary variable γ ∈ {0, 1} with nonzero probabilities, if f (Z) is p 1 -local (or equivalently p 2 -local), then f (Z) is p 12 -local.
Proof. We again work with the contrapositive definition of p-locality, observing that:
where the third implication follows from the fact that if the sum of two nonnegative quantities is zero, then both quantities are zero, and the final implication holds from the p 1 -locality of f (Z).
For several of these properties we enforce mild regularity constraints on the density. This is because we want the integral of the squared score being equal to zero to imply that the score itself is equal to zero. A sufficient condition for this is that the joint density function and partial derivatives w.r.t. Θ k ∀k are, for every fixed value of Z's discrete elements, continuous functions of Z's continuous elements.
Consider the following example of a simplified linear recurrent neuron model with synaptic weight matrices W. The joint distribution is given by:
where W is assumed to be a symmetric matrix. To see why this probability distribution is relevant for neuroscience, we first note that E is a linear, continuous analog of the Hopfield energy function, which is also used for discrete-valued Boltzmann machines. There are two critical differences between this probability distribution and the linear feedforward network explored in Section 2.4: first, this distribution corresponds to the undirected graphical model shown in Figure 1c, as opposed to the DAG shown in Figure 1a; second, the marginal distribution is not a free parameter that we can choose with convenient factorization properties if we want our joint distribution to give us a version of p-locality that corresponds with our concept of biological locality. For undirected graphical models, one typically is required to define the joint distribution first, and compute conditional distributions explicitly through Bayes theorem or approximate through some form of MCMC sampling. In our case, we can see that p(r|W) corresponds to the steady-state distribution of a stochastic differential equation using E to perform Langevin sampling:
where here B i (t) corresponds to uncorrelated Brownian noise injected into the system. These stochastic sampling dynamics correspond to a noisy linear recurrent network. Therefore, p(r|W) corresponds to the steady-state stimulus response distribution of a linear recurrent network.
Let's ask: for which neural indices k can we have ∂ ∂r k f Wij (r) ̸ = 0 so that the function f is still p-local? For f to remain p-local, we would need
For the definition of p-locality to conform to our intuitions about biological locality, we would expect the only allowable variables to be the pre-and postsynaptic neurons r i and r j -we will show that including any other variable will violate p-locality. To see why, suppose k ̸ = i, j. Note that we can decompose E as:
Under this decomposition, E k has no dependency on W ij , and E ̸ =k has no dependency on r k . Now we're in a position to demonstrate that for any choice of k such that k
Because the conditional distribution has no dependency on W ij , then the Fisher Information is also 0, which concludes the demonstration:
To summarize, for our simple example, any parameter update for W ij that depends on the activity of any neuron r k that is not the pre-or postsynaptic neuron (k ̸ = i, j) cannot be p-local. Alternatively, since this is an undirected graphical model, we can also inspect its corresponding graph (summarized in Figure 1c.). To verify that the graph in Figure 1c. corresponds to our network, observe that our probability distribution factorizes according the cliques of the graph [42] as follows:
Looking at the graph, we can verify by inspection that the only neighbors of W ij are r i and r j , which confirms our detailed analysis by Property 2.2.
Proof. The REINFORCE derivation proceeds as follows: suppose that we have some probabilistic formulation of a neural network and incoming sensory stimuli p(X|Θ) and some probabilistic reward function p(R|X) dependent on the stimuli and neural responses. We want to maximize expected reward:
If we want to modify our parameters Θ in order to improve performance, we take steps in an approximation of the direction of the gradient of the objective E[R].
where in this last step we have employed a Monte Carlo approximation of the expectation, where R (k) and X (k) ∼ p(R, X). This update function:
However, as we know, this update is the product of a score function with a marginal parameter distribution that we have assumed factorizes, which we know to be p-local by Property 2.7, with a scalar reward R. In this case, one could postulate that reward information is projected broadly to many synapses in the neural network via a neuromodulatory pathway (Figure 2a). We see that f (R, X) = h(R, g(X)) if we take h(a, b) = a×b and g(X) = ∂ ∂Θ log p(X|Θ); we further see that g is p-local, and hence f (R, X) is by Definition 2.3 Rp-local. This might seem contrived, because any function is Sp-local for some sufficiently broad choice of S. However, we have shown here that the REINFORCE algorithm is Rp-local for any choice of p with a marginal parameter distribution that factorizes (an easy constraint to satisfy for directed graphical model architectures). This means that we can make any of a huge variety of neural network or probabilistic model choices and still have the REINFORCE algorithm obey the same notion of locality, without having to modify our definition post-hoc.
MLE is a highly popular machine learning method and the fundamental basis for several subsequent normative plasticity algorithms. This algorithm involves fitting a model, p m (X|Θ), to an empirical data distribution, p d (X).
Proof. We proceed by first deriving the MLE update function. The objective function for maximum likelihood estimation is given by the KL divergence between an empirical data distribution, p d (X) and a probabilistic model of the data p m (X|Θ). We have:
We want to minimize this objective function, which we do by gradient descent:
where X k ∼ p d (X), and in the last approximate equality we have used a Monte Carlo sampling integral approximation. This update exclusively contains the score function of p m , so by Property 2.7, the update is p m -local.
MLE estimation runs into difficulties when attempting to fit latent variable models, (e.g. when
, where X h are latent variables within the model distribution that 'explain' observed data X o . Latent variable models are extraordinarily powerful, and appear in computational neuroscience in a variety of forms, including but not limited to factor analysis, hidden Markov models, and Kalman Filters [43]; for these models, we will take X = [X o , X h ]. Instead of performing explicit MLE, when fitting latent variable models one usually resorts to some variant of the Expectation-Maximization (EM) algorithm [44]. Here, we show that a particular variant of the EM algorithm, called Generalized EM (GEM) [45], is p m -local in the same way as MLE.
GEM gains computational benefits by substituting (by any of a variety of methods) an approximate posterior distribution p d (X h |X o ) for the true, but typically intractable, model posterior p m (X h |X o ) via minimizing a variational free energy [45]. However, GEM is not just a convenient model-fitting algorithm: in subsequent sections, we will show that the p m -locality of GEM explains why several popular normative plasticity algorithms produce biologically plausible updates.
the GEM algorithm minimizes an upper bound (the variational free energy).
where the inequality follows from the positivity of the KL divergence. Here, p d (X h |X) is an approximate inference distribution. Different choices of how this distribution is selected/optimized can produce very different learning algorithms, with varying degrees of biological plausibility. Obviously, the loss is minimized with respect to
(where Θ 0 = Θ prior to optimization wrt Θ). This choice corresponds to GEM [45]. For now, we will not concern ourselves with how p d (X h |X) is selected-instead, we will focus on the locality properties of gradient updates of this loss with respect to Θ m .
Having packaged hidden and observed variables together (X = [X o , X h ]), our derivation proceeds exactly the same as for MLE:
where as with the MLE update, X k ∼ p m (X|Θ). This update is p m -local for the same reason that the MLE update is.
As an additional note, if one takes the approximate posterior p d (X h |X o ) to be given by:
where δ(•) indicates a Dirac delta distribution and Xh (X o ) indicates a set of observation-dependent mean parameters, then we recover the predictive coding family of algorithms [35]. Typically, in this context for a given observed stimulus
) is estimated by reparameterization and gradient descent with respect to Xh (X o ) (a mean parameter that is observation-dependent), which-for clever choices of p m -loosely resembles the dynamics of a recurrent neural network relaxing to a stimulus-conditioned equilibrium state [33]. After estimating Xh (X o ), parameters Θ are updated as in GEM. Therefore, the derivation above also applies to predictive coding algorithms, which are consequently also p m -local. Theorem E.4. If p(Θ) = k p(Θ k ), the PC update given by A P C (p m (X|Θ), p d (X)) is p m -local.
Unlike the previous three examples, which only require sampling from the p d distribution and only calculate parameter updates according to the p m distribution, the Wake-Sleep algorithm parameterizes both distributions and jointly samples from a mixture of the two distributions across its 'wake' and 'sleep' phases. As we will see, this will mean that the Wake-Sleep algorithm will end up being γp md -local, where γ is the binary variable that controls whether the system is in its 'wake' or 'sleep' phase.
k ) , the Wake-Sleep estimator given by A W S (p m (X|Θ),
Proof. Our updates use a similar loss to the GEM algorithm, namely we take:
where ∆Θ W S is given by:
Here, γ k , X ∼ p md (X, γ|Θ, Θ (d) ), whereas for GEM, we sampled only from p d . Because each term of this update is 0 if γ k ̸ = 1, this update is still an unbiased estimate of the gradient [46] and is effectively equivalent to the GEM update, except that it allows the system to alternate between sampling from p m and p d . This alternation is useful because it will allow also calculating parameter updates for ∆Θ (d) . For Θ (d) , we optimize the reverse KL-divergence 4 ; by a directly analogous derivation to Eq. E.12, the update is given by:
Now, these updates contain the score function for both p m and p d , as well as the scalar mixture variable γ. As a consequence, both updates are γp md -local, via Properties 2.7 and 2.8 (slightly more precisely, ∆Θ is γp m -local, and ∆Θ (d) is γp d -local).
The impression learning parameter update [23] is closely related to the Wake-Sleep parameter update, and is consequently also γp md -local. What distinguishes IL from WS is the use of rapid alternations in the gating signal γ t within a single trial with T time steps. Here, X = [X 0 , ..., X T ], γ = [γ 0 , ..., γ t ] and p md (X|γ, Θ)
is a mixture distribution in which γ t alternates between 0 and 1, sampling from either p m or p d at the time step t, respectively.
Theorem E.6. If p(Θ) = k p(Θ k ), the impression learning estimator given by A IL (p m (X|Θ), p d (X|Θ)) is γp md -local, where p md = M ix p m (X|Θ), p d (X|Θ (d) ) p(Θ).
Proof. Similar to WS, the update is given by:
where in this last equality, X ∼ p md (X), and we are performing a single-sample gradient approximation. It is worth noting that unlike in the Wake-Sleep algorithm, here γ t is not a constant throughout time. Instead, γ t alternates between 0 and 1 with 'phase duration' K, i.e. γ t+1 = 1 -γ t if mod (t, K) = 0, and γ t+1 = γ t otherwise. The IL update is the score function of p md (X|1 -γ, Θ), which has identical dependencies to the score function of p md (X|γ, Θ) (only a change from γ → 1 -γ has occurred). Therefore, if p(Θ) = k p(Θ k ), this parameter update is γp md -local by Property 2.7.
While the GEM learning update is provably p m -local, it is also predicated on the assumption that the parameter marginal distribution factorizes as p(Θ) = k p(Θ k ), which as we note in Appendix D can be difficult to ensure for even simple undirected graphical models. An extension of the GEM algorithm, we can show that the CD algorithm is γp md -local (as opposed to just p m -local) under less restrictive assumptions. The cost of this is that CD learning usually requires costly MCMC sampling from both the posterior distribution p m (X h |X o , Θ) and the full joint distribution p m (X h , X o |Θ).
Theorem E.7. The CD update given by A CD (p m (X|Θ), p d (X)) is γp md -local, where p md = M ix (p m (X|Θ), p d (X)) p(Θ).
As mentioned above, for GEM the most natural choice for p d (X h |X) is given by p m (X h |X, Θ 0 ). As we demonstrated, parameter updates calculated according to this rule will be γp md -local, but there are two important caveats. First, for GEM to produce biologically plausible updates, we still need a biological system that can sample from p m (X h |X, Θ 0 ). Second, it is important to remember that we are only guaranteed that the score function is guaranteed to be p-local if the marginal parameter probability distribution factorizes as p(Θ) = k p(Θ k ). For a DAG, it may be difficult to satisfy the first condition without approximation (given by the Wake-Sleep algorithm, for instance), whereas for an UG, it may be difficult to satisfy the second condition, as we saw in Section D. To make our update γp md -local for an undirected graphical model like the Boltzmann machine, we will require an extra step that we outline here to use the energy function Property 2.6 rather than the score function Property 2.7.
Having committed to working with an undirected graphical model, instead of sticking to the original GEM update, here we break apart the probability distribution as:
where X k is sampled from p md . Now, by Property 2.6 and Property 2.8, this update is γp md -local. This is the Boltzmann machine learning algorithm [26], where the clamped and unclamped phases are alternated between stochastically [46]; for our previous linear Boltzmann example, in which X = r and Θ = W, the derivative of the energy function (Eq. D.3) with respect to a parameter W ij is r i r j , demonstrating that updates correspond to two phases of updates: one in which r o is clamped to a data distribution given by p d (r o ) for some r o ⊆ r and Hebbian updates are positive, contrasted with an unclamped phase in which updates are negative. Note that for this model approximate sampling from a posterior distribution p m (r ̸ =o |r o , W)p d (r 0 ) is no more difficult than sampling from the joint distribution: one simply holds r o fixed to an environmental data sample and performs Langevin sampling on all other variables.
Though the derivation and setup of the equilibrium propagation algorithm [27] is very different from Contrastive Divergence, the functional form of the derived update is very similar. While equilibrium propagation typically operates on deterministic networks, here we will provide our derivation for the stochastic version with an energy function defining a joint distribution over Θ and X (as in Section D), which is somewhat more straightforward to fit into the p-locality framework.
Suppose that we have a probabilistic energy-based model whose energy function is given by:
where ϵ ∈ {0, β}, where β ≪ 1, where L(Z) is the loss function that the parameter updates are optimizing. This can be thought of as a 'soft-clamped' system, in which nonzero ϵ pushes the system towards slightly better performance. Intuitively, the EP parameter update attempts to change network dynamics so that the unclamped system is nudged towards the slightly better performing soft-clamped system. Then we have the following theorem: Theorem E.8. The EP update given by A EP (p(X|Θ, is γp md -local, where p md = M ix (p(X|Θ, ϵ = β), p(X|Θ, ϵ = 0)) p(Θ).
The parameter update for equilibrium propagation is given by:
where for the final equality we are using a sampling-based approximation in which we are sampling from X k ∼ p(X|Θ, ϵ k ), and ϵ k /β ∼ Bernoulli(0.5). This is almost identical to the Contrastive Divergence update, except that rather than clamping neural activities to a target output, they are slightly biased towards better performance. Because this is the combination of the derivative of the energy function with a mixture variable γ = ϵ k /β, by Properties 2.6 and 2.8, this update is γp md -local where p md = M ix (p(X|Θ, ϵ = β), p(X|Θ, ϵ = 0)) p(Θ).
E.9 Winner-take-all STDP While Contrastive Divergence uses MCMC sampling to approximate the GEM update, Nessler et al. [48] use a particular generative model for which the posterior can be analytically calculated and resembles a simple winner-take-all neural circuit. Then, the authors derive their STDP parameter update as an approximation to the GEM algorithm. Because of this, one might imagine that the derived STDP update may, like the GEM algorithm, be p m -local. We will see below that this is the case.
First, we define the generative model p m used in the paper:
where Z is the normalizing constant and the r and s vectors contain binary random variables. Furthermore, in the network only one neuron r is assumed to fire at any given time (r i ̸ = 0 ⇔ r k = 0 ∀k ̸ = i). The inference distribution, conditioned on a stimulus s can be calculated as follows:
This probability distribution can be interpreted as a kind of winner-take-all computation, dominated by the neuron with the greatest input current [48]. Samples from this distribution are used to compute the weight updates:
where c is a positive constant. We now prove the following theorem assessing the p-locality of this distribution: Theorem E.9. The STDP update given by A ST DP (p(X|Θ)) is γp m -local.
Proof. To see this, first note that the gradient of the energy function E(r, s, W) with respect to the parameters is p m -local by Property 2.6. Therefore, any variables contained within this will also be permissible under p m -locality.
For W ij , we have: ∂E(r, s, W) ∂W ij = -r i s j , (E.34) so that we know r i and s j are permissible for ∆W ij under p m -locality; further, the value of a parameter itself, W ij , is always allowed under p m -locality. These are the only variables on which ∆W ij depends, so this update is p m -local.
For W i0 , we have:
so that we know r i is permissible for ∆W i0 . By the same reasoning, this update is also p m -local. Since all updates are therefore p m -local, we may conclude that the full algorithm is p m -local. However, this proof does not have the same level of generality as for the previous algorithms, because the algorithm is only defined for a single winner-take-all network model. i is given by their respective gradient approximations.
Proof. As a first step, for clarity purposes we will demonstrate that backpropagation [28], is not p-local with respect to the simple feedforward neural network architecture we outlined above; we will subsequently demonstrate that it and its approximations do satisfy a particular notion of Sp-locality. For a scalar loss function L(r L ) and a single parameter W (l) ij , the backpropagation gradient is given by the negative gradient of the loss with respect to the parameter of choice, using the reparameterization trick [25,24] to take for a single sample of {r (l) } l=0:L the mean (noiseless) mapping from r (l-1) → r (l) to be r(l) = h(W (l) r (l-1) ), so that by Eq. 5, we have r (l) = r(l) +ση (l) . By the chain rule, gradient descent gives:
36) Based on our analysis in Section 2.4, this update function is clearly not p-local, because the update depends on firing rates r (k) for k > l. However, while backpropagation is not in general p-local, any algorithm can be Sp-local: for example, if we take S = Z, then by Definition 2.3, any parameter update can contain any variable in the graphical model p(Z). Taking S = Z is inherently vacuous: Sp-locality is only conceptually useful if we can cleanly reduce the number of variables included in S for a broad set of biologically relevant neural architectures. Fortunately, for backpropagation operating on a feedforward neural network governed by Equation 5, we do not need to include every variable in the network. By Equation E.36, we see for our example feedforward neural network that: ∆W (l) ij ∝ -dL dr (l) i dr (l) i dW (l) ij , (E.37) where dL dr (l) i
= dL(r (L) )
dr (L) L k=l+2
dr (l+1) (r (l) ) dr
is the derivative of the global loss function with respect to the individual mean neuron activation r(l) i . Interestingly,
-the derivative of the mean parameter for neuron r i -is a function of only the parents of r (l) , which are therefore the coparents of W (l) ij . To verify that this particular component of the weight update is p-local, we can compare its dependencies to the score function, which is in this case p-local by Property 2.7. As noted in Section 2.4, the score function is given by: 1) . Because the score function is p-local, any variables that it depends on are permissible for p-local updates. The score function depends on r (l) i and r (l-1) , whereas
ij is e (l) i p-local. Importantly, this does not mean that backpropagation is biologically plausible: this notion of locality provides no clues as to how e (l) i could be calculated or approximated in the brain, and an explicit calculation of gradients could not be possible due to the weight transport problem [20]. There are many recent models that account for how e (l) i could be approximated by an approximate credit assignment signal ê(l) i involving either random feedback synapses that project errors backwards through the network (feedback alignment [29]) or feedback synapses that dynamically adjust through local synaptic mechanisms so that ê(l) i provides an unbiased approximation (e.g. weight mirror or Kolen-Pollack alignment [30], and BurstProp [31]). Each of these algorithms decomposes into a nonlocal feedback term ê(l) i and a p-local term in exactly the same way, and are consequently ê(l)
i p-local.
Consider an autonomous recurrent neural network whose directed acyclic graphical model is provided by the following equations (we will ignore stimulus-dependence for notational simplicity):
where p(r(0)) corresponds to some initial distribution of activity states. This probability distribution of firing rates corresponds to the following neural sampling dynamics:
where η ∼ N (0, 1). For this model, we have the following theorem: Theorem E.11. If p(Θ) = k p(Θ k ) and p(X|Θ) is defined by Eq. E.39 (with X = r and Θ = W), the RTRL update for W ij with a loss L(r(T )), given by A RT RL (p(r|W), L(r)) is eJp-local, where e = ∂L(r(T ) ∂r(T ) , and J = {J(t) = ∂r(t,r) ∂r(t-1,r) }.
Proof. The directed graphical model corresponding to these dynamics is depicted in Figure 1b: as with backpropagation, we will use the score function for our graphical model to identify permissible variables. For a single synapse, the score function is given by:
where V i (t) = W i: r(t -1). Thus p-local parameter updates for W ij may include W i: , r i (t) and {r k (t -1) : W ik ̸ = 0} ∀t. We will now compare these allowed variables to the RTRL update. As with backpropagation, we take r(t, r) = h(Wr(t -1)), so that r(t) = r(t, r) + ση. The RTRL update minimizes a loss L(r(T )) via the chain rule [49,21]:
this second equation provides a recursive update equation which can be stored online as a trial progresses. The g(r(t -1)) term is p-local, because it appears in Eq. E.43. However, ∂r(t,r) ∂r(t-1,r) , an N × N Jacobian matrix, is not p-local, since it depends on all neurons in the network r(t -1) as well as all parameters W-neurons that do not directly synapse onto neuron r i and weights W kl for k ̸ = i are excluded from p-local updates by Property 2.1 according to the DAG defined by Eq. E.39. Furthermore, as we have seen with backpropagation, in general the credit assignment signal ∂L(r(T )) ∂r(T ) is not p-local. Therefore, to characterize the Sp-locality of RTRL, we will have to proceed similarly to backpropagation, and define auxiliary variables to include in the set S.
As with backpropagation, we define the auxiliary random variable e = ∂L(r(T )) ∂r(T ) . Because we have found the Jacobians to also violate p-locality, we will also define the set of auxiliary variables J = {J(t) = ∂r(t,r) ∂r(t-1,r) }. With these auxiliary variables, we can see that ∂r(t,r) ∂Wij is Jp-local ∀t, and consequently, the RTRL update is eJp-local. This is, of course, not biologically plausible in any way. The set J allows the parameters to have access to the state of the entire network, at all time points, even from neurons that do not have any direct connections to the neuron whose synapse is being updated. Further, the entire error vector e is required to compute the update. This is even less plausible than backpropagation, which only required access to e i . However, the RTRL update is an important baseline for analyzing the locality properties of other learning algorithms that are constructed as approximations of it, namely e-prop and RFLO. E.12 e-prop Theorem E.12. If p(Θ) = k p(Θ k ) and p(X|Θ) is defined by Eq. E.39 (with X = r and Θ = W), the e-prop update for W ij with a loss L(r(T )), given by A ep (p(r|W), L(r)) is e i p-local, where e i = ∂L(r(T ) ∂ri(T ) .
Proof. For e-prop [37], we will also consider networks constructed according to Eq. E.39. The update is almost identical to the RTRL update, but several terms will be discarded, allowing the update to be e i p-local, as opposed to the eJp-local update given by RTRL. The update is as follows:
This update combines the neuron-specific credit assignment signal e i with a local 'eligibility trace' ∂ri(t,r) ∂Wij which performs approximate credit assignment by filtering and summing coactivity between neuron i and neuron j across timesteps. It is worth noting that the particular functional form of this eligibility trace is determined by our simplified RNN dynamics (Eq. E.39), which causes coactivity from previous timesteps to decay exponentially in proportion to the magnitude of the autapse W iialternative neural network dynamics using continuous-time dynamics, or adaptive neural firing thresholds may alter the functional form of the eligibility trace [37], but do not fundamentally alter the p-locality properties of the update. Now, we only need to show that the eligibility trace is p-local.
As with RTRL, we can observe that h ′ (V i (t)) and r j (t -1) both appear in the score function for our RNN (Eq. E.43) for all timesteps, as does W ii ⊂ W i: . Because the score function is p-local, we know that these variables are all allowed under p-locality. The eligibility trace only depends on these terms, from both the current time step and, recursively, from previous timesteps. Therefore, the eligibility trace is p-local. The e-prop update is a multiplication between e i and the eligibility trace, so by Def. 2.3 the update is e i p-local.
The RFLO update [36] is nearly identical to the e-prop update, except we replace e i with an approximate credit assignment signal êi (which replaces symmetric feedback weights with random connections, similar to Feedback Alignment).
The update is given by:
r) ∂W ij (E.49) ∂r i (t, r) ∂W ij = h ′ (V i (t))W ii ∂r i (t -1, r) ∂W ij + h ′ (V i (t))r j (t -1). (E.50)
Following exactly the same reasoning as with e-prop, we may show that this update is êi p-local. Theorem E.13. If p(Θ) = k p(Θ k ) and p(X|Θ) is defined by Eq. E.39 (with X = r and Θ = W), the RFLO update for W ij with a loss L(r(T )), given by A RF LO (p(r|W), L(r)) is êi p-local, where e i = ∂L(r(T ) ∂ri(T ) .
The FOLLOW algorithm [50] is defined in terms of a particular continuous-time LIF circuit with postsynaptic potential kernels. For simplicity, we will focus our analysis on a linear version of the same circuit, disregarding the dynamic postsynaptic potentials and input stimuli. Disregarding these features does not affect the p-locality properties of the FOLLOW algorithm, but it would certainly degrade its performance on tasks.
The network dynamics are given by:
r(t), e(t)) + ση (E.51) = (1 -∆t τ )r(t) + 1 τ Wr(t) + kW f b e(t) ∆t + ση, (E.52)
where W f b is an N × N o random feedback weight matrix, k is a positive constant, and e(t) is an N o -dimensional error feedback vector delivered at every timestep, with N o the number of output dimensions. Because we are in continuous time, we will assume that η ∼ N (0, ∆t).
Similar to RTRL, we can write the probability distribution for the network as:
p(r|e, W) = p(r(0))
t N i=1 p(r i (t + ∆t)|r(t), e(t), W) (E.53) p(r i (t + ∆t)|r(t), e(t), W) ∼ N (h(r(t), e(t)), σ 2 ∆t), (E.54)
where p(r(0)) is some initial distribution of firing rates. Further, we can assume that the distribution of errors at timestep t + ∆t has any arbitrary distribution p(e(t + ∆t)|r(t)).
The update for weight W ij is given by:
r j (t). (E.55) Therefore, only the postsynaptic error current and presynaptic input are necessary to update the weights for a given synapse in this type of network. Below, we will show that this update is p-local. Theorem E.14. If p(Θ) = k p(Θ k ) and p(X|Θ) is defined by Eq. E.53 (with X = {r, e} and Θ = W), the FOLLOW update for W ij , given by A F W (p(r, e|W)) is p-local. Proof. To see that this is true, we need only show that the variables included in ∆W ij are subsets of the variables included in the score function ∂ log p(r,e|W) ∂Wij . These variables are permissible for p-local updates by Property 2.7. The score function is given by: ∂ log p(r, e|W) ∂W ij = t ∂ log p(r(t + ∆t)|r(t), e(t), W) ∂W ij + t ∂ log p(e(t + ∆t)|r(t)) ∂W ij (E.56) = t ∂ log p(r i (t + ∆t)|r(t), e(t), W) ∂W ij (E.57) = t r i (t + ∆t) -r i (t) + ∆t τ -r i (t) + W i: r(t) + kW f b i: e(t) ∆tσ 2 r j (t).
(E.58) Therefore, for weight ∆W ij , the permissible variables include: r i (t) ∀t, any r k (t) such that W ik ̸ = 0 (∀t), any e k (t) such that W f b ik ̸ = 0 (∀t), and the parameters W i: and W f b i: . The parameter update requires only r j (t) and W f b i: e(t), which is a subset of these permissible variables. Therefore, the update is p-local.
It is very important to clarify the exact relationship between p-locality and biological plausibility. Except for some network-wide variables that a theoretician may decide to allow through a particular choice of Sp-locality, we have generally shown that p-locality is overly permissive, in that a particular choice of p may allow parameter updates to include variables that an individual synapse may not have access to. Furthermore, p-locality does not restrict the network architecture defined by p to be biologically plausible. The best way to interpret p-locality is as follows: if p(X, Θ) defines a biologically plausible architecture and an algorithm A is p(X, Θ)-local, then the parameter update provided by A(p(X, Θ)) will be biologically plausible. There are many network architecture and parameter update combinations that may be biologically plausible without being proven p-local (e.g. explicit approximations to backpropagation [29,30,31]), as there are many combinations that are p-local without being biologically plausible. Below, we will show two important instances in which p-locality does not properly diagnose a combination of network architecture and parameter update as biologically implausible.
The first example is pervasive in neural network models of the brain: networks frequently violate Dale's law, which states that neurons in a neural network are (for the most part [51]) either excitatory (outgoing weights are positive) or inhibitory (outgoing weights are negative), but not both. In fact, in the simple network example we have provided (Section 2.4), neural firing rates are not constrained to be strictly positive, and outgoing synaptic weights are not sign-constrained. For this biologically implausible architecture, p-locality defines which variables are allowed to be included in individual parameter updates in a way that is sensible (allowing only variables involving the postsynaptic firing rate and the firing rates of all pre-synaptic neurons), but it says nothing about the aforementioned implausibilities of the network architecture. Similarly, the linear Boltzmann machine example provided in Appendix D does not constrain firing rates to be positive, and requires symmetric weights (W ij = W ji ), which could not satisfy Dale's law while allowing connections between an inhibitory neuron i and an excitatory neuron j (W ij > 0 while W ji < 0 ⇒ W ij ̸ = W ji ).
These examples illustrate an important fact: p-locality focuses on the plausibility of updates given an architecture that has been predetermined to be acceptable. However, it is worth noting that if we were to impose these additional constraints for the proposed networks, the accepted variables determined by p-locality would not change.
Another important caveat when working with p-locality is that the random variables Z have to correspond to the relevant biophysical quantities of interest, e.g. neural firing rates X and synaptic weights Θ. If this is not the case, then p-locality can easily defy standard notions of biological plausibility. For instance, if we define a probability distribution in terms of a 3-layer neural network: p(X|Θ) ∼ N (r 2 (r 1 (r 0 (Θ))), σ 2 ), (F.1)
Then the score function of this distribution is given by: ∂p(X|Θ) ∂Θ = d log p(X|Θ) dr 2 dr (2) dr (1) dr (1) dr (0)
This equation depends on X, which is the output of the network, even though Θ parameterizes r(0) . Therefore, if the random variables had been defined in as in Section 2.4, then this update would not be p-local. However, because it is the derivative of the score function, for any independent marginal p(Θ), it is p-local for this choice of random variables. Therefore, it is important when working with an algorithm such as Wake-Sleep or REINFORCE, that one chooses a conditional probability distribution p(X|Θ) that captures biologically plausible dependencies. When this is not done, as in [52,53], the resulting updates have no correspondence to synaptic plasticity rules.
Note that this fact does not undermine the utility of p-locality as a concept. Our proofs for algorithms in Appendix E apply for any p(X|Θ), as long as p(Θ) factorizes to i p(Θ i ). Therefore, algorithms that have universal p-local properties will respect the variable dependencies implied by p(X|Θ) whether this distribution is plausible or not, which means that the algorithms will respect variable dependencies for all plausible network architectures.
We define dynamics in terms of probability distributions because many canonical normative plasticity models are stochastic; deterministic computation graphs are a limit case of our approach.
A rigorous discussion of why this optimization process is sensible is beyond the scope of this manuscript. See[22,47] for more detail.