A place field is canonically described as a localized region in an environment where a hippocampal neuron's firing rate is maximal and robust across trials (O'Keefe & Dostrovsky, 1971;O'Keefe, 1978). Classically, each neuron has a unique spatial receptive field such that the population activity can describe an animal's allocentric position within the environment (Moser et al., 2015). Ablation studies demonstrate that the hippocampal representation is useful for learning to navigate to new targets (Morris et al., 1982;Packard & McGaugh, 1996;Steele & Morris, 1999). Importantly, each field's spatial selectivity evolves with experience in a new environment before stabilizing in the later stages of learning (Frank et al., 2004). Specifically, a high density of place fields emerge at reward locations (Gauthier & Tank, 2018;Lee et al., 2020;Sosa et al., 2023), place fields elongate backward against the trajectory (Mehta et al., 1997;Priestley et al., 2022), and individual field's spatial selectivity continues to change or "drift" even when animals demonstrate stable behavior (Geva et al., 2023;Krishnan & Sheffield, 2023;Kentros et al., 2004;Mankin et al., 2012;Ziv et al., 2013). Although disparate mechanisms have been proposed to model these phenomena, a framework that can unify these phenomena and clarify their computational role remains elusive. Here, we propose a single normative model for spatial representation learning , based on the hippocampal CA1 given its role in representing salient spatial information (Dong et al., 2021;Dupret et al., 2010). Our key contributions are:
• We develop a two-layered reinforcement learning model to study spatial representation learning by place fields (Fig. 1A). The first layer is a population of Gaussian radial basis functions that transform continuous spatial information into a relevant representational substrate or "state", which feed into an actor-critic network in the second layer that uses these representations to learn actions that maximize cumulative discounted reward. Besides the actor and critic weights, each place field's amplitude, center and width is optimized by the Temporal Difference error.
• Our model recapitulates three experimentally-observed neural phenomena during task learning: (1) the emergence of high place field density at rewards, (2) elongation of fields against the trajectory, and (3) drifting fields that do not affect task performance.
• We analyze the factors that influence these representational changes: a low number of fields drives greater spatial representation learning, the mean population firing rate reflects the value of that location, and increasing 1 noise magnitude during field parameter updates causes a monotonic decrease in population vector correlation but non-monotonic change in behavior.
• We demonstrate that optimizing place field widths and amplitudes enhances reward maximization and policy convergence. However, field parameter optimization alone is insufficient for learning to navigate to new targets. Introducing noisy field parameter updates improves new target learning, suggesting a functional role for noise.
Anatomically constrained architecture for navigation.
Learning to navigate involves the hippocampus encoding spatial information and its strong glutamatergic projections to the striatum (Lisman & Grace, 2005;Floresco et al., 2001). The ventral and dorsal regions of the striatum are associated with value estimation and stimulus-response associations, functioning similarly to a critic and an actor, respectively (Niv, 2009;Joel et al., 2002;Houk et al., 1994).
Additionally, dopamine neurons in the Ventral Tegmental Area influence plasticity in the striatal synapses (Reynolds et al., 2001;Russo & Nestler, 2013). This anatomical insight has led to the design of a biologically plausible navigation model, where place fields connect directly to an actor-critic framework, and synapses are modulated by the TD error (Arleo & Gerstner, 2000;Foster et al., 2000;Frémaux et al., 2013;Brown & Sharp, 1995;Kumar et al., 2022). Recent evidence shows direct dopaminergic projections to the hippocampus to modulate place cell activity, strengthening the case for navigation models with adaptive place fields (Palacios-Filardo & Mellor, 2019;Krishnan et al., 2022;Kempadoo et al., 2016;Sayegh et al., 2024). How upstream information from the entorhinal cortex influences place field representations for policy learning needs clarity (Fiete et al., 2008;Bush et al., 2015). As new experiments challenge the canonical definition that a place cell only has one place field (Eliav et al., 2021), we study spatial representational learning using Gaussian place fields, instead of place cells.
Field density increases near reward locations. Density traditionally refers to the number of field centers of mass in a location. However, we also consider changes in the mean population firing rate, which includes variations in each field's width and amplitude. As animals learn to navigate towards a reward, a high density of place fields emerge at reward locations (Gauthier & Tank, 2018;Lee et al., 2020;Sosa et al., 2023). Reward location based reorganization was observed in hippocampal CA1 and not in CA3 (Dupret et al., 2010). Interestingly, a recent study showed that place fields initially coding the reward location shifted backwards against the trajectory causing a decrease in reward coding fields, suggesting of a representation predictively coding for reward (Yaghoubi et al., 2024).
Fields learn to encode future occupancy. As animals traverse a 1D track, most CA1 fields increase in size and their center of mass shift backwards against the trajectory of motion (Mehta et al., 1997;Frank et al., 2004;Priestley et al., 2022). A proposal for this behavior is that fields initially coding for location x t are learning to also encode the previous location x t-1 , hence predictively coding for location occupancy p(x t+1 |x t ) (Mehta et al., 2000;Stachenfeld et al., 2017). While algorithms such as the successor representation (Dayan, 1993) learn to predict the transition structure (Gershman, 2018), the representation is dependent on a predefined navigation policy. Hence, a complete normative argument-including policy learning-for why fields exhibit this behavior is still lacking.
Fields drift during stable behavior. After animals reach a certain performance criterion in navigating to a reward location, the spatial selectivity of individual place fields changes across days, even though animals exhibit stable behavior (Kentros et al., 2004;Mankin et al., 2012;Ziv et al., 2013;Geva et al., 2023;de Snoo et al., 2023). A proposal is that these fields continue to drift within a degenerate solution space while the overall representational manifold or the chosen performance metric remains stable (Qin et al., 2023;Pashakhanloo & Koulakov, 2023;Masset et al., 2022;Kappel et al., 2015;Rokni et al., 2007). Another proposal is that compensatory synaptic plasticity adjusts the readout to maintain stable decoding over time (Rule et al., 2020;Rule & O'Leary, 2022). However, a model that demonstrates stable navigation learning behavior with drifting fields is absent, and the functional role of drift remains unclear.
Most navigational experiments involve an animal moving from a start location to a target location to receive a reward, either in a one-dimensional (1D) track or a two-dimensional (2D) arena. Similarly, our agents receive their true position at every time step (t) described by the variable (scalar x t in 1D, vector x t in 2D), and have to learn a policy (π) that specifies the actions to take (g t ) to move from a start location (e.g. x start = -0.75, Fig. 1A green dash) to a target with reward values following a Gaussian distribution (x r = 0.5, σ r = 0.05, Fig. 1A red area). The agent outputs a one-hot vector g t (left or right in 1D and left, right, up or down in 2D), which causes its motion to be discrete, similar to a trajectory in a grid world. To model smooth trajectories in a continuous space as an animal's behavior (Foster et al., 2000;Frémaux et al., 2013;Kumar et al., 2022;2024), we use a low-pass filter to smooth g t using a constant α env = 0.2 after scaling for maximum displacement using v max = 0.1:
To track an agent's reward maximization performance during navigational learning we compute the cumulative discounted reward (G = T t=0 T k=0 γ k r t+1+k ) for the entire trajectory for each trial using γ = 0.9 as the discount factor, which is similar to tracking the cumulative reward. The agent will continue to receive rewards by staying at the target. Hence, the trial is terminated when either the maximum trial time is reached T max or when the total reward achieved T t=0 r t reaches a threshold R max .
The agent represents space through N place fields, which have spatial selectivity modeled as simple Gaussian bumps:
(2) where α, λ and σ set the amplitude, center, and width respectively. Two types of place field distributions were initialized to tile the environment: (1) Homogeneous population with constant values for amplitudes α i = 0.5, widths σ i = 0.1, and centers uniformly tiling the environment λ = [-1, ..., 1] (Foster et al., 2000;Frémaux et al., 2013;Kumar et al., 2022;2024). ( 2) Heterogeneous population with amplitudes, widths and centers drawn from uniform random distributions between [0, 1], [10 -5 , 0.1], [-1, 1] respectively. These ranges are consistent with experimental data where place fields were 20 cm to 50 cm wide in small to medium environments (Lee et al., 2020;Frank et al., 2004;Mehta et al., 1997;Sosa et al., 2023). 2D place fields have scalar amplitudes, two dimensional vectors for center, and square covariance matrices for the width (Menache et al., 2005). Refer to App. A for further details.
To model an animal's trial-and-error based learning behavior, we adopt the reinforcement learning framework, specifically the actor-critic (Arleo & Gerstner, 2000;Brown & Sharp, 1995;Foster et al., 2000;Frémaux et al., 2013;Kumar et al., 2022;2024). The critic linearly weights place field activity using a vector w v i to estimate the value of the current location:
(3) The value of a location corresponds to the expected cumulative discounted reward for that location. The actor has M units, each specifying a movement direction. In the 1D and 2D environments, M = 2 and M = 4 respectively to code for opposing directions in each dimension e.g. left versus right and up versus down. Each actor unit a j linearly weights the place field activity such that the matrix W π ji computes the preference for moving in the j-th direction:
) with the probability of taking an action computed using a softmax. A one-hot vector g j is sampled from the action probability distribution P as in Foster et al. (2000), making this policy stochastic. w v i and W π ji were initialized by sampling from a normal distribution N (0, 10 -5 ), with other initializations (uniform or all zeros) reproducing similar results.
The objective of our agent is to learn a policy π that maximize the expected cumulative discounted reward
over all trajectories τ . To achieve this goal, our agent uses the standard actor-critic algorithm using the temporal difference residual (refer to App. A):
(5) which reduces variance and speeds up policy convergence (Sutton & Barto, 2018;Dayan & Abbott, 2005;Wang et al., 2018;Schulman et al., 2017;Mnih et al., 2016). The TD residual is also biologically relevant, as the responses of midbrain dopamine neurons resemble TD reward prediction error (Schultz et al., 1997;Starkweather & Uchida, 2021;Gershman & Uchida, 2019;Amo et al., 2022;Montague et al., 1996). The actor learns a reward maximizing policy by ascending the gradient of the policy log likelihood, modulated by the TD residual. To accurately estimate the value function and critique policy learning using the TD error, the critic minimizes the squared TD error L = E τ 1 2 δ 2 t . As our agent uses a single population of place fields, these fields must learn spatial features that enhance both policy and value learning. The field parameters θ = {α, λ, σ} and the policy weights W π , w v are updated by gradient ascent using a joint objective modified from Wang et al. (2018):
with ∇ w v J T D = 0 and ∇ W π L = 0. We estimate all parameter gradients online, and provide the explicit update equations for each parameter in App. A. The learning rates for the actor-critic and place field parameters can be the same (Fig. S13). For theoretical analysis, we assume a separation of timescales between learning the actor-critic weights and updating place field parameters (App. B).
We first examine the neural phenomenon where a high field density emerges at the reward location over the course of learning. Field density is defined by the distribution of place field center of mass (COM) (Lee et al., 2020), which we estimate using Gaussian kernel smoothing.
Figure 1B shows how our agent's track occupancy (p(x)), field density (d(x)), mean firing rate (f (x)), and individual field's spatial selectivity (ϕ(x)) change when learning to navigate in a 1D track from the start x start = -0.75 to the target at x r = 0.5, when only optimizing place field centers (∆λ). In the early stages of learning (T = 50 to T = 1200), the agent with a homogeneous place field population spends a higher proportion of time at the start location with only sporadic exploration towards the reward. Despite this behavior, a high field density and mean firing rate emerges at the target after a few trials. Individual fields at the reward location shift closer to the target, as seen in Sosa et al. (2023), in contrast to fields at non-rewarded locations. As learning progresses and the agent spends a higher proportion of time at the reward (T = 12000), field density and mean firing rate at the start location also begins to rise, replicating the two-peaked field distribution in Gauthier & Tank (2018), with some fields shifting backwards from the reward location towards the start as in Yaghoubi et al. (2024). Figure 1B (right) also shows that a high field density at the reward location followed by the start location robustly emerges in agents initialized with a heterogeneous place field population, and when all the field parameters (λ, α, σ) as well as the actor (W π and critic (w v ) parameters are optimized using the same learning rate (η = 0.0005).
Similar field dynamics are observed in a 2D arena with an obstacle where agents have to navigate to a target from a starting location (Fig. 1C). When optimizing all the place field parameters in a homogeneous population, a high field density rapidly emerges at the reward location to increase goal representation (number of COM within 0.25 unit radius from target center) as seen in (Dupret et al., 2010), followed by gradual reorganization of field density backward along the agent's trajectory.
Increasing the number of fields in a heterogeneous place field population reduced the average density and mean firing rate (Fig. 1D, Fig. S1) that emerges near the reward location. This is because as the number of fields increase, the agent goes into a weak feature learning regime (Fig. S4) in which feature learning does not contribute to additional advantage. Conversely, the density and mean firing rate are proportional to the reward magnitude, and inversely proportional to the reward location width as a narrower target might require higher discriminability for the agent to maximize rewards.
To understand why place fields exhibit these dynamics, we perform a perturbative approximation to the place field parameter changes under TD learning updates (Menache et al., 2005;Bordelon et al., 2024). In this approximation, we assume that the change to the field parameters is small, controlled by the number of fields, and by the large separation between learning rates. Focusing on the place field centers, we derive in App. B the approximation where η λ = 0.0001 is the learning rate for the field centers and η = 0.01 is the learning rate for the critic weights such that η λ ≪ η:
Under this approximation, each field's center shifts proportionally to the squared magnitude of the critic weights (w 2 v ), implying that fields at locations with a high value will shift at a faster rate compared to locations with a low value. In addition to the value of a location, the agent's start location (modeled as a Gaussian with mean μx = -0.75 and spread σ x ) and the mean field center location λ over time under the policy influence each field's displacement. As the reward location is visited frequently, we expect λ ≈ 0.5. As the term within the square bracket changes sign depending on the field location, only the fields near the reward location will shift towards the reward, while the rest of the fields will move towards the start location. Due to these influences, the field density at the reward location will increase first followed by a gradual increase in start location (Fig. 1B,E). Additional approximations are needed to model the agent's trajectory and improve the simulation-theory fit for place field centers (See App. B).
A similar perturbative analysis for amplitude yields:
when η α ≪ η and where η α = 0.0001 is the learning rate for the α parameters. Thus, place fields will be amplified at a rate similar to learning the value function, causing fields at the reward location to be amplified first, followed by the start location (Fig. 1F, Fig. S1). Therefore, these approximations predict fields shifting to the start and reward location with field amplification at locations of high value.
We now turn to the next phenomenon where place field sizes increase and their centers shift backward against the movement direction as animals learn to navigate. This behavior suggests predictive coding for future occupancy, which can be learned through Hebbian association of fields (Mehta et al., 2000), or through the successor representation (SR) algorithm, which minimizes state prediction error for each place field to learn the transition probabilities (Stachenfeld et al., 2017). Here, we show that our proposed Reward Maximizing (RM) agent also recapitulates field elongation in a 1D track and 2D arena around an obstacle.
For comparison, we developed two agents: A) an SR agent that learns the transition probabilities in parallel to policy learning (Fig. 2A). The SR agent has a similar architecture to our (RM) agent (Fig. 1A), with two key differences: 1) It has one set of place fields with fixed parameters, and only the synapses from these place fields to the actor-critic are optimized for policy learning. 2) There is a separate set of N successor place fields ψ(x) that receive input from the fixed place fields via synapses U which are optimized using the SR algorithm (App. C). We compare the learned successor place fields to the learned place fields in our RM model, referring to them henceforth as place fields. B) a Metric Representation (MR) agent (Fig. 2B) that estimates its current coordinates in an environment (z t ). This metric representation enables navigation to recalled targets by vector subtraction (Foster et al., 2000;Kumar et al., 2024). The coordinate readout weights and place field parameters are updated by gradient descent to minimize the path integration-derived TD error
while only the actor and critic readout weights are updated to learn a policy. This objective allows place fields ϕ(x) to reorganize even in the absence of rewards. See App. D for the derivation.
All three agents (SR, MR, RM) recapitulate the phenomena seen in (Mehta et al., 1997): on average, place fields increase in size over learning (Fig. 2C), and the center of mass (COM) shifts backwards from their initialized positions (Fig. 2D). However, the place field dynamics evolve differently. All agents initially spend a high proportion of time at the start location and gradually learn a policy to spend a higher proportion of time at the reward (Fig. 2E, Fig. S5E).
The SR, by design, tracks the transition probabilities of the agent's policy. Consequently, the SR population mean firing rate f ψ (x) closely aligns with the agent's probability of being in a location p SR , showing a high positive correlation (Fig. 2F, blue). Since the MR representation is modulated by the agent's displacement a t , fields reorganize more at the start location since displacement is nonzero, causing a higher mean firing rate. Conversely, displacement becomes zero at the reward location as the agent comes to a stop to maximize rewards, causing low field reorganization at the reward. Hence, MR fields f ϕ M R (x) become positively correlated with p M R at the start location, but do not fully align with the agent's time spent at the reward location (Fig. 2F, green). Conversely, during early learning, the RM agent exhibits a high population mean firing rate f ϕ RM (x) at the reward location, which contrasts sharply with the proportion of time spent at that location, leading to a highly negative correlation between f ϕ RM (x) and p RM (Fig. 2F, orange). Interestingly, in the later phase of learning, f ϕ RM (x) and p RM become positively correlated.
The mean firing rates learned by the SR and RM agents become negatively correlated during the early learning phase but become positively correlated at the later learning phase (Fig. 2G, blue). Conversely, the mean firing rate correlation decreases monotonically towards zero for the MR and RM agents (Fig. 2G, green), while the correlation between SR and MR increases due to the alignment at the start location in the early learning phase before becoming uncorrelated in the later learning phase. A similar change in correlation is observed when comparing the individual field selectivity, and the spatial representation similarity matrix (Fig. S5F,G). Hence, while all three algorithms demonstrate qualitatively similar neural phenomenon, the dynamics of learning these representations are different, with SR and RM agents eventually learning similar spatial representations.
In a 2D arena with an obstacle, the three agents show field elongation against the movement direction (Fig. 2H) while also accounting for the blockage of path by the obstacle. The RM agent shows a significantly larger elongation of fields to span the entire corridor while the elongation of fields by SR is subtle and field elongation by MR is more pronounced only at the start location. See Fig. S6 for summary statistics.
The third phenomena that the model captures has been described as representational drift, where the agent demonstrates stable behavior but the spatial selectivity of individual place fields change over time (Fig. 3A, Fig. S8G), as seen in Ziv et al. (2013). Although our agent uses a stochastic policy, both the navigation behavior (Fig. 3E, blue) and the population vector (PV) correlation (Fig. 3C, blue) are extremely stable.
To drive larger variability in the representation, we introduced Gaussian noise (ξ t ) of varying magnitudes during place field parameter updates at every time step (θ t+1 = θ t + ξ t , see App. E). Increasing the noise magnitude led to a faster decrease in PV correlation but also disrupted agents' policy convergence for magnitudes greater than 10 -3 (Fig. 3E purple, Fig. S7). Hence, we consider the noise magnitudes between 10 -4 and 10 -3 to be relevant. As the noise magnitude increases, agent's reward maximization behavior remains stable (Fig. 3E) while the PV correlation decreases rapidly (Fig. 3C). This demonstrates that agents can optimize their policies to maintain stable behavior even though individual spatial selectivity is changing. Interestingly, the spatial representation similarity matrix remains more stable than PV correlation (Fig. 3B), even at higher noise magnitudes (Fig. 3D), and when the agents are not explicitly optimizing for representational similarity (Qin et al., 2023).
Unlike noisy field parameter updates, adding noise to the actor and critic synapses (w v t+1 = w v t + ξ v t and W π t+1 = W π t + ξ π t ) caused the reward maximization behavior, representation similarity and PV correlation to change at similar rates (Fig. S7), which is not as consistent with experiments (See Fig. S9 for comparisons to data).
We quantified this drifting behavior at the level of individual neurons by summing the normalized (between [0, 1]) variance in each field's parameters ( V ar( θ) = V ar( α) + V ar( λ) + V ar(σ)) across learning trials, and comparing this against the mean amplitudes for each field. When no Gaussian noise is added (Fig. 3F, blue), fields with a higher mean amplitude showed a higher variance in its parameters, which is expected since fields with a higher amplitude are more likely to be involved in policy learning. Conversely, with a small Gaussian noise, we see the opposite trend where fields with a smaller mean amplitude showed a higher variance in parameters while fields with a higher mean amplitude were more stable (Fig. 3F, orange). At smaller noise magnitudes, there is a strong positive correlation between higher amplitude fields and the magnitude of actor and critic readout weights (Fig. S8). This suggests that high-amplitude fields are more involved in policy learning and thus more stable, whereas less important fields can alter When σnoise = 0.0001, we see the opposite trend, where fields with a larger amplitude are more stable than fields with a smaller amplitude, suggesting fields with larger amplitudes acting as anchored representations in an environment. Refer to Fig. S8 for other σnoise values. Shaded area is 95% CI over 10 seeds.
their spatial selectivity to maintain stable behavior (Fig. 3E, orange), consistent with Qin et al. (2023). Increasing the noise magnitude beyond this magnitude causes place fields with large magnitudes and more involved in policy learning to drift more (Fig. 3F, purple), causing unstable behavior (Fig. 3E, purple). As the reward-maximizing model recapitulates experimentally-observed changes in place fields, it is natural to ask what computational advantage representation learning might offer during reinforcement learning. To probe the contributions of each field parameter to policy learning, we perform ablation experiments. These ablations are particularly important due to the parameter degeneracies in the model: one can trade off the place field amplitudes and the critic and actor weights.
We first considered the task of navigating to a single fixed target. Agents with fixed place fields attained the lowest navigational performance with cumulative reward G plateauing at G = 33 (Fig. 4A), and showed the slowest policy con-vergence even as the number of fields increased (Fig. 4B). Optimizing place field widths (σ) contributed to the greatest improvement in maximum reward and largest decrease in the number of trials needed for policy convergence (Fig. 4A-B). Optimizing place field amplitudes (α) contributed to the next most significant improvement (Fig. 4A-B). Interestingly, place field center (λ) optimization did not contribute to a significant improvement in performance, and in fact caused a decrease in reward maximization performance and speed of policy convergence when optimized together with the amplitude parameter. Hence, optimizing field widths followed by amplitudes and lastly centers significantly improved agent's reward maximization performance and increased the speed of policy convergence.
Optimizing field parameters using the auxiliary metric representation (MR) objective, inspired by Fang & Stachenfeld (2023), marginally improved policy learning (Fig. S15). However, as the number of place fields increase (Fig. 4B), the computational advantage afforded by place field optimization extinguishes. Nevertheless, optimizing all the parameters in a small number of fields, e.g. 8, leads to a similar rate of policy convergence than with a larger number of randomly initialized fields e.g. 128, which hints that representation flexibility could allow efficient learning in systems with few neurons.
We now turn to the influence of noisy fields when learning to navigate to new targets, inspired by Dohare et al. (2024). Agents now have to navigate from the same start location to a target that repeatedly changes location. Although all agents learned to navigate to the first (x r = 0.5) and the second (x r = 0.0) targets equally well, agents without noisy field updates struggled to learn the next three targets, and achieved a lower average cumulative reward (Figure 4C, blue). Increasing the noise magnitude led to a monotonic improvement in new target learning. Some place fields coding for the initial reward location shifted to code for the new reward location (Fig. S3), replicating the behavior of place cells that selectively code for rewards (Gauthier & Tank, 2018;Sosa et al., 2023). This behavior was suppressed in agents without noisy field parameter updates. However, noise magnitudes beyond a threshold (σ noise = 0.001) caused average cumulative reward to decrease. These results suggests that there is a functional role for noise, especially for new target learning. We see a similar improvement in reward maximization performance with noisy field updates in a 2D arena with an obstacle when we either change the target or the obstacle location (Fig. S12).
We present a two-layer navigation model which uses tunable place fields as feature inputs to an actor and a critic for policy learning. The noisy parameters of the place fields and the policy and value function learn to maximize rewards using the temporal difference (TD) error. Our simple reinforcement learning model reproduces three experimentallyobserved neural phenomena: (1) the emergence of a high place field density at rewards, (2) enlargement of fields against the trajectory, and (3) drifting fields without influencing task performance. We analyzed the model to understand how the TD error, number of place fields and noise magnitudes influenced place field representations. Lastly, we demonstrate that learning place field representations with noisy field parameters improves the rate of policy convergence when learning single and multiple targets.
Our goal was not to replicate every known mechanistic detail about place fields, but come up with a minimal biologically grounded model that captures as many phenomena as possible. This required deliberate decisions to omit certain granularities. While this necessitates some disconnect with neural mechanisms, it allows parsimony (Van Vreeswijk & Sompolinsky, 1996;Burak & Fiete, 2009;Hopfield, 1982) and interpretability (Bordelon et al., 2024) to make experimentally testable predictions for neural systems (Montague et al., 1996;Schultz et al., 1997;Kumar et al., 2024). For instance, our model gives an alternative normative account for field elongation against the trajectory, which can be contrasted with the successor representation algorithm (Raju et al., 2024;Kumar et al., 2024). As field dynamics are different in these two models, they could be distinguished by experiments that track fields over the full course of learning (Fig. 2C-E, Fig. S6). Furthermore, place field width and amplitude optimization increases maximum cumulative reward and accelerates policy convergence (Fig. 4A-B).
Most models that characterized representational drift were not studied under the context of navigational policy learning (Masset et al., 2022;Pashakhanloo & Koulakov, 2023;Ratzon et al., 2024). We showed that increasing the noise magnitudes caused different drift regimes (Fig. 3F; Fig. S9D), and at very high noise levels navigation behavior started to collapse (Fig. 3C, Fig. S7). Importantly, we showed that fields in the noisy regime allowed agents to consistently learn new targets in both 1D (Fig. 4C) and 2D (Fig. S12A-B) environments, without getting stuck in local minima. The biological origins of adding noise to place field parameters can be attributed to noisy synaptic plasticity mechanisms (Mongillo et al., 2017;Kappel et al., 2015;Attardo et al., 2015). Other mechanisms such as unstable dynamics in downstream networks (Sorscher et al., 2023) and modulatory mechanisms such dopamine fluctuations (Krishnan & Sheffield, 2023) could adaptively control drift rates. A difficult experiment that could directly verify our model is to induce or constrain place field drift rates in animals and determine how this perturbation influences new target learning. How fluctuations in dopamine, stochastic actions and stochastic firing rates within place fields drive drift rates needs to be explored. The current model provides a starting point for this investigation.
The proposed model is not without limitations. First, we modeled single peaked place fields instead of the complex representations resulting from single "place" cells, which can be multi-field and multi-scale. Nevertheless, the proposed online reinforcement learning framework is general enough to accommodate other models of place cell description (Mainali et al., 2024;Sorscher et al., 2023) e.g. Fig. S14, and can be extended to study representation learning in other brain regions e.g. medial entorhinal (Boccara et al., 2019;Wen et al., 2024) or posterior parietal (Suhaimi et al., 2022) cortex.
Next, place field parameters are optimized by backpropagating the temporal difference error through the actor and critic components (Fig. S15). Since the motivation was to develop a normative model whose objective was to maximize rewards, this was a reasonable starting point. However, this model must be extended using biologically-plausible learning rules (Miconi, 2017;Murray, 2019;Lillicrap et al., 2016;Nøkland, 2016;Overwiening et al., 2025) before it can in any way be considered mechanistic (Lee et al., 2024;Starkweather & Uchida, 2021;Krishnan et al., 2022;Kempadoo et al., 2016;Edelmann & Lessmann, 2018).
Although we explored a simple non-reward-dependent objective to drive place field reorganization, extending the model to other auxiliary objectives (Low et al., 2018;Schaeffer et al., 2022) to understand their influence in representation learning for policy learning is the next step. While our computational experiments successfully demonstrated the model's effectiveness in reproducing three disparate phenomena, further work should test its robustness across other reinforcement learning algorithms e.g. policy gradient (Kumar & Pehlevan, 2024) and network architectures (Team et al., 2021;Kumar et al., 2025). Additionally, we need to explore how place field reorganization scales in larger, more complex environments (Hill et al., 2020;Lin et al., 2023;Nieh et al., 2021;Kumar et al., 2024) beyond the few environments we considered.
The code for initializing and training the model in 1D and 2D environments, along with the code for analyzing neural phenomena and generating all figures, will be available on GitHub upon acceptance.
The agent contains N place fields. In a 1D track, each place field is described as
with α, λ and σ describing the amplitude, center and width, adapted from Foster et al. (2000); Kumar et al. (2022;2024). Most of the simulations were initialized with amplitudes α i = 0.5 and widths σ i = 0.1, with centers uniformly tiling the environment λ = {-1, ..., 1}. Nevertheless, similar representations emerge for amplitudes drawn from a uniform distribution between [0, 1] and widths uniformly drawn between [0.01, 0.25]. This parameter initialization was used for ablation studies in Fig. 4. In a 2D arena, each place field is described as
where Σ i is a 2x2 covariance matrix, adapted from Menache et al. (2005). The off-diagonals were initialized as zeros and diagonals initialized to match the variance in the 1D place field description, i.e. Σ ii = 0.1 2 to ensure field widths are consistent in 1D and 2D.
The objective of the model is to learn a policy π parametrized by W π and spatial features ϕ parameterized by θ that maximizes the expected cumulative discounted rewards over any trajectory τ = {x 0 , g 0 , r 0 , x 1 , ...x T , g T , r T , x T +1 } in a finite-horizon setting, modeling the trial structure in neuroscience experiments
where γ is the discount factor, r t is the reward at time step t after choosing an action g t at state x t , and the time horizon T is finite with trials ending after a maximum of 100 steps in the 1D track and 300 steps in the 2D arena.
To maximize the cumulative reward objective, we perform gradient ascent on the policy and place field parameters,
where η θ and η are learning rates for θ and W π respectively. The gradients are derived using the log-derivative trick,
where the trajectory τ describes the state to state transitions. We expand the above using the Markov assumption that the transition to future states depend only on the present state and not on the states preceding it,
Since the gradients are not dependent on the state transitions, the last line excludes them. Substituting Eq. 19 into Eq. 16 yields
which completes the full derivation of the policy gradient theorem (Sutton et al., 1999;Sutton & Barto, 2018). The policy gradient objective was used by Kumar & Pehlevan (2024) to optimize the policy and place field parameters. However, this learning signal requires an explicit reward and policy gradient methods are slow to converge as they suffer from high variance due to:
• Monte Carlo sampling: Agents need to sample an entire episode to estimate the expected return E τ [G t = r t + γr t+1 + γ 2 r t+2 + ...] before updating the policy. This can introduce significant variance because the estimate is based on a single path through the stochastic environment, which may not be representative of the expected value over many episodes.
• No Baseline: The basic policy gradient algorithm computes the gradient solely based on the return G from each trajectory. By introducing a baseline (either constant b or dynamically evolving b t e.g. value function v t ), which estimates the expected return from a given state, the variance of the gradient estimate can be reduced, because now the policy learns which action is better than the previous (concept of using an Advantage A t instead of rewards).
Value based methods (Sutton & Barto (2018), Chapter 3.5) were introduced to address some of these issues. For instance, instead of sampling returns G t , value functions V t learn to estimate the expected returns at each time step t:
which can reduce the variance during credit assignment. The combination of policy gradient with value-based methods lead us to the Actor-Critic algorithm.
The optimal value function V t reflects the true expected cumulative discounted rewards, hence the policy gradient objective can be rewritten as
(24) which yields the following self-consistency equation r t + γV t+1 -V t ≡ 0 , (25) as argued by Sutton & Barto (2018); Frémaux et al. (2013).
Alternatives to policy gradient algorithms propose subtracting a baseline which can be a fixed constant b or a dynamically changing variable b t . Since we have the value function V t we can modify the objective to be
which gives us the Advantage function (Mnih et al., 2016;Schulman et al., 2015). This reduces the variance as the policy has to learn to select actions that gives an advantage over the current value function. We get a learning objective function that is an analogue to maximizing the expected cumulative discounted returns while subtracting a baseline Eq. 11.
However, we have assumed that we are given the optimal value function V t to critique the actor if it is doing better or worse than before. Instead, we can learn to estimate the value function v t using a critic by minimizing the Temporal Difference error
The critic can learn to approximate the true value function by minimizing the mean squared error between the true value function V t and the predicted v t , or the temporal difference error δ t
Since we do not have the optimal value function V t , we can approximate it by bootstrapping the estimated value function v t and ensuring that we do not take gradients with respect to the time shifted value estimate v(x t+1 ) using a stop gradient (sg):
We minimize the temporal difference error using gradient descent for the critic to estimate the value function
Notice the additional negative sign that pops out when you take the derivative of δ only with respect to
since r t and v t+1 are treated as constants, we do not take their derivatives. Since we do not have the optimal value function V t but a biased estimate v t , we can use the temporal difference error as our reward maximization objective
As the value estimation becomes closer to the optimal value v t → V t , this objective becomes similar to the advantage objective J T D → J A . Note that we are not directly maximizing the TD error during policy learning. Rather, we want to optimize the policy π and place field parameters θ by gradient ascent, using the biased estimate of the advantage function
An alternative interpretation is that during policy learning, the agent learns a policy to maximize the difference between the actual reward and the estimated value
In our model (Fig. 1A), actor W π and critic w v weights are optimized separately, while the place field parameters θ overlap. The actor uses gradient ascent for Eq. 27, and the critic employs gradient descent for Eq. 35. Since we have a single population of place fields, we optimize these parameters to support both objectives. Thus, we derive a combined objective function to update W π , w v , and θ in a single gradient pass
where ∇ w v J T D = 0 and ∇ W π L T D = 0 since the respective objectives are not parameterized by w v and W π respectively. This means that W π is tuned to maximize J T D , w v is tuned to minimize L T D and θ is tuned to balance both the objectives.
Since most optimizers e.g. in Tensorflow, PyTorch perform gradient descent, not ascent, we can minimize the negative policy gradient Eq. 27, which is equivalent to the negative log likelihood
which is the same update rule used in Wang et al. (2018); Mnih et al. (2016) to train the actor and critic separately while the feature parameters are trained jointly.
It is also possible to initialize two separate populations of place fields, each for the actor and critic. Alternatively, we only optimize place field parameters using the actor's objective while the critic uses the spatial features to learn the value function.
The converse is also possible where the place field parameters and critic weights are optimized to minimize the TD error while the actor learns a policy without optimizing the spatial representations, as we did in the perturbative approximation (App. B). From numerical experiments, optimizing place field parameters using both the actor and critic objectives allowed the agent to achieve the fastest policy convergence and highest cumulative reward performance (Fig. S15).
Now, we derive an online implementation of Eq. 6 which is the same as Eq. 42, so that all parameters are updated at every time step. Extending from Foster et al. (2000); Kumar et al. (2022), the actor and critic weights are updated according to the gradients
where gt,j = g t -P and η = 0.01. The gradient updates for place field parameters follow
where we use a significantly smaller learning rate η θ = 0.0001 so that the spatial representation evolves in a stable manner.
Specifically, each field parameter is updated according to
where δ bp i,t is the TD error gradient that has been backpropagated through the actor and critic weights. Using just the w v i (t) or W π ji weights alone to backpropagate the TD error influences the representation learned by the place field population and ultimately the navigation performance (Fig. S15).
There are two ways to optimize the place field width parameter. The first and straightforward method is to update the width parameter according to
where k = 1 in a 1D place field. In a 2D place field with k = 2, we can update the diagonal elements in the 2D matrix while keeping the off-diagonals to zeros as in Menache et al. (2005). However, fields will only elongate along each axis. Instead, in our simulations, we optimized the off-diagonals using the same gradient flow equations. However, we needed to include additional constraints so that each place field's covariance matrix remains 1) symmetric, 2) bounded, and 3)positive semi-definite to perform matrix inversion. Specifically, the covariance matrix was bounded between [10 -5 , 0.5] to prevent exploding widths and gradients.
Under the condition that α 0 = 1 and M = M ⊤ we can work out an exact expression in terms of the eigendecomposition
we can approximate this at late times as
As t → ∞ we can approximate this as lim t→∞ θ(t) ≈ 2(w V ) 2 . This indicates that neurons which are heavily involved in the reproduction of the value function are upweighted in their amplitude.
Based on the place field center update equation and rewriting the terms as above,
We need to compute an average over spatial positions. We approximate the space position early in training as a Gaussian with mean s 0 and variance σ 2
x
where
x μx is the mean value of x obtained by the above Gaussian integral under the approximation that p(x) ∼ N (μ x , σ 2
x ). Approximating λ j as the mean position of the tuning curves λ we obtain the following prediction
Following the solution in Eq. 63, we can approximate this at late times as
Hence, in addition to the value of a location, three additional factors influence each field's displacement.
where λ is the agent's expected location sampled from its policy, μx = -0.75 is the starting location and σ x is the estimated spread of the trajectory. This analysis suggests that fields will be influenced by both the start location and the location where the agent spends a higher proportion of time at. In later learning phases, this will be the reward location λ = 0.5. Consequently, only the fields near the reward location will shift towards the reward, while the rest of the fields will move towards the start location. We illustrate this perturbative approximation at early and late times of training in Figure 5. The theory is quite accurate early in training, but fails at sufficiently long training time.
The generalized temporal difference error is given by
with M i representing the predicted successor representation and ϕ(x) representing the initialized place field representation that is not optimized.
The successor representation is computed using a summation of the place fields with a learned matrix U that is positively rectified. The rectification is necessary to have a non-negative representation.
The matrix U is initialized as an identity matrix and is updated using a two-factor rule using the TD error as in Gardner et al. (2018).
The hippocampus is known to learn and represent spatial maps even in the absence of rewards, enabling rapid navigation to new locations when required (Tolman, 1948;Steele & Morris, 1999;Tse et al., 2007). This requires reorganization of place fields in non-rewarded conditions, which has been proposed as a mechanism for learning a predictive map that estimates future spatial occupancy (Mehta et al., 1997;Stachenfeld et al., 2017). To describe this non-reward-based reorganization, the successor representation algorithm (Dayan, 1993) has been used. More recently, an auxiliary predictive objective has been proposed (Fang & Stachenfeld, 2023).
Here, we present a simple predictive objective for place field reorganization that is independent of rewards. We introduce a previously described objective called the Metric Representation (MR), which learns a low-dimensional representation of an environment using place field activity and a biologically plausible learning rule that is modulated by a path integrationderived Temporal Difference error. This representation allows an agent to predict its current coordinates z(x t ) and perform vector subtraction to rapidly navigate to recalled goals (Foster et al., 2000;Kumar et al., 2024). However, representation learning was not studied using this objective. Recently, a similar objective was proposed to learn a spatial map using local learning rules, although as a high-dimensional representation (Stöckl et al., 2024).
The dimensionality of the coordinate prediction z(x t ) is equal to the dimensionality of the environment, calculated through a linear summation of place field activity:
When the agent accurately predicts its coordinates in the environment, the following path integration-derived self-consistency equation holds:
where a j (x t ) is the true displacement of the agent in the environment. However, if the prediction is inaccurate, Eq. 77 can be reformulated into a temporal difference error for each dimension j of the environment as described by Foster et al. (2000); Kumar et al. (2024):
(78) This one step prediction error (χ t ) can be expressed as a loss function, similar to Fang & Stachenfeld (2023) without the temporal discounting factor:
which can be minimized by gradient descent by optimizing both the coordinate readout weights (W M R ) and place field parameters (θ ∈ {α, λ, σ}):
The gradient updates were implemented in an online manner:
We can analyze how the different temporal difference residues (both the canonical reward-dependent and newly proposed metric representation-based) influence place field reorganization and agent policy learning performance by propagating the residues through a combination of actor, critic, and metric representation weights:
This can be done by setting the weighting of each component β v , β π , β M R ∈ {0, 1}. Refer to Fig. S15 for policy convergence performance in both the 1D and 2D environments when using different combinations to learn place field representations.
To induce drift, we independently introduced noise to field amplitudes, centers and width, as well as the synapses to the actor and critic (θ ∈ {α, λ, σ, w v , W π }).
where the noise term ξ t are independent Gaussian noises with zero mean and magnitude σ noise ∈ {10 -6 , 10 -1 }. We performed a noise sweep to determine how increasing the noise magnitude affected the agent's reward maximization behavior, population vector correlation and representation similarity. Refer to Fig. S7. , field density which is the number of Center of Mass (COM) in a location after smoothing with a Gaussian kernel density estimate (gKDE) (gKDE(COM ), blue) and, the frequency of being in a location (pRM (x)), when optimizing different combinations of field parameters (α, λ, σ) during reward maximization (RM). The location in which the highest value for mean firing activity, field density and frequency is attained is indicated by a red, blue and black vertical dash line respectively. Optimizing a (A) small number (N = 16) and (B) large number of place fields yields a similar high mean firing rate at the reward location followed by the start location. However, the field density evolves differently when in the low field regime, (A) a high density emerges at the reward location in the early stages of learning, but it shifts to the start location at later stages of learning. This effect was observed in a recent experiment where place fields which initially encoded the reward location, gradually shifted backward towards the corresponding start location. This shift led to a decrease in place fields specifically coding for the reward, suggesting that the hippocampal representation reorganizes to predictively code for the reward (Yaghoubi et al., 2024). Whether experiments demonstrate such misalignment between place field density and mean firing rate needs to be analyzed. Based on the ablation studies (Fig. 4A,B), mean firing rate will be a stronger indicator of learning performance than field density. (B) In the high field regime, a high field density at the reward location remains stable throughout learning. Note that COM changes only when the place field centers are optimized (∆λ). Distribution is shown for a single seed run for a homogeneous place field population that has been initialized by with equal spacing between field centers (λ ∈ [-1, 1]), equal amplitude (α = 0.5) and width (σ = 0.01). Refer to Fig. S2 for general place field reorganization over different seeds. . Noise amplitude monotonically influences population vector correlation and agent performance. Adding Gaussian noise with increasing magnitude [5x10 -7 , 10 1 ] either in field parameters (α, λ, σ) or Actor-Critic (Wπ, wv) influences the variance in Population Vector Correlation (RP V , blue), Spatial Representation Similarity which is the dot product of field activity (RRS, orange) and cumulative discounted reward (G, green). Low variance of RP V and RRS indicates high correlation as learning progresses. Low variance in G indicates stable performance. When G increases before decreasing as the noise amplitude increases, agent's navigation performance collapsed and the agent achieves 0 reward with low variance. A high ratio of variance in population vector correlation and reward maximization behavior (RP V /G, red) indicates that there is an optimal noise amplitude which causes high variance in population vector correlation (low PV correlation) while demonstrating stable performance. A similar analysis can be performed using representational similarity (RP V /RRS, purple) to determine the optimal noise amplitude for high variance in population vector correlation but stable representation similarity as seen in Qin et al. (2023). Note that our agents are only optimizing for navigation behavior instead of representation similarity. When a small amount of noise is added, fields with a larger mean amplitude show a smaller variance in change in parameter while fields with a smaller amplitude show higher variance. Conversely, when the magnitude of noise is further increased (purple), fields with a higher amplitude show higher variance in its parameters. (C) The correlation between mean amplitude and the magnitude of the readout weights (sum over all actions for squared actor weights and squared critic weights) is high and positively correlated when the noise magnitude is low. This correlation decreases and becomes weakly positive when σnoise = 0.001. This supports the claim that in the low noise regime, fields with a high amplitude are more involved in policy learning and hence drift less or are more stable to maintain performance integrity. (D) Population vector correlation decreases at a faster rate than the similarity matrix when noise magnitude increases. (E) Representation similarity correlation decreases as the noise magnitude increases, but at a slower rate than PV correlation. (F) Proportion of fields that are active (average fraction of fields with firing rate less than 0.05, 0.1,0,25) continues to increase with higher noise magnitude. (G) Introducing Gaussian noise with zero mean and variance N (0, 0.00025) to place field parameters during updates θt+1 = θt + ξt caused each place field's center, firing rate and width to fluctuate as trials progressed. See App. E for details. This causes each field's spatial selectivity to change over time. Specifically, each field's centroid (λ) shifted from its initialized location, firing rates fluctuated (α 2 ) causing fields to gain or lose selectivity, and most fields increased in size (σ 2 ) while some did not. The first two were observed by Qin et al. (2023) who analyzed Gonzalez et al. (2019). Each color corresponds to the dynamics of a specific field, with 5 example fields shown. We evaluated the speed of policy learning when optimizing (A) heterogeneously distributed place field population in the 1D track and (B) homogeneously distributed place field population in the 2D arena using: (1) fixed place field parameters (blue), (2) backpropagating the TD error δt through the actor weights W ⊤ π gt (βπ = 1, βv = 0, βMR = 0, orange), (3) backpropagating the TD error through the critic weights w v (βπ = 0, βv = 1, βMR = 0, green), (4) backpropagating the path integration derived TD error χt through the metric representation weights WMR (βπ = 0, βv = 0, βMR = 1, red) while learning the value function and policy by optimizing only the readout critic and actor weights, (5) backpropagating the TD error through both the actor and critic weights, otherwise called the Reward Maximization agent (βπ = 1, βv = 1, βMR = 0, purple), (6) backpropagating the TD error through both the actor and critic weights and the path integration based TD error through the metric representation weights, (βπ = 1, βv = 1, βMR = 1, brown). The combined RM+MR objective used for place field parameter optimization achieved the fastest policy learning, similar to Stachenfeld et al. (2017) when the number of fields was low (N = {4, 8, 16, 32} in 1D and N = {4, 8} in 2D). With more fields, the reward maximization agent (RM , purple) was almost as effective as the combined objective (RM + M R, brown). Optimizing place field parameters using only the actor weights led to the slowest policy convergence (orange), nevertheless faster than using fixed place fields. The same learning rates were used when the number of fields were increased. Hence, tuning learning rates should improve the stability of policy learning, especially in the 2D arena for the agent with the combined RM+MR objective. Shaded area indicates 95% CI over 50 random seeds.
The John A. Paulson School of Engineering and Applied Sciences, Harvard University
Center for Brain Science, Harvard University
The Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University
Society of Fellows, HarvardUniversity. Correspondence to: M Ganesh Kumar <mganeshkumar@seas.harvard.edu>.Proceedings of the 42 nd International Conference on MachineLearning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).