Thermodynamics is the branch of physics which is concerned with the relation between heat and other forms of energy. Historically, it was born of the quest to quantify the maximal efficiency of heat engines, i.e., the maximal ratio of the total work output over the total heat input to a thermodynamic system. This was accomplished in the celebrated work of Carnot [1], [2] where, assuming that transitions take place infinitely slowly, it was shown that the maximal efficiency possible is η C = 1 -T c /T h (Carnot efficiency), where T h and T c are the absolute temperatures of two heat reservoirs, hot and cold respectively, with which the heat engine alternates contact.
Somewhat inadvertently, Carnot's work gave birth to the second law of thermodynamics, which affirms that the total entropy of a system can never decrease, and whose most prominent consequence is to highlight the arrow of time. Specifically, it states that the work output -W can not exceed the free energy difference between the initial and terminal states of the thermodynamic system -∆F , that is,
In Lord Kelvin's words, the second law of thermodynamics amounts to the impossibility of a self-acting machine, unaided by any external agency, to convey heat from one body to another at a higher temperature [3].
Soon after Lord Kelvin's assertion, Maxwell's far reaching thought experiment that involved a demonic creature [4], pointed to ways to generate a temperature gradient by sorting particles in a thermodynamic ensemble based on velocity measurements. The apparent paradox was not resolved until, a century later, Rolf Landauer affirmed that information is physical [5]. Starting from the basic assumption that information must be stored somewhere, he was able to link the loss of information with the work performed.
The relation between information and work gradually became a central theme of stochastic thermodynamics [6], [7], [8], [9], [10], [11] -a field shaped in the past two decades to study thermodynamic transitions taking place in finite time. To this end, thermodynamic ensembles are modeled via stochastic differential equations and notions of work and heat are described at the level of individual trajectories of the ensemble. Ideas from stochastic control were naturally brought in and the second law was extended to include discrete time measurements, as well as continuous ones, both for quantum systems [12], [13], [14] and classical systems under feedback cooling [15], [16]. In these studies, a generalized version of the second law has taken the form:
where I represents the information utilized in effecting a thermodynamic transition. Information engines that work without temperature gradient and only fueled by information soon followed [17], [18], [19].
The present work aims to develop further this circle of ideas within a stochastic controls perspective. Specifically, we derive tighter forms of the second law for over-damped systems in general, modeled by Langevin equations and subject to continuous nonlinear measurements. Moreover, in the setting where the ensemble is seen as the medium of a thermodynamic engine and where performance is measured by power drawn, detailed expressions for maximal power and efficiency are derived in the setting of linear-dynamics with Gaussian-distributions.
The exposition proceeds as follows. Section II provides a preamble on optimal mass transport -a theory that constitutes the template for optimal control of probabilistic ensembles. Section III explains the stochastic model of a thermodynamic engine, the energy exchange mechanism, and the form of the second law in the absence of feedback.
Section IV extends the second law to the case when information from a single measurement becomes available. Section V contains our main results on operating a thermodynamic engine with nonlinear continuous time measurements and a form of the second law that applies in this case. Section VI details expressions for maximal power and efficiency of the linear Gaussian information engine. Finally, Section VII provides perspective and research directions.
We outline certain geometrical notions from optimal mass transport [20] that play an essential role in the present paper. Given probability distributions p 0 and p f on R,
where Π(p 0 , p f ) denotes the set of joint probability distributions on R × R with p 0 , p f as marginals, defines the so-called 2-Wasserstein distance (metric). It turns out that W 2 (p 0 , p f ) makes probability distributions into a geodesic space. In turn, geodesics correspond to (optimal) flows between endpoint distributions that provide an alternative expression for W 2 (p 0 , p f ). Specifically, the time-varying probability distribution p(t, x), driven by the velocity field v(t, x) via the continuity equation
represents an action integral for the flow p(•, x). A celebrated result by Benamou and Brenier states min
as a minimal over the set of paths connecting p 0 to p f .
In this paper particles are governed by the overdamped Langevin dynamics (one-dimensional, for simplicity)
where X t ∈ R denotes the location of a particle, p 0 the initial distribution of an ensemble, γ the viscosity coefficient of the ambient medium, k B the Boltzmann constant, T the temperature of a heat bath, B t a standard Brownian motionfoot_0 that models the thermal excitation from the heat bath, and U (t, x) a time-varying potential exerting a force -∇ x U (t, x) on a particle at location x ∈ R. The potential function U (t, x) is externally controlled and exchanges work with the particle. The work performed on the particle, during the interval [0,
The average work is
where the probability p(t, x) of the particle X t evolves according to the Fokker-Planck equation
where we introduced the effective velocity field
In order to state the second law of thermodynamics, we introduce the notion of free energy corresponding to a potential function U and a probability distribution p, namely [11],foot_2
The first term represents the energy and the second term represents the negative of entropy, while together, F relates to the relative entropy between p and the Boltzmann distribution corresponding to the potential. The following proposition relates the average work over the interval [0, t f ] to the free energy difference between the initial and final states, giving a version of the second law of thermodynamics. Proposition 3.1: For the over-damped Langevin dynamics (3), the average work satisfies the identity,
and the bound
where
Remark 3.1: The second term in the identity ( 6) is equal to the action integral (1) and represents the dissipation along the thermodynamic transition. According to (2), its minimum is the Wasserstein distance between the end-point distributions, concluding (7). The bound is tight and can be achieved by transporting along the geodesic with constant velocity. In the quasi-static limit, as t f → ∞, the dissipation term vanishes, leading to the classical statement of the second law W ≥ ∆F . As a result, the bound ( 7) is interpreted as refinement of the second law for finite-time transitions. It was obtained in [26] for Gaussian setting and generalized in [27] to arbitrary distributions.
We now extend the second law (i.e., the bound (7)) to the case where access to a single noisy measurement of the particle's location is available. Thus, assume we have access to noisy measurement Y of the initial particle location X 0 .
We utilize the measurement Y to modify our control in U , denoted U Y . The expected work conditioned on Y is
The information in Y allows extracting work, and this additional work is characterized in terms of the mutual information between X t and Y ,
Here, H(X t ) and H(X t |Y ) are the entropy of X t and the conditional entropy of X t given Y respectively, defined as
where p Xt,Y denotes the joint distribution of (X t , Y ), p Xt|Y the conditional, and p Xt and p Y the marginals.
The following proposition states an extension of the second law. In order to compare to the case with no measurement, we set the initial and final potential to a fixed function U 0 and U f respectively. Note that the potential function is allowed to have discontinuous jump at initial and final time.
Proposition 4.1: Consider a particle governed by the overdamped Langevin dynamics (3), and access to a noisy measurement Y of initial particle location X 0 . Fix the initial and final potential functions U 0 and U f , respectively. Then, the average work satisfies the bound
where ∆F = F (U f , p Xt f ) -F (U 0 , p 0 ).
Remark 4.1: Compared to (7), the new bound (9) contains an additional term k B T (I(X 0 ; Y ) -I(X t f ; Y )). This term quantifies the amount of information by measuring Y that is actually being used as the particle transitions from X 0 to X t f . In the case where the system undergoes cyclic transitions, and therefore ∆F = 0, the information term provides the maximum amount of work that can be extracted from a single heat bath with constant temperature using feedback. The thermodynamic system under such a feedback cycle is referred to as an information machine [18].
Remark 4.2: Compared to earlier literature, and specifically the bound
Eq. ( 1)], our bound is tighter and involves two additional terms. The additional term involving the Wasserstein distance characterizes the minimum dissipation in the process. The additional term k B T I(X f ; Y ) contains the information that has not been used at the end of the process and cannot be transformed to work. Assuming the system converges to a steady state independent of Y , both of these terms will tend to zero as t f → ∞.
Proof: The conditional probability distribution p Xt|Y satisfies the Fokker-Planck equation for t ≥ 0,
where
Upon expressing the derivative of the free energy as
and integrating over the time interval [0, t f ],
where
The expected free energy at the initial time is
where we used that I(X 0 ; Y ) = H(X 0 ) -H(X 0 |Y ). Then, with a similar conclusion for the expected free energy at t f ,
It now remains to bound the dissipation term from below. For a fixed value of the measurement Y ,
because of (10) and the Benamou-Brenier result (2). In addition, the expectation of the Wasserstein distance, over the measurement Y , satisfies the lower bound
). This bound is obtained using the standard dual formulation of the Wasserstein distance as a sup over linear functional of the marginals. Interchanging the expectation and sup results in this lower-bound and concludes the result.
We now consider the case of having access to a continuous stream of measurement given by
where h(•) is the observation function, {V t } is a standard Brownian motion representing the noise in measurements, and σ v is the strength of noise. We assume that {V t } and {B t } are mutually independent processes. The expected work conditioned on the measurement history, i.e. the filtration Z t generated by the observation process {Z s ; s ∈ [0, t]}, is
where we used the notation U Zt (t, X t ) to indicate that the potential function at time t may depend on the history of observations up to that point. Similar to the single measurement case, this information can be used to extract work from the system. The information in the continuous-time setting is characterized by the mutual information between the random processes X 0:t f and Z 0:t f . For the particular observation model (12), the mutual information is given by [28] I(X 0:
where ĥt := E[h(X t )|Z t ].
Proposition 5.1: Consider the particle governed by the over-damped Langevin dynamics (3) and access to a continuous stream of measurements according to (12). Assume the initial and terminal potential functions are fixed to U 0 and U f respectively. Then,
where
Remark 5.1: The notion of information in the continuous measurement case involves the mutual information between the particle location and the measurement I(X 0:t f ; Z 0:t f ), as well as the remaining information I(X t f ; Z 0:t f ) that has not been used. This result provides the first and tightest analysis for the role of information for feedback systems under continuous nonlinear observation models.
Proof: The conditional probability distribution p Xt|Zt evolves according to the Kushner-Stratonovich equation [29]
where dξ t = dZ t -ĥt dt is the innovation process and
Differentiating the free energy
Integrating over the interval and taking the expectation yields
where
) and we used the fact that ξ t behaves as a Brownian motion under conditional expectation [29,Lemma 5.6]. Using the definition (13) and applying the relationship (11) for the expected free energy at the final time concludes
It remains to obtain a lower-bound on the dissipation term. By Jensen's inequality
where we introduced v(t, x) := E[v Zt (t, X t )|X t = x]. Upon taking the expectation and integrating over the time interval,
The proof follows by showing that the velocity field v(t, x) generates the flow for the marginal distribution p Xt , i.e. that
In order to do so, we take the expectation of both sides of equation ( 15) and use the identities
as well as cancel the mean-zero term multiplied by dξ t .
The efficiency for information engines is defined [18] as the ratio between the work output and the amount of information that is available to be used. Thus, in our case,
In light of ( 14), the efficiency is always smaller than 1. It is also noted that, in order to achieve maximal efficiency, it is necessary that I(X t f ; Z 0:t f ) = 0, and thereby, that all available information has been used within the interval.
We now focus on the case of a quadratic potential function
, where the location r t of the center of the potential represents the control input while the intensity q 0 remains constant. We assume access to continuous measurements of the particle with observation function h(x) = x. Thus, the dynamics for the particle and the observation are
The objective is to maximize the work output during a cycle of period t f by designing the control input r t . We assume boundary condition r 0 = r t f = 0. We also assume that the initial probability distribution is at equilibrium to disregard any amount of work that can be extracted if the system is not prepared at equilibrium. For the initial potential U 0 (x) = q0 2 x 2 , the equilibrium distribution is Gaussian N (0, Σ 0 ) with variance Σ 0 = k B T q0 . In this special linear Gaussian case, the conditional probability distribution of X t given the observations is Gaussian N (m t , Σ t ), where the mean and variance evolve according to Kalman-Bucy filter equations [30]
and dξ t = dZ t -m t dt is the innovation process. In this special case, the work input to the system is
and the conditional expectation of work given the observations is
where we replaced X t with its conditional expectation m t . Upon integration by parts and utilizing the boundary conditions r 0 = r t f = 0,
Finally, taking expectation, the second term disappears and, in order to maximize work output, we end up with the following stochastic optimal control problem
where we introduced the control input u t = r t 1 2 m t . Proposition 6.1: Consider a particle governed by the overdamped Langevin equation with quadratic potential and a linear observation model (17), and assume that the boundary conditions r 0 = r t f = 0 and the equilibrium initial distribution N (0, k B T q0 ) hold. The maximum work output over [0,
and the optimal control is given by r t = ( 1 2 -P t )m t , where m t and Σ t are the conditional mean and variance of X t given by the Kalman-Bucy filter equations (18) and
or, in closed form,
Moreover, the efficiency at maximum power is
Proof: We use the following candidate value function V(t, m) = P t m 2 + Q t where P t and Q t are time varying parameters to be determined later. Express the objective function as
Upon adding the zero term t f 0 dV(t, m t )-V(t f , m t )+V(0, m 0 ) = 0 to J, and using
we arrive at
Now we use our freedom to specify P t and Q t to make the first term a complete square and second term zero.
We also set the terminal condition to zero to make V(t f , m t f ) = 0. The resulting expression for J, after taking the expectation, is
The term V(0, m 0 ) does not depend on u. Therefore, the optimal control is u t = -q0 γ P t m t , and the optimal value is
where we used m 0 = 0. The result of the proposition follows by noting
dt and changing P t → q0 γ P t . Remark 6.1 (Steady-state analysis): Explicit formulas for average power -W * t f and efficiency is obtained in steady-state as t f → ∞. The steady-state average power is
where SNR = 2γk B T q 2 0 σ 2 v represents the signal to noise ratio. The limit is obtained using the steady-state values P ss = -1 2 and
). In particular, as σ v → ∞, power converges to zero (because in this case, effectively, no information is available), and as σ v → 0, power attains its maximum value q0k B T γ . The efficiency at steady state becomes
Note that as σ v → ∞, the efficiency goes to 1. However, as σ v → 0, the efficiency converges to 0, since the available Steady-state values for maximum power and efficiency for a linear Gaussian over-damped information machine, as a function of the measurement noise. information is infinite. We numerically illustrate power and efficiency tradeoffs as functions of σ v in Figure 1.
Remark 6.2: The work presented in this section parallels the work of Sandberg etal. [21], but the model and approach are fundamentally different. A major difference is on definition of work (4) as well as the nature of the control variable. In spite of the differences, we arrive at the qualitatively similar results on power and efficiency (c.f. [21, Figure 3]).
After more than a century of debate the celebrated paradox of Maxwell's demon led to the deep insight that information is physical and can be traded for work. From this vantage point several authors sought to quantify the relation between work, heat, dissipation, entropy and information (e.g., [21], [17], [18], [11]). In a similar spirit we quantify the work that can be drawn from a thermodynamic ensemble in contact with a heat bath of fixed temperature, as information on the state of the ensemble becomes available either at one point in time (Proposition 4.1) or via monitoring over a finite interval (Proposition 5.1). The bounds that we obtain are tighter than in earlier works, as they account for the dissipation over finite-time transitions as well as for the information that has not been utilized at the end of the operating interval. Further, we work out explicitly the optimal control protocol to extract maximal work in the linear-Gaussian case (Proposition 6.1) that illustrates the apparent trade-off between power and efficiency (cf. 1).
Extension of results to nonlinear dynamics requires solving a stochastic optimal control problem with partial observations which, in general, is challenging. However, intuitively, the bounds inform that the optimal protocol needs to trade-off a dissipation cost, that depends on the Wasserstein length of the state trajectory, against the utilized information.
Formally, the standard Brownian motion satisfies E[dBt] = 0 and obeys the Itô rule (dBt)
= dt and dBtdt = 0. 2 This definition of work is standard in stochastic thermodynamics, but differs from the one in[21]. See also[22],[23],[24],[25].
This is a notion of non-equilibrium free energy, since p does not need to be the Boltzmann distribution p ∝ exp(-U k B T )