Prior knowledge and assumptions are central to many statistical problems, and in practice it is important to assess their impact on the final inference. When such an assessment is missing, it can be difficult to tell whether the results could be reproduced with a different prior, or whether similar studies could be made more efficient by incorporating existing information that was neglected. Of course, each researcher chooses whichever prior seems most appropriate to them, but reporting the impact of the prior allows others, and even the original researchers, to better interpret the results. These considerations are important in many scientific studies. For example, Chen et al. (2019) propose a Bayesian analysis of the brightnesses of a large collection of stars based on a multi-telescope astronomical dataset, and highlight that scientific prior distributions provide key information about each of the specific instruments and play a substantial role in the final inference. For both the scientists directly involved in the study, and also others who rely on their work, it is important to understand the role of the prior distributions used, e.g., do the priors associated with one particular instrument have a much greater impact on the inference than those for other instruments?
One appealing and interpretable way to assess prior impact is to provide a measure of the effective prior sample size (EPSS), i.e., the approximate number of observations to which the information in the prior is equivalent. Gaussian conjugate models offer a canonical example: with observed data y i iid ∼ N (μ, σ 2 ), for i = 1, . . . , n, and conjugate prior distribution μ ∼ N (μ 0 , σ 2 /r), the posterior distribution of μ is N (w n ȳn + (1w n )μ 0 , σ 2 /(n+r)), where w n = n/(n+r). Based on the denominator n+r in the expression for the posterior variance, the effect of the prior appears to be equivalent to that of r samples, so we say that the EPSS is r. However, this formulation faces two challenges: (a) it is not immediately clear how to generalize beyond conjugate models; and more importantly, (b) when μ 0 is arbitrarily different to ȳn , the prior impact on the posterior mean is arbitrarily large, and is therefore clearly not equivalent to that of r samples.
EPSS measures have gained substantial attention in the literature, and a number of strategies have been proposed in response to the two challenges above, e.g., Clarke (1996), Morita et al. (2008), and Morita et al. (2010). Most of the strategies proposed rely on a comparison between the actual prior π and a default or baseline prior π b , e.g., the improper prior π b (μ) ∝ 1 would be a natural choice for the baseline prior in the Gaussian conjugate model above. This comparative information approach is necessary because there is no universal "non-informative" prior against which to measure prior impact, and Bayesian inference cannot be conducted without a prior. Early generalizations along these lines sought to match the prior π to a hypothetical posterior distribution constructed using the baseline prior π b and some hypothetical previous samples, that is, they interpreted the prior π as the posterior from a previous analysis. The EPSS is then defined as the number of observations used in the hypothetical posterior distribution, e.g., Clarke (1996) and Morita et al. (2008). These approaches successfully generalize the notion of EPSS, but do not address concern (b) regarding the real impact of the prior when the data mean and prior mean differ substantially. Indeed, these methods do not consider the observed data or the real posterior distribution at all. Reimherr et al. (2014) instead suggested minimizing the discrepancy between two posterior distributions, one using the real prior π and the other using the baseline prior π b . In this case the EPSS is defined as the difference in the number of samples used by the two posteriors. Similar ideas have also been proposed in slightly different contexts, e.g., see Lin et al. (2007) and Wiesenfarth and Calderazzo (2019). The Reimherr et al. (2014) method offers many improvements over early approaches and goes beyond simply capturing the variance of the prior; it also partially quantifies the impact of the prior location. However, it averages over the data using the bootstrap, and therefore does not quantify the impact of the prior for the specific analysis carried out with the observed data at hand, which is of most interest in practice.
Another recent approach introduced by Neuenschwander et al. (2020) defines the EPSS as the expected local-information-ratio (ELIR), i.e., the prior mean of the ratio of the prior information and Fisher information (of a single observation). This approach has the elegant property that a sample of size n from the posterior predictive distribution has an effective sample size of n plus the EPSS. However, similarly to the strategies already mentioned, this method does not take into account the observed data and therefore does not fully capture the impact of the prior on the Bayesian analysis at hand.
In this paper, we follow a similar approach to Reimherr et al. (2014) but propose a new EPSS measure which addresses the above limitations by conditioning on the observed data, and thereby directly quantifies the prior impact for the actual analysis performed. Our measure is called the mean Observed Prior Effective Sample Size (mOPESS), where 'Observed' indicates that the observed data is treated as fixed, and 'mean' refers to an average over additional future samples drawn from the posterior predictive distribution (the purpose of which will become clear). This new measure was inspired by the work of Efron and Hinkley (1978) which highlighted that observed Fisher information is sometimes more useful than expected Fisher information. By providing an explicit definition of the mOPESS in terms of future observations, we also identify the real-world estimand of interest, which we call the Observed Prior Effective Sample Size (OPESS), i.e., a quantity that would be realized if future samples were actually collected. The interpretation of the OPESS is essentially the number of additional samples that must be combined with the baseline prior π b in order to obtain similar inference to that under our actual prior π. In other words, the OPESS communicates how much sampling effort is saved by having access to the information in the prior π, rather than only the default information captured by π b . Further appealing properties of our mOPESS measure include a Bayes estimate interpretation and no lower limit on the observed data sample size n. The latter property is important because prior impact is often most pronounced, and therefore of most interest, when the sample size is small. In contrast, Reimherr et al. (2014) require n to be large because their method relies on the bootstrap and an accurate estimate of the "true parameter" value, see Section 2.2 for a review. In summary, our approach represents a substantially improved method for quantifying prior impact in practice, and its real-world interpretation makes it a valuable tool for clearly reporting the contribution of priors in Bayesian analyses.
One possible limitation of our approach is the need for a baseline or default prior against which to compare the prior at hand. However, in our opinion, a key purpose of an EPSS measure is to communicate the impact of a prior, and for this objective comparing to a standard prior that many researchers are already familiar with is in fact a strength rather than a weakness. Of course, if necessary our mOPESS measure could be computed for several different baseline priors, but we suspect one version will usually provide a sufficient summary of prior impact. As mentioned above, most of the existing literature on EPSS measures has similarly concluded that comparing against a baseline prior is desirable or necessary or both. This paper is organized as follows. Section 2 briefly overviews some topics from the broader literature connected to EPSS measures and their uses, then summarizes the EPSS methods on which we build, and lastly provides a motivating Gaussian example to illustrate our mOPESS measure. Section 3 defines the mOPESS and discusses its computation, first in general and then in the specific case of the motivating Gaussian example introduced in Section 2.3. Section 4 provides intuition and theory supporting our method. Section 5 provides additional numerical results in the form of non-conjugate Gaussian, Beta-Binomial, and regression model examples. Section 6 provides a summary. Proofs are given in the appendices.
To provide greater context for the importance of EPSS measures, we now briefly discuss several concepts that have been studied in the literature on prior distributions, and highlight their connections to EPSS measures.
Prior-likelihood conflict, e.g., Evans et al. (2006); Bousquet (2008); Walter and Augustin (2009); Nott et al. (2020Nott et al. ( , 2021)). The topic of prior likelihood-data conflict is very much related to EPSS measures in the sense that such conflict can indicate that the prior distribution has a large influence on the final analysis. On the other hand, measures of EPSS are not restricted to quantifying prior-likelihood conflict: in the case of prior-likelihood alignment or weakly informative priors, the mOPESS measure proposed here gives non-zero values which describe the extent to which the prior is facilitating the inference, e.g., by increasing posterior concentration.
Sample size determination, e.g., Wang et al. (2002); Sahu and Smith (2006); Clarke et al. (2006); Gupta et al. (2016). In clinical trials, a classical problem is to determine the sample size needed to achieve a certain power when performing a hypothesis test for the presence of a treatment effect, see Sahu and Smith (2006) for more detailed discussion and examples. In its classical form this line of research does not usually emphasize prior impact and is not closely related to our work. However, the concepts of prior-likelihood conflict and EPSS are important for sample size determination in adaptive clinical trials, where priors are typically constructed from an interim analysis, e.g., Hobbs et al. (2013) uses effective historical sample size (EHSS) to determine a randomization procedure for allocating patients. Similarly, Wiesenfarth and Calderazzo (2020) compare EPSS measures that quantify prior information in terms of historical/external samples (e.g., Morita et al., 2008) or current/new samples (e.g., Reimherr et al., 2014); and tailor the method in Reimherr et al. (2014) to the adaptive design setting. Wiesenfarth and Calderazzo (2020) specifically investigates priors that adaptively discard prior information in the case of prior-data conflict, e.g., robust mixture (Berger et al., 1986), power (Ibrahim et al., 2015) and commensurate (Hobbs et al., 2012) priors. Our mOPESS measure could be applied in such studies to identify other similarly "adaptive" priors and to better quantify their impact in Bayesian clinical trials.
Bayesian prior sensitivity analysis, e.g., Berger (1990); Weiss (1996); Roos et al. (2011Roos et al. ( , 2015)). This line of work is aimed at assessing the impact on the posterior inference when the prior distribution is perturbed, thus possibly identifying parameters for which the posterior inference is highly sensitive to changes in the prior. In our work, we only assess the impact of the given prior on the current inference, as compared to a baseline prior, and do not consider whether this impact would be different for other similar priors. On the other hand, if posterior inference substantially varied across a group of priors, then our mOPESS measure would likely be high for some or all of the priors in question, and so our measure could be used to detect this type of sensitivity.
Prior construction for objective Bayes, e.g., Kass and Wasserman (1996); Ghosh et al. (2011b); Berger et al. (2015); Consonni et al. (2018); Leisen et al. (2020). For example, Ghosh et al. (2011a) constructs "objective priors" by maximizing an approximate expression for the distance between the prior and the posterior under a general divergence criterion. In general this line of research focuses on constructing priors in such a way as to avoid the subjective nature of Bayesian inference, e.g., ensuring the prior has little influence in cases where the data contain substantial information. Our approach is about assessing the prior influence and giving an intuitively quantitative measure of the impact of an informative prior or subjective prior. Moreover, as opposed to approximate measures based on asymptotic expansions (e.g., Ghosh et al., 2011a), our method works on the exact posteriors and is particularly designed for finite sample settings, in which prior impact is potentially substantial. In our approach, reference priors (typically, but not necessarily, non-informative) are only used as a baseline to compare our working priors against. On the other hand, our mOPESS measure could likely be used to assess whether a prior is a suitable alternative to the current non-informative prior of choice.
Suppose that π is our prior distribution for a collection of unknown parameters of interest θ ∈ Θ. Let π b be a baseline prior and x = {x 1 , . . . , x n } be unknown hypothetical previous data with probability density f (x|θ). Imagine that our real prior π is the posterior distribution q b (•|x) ∝ f (x|•)π b (•) computed using the unknown hypothetical dataset x. Under this formulation, Clarke (1996) considers
where KL(g, h) denotes the Kullback-Leibler divergence (KL divergence) defined as Θ log(g(θ)/h(θ))g(θ)dθ, and X is the support of f (for simplicity we assume X to be the same for all θ ∈ Θ). In words, the approach of Clarke (1996) is to find the hypothetical dataset x * that, when combined with the baseline prior π b , produces the posterior distribution q b (•|x * ) with minimum KL divergence from our true prior π. The EPSS can then be quantified as the number of individual observations contained in x * . Note that the density f is a user specified hypothetical distribution for prior data, and is not necessarily the same as the model for any actual data.
The above approach is distinguished from most other methods (such as those mentioned below) in that it gives a specific dataset x * which represents the information in the prior. An advantage of this approach is that x * can potentially capture other aspects of the information contained in π in addition to the EPSS. However, x * has no concrete relation to the likelihood or data at hand, which we consider to be a drawback, at least when the impact of the prior on a specific analysis is of primary interest. Morita et al. (2008) adopt a similar approach but measure distance using the difference of the second derivative of the log densities rather than KL divergence. Furthermore, to avoid the peculiarity of reporting a specific dataset x * , and to take account of uncertainty regarding the hypothetical dataset; they take an expectation over x, i.e., they compute
. This treatment of the hypothetical previous data x may be preferable to that of Clarke (1996). But for the purpose of assessing prior impact on a specific analysis, the Morita et al. (2008) method suffers from the same fundamental problem of not taking the likelihood of any actual data into account.
To address this limitation, Reimherr et al. (2014) introduced the notion of priorlikelihood discordance and incorporated it in their measures of EPSS. The key change they proposed was to compare two posterior distributions rather than comparing a prior to a (hypothetical) posterior. To make the comparison, under each prior π, they consider the expected mean squared error (MSE) when a draw from the posterior is used to estimate the true parameter θ T , i.e.,
where the "posterior MSE", as defined in Reimherr et al. (2014), is
Reimherr
where θ is the maximum likelihood estimate of θ T based on y, and Û is computed by averaging over datasets of size k drawn from the empirical distribution (hence the constraint that k n). By a slight abuse of terminology, we refer to their averaging method as bootstrapping (as they do). One of the novel aspects of this formulation is that their estimate of EPSS, r, is allowed to be negative. This is helpful when, for example, we are trying to assess if π is a low-information prior and therefore might feasibly have less impact than π b .
The approach of Reimherr et al. (2014) described above has a number of advantages over earlier methods: (i) it focuses on the impact of the prior on posterior inference;
(ii) it incorporates the likelihood, although for reduced data size; and (iii) it proposes a potentially reasonable method for generating datasets to combine with π and π b (bootstrapping). There are however still some limitations of their approach. Firstly, their method averages over the data and therefore their measure of EPSS does not tell us what the impact of the prior is on the inference using the observed data y, which is of most interest in practice. Secondly, their approach relies on bootstrapping the data and estimating θ T which both require n to be large, but the impact of a prior is usually greatest and of most interest when n is small. Lastly, their use of MSE is not necessarily the best way of quantifying the difference between two posterior distributions and therefore the prior impact. Indeed, there is in fact no reason to introduce the notion of a true parameter value θ T in order to measure prior impact.
Suppose that we have observed data
∼ N(μ, σ 2 ), for i = 1, . . . , n. Assume that σ 2 is known, that our prior for μ is a conjugate prior, denoted π(μ) ≡ N (μ 0 , λ 2 0 ), and that the baseline prior is π b (μ) ∝ 1. Suppose that μ = μ 0 = 0, σ 2 = 1, and λ 2 0 = 0.1. We now use this example to illustrate the behavior of our EPSS measure, which is called the mean Observed Prior Effective Sample Size (mOPESS). All mathematical and computational details are deferred to Section 3.
The top left panel of Figure 1 shows the mOPESS plotted against the data mean ȳn . It can be seen that the mOPESS increases with the difference |ȳ nμ 0 | between the prior mean and data mean, which is a key feature of our proposed measure of EPSS. The top right panel of Figure 1 shows the quantiles of the Observed Prior Effective Sample Size (OPESS), i.e., it summarizes the distribution of which the mOPESS is the mean for any given ȳn . The OPESS represents a real-world quantity (defined in Section 3.1) which captures the impact of the prior, and can in principle be observed by collecting more samples. Variability in the OPESS for a fixed value of ȳn indicates genuine uncertainty about the future observations. The mOPESS (i.e., the mean of the OPESS distribution) averages over this uncertainty and is our preferred single number summary of prior impact.
The bottom left panel of Figure 1 corresponds to the point indicated by a "+" symbol in the top left panel, i.e., one of two points with the largest value of |ȳ nμ 0 |. In particular, the bottom left panel shows the priors π and π b , as well as the corresponding posterior distributions denoted q n and q b n , respectively. Note that the posterior distribution q n is pulled towards zero by the informative conjugate prior π. The top left panel shows that in this case the mOPESS is larger than for values of ȳn that are closer to the prior mean μ 0 = 0. In particular, the mOPESS has a value of about 14.5, which has the interpretation that on average an investigator using q b n would need to collect 14.5 additional samples to obtain similar inference to an investigator using π, for the specific value of ȳn currently at hand, i.e., for ȳn ≈ -0.6.
The bottom right panel of Figure 1 illustrates the case where |ȳ nμ 0 | is smallest across the points plotted in the top left panel, i.e., the point indicated by a cross in the top left panel. From the bottom right panel we can see that in this case both Figure 1: (Left top) mOPESS as a function of ȳn . The solid curve shows a LOESS (LOcal polynomial regrESSion) fit to the points. (Right top) 95% quantile (long dash curve), median (short dash curve), and 5% quantile (dash-dot curve) of the OPESS as a function of ȳn . The solid curve is the same as in the left panel. (Left and right bottom) posteriors q n and q b n (dashed lines) and priors π and π b (solid lines) for the dataset indicated by a green "+" symbol and a cross, respectively, in the top left plot.
posteriors are centered very close to zero. Specifically, the posteriors have substantial overlap because the conjugate prior π is centered at μ = μ 0 = 0 ≈ ȳn and so does not cause q n to have a substantially different mean to q b n , only a smaller variance. Returning to the top left panel we can see that this case corresponds to a mOPESS value of around 10.5, which is one of the smallest among the points plotted.
In conclusion, we can see that the mOPESS is larger the further ȳn is from μ 0 = 0 and that this is because the prior has more impact on the posterior in these cases. Thus, at least in this simple example, our mOPESS measure of EPSS seems to have an intuitive interpretation that well captures the way the prior impact changes with the observed data.
Our guiding intuition is that we want to know how many extra samples are needed to obtain similar inference under the baseline prior as that under our informative prior. To that end, for each m = n + r, where r ∈ Z ≥0 , we introduce a hypotheti-
The superscripts '(m)' are necessary because we do not assume that x (m+1) = x (m) ∪ {x m+1 }, for reasons to be explained in Section 3.2. If x (m) was known for all m then intuitively we would choose the EPSS to be r = mn for the m that minimizes the distance between the two posterior distributions
, where π is our real prior whose EPSS is to be measured, π b is the baseline prior, and f is the model. We denote the distance for a given m by D(q b (•|x (m) ), q(•|y)). The collection of expanded datasets is denoted by x L = {x (n) , x (n+1) , . . . , x (L) }, where L is the maximum feasible value of m, or in other words Ln is the maximum feasible magnitude of the EPSS associated with π.
We must account for the possibility that our prior π is in fact less impactful than the baseline prior π b . This happens when the prior π is more diffuse than the baseline π b or is similarly diffuse but has greater location agreement with the data than π b . Thus, we also consider the alternative distance D(q b (•|y), q(•|x (m) )), where the extra hypothetical samples are combined with our real prior π rather than with the baseline π b . Again we do not assume that x(m) = x (m) , for reasons to be explained in Section 3.2. We write (n+1) , x(n+1) , . . . , x (L) , x(L) } to denote all the future samples combined, and for conciseness introduce the notation D(m) and D(m) as shorthand for the distances D q b (•|x (m) ), q(•|y) and D q b (•|y), q(•|x (m) ) , respectively. We can now define the underlying quantity of interest.
L , the Observed Prior Effective Sample Size (OPESS) is
where
The OPESS is negative when S n (x all L ) = -1 because this suggests that π is less informative than the baseline prior π b . In practice, the future samples x all L are unknown and therefore the OPESS must be estimated. The mOPESS defined in Definition 3.2 below is simply the posterior mean of the OPESS, and as such provides a convenient estimate of the OPESS. Posterior quantiles of the OPESS distribution and other summaries could also be reported to provide a measure of uncertainty.
where X is the domain of x all L and p(•|y, π) is the corresponding posterior predictive distribution.
In practice, it makes sense for the posterior predictive distribution in Definition 3.2 to factorise, i.e.,
as we now explain. A researcher who does not want to use π would not collect many additional samples and then attempt to find the m to minimize the distance between their posterior and q(•|y). Instead, the researcher would simply collect a fixed number of additional samples r = mn (fixed in the sense that no minimization of the distance to q(•|y) is performed). Therefore the correct question to ask when trying to estimate the OPESS of π is as follows: if there are multiple independent researchers each of whom chooses a different value of r, then whose inference will most closely agree with our inference? This is why, for the mOPESS to correspond to normal scientific procedure, the hypothesized future samples x (m) (and x(m) ) need to be conditionally independent across values of m, given θ. We avoid assuming that xL = x L for essentially the same reason: for the interpretation of the mOPESS to correspond to normal scientific procedure, we cannot assume that each individual researcher computes both q b (•|x (m) ) and q(•|x (m) ) and then decides which to use depending on whether D(m) or D(m) is smaller.
We instead assume there are two researchers in the population for each value of m, one who computes q b (•|x (m) ) and one who computes q(•|x (m) ), and since the researchers will likely have different laboratories it is natural to assume that x (m) and x(m) are independent. Unconditionally, all the future samples are dependent, which corresponds to the real-world in that all additional samples collected would be generated using the same underlying, but unknown, value of θ.
In this paper we set the discrepancy measure D to be the 2-Wasserstein distance, and from hereon replace "D" by "W 2 " in our notation. For p ≥ 1, let u and v be probability measures defined on M with finite p th moment. The p-Wasserstein distance between u and v is defined as
where Γ(u, v) denotes the set of measures on M×M with marginals u and v respectively, and d is a metric on M. Conveniently, in the case of multivariate Gaussian distributions the 2-Wasserstein distance can be computed in closed form, and more generally there are efficient software packages for approximating it given samples from the two distributions at hand, e.g., Schuhmacher et al. (2019). On the other hand, our framework is general and other measures of posterior discrepancy could also be used, e.g., Kullback-Leibler Algorithm 1: General procedure for computing the mOPESS.
Step 1: Compute q b n ≡ q b (•|y) and q n ≡ q(•|y).
Step 2: For j = 1, . . Step 3: Report the (estimated) mOPESS:
(KL) divergence or mean squared error as adopted by Clarke (1996) and Reimherr et al. (2014), respectively. General f -divergences (Ali and Silvey, 1966;Sason and Verdú, 2016), of which KL divergence is a special case, provide further options.
Algorithm 1 summarizes how to estimate the mOPESS in practice. In Step 2, S denotes the number of realizations of the OPESS simulated. The procedure is widely applicable and can be implemented for a large family of models beyond the specific cases considered in this paper. Naturally, we use analytical forms of the posterior distributions and the Wasserstein distances when available; otherwise, we use approximation strategies such as importance sampling or Markov chain Monte Carlo (MCMC) methods (e.g., Marin and Robert, 2007;Liu, 2008;Brooks et al., 2011). For convenience, in Algorithm 1 and the remainder of this paper, we use mOPESS to refer to the estimate M n = 1
Algorithm 2 provides a detailed version of Algorithm 1 for the case of the Gaussian example introduced in Section 2.3. We applied Algorithm 2 to 300 simulations of ȳn , and set S = 10,000 in Step 2, i.e., for each value of ȳn , the value of M n was computed via 10,000 Monte Carlo samples of x all L . The left panel of Figure 2 shows the posteriors q n and q b n (dashed lines) for a single example observed value of ȳn , with n = 20. The priors π and π b are also plotted (solid lines). The right panel of Figure 2 shows the distribution of the mOPESS, i.e., of M n , across 300 datasets. Interestingly, in the current context M n is quite variable and is always higher than the nominal EPSS of 10 (vertical line). The nominal EPSS is 10 because of the following three information based analogies between the prior and data: (i) if n = 10 then the Fisher information is n/σ 2 = 1/λ 2 0 = 10, (ii) if n = 10 then ȳn ∼ π, and (iii) for any n, the posterior distribution is N (0, σ 2 /(n + 10)).
In Section 4.1 we illustrate that there is a good explanation for this disagreement with the nominal EPSS: classical information measures consider the prior in isolation Algorithm 2: mOPESS computation for Gaussian example.
Step 1: Compute the initial posterior distributions
where
Step 2: For j = 1, . .
Part a: Generate extra samples x all L = x L ∪ xL by drawing μ * ∼ q n and then drawing x all
Part c: For m = n, . . . , L, compute the distances
n given by (3.1).
Step 3: Report the (estimated) mOPESS:
and can typically only correspond to the prior impact if there is no data. As soon as some data are collected there is always some disagreement between the prior and the data and therefore M n is usually greater than the nominal EPSS, at least in the current Gaussian conjugate model example. On the other hand, in the right panel of Figure 1, the 5% quantile of M n (x all L ) (dash-dot curve) shows that there often exist Figure 2: (Left) posteriors q n and q b n (dashed lines) and priors π and π b (solid lines) for a single simulated dataset y. (Right) distribution of the mOPESS (M n ) across 300 simulated datasets. The vertical line shows the nominal EPSS of 10. some realizations of x all L such that the value of the OPESS M n is less than the nominal EPSS. Indeed, q b m may by chance be closest to q n after mn < 10 additional samples.
In this section we focus on the Gaussian conjugate model introduced in Sections 2.3 and 3.3, for which we derive theoretical results to justify our proposed method. These results can be seen as general in the sense that the Bernstein von Mises Theorem ensures that the posterior distribution is asymptotically Gaussian under mild conditions, see Van der Vaart (2000) and references therein. On the other hand, the main role of these results is to provide a foundation for understanding and developing the mOPESS, and we emphasize that our method does not rely on asymptotic posterior normality or consistency of the MLE (Maximum Likelihood Estimate). Indeed, our approach is distinguished from many existing methods in that it is designed for small or moderate sample size settings, in which the prior can substantially impact the inference.
Let r = mn and denote the additional samples collected by s
Recall that in our approach described in Section 3.1 the future samples are drawn from the posterior predictive distribution (3.3)-(3.4) under π, meaning that
In contrast to our approach, Morita et al. (2008) sample from the distribution of hypothetical previous data and Reimherr et al. (2014) bootstrap the observed data. Our proposed sampling method is therefore not the only option and in order to provide justification for our choice it is instructive to consider the behavior of M n under several sampling methods. To investigate this, Proposition 4.1 below considers the case where the empirical mean of the future samples sr is exactly equal to its theoretical mean, denoted γ, e.g., γ = μ n (the posterior mean) under our approach of sampling the future observations from the posterior predictive distribution. If the behavior of M n for sr = γ does not make sense then there is little hope that the corresponding sampling method is useful, and if it does make sense then the investigation may offer valuable insights. The proof of Proposition 4.1 is given in Appendix A by Jones et al. (2021). Result (a) of Proposition 4.1 corresponds to our proposed method of sampling the future samples from the posterior predictive distribution (3.3)-(3.4). The result is consistent with the top right panel of Figure 1 in Section 2.3, which shows that the median (dashed curve) value of M n is always equal to or greater than z. To gain further intuition consider the distance W 2 (m) under the conditions of Proposition 4.1:
Inspecting (4.4) reveals that the second term captures the nominal EPSS: setting m = n + z makes the standard deviation of the baseline posterior π b m match that of the conjugate posterior q n , so the second term of (4.4) equals zero, and thus giving z extra samples to the baseline prior minimizes the second term. However, the first term of (4.4) reveals that there is an intuitive reason for the value of M n to often be larger than z: disagreements between the prior and the data as captured by (μ nȳn ) 2 = (z/(z + n)) 2 (μ 0ȳn ) 2 mean that the two posteriors will not be centered in the same location, and the (n/m) 2 term in (4.4) suggests that greater agreement is expected to be obtained by adding further samples to the baseline prior, i.e., by increasing m. Thus, our definition of M n correctly identifies that simply reporting the classical information content of the prior as determined by its standard deviation is not sufficient: we must also take into account the impact of the prior location relative to the data. Of course, the results in Proposition 4.1 also take account of W 2 (m), see Appendix A by Jones et al. (2021) for details.
Bayesian methodology stipulates that the extra samples must be drawn from the posterior predictive distribution, as above, but results (b) and (c) of Proposition 4.1 additionally reveal that several other natural approaches do not work well. Result (b) supposes that γ = ȳn which corresponds to the case where the future observations are sampled from the empirical distribution (bootstrap) or from the baseline posterior predictive distribution, i.e., the posterior predictive distribution under π b and conditional on only y. The first part of result (b), M n = z for ȳn ≈ μ n , is similar to what is seen in Figure 1. However, the second part of result (b), M n < 0 for large |ȳ nμ n |, does not make sense in the current scenario firstly because z > 0 secondly because intuitively the prior impact is large when ȳn is far from μ n . Reimherr et al. (2014) avoided this problem by defining the EPSS so that negative values convey a disagreement between the prior and the data, but there are limitations of their approach as discussed in Section 2.2. Furthermore, there is always some disagreement between the data and the prior so we find it conceptually more appealing to always have positive prior impact (unless our prior is less informative than the baseline).
Result (c) of Proposition 4.1 corresponds to the case where the additional samples are drawn from the conjugate prior distribution π: just as some might argue that the future data sampling method should not be "contaminated" by the prior, others may argue that it should not be "contaminated" by the data! Under the scenario of the proposition, this sampling scheme yields M n = z, which is at least never negative. However, simply recovering the nominal EPSS regardless of the magnitude of |μ 0ȳn | does not convey differences in the impact of the prior, which is the purpose of having a measure of prior impact. In summary, drawing the extra samples from the posterior predictive distribution (3.3)-(3.4) seems to yield the most reasonable behavior.
To study the variation in the OPESS for a given observed dataset, we now derive the theoretical distribution of the OPESS conditional on y for the Gaussian conjugate posterior example discussed in Sections 2.3 and 3.3. More generally, the distribution of the OPESS will typically be hard to derive, but it can be empirically approximated, see Algorithm 1 Step 2. Lemma 4.1 below gives the distribution of the distances W 2 (m) and W 2 (m) conditional on ȳn and μ (drawn from q n , given by (3.7) in Algorithm 2). The proof is given in Appendix B by Jones et al. (2021). We condition on both ȳn and μ because then the two distances are independent which facilitates derivation of the OPESS distribution. The distance distributions conditional on only ȳn are given in Appendix D by Jones et al. (2021). Lemma 4.1 states that both distances follow shifted non-central χ 2 distributions, whose non-centrality parameters depend on μ through λ m and δ m (given in the lemma statement). ∼ N(μ, σ 2 ), for i = n + 1, . . . , m. Then we have
where
and
where
conditional on ȳn and μ, W 2 q b m , q n and W 2 q m , q b n are independent.
Theorem 4.1 below gives the posterior distribution of the OPESS conditional on ȳn . The proof is given in Appendix C by Jones et al. (2021). In the theorem statement, v denotes a possible value of M n (e.g., in the notation P (M n = v|ȳ n )), and t is a dummy variable for the distance corresponding to M n = v, i.e., the distance W 2 (n + v), if v ≥ 0, and the distance W 2 (n + |v|), otherwise. The result gives a separate expression for the case M n = 0 because when m = n the distance between the posteriors (i.e., W 2 (n)) is not random, meaning that an integral over the distance dummy variable t is not required. In the case v ∈ Z/{0}, the integrands specified are tractable because the products are truncated at M (t) and M (t) (defined in Appendix C by Jones et al. (2021)), which are finite for all values of t ≤ σ 2 /(n + z). m , q n and W 2 (m) = W 2 q m , q b n , respectively, as given in Lemma 4.1. Lastly,let g(t,μ,v, M, M ) denote the function that gives P (min
where t m = (tc 2 m )/τ m and tm = (t -c2 m )/κ m , and M and M are known functions (see Appendix C by Jones et al. (2021)). Then
m otherwise, and M (t) and M (t) are finite integers for all values of t ≤ σ 2 /(n + z).
Figure 3 shows two examples of the conditional posterior distribution of the OPESS given ȳn and μ, where ȳn = μ = 0 in the left panel and ȳn = μ = 0.45 = 2σ/ √ n in the right panel. We plot the conditional posterior distribution to gain intuition about how the particular draw of μ from π impacts the conditional distribution of the OPESS. This is important because in reality the value of μ is fixed but unknown, and it is therefore valuable to understand how the distribution of the OPESS changes when we simulate the future samples based on different fixed choices of μ. In Figure 3, the linedot density is a close Monte Carlo approximation to the theoretical conditional density of the OPESS given ȳn and μ, and was obtained by simulating from the theoretical conditional distributions of W 2 (m) and W 2 (m) and averaging the resulting values of the integrand given in Theorem 4.1 (except in the case P (M n = 0) for which no Monte Carlo approximation is needed). The red crosses show the empirical distribution of the OPESS obtained by directly applying the first two steps of Algorithm 1, except with the modification that μ is fixed in Step 2(a). Figure 3 illustrates that for ȳn (and μ) farther from the prior mean μ 0 = 0 (right panel) the conditional OPESS distribution has larger mean and is more right-skewed. This corroborates the numerical results seen in Figure 1. For some values of ȳn and μ the conditional posterior distribution of the OPESS is bi-modal, with one mode at positive values and one at negative values (not shown). For other ȳn and μ, there is a mode at M n = 0, which is prominent in examples where the mOPESS is less than the nominal EPSS. Such scenarios are discussed further in Section 5.2.
We now investigate the properties of Algorithm 1 when using a non-conjugate prior distribution for μ. The setting is identical to that outlined in Section 2.3, except that we set the informative prior to be a t-distribution, π t (μ) ≡ T (μ|ν, μ 0 , λ 2 0 ). The degreesof-freedom parameter, ν, controls the heaviness of the tails, and μ 0 and λ 0 are location and scale parameters, respectively. We set μ 0 = 0 and λ 2 0 = 0.1, and investigate two choices of ν, namely, ν = 4, 100. As ν → ∞ the t-distribution density converges to a Gaussian density, so for large ν we expect the relationship between the mOPESS and ȳn to be similar to that seen in the top left-hand panel of Figure 2. However, for relatively small values of ν, we expect the relationship between the mOPESS and ȳn to be different, because in such cases both the posterior variance and the posterior skewness have nonnegligible dependence on ȳnμ 0 (in contrast to the conjugate Gaussian example of Section 2.3).
The top-left panel of Figure 4 shows the relationship between the mOPESS and ȳ, for the t-distribution prior π t . As in Section 2.3, the values of ȳ were chosen to be the quantiles (k -0.5)/100, for k = 1, . . . , 100, of its distribution, namely a zero mean Gaussian with variance 1 n . The relationship for ν = 100, indicated by a solid red line, mimics the quadratic curve shown in the top-left panel of Figure 1, but we see a concave relationship for ν = 4, represented by a dashed black line. The latter pattern shows that, for ν = 4, the mOPESS is smaller when |ȳ nμ 0 | is larger. This result can be explained by the fact that the posterior variance of μ increases with |ȳ nμ 0 |, see the upper right-hand panel of Figure 4 (dashed black line). We computed the posterior variance for each (ν, ȳn ) pair with Monte Carlo integration using 5×10 5 posterior draws generated by RStan (Stan Development Team, 2020b,a). In the case of ν = 100 (solid red line), the posterior variance of μ is nearly constant for all values of |ȳ nμ 0 |, and To gain further insight, we define the equivalent nominal EPSS of the t-distribution prior by matching posterior variances, i.e., we define it to be the nominal EPSS of the conjugate Gaussian prior π which yields the same posterior variance as under the tdistribution prior π t . The bottom left panel of Figure 4 shows that when ν = 4 and ȳn = -0.6, the equivalent nominal EPSS is 4 (dashed black line), whereas for ȳn near 0 it is 10. In contrast, the equivalent nominal EPSS of a t-distribution prior with ν = 100 (solid red line) is 10 for all ȳn ∈ [-0.6, 0.6], as in the case of a conjugate Gaussian prior with z = 10 (shown as a dotted blue line overlapping with the solid red line).
The pattern described above suggests that subtracting the equivalent nominal EPSS from the mOPESS might shed more light on the comparison between the cases ν = 4 and ν = 100. We call the quantity resulting from this subtraction the excess mOPESS. The bottom right-hand panel in Figure 4 shows the excess mOPESS, and reveals that the excess increases with |ȳ nμ 0 |. Indeed, the relationship is now seen to be what we may have initially expected: a t-distribution prior with ν = 4 (dashed black line) has a qualitatively similar, but smaller, impact than a t-distribution prior with ν = 100 (solid red line).
A third factor that is unexplored in the tetraptych above is the posterior skewness of μ under the two values of ν. As |ȳ nμ 0 | increases, so too does the skewness of the posterior for μ. Furthermore, the sensitivity of the posterior skewness to the quantity ȳnμ 0 is a function of the degrees of freedom parameter ν. Skewness of the posterior also impacts the OPESS. If we generate a μ from the tail of π t n , then M n is nearly certain to be negative. For example, suppose that ȳn < μ 0 . Then our posterior π t n will be rightskewed. Consequently, Algorithm 1 will tend to generate more posterior draws from the region μ E[μ|ȳ n , π t ] than from the region μ E[μ|ȳ n , π t ]. In the former region, P (W 2 (m) < W 2 (m)|μ, ȳn , π t ) is small because the extra samples generated from the predictive distribution will decrease W 2 (q m , q b n ), but will increase W 2 (q b m , q n ), leading to a negative value of M n . Because the magnitude of the skewness decreases with |ȳ nμ 0 |, smaller values of |ȳ nμ 0 | yield fewer negative M n values.
Suppose {y i , 1 ≤ i ≤ n} are independent observations taking values in the set {0, 1}. The unknown parameter θ is the probability that y i = 1. We set the informative prior to be π(θ) ≡ Beta(α, β), where α, β are known hyperparameters, and the baseline prior to be
∼ Bernoulli(θ) for i = n+1, . . . , m (but since θ is unknown it is drawn from its posterior distribution when computing the mOPESS, see Algorithm 1 Step 2(a)). Let q A n and q B m denote the posterior distribution using the original data y = (y 1 , . . . , y n ) T and the expanded dataset x (m) , respectively, under priors A and B. Also define (F A n ) -1 and (F B m ) -1 to be the quantile functions associated with these posterior densities. Then it can be shown that the 2-Wasserstein distance between q A n and q B m is
, see
Theorem 2 in Cambanis et al. (1976). Unfortunately, this distance cannot be expressed in closed form in the case of Beta distributions, but it can be approximated to high precision using numerical integration, which is the approach we take.
In our simulations we set α = β = 5, which corresponds to a nominal EPSS of α + β -2 = 8. The subtraction of 2 highlights that the standard nominal EPSS is relative to the prior sample size of the flat prior Beta(1, 1), which is also our baseline prior π b . To investigate the prior impact across different datasets, we sample 1,000 datasets of size n = 20 with replacement from the sex ratio dataset presented in Section 2.4 of Gelman et al. (2013). The sex ratio dataset consists of the biological sexes of 980 babies born to mothers with placenta previa: 437 of the babies are female, a proportion of 0.446 of the total.
The top left panel of Figure 5 shows the mOPESS values obtained across the 1,000 simulated datasets. The M n estimates have a similar pattern as in the Gaussian conjugate model of Section 2.3, except that M n is less than the nominal EPSS for datasets with ȳn = 0.5. The top right panel of Figure 5 shows that the median of the posterior distribution of M n is similar to the mean. It also shows that the posterior distribution is much wider for datasets with means near 0.1. This is due to the fact that the posterior distribution of θ is strongly right skewed when ȳn is much less than the prior mean of Figure 5: (Top left) mOPESS as a function of ȳn . Each point corresponds to one of the 1,000 simulated datasets. (Top right) Quantiles of the posterior distribution of M n as a function of ȳn including the median (dashed black curve), 95% quantile (long dash green curve), and 5% quantile (dash-dot blue curve). The horizontal solid line shows the nominal EPSS of 8. (Bottom left and right) Comparison of the posteriors q n and q b n and their respective priors in the case where ȳn = 0.1 and ȳn = 0.5, indicated in the top left panel by a green "+" and cross, respectively. 0.5. The skewness is in turn reflected in the future observations x all L , which are simulated conditional on a draw of θ, and this results in large posterior uncertainty for M n . In particular, the right-skewness of q b n means that a large number of future samples can be needed to move the distribution to the right, resulting in some large OPESS values, and the right-skewness of q n means that future samples to the left can substantially shift it, which results in some negative OPESS values. That posterior skewness is reflected in the resulting OPESS distribution is a strength of our approach. Indeed, the large spread in the posterior distribution of M n corresponds to genuine uncertainty about the number of extra samples that need to be collected in order to minimize the distance between q n and q n (or q m and q b n ). Next, we examine the relationship between the mOPESS and the nominal EPSS in two cases, namely, those where the prior mean and ȳn are highly discrepant and perfectly aligned, respectively. The top left panel of Figure 5 indicates a simulated dataset for which ȳn = 0.1 (green "+"), and the bottom left panel shows the corresponding initial posterior distributions q n and q b n (as well as the prior distributions). In this case, the mOPESS value is high (around 12) because the initial posteriors are very different. The bottom right panel of Figure 5 shows an analogous plot for a dataset with ȳn = 0.5, i.e., that labeled with green cross in the top left panel. In this case, the initial posteriors are very similar, which is why the mOPESS value is low (approximately 7.5, the variation across datasets is due to Monte Carlo error). In particular, the mOPESS value is less than the nominal EPSS of 8, a phenomenon that did not occur in the Gaussian conjugate model example of Section 2.3 for any of the datasets considered.
The low mOPESS value occurs here due to circumstances that arise, in this case, due to the discreteness of the data and the future data, as we now explain. The data mean ȳn exactly matches the mean of the prior π, which in turn makes the means of q n and q b n exactly equal. Thus, q n and q b n are very similar to begin with, and it is unclear whether adding more samples to one of these posteriors will further reduce the distance between them. Adding more samples to the baseline posterior q b n could reduce the width of the distribution, and therefore may lead to greater agreement with q n . However, the discrete nature of the data means that one additional sample with value one or zero will necessarily move the posterior mean away from 0.5, therefore potentially increasing the 2-Wasserstein distance between the two posteriors. Of course, if we draw an even number of extra samples then their average may be close to 0.5, so a reduction in the width of the baseline posterior q b n may be achieved without any substantial change in the mean. However, based on the nominal EPSS value, the approximate number of extra samples needed for matching the posterior widths is 8, but the probability of achieving an average of 0.5 (or very close to this) when drawing around 8 samples is not sufficiently high, and consequently the distance W (n) is often smaller than W 2 (m) and W 2 (m) for all m > n. Thus, for many simulations of x all L , we have M n = 0, meaning that the mOPESS M n is shrunk towards zero.
In summary, the mOPESS may be less than the nominal EPSS when the means of the initial posteriors q n and q b n are very similar relative to the size of z. In particular, adding extra samples to one posterior may not reduce the 2-Wasserstein distance between the two posteriors because: (i) if few extra samples are added then the variability in their mean can introduce discrepancies between the posterior means, and (ii) adding many extra samples will introduce discrepancies in the spreads since the initial discrepancy will be over-corrected. Thus, the smallest distance between the posteriors may often be achieved when M n = 0, in which case M n will be small in magnitude. Note that this phenomenon can occur in the Gaussian conjugate model example if n z (whereas in Section 2.3 we set n = 2z).
We now consider the setting of a simple linear regression model:
, ω i ∼ N (0, 1), (5.1) for i = 1, . . . , n, where σ 2 is known, and β = (β 1 , β 2 ) are the unknown model parameters. We note that in simple linear regression models, distributional assumptions on covariates are typically not made. We assume that the distribution of the covariates ω i is known in order to simplify the algorithm for generating hypothetical samples. Let right panel in Figure 7, which shows the mOPESS versus ( βn ) [2]γ 0 , we see that the maximum mOPESS occurs at the maximum observed value of ( βn ) [2]γ 0 (because the maximum value of this discrepancy is large compared to that for ( βn ) [1]μ 0 ). In summary, the mOPESS generalizes to two dimensions as we would expect it to, with joint dependence on βnη 0 .
SummaryIn this paper, we have proposed the mean Observed Prior Effective Sample Size (mOPESS) as a measure of the impact of the prior distribution on the Bayesian analysis at hand. Our measure is different from other methods proposed in literature in that we condition on the observed data, instead of averaging over the data. Furthermore, we do not rely on asymptotic results, meaning that our method can be applied to small or moderate sample size settings, where the prior impact is largest and of most interest in practice.