In [1] the authors showed that the Cramér Rao bound (CRB) [2], [3] on the variance of an unbiased estimator of a parameter θ i in the measurement model y ∼ N n (x(θ), σ 2 I), x ∈ R n , θ ∈ R p , p ≤ n could be written as
where g i = ∂x(θ) ∂θi characterizes the sensitivity of the mean to the i th parameter, G = [g 1 , . . . , g p ], G i consists of all columns of G except g i , P ⊥ Gi = I -P Gi , and P Gi is the orthogonal projection onto the subspace G i . The denominator in (1) is the Euclidean inner product g i , P ⊥ Gi g i , and the geometry is shown in Fig. 1. This result raises the question of whether there exists a more general version of the geometry illustrated in Fig. 1. In fact, we were motivated to find a similar geometry for the case where θ parameterizes the covariance matrix in the multivariate normal model. With this motivation, our ambition in this paper is to illuminate the geometry of the Cramér-Rao bound in This work is supported in part by NSF under grant CCF-1712788, and by the Air Force Office of Scientific Research under award number FA9550-14-1-0185. Consequently the U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. We start with a two channel linear estimation problem and derive the geometry of a lower bound on the Grammian of the estimation error. Then, we exploit this estimation result to discuss the geometry of quadratic covariance bounds. The quadratic covariance bounds can be derived as bounds on the minimum error covariance when linearly estimating the centered error scores from centered measurement scores. Different classes of quadratic covariance bounds are characterized by their associated measurement scores. The conceptual framework is the Hilbert space of second order random variables, but when specialized to the Cramér-Rao bound in a multivariate normal model, the Hilbert space inner products reduce to Euclidean inner products.
Let H be a Hilbert space, with the inner product •, • : H × H → C, and the associated norm • . For any positive integer q denote the direct sum of q copies of H by H q . H q is a Hilbert space with an appropriate choice of inner product. For any ordered set of vectors u = (u 1 , . . . , u q ) ∈ H q , the Gram matrix, or Grammian K(u) ∈ C q×q is defined to have elements
N o w ,g i v e na n o t h e r v=(v 1 , . . . , v p )∈ H p , w ed e fi n et h e c r o s sG r amm a t r i xK( u ; v )b e tw e e n t h e tw os e t sa s t h eq ×p m a t r i xw i t he l em e n t s K( u ; v ) i j = u i , v j .
(
T h es t a n d a r d i n n e rp r o d u c to nH q i sT r ( K( u ; v ) ) .
W i t h a n ym a t r i x T∈C m×P w e c a n a s s o c i a t e a l i n e a ro p e r a t o r L T : H p → H m g i v e nb y L T v= p j = 1 t 1 j v j , p j = 1 t 2 j v j , . . . ,
w h e r e{ t i j }a r e t h ee l em e n t so fT.
T h e G r am m a t r i xK( u )a n d t h ec r o s s G r am m a t r i xK( u ; v ) h a v e t h e f o l l o w i n gp r o p e r t i e s :
1 )K( u ) 0 .
2 )K( u ; v )= K H ( v ; u ) .
3 )F o rc 1 , c 2 ∈C, K( c 1 u 1 +c 2 u 2 ; v 1 ) =c 1 K( u 1 ; v 1 )+ c 2 K( u 2 ; v 1 ) .
4 )F o ra r b i t r a r yT 1 , T 2 ∈C m×q a n dT 3 ∈C ,w eh a v e
EX P ER IM EN T C o n s i d e ratw oc h a n n e le s t im a t i o np r o b l em w h e r et h ee l em e n t so f u =(u 1 , . . . ,u q )a r et ob ee s t im a t e df r omt h e e l em e n t so fv =(v 1 , . . . , v p ) .F o rs im p l i c i t y ,l e tK uu = K
w h e r eI q i s t h e i d e n t i t y m a t r i xo fs i z e q .T h e r e f o r e ,d e fi n i n g
T h u s ,L KuvK 1 vv vi s t h eb e s t l i n e a re s t im a t o ro f t h ee l em e n t s o fu=(u 1 , . . . ,u q )f r om t h ee l em e n t so fv=(v 1 , . . . , v p ) .
w h e r e t h e l a s t i n e q u a l i t yc om e s f r om t h e f a c t s t h a tK( v ; e )= 0 , a n d (
vv vi s t h eo r t h o g o n a lp r o j e c t i o no f uo n t o t h e s u b s p a c e s p a n n e db yt h ee l em e n t so fv , w h i c h w e w r i t ea sP v u = L KuvK 1 vv v .F u r t h e rm o r e , K(
w h e r e t h es e c o n de q u a l i t yc om e sf r omp r o p e r t y4 i nS e c t i o n I I ,a n d
T h e r e f o r e ,w ec a nd e c om p o s e t h eG r amm i a nK uu a s
N o w ,c o n s i d e r t h es p e c i a lc a s e w h e r e K u v = [I q 0 q ×( p -q ) ] f o rs om e p>q .F r om ( 1 2 )w eh a v e
w h e r e K -1 v v q q i s t h en o r t hw e s t q ×qb l o c ko fK
T h e r e f o r e , f r om ( 1 3 )a n d ( 1 4 ) ,w eh a v e
w h e r e P ⊥
v 2 i st h eo r t h o g o n a lp r o j e c t i o no fv 1 =(v 1 , . . . , v q )o n t ot h e s u b s p a c es p a n n e db yt h ee l em e n t so fv 2 =(v q + 1 , . . . , v p ) . E q u a t i o n ( 1 5 ) i so u rm a i n r e s u l t .T h e e l em e n t so fK a r eH
T h ec om p o s i t ec o v a r i a n c e m a t r i x f o r[ e T θ ( y ) , s T θ ( y ) ] T i s (
w h i c him p l i e st h a tt h ee l em e n t s o f e θ ( y )b e l o n gt ot h e s u b s p a c e s 1 , . . . , s m .
A sa s p e c i a le x am p l eo f t h eg e om e t r i c a l i n t e r p r e t a t i o n i n ( 1 9 ) , w ec o n s i d e r t h eC r amé r -R a ob o u n do n t h ee r r o rc o v a r i a n c eo f a nu n b i a s e de s t im a t o ro f t h ep a r am e t e r sθ= [θ 1 , . . . , θ q ] T ∈ R q .T h eF i s h e rs c o r e i sd e fi n e da s
w h i c hh a sz e r o m e a n [ 8 ] .T h u s , t h ec e n t e r e dF i s h e rs c o r e i s s ( y , θ )= σ θ ( y ) ,a n d t h eF i s h e r i n f o rm a t i o nm a t r i x i s J ( θ )= E[ s ( y , θ ) s H ( y , θ ) ] .F r om t h ep r o p e r t i e so f t h e s c o r e f u n c t i o n i n( 2 1 ) ,t h es e n s i t i v i t y m a t r i xi sT( θ ) =[ I q 0 q ×( p -q ) ] .
T h e r e f o r e , t h eg e n e r a l r e s u l to f ( 1 9 )s p e c i a l i z e s t o
w h e r e J -1 ( θ ) q q i s t h e q×qn o r t hw e s tb l o c ko f i n v e r s eo f t h eF i s h e r i n f o rm a t i o nm a t r i x J -1 ( θ ) .B u t f r om ( 1 5 ) , t h i sm a y b ew r i t t e na s C o v θ { ĝ ( y ) } J -1 ( θ ) q q = K( P ⊥ s 2 s 1 )
w h e r e s 1 =(s 1 , . . . , s q ) , s 2 =(s q + 1 , . . . , s p )a n dP ⊥ s 2 s 1 = s 1 -P s 2 s 1 ,a n dP s 2 s 1 i st h eo r t h o g o n a lp r o j e c t i o no ft h e e l em e n t so fs 1 o n t o t h es u b s p a c es p a n n e db y t h ee l em e n t so f s 2 .T h em a t r i x K( P ⊥ s 2 s 1 )i s t h eG r amm i a no fe s s e n t i a ls c o r e s s 1 ,a f t e rp r o j e c t i o no n t ot h es u b s p a c es p a n n e db yn u i s a n c e s c o r e s s 2 .T h ee l em e n t so f K a r eH i l b e r t s p a c e i n n e rp r o d u c t s d e fi n e db y ( 1 7 ) .T h eC r am é r -R a ob o u n do n t h ee r r o rv a r i a n c e o fa nu n b i a s e de s t im a t o ro f t h ep a r am e t e rθ 1 i s v a r θ { θ 1 ( y ) }≥ J -1 ( θ ) 1 1
where s 2 = (s 2 , . . . , s p ). Importantly, the denominator in ( 24) is a Hilbert space inner product defined by (17).
We demonstrate two examples for which there exists a Euclidean space geometry counterpart for the Hilbert space geometry of the Cramér-Rao bound.
Example 1: Complex multivariate normal measurements with parameterized mean.
Assume the measurement y is a proper random vector distributed as CN n (x(θ), C), x ∈ C n , and θ ∈ R p . Let q i = C -1/2 ∂x(θ) ∂θi and define g i = (q T i , q H i ) T . The (i, j) th element of the the Fisher information matrix is the Euclidean inner product of the g i and g j [1], [8]. That is
Define G 1 = [g 1 , . . . , g q ], G 2 = [g q+1 , . . . , g p ]. From (23), the Cramér-Rao bound on the error covariance of an unbiased estimator θ(y) of θ = [θ 1 , . . . , θ q ] may be written as
where
is the orthogonal projection matrix onto the subspace spanned by the columns of G 2 . In this case the Hilbert space inner products of (23) are computed as Euclidean inner products in C 2n .
Example 2: Complex multivariate normal measurements with parameterized covariance. Assume the measurement y is a proper random vector distributed as CN n (m, R(θ)), m ∈ C n , and θ ∈ R p . Let D i = R -1/2 (θ) ∂R(θ) ∂θi R -1/2 (θ). The (i, j) th element of the the Fisher information matrix may be written as an inner product of D i and D j [8]. That is
Define D 1 = (D 1 , . . . , D q ), D 2 = (D q+1 , . . . , D p ). Again, from (23), the Cramér-Rao bound on the error covariance of an unbiased estimator θ(y) of θ = [θ 1 , . . . , θ q ] may be written as
where the orthogonal projection P ⊥ D2 D 1 , and K(P ⊥ D2 D 1 ) in (28) are defined with respect to the inner product in (27). Again, the Hilbert space inner products of (23) are replaced by the Euclidean inner products in C n×n defined in (27).
A general class of quadratic covariance bounds on estimation error covariance may be represented as the Grammian of the error score after projection onto the space orthogonal to the subspace spanned by the measurement scores. This is the Hilbert space picture, as the Grammian is defined with respect to inner products in a Hilbert space of second order random variables. This geometric result may be applied to a large class of quadratic covariance bounds such as Barankin, Cramér-Rao, and Bhattacharyya bounds, by considering their corresponding measurement scores. In the case of Fisher score, the bound is determined by the inverse of the Grammian of essential scores after projection onto the subspace orthogonal to the subspace spanned by the nuisance scores, a result that clarifies the influence of nuisance parameters on parameter estimation.