The study of Whittaker and degenerate Whittaker models for representations of reductive groups over local fields evolved in connection with the theory of automorphic forms (via their Fourier coefficients), and has found important applications in both areas. See for example [15,16,19,29,32,35,49,53,59,61].
Let F be either R or a finite extension of Q p , and let G be a finite central extension of the group of F -points of a connected reductive algebraic group defined over F . Let Rep ∞ (G) denote the category of smooth representations of G (see Section 2.3 below). Let g denote the Lie algebra of G and g * denote its dual space. To every coadjoint nilpotent orbit O ⊂ g * and every π ∈ Rep ∞ (G) we associate a certain generalized Whittaker quotient π O (see Section 2.5 below). Let WO(π) denote the set of all nilpotent orbits O with π O = 0 and WS(π) the set of maximal orbits in WO(π) with respect to the closure ordering.
We recall the notion of admissible nilpotent orbit. It has to do with splitting of a certain metapletic double cover of the centralizer G ϕ for any ϕ in the orbit. We also define a weaker notion of a quasi-admissible orbit. See Section 2.6 below for both notions.
For split p-adic groups, admissibility is also related to the notion of a special nilpotent orbit in the sense of Lusztig (see Section 6.2 below). In particular, for p-adic classical groups the two notions are equivalent ( [44]). Then O is a quasi-admissible orbit.
Note that in the Archimedean case, O is not always admissible, e.g. for minimal representations of U (2, 1), see Section 6.2 below. The notions of admissible and quasi-admissible also differ for the split real forms of E 7 and E 8 , though we do not know whether the non-admissible quasi-admissible orbits appear in Whittaker supports of representations. For the symplectic and orthogonal groups the two notions are equivalent. Proposition 1.2 (Section 6.2). -Let G be either Sp 2n (F ), or O(V ) or SO(V ) (for a quadratic space V over F ), and let O ⊂ g * be a nilpotent orbit. Then the following are equivalent.
We deduce this proposition from [44,50]. It is possible that the notions of admissible and quasi-admissible are equivalent for all G in the case when F is non-Archimedean. In this case, and under the additional assumption that G is classical, it is shown in [31,41] that all O ∈ WS(π) are admissible, for any π ∈ Rep ∞ (G). For exceptional G, a slightly weaker result is shown in [31].
For p-adic F it is conjectured that if G is algebraic then all orbits in WS(π) are special. For classical G this follows from [31,41]. For G = G 2 (F ) this follows from [31,37]. For F = R the analogous statement does not hold. Namely, [58] constructs a small unitary irreducible representation π of G 2 (R). We show in Section 6.2 that WS(π) is also small, i.e. consists of the minimal orbit for G 2 . This orbit is non-special but admissible. For classical algebraic groups over all local fields, all special orbits are quasiadmissible. It is possible that this holds for all groups.
It is quite probable that if G is algebraic and π is admissible and has integral infinitesimal character then all O ∈ WS(π) are special, cf. [4, Theorem D] and [5,Theorem 1.1].
We also prove that for quasi-cuspidal π ∈ Rep ∞ (G), the orbits in WS(π) are F -distinguished. Here, F is non-Archimedean, and π is quasi-cuspidal if all its Jacquet reductions vanish (i.e. r P π = 0 for any parabolic subgroup P ⊂ G) and a nilpotent orbit O ⊂ g * is F -distinguished if the corresponding orbit in g does not intersect the Lie algebra of a Levi subgroup of any proper parabolic subgroup P ⊂ G defined over F . For classical G, it was shown in [41] that all the orbits in the Whittaker support of all tempered admissible (finitely generated) π are Fdistinguished. For a similar result in the case F = R see [26].
One of our basic tools is Lemma 4.8 below, that follows from the Stonevon-Neumann theorem. An analogous lemma first appeared in the non-Archimedean case in [17,Lemma 2.2], and is sometimes referred to as the "root exchange" Lemma. We also use [6,Lemma 5.10] (see Lemma 4.5 below) and its Archimedean analog Proposition 3.1 that we prove in Section 3 below using some properties of modules over algebras of Schwartz functions established in [9]. Two more central tools we use are the deformation technique of [20] and the notion of quasi-Whittaker quotients introduced in this paper, see Section 4.1 below. For our strategy of proof see 1.4 below.
Let γ = (f, h, e) be an sl 2 -triple. Let G γ be the centralizer of γ in G, and G γ be its metaplectic cover (see Section 2.6 below). Let ϕ ∈ g * be given by the Killing form pairing with f . Then G γ acts on the generalized Whittaker quotient π ϕ = π γ (see Section 2.6 below).
We denote by M γ the subgroup of G γ generated by the unipotent elements. Let M γ denote the preimage of M γ under the projection G γ → G γ .
Theorem 1.4 (Section 5.1). -Let π ∈ Rep ∞ (G) and assume that G • ϕ ∈ WS(π). Then (i) If F is non-Archimedean then M γ acts on π ϕ by ± Id. (ii) If F is Archimedean then the action of M γ on the dual space π * ϕ is locally finite.
Let S ∈ g be such that the adjoint action ad(S) diagonalizes over Q and ad(S) * (ϕ) = -2ϕ. We will call such pairs (S, ϕ) Whittaker pairs. Following [42] we attach to (S, ϕ) a certain degenerate Whittaker quotient π S,ϕ . If S = h then this is the generalized (a.k.a. neutral) Whittaker quotient (see Section 2.5 for the definitions).
Theorem 1.5 (Section 4). -Let π ∈ Rep ∞ (G) and let (S, ϕ) be a Whittaker pair such that G • ϕ ∈ WS(π). Then π S,ϕ = 0. Moreover, if F is non-Archimedean then the epimorphism π ϕ π S,ϕ constructed in [20, Theorem A] is an isomorphism.
The special case of p-adic F and admissible π follows from [20,42,57]. Another natural question to ask is: given WS(π), what smaller orbits lie in WO(π)? In the case of GL n the answer is: all the orbits lying in the closure of orbits in WS(π). For general reductive groups we deduce from Theorem 1.5 a partial result, see Theorem 7.4 below.
We also provide global analogs of Theorems 1.1-1.5, see Section 8 below. These analogs generalize several results from [8,18,31,54]. In particular, this puts restrictions on Whittaker supports of cuspidal representations, confirming conjectures from [15,Section 4]. We also provide an analog of Theorem 7.4 and deduce the following corollary. For a certain analogous statement for SL n (K) see Corollary 8.10 below. For other groups, we only have a partial result (see Theorem 8.9 below).
Let us summarize some open questions that arise naturally from the results discussed above.
(1) Over a non-archimedean F , do there exist representations of linear reductive groups with non-special Whittaker support? (2) Over a non-archimedean F , are all special orbits admissible? (3) Analogs of the two questions above over a global field K. (4) Over R, do non-admissible quasi-admissible orbits of split groups appear in Whittaker supports of representations? (5) Over all fields, are all special orbits quasi-admissible? (6) Given WS(π), how does WO(π) look like? Another deep conjecture, posed in [42], says that for any irreducible π, all the orbits in WS(π) lie in the same orbit over the algebraic closure. This is conjectured for all global and local fields, but known only in some examples, in particular for GL n , see [42, Chapter II] for the non-Archimedean case and [52,Theorem D] and [20,Theorem B] for the Archimedean case. For further open questions we refer the reader to [15,Section 5], and [24, Section 1].
In Section 2 we give the necessary preliminaries on sl 2 -triples, smooth representations, oscillator representations, Schwartz induction, generalized and degenerate Whittaker models and covering groups.
In Section 3 we prove several statements on non-generic π ∈ Rep ∞ (B).
Here, B is a Borel subgroup of the metaplectic group SL 2 (R), and we say that π is non-generic if it has no non-zero functionals equivariant under the nilradical N of B by a non-trivial unitary character. Over p-adic fields, [6,Lemma 5.10] implies that the action of N on π is trivial. Over R we prove in Proposition 3.1 that for any non-generic π, the action of the Lie algebra n of N on π * is locally nilpotent.
In Section 4 we prove Theorem 1.5. The proof uses the epimorphism W ϕ W S,ϕ constructed in [20]. Here, W ϕ and W S,ϕ are degenerate Whittaker models, that define the quotients π ϕ and π S,ϕ as the coivariants π ϕ = (W ϕ ⊗ π) G and π S,ϕ = (W S,ϕ ⊗ π) G (see Section 2.5 below). Let us recall the construction of the epimorphism W ϕ W S,ϕ . One can show that S can be presented as h + Z, where h is a neutral element for ϕ and Z commutes with h and ϕ. Consider a deformation S t = h + tZ, and denote by u t the sum of eigenspaces of ad(S t ) with eigenvalues at least 1. We call a rational number 0 < t < 1 regular if u t = u t+ε for any small enough rational ε, and critical otherwise. Note that there are finitely many critical numbers, and denote them by t 1 < • • • < t n . Denote also t 0 := 0 and t n+1 := 1. For each t we define two subalgebras l t , r t ⊂ u t . Both l t and r t are maximal isotropic subspaces with respect to the form ω ϕ , r t contains all the eigenspaces of Z in u t with positive eigenvalues and l t contains all the eigenspaces with negative eigenvalues. Note that the restrictions of ϕ to l t and r t define characters of these subalgebras. Let L t := Exp(l t ) and R t := Exp(r t ) denote the corresponding subgroups and χ ϕ denote their characters defined by ϕ. The Stone-von-Neumann theorem implies
. This is an analog of the root exchange Lemma [17,Lemma 2.2]. We show that for any 0 i n, r ti ⊂ l ti+1 . This gives a natural epimorphism
Altogether, we get
This sequence of epimorphisms naturally defines a sequence of epimorphisms
We see that for each i, π St i+1 ,ϕ is the quotient of π St i ,ϕ by the group A i := L ti+1 /R t , that we show to be commutative. By Proposition 3.1 and [6, Lemma 5.10] discussed above, in order to prove the theorem it is enough to show that π St i ,ϕ is a non-generic representation of A i . For that purpose we show that every unitary character of A i is given by some ϕ ∈ g * with ad * (S ti+1 )ϕ = -ϕ such that ϕ does not lie in the tangent space to O at ϕ. We then define a quasi-Whittaker quotient π St i+1 ,ϕ,ϕ , and show that its dual is the space of (A i , χ ϕ )-equivariant functionals on π St i ,ϕ . Then we generalize (1.2) to quasi-Whittaker quotients, construct some additional epimorphisms and deduce the vanishing of π St i+1 ,ϕ,ϕ from the vanishing of π O for all O = O with O ⊂ O . We find quasi-Whittaker quotients to be an important new notion. For an additional evidence for that see Remark 4.7.
To prove Theorem 1.4 we show in Section 5.1 that the action of any subgroup of G γ isomorphic to a quotient of SL 2 (F ) is locally finite. By a corollary from Proposition 3.1 and [6, Lemma 5.10] it is enough to show that it is non-generic. To show that we choose an sl 2 -triple (e , h , f ) in the Lie algebra of such a subgroup and let ϕ ∈ g denote the nilpotent element given by the Killing form pairing with f . Consider the deformation S t := h + th . For t > 1/2, e acts trivially on the Whittaker quotient π St,ϕ . We show that the action of e commutes with the maps in (1.2), and deduce that π St,ϕ has no (e , ϕ )-equivariant functionals for any t 0.
In Section 5.2 we deduce Theorem 1.3 from Theorem 1.5, by way of contradiction. Namely, for any not F -distinguished O we find a Whittaker pair (S, ϕ) with ϕ in O such that π S,ϕ is a quotient of a Jacquet module of π, and thus vanishes. By Theorem 1.5 we get O / ∈ WS(π). In Section 6 we discuss quasi-admissible orbits. In Section 6.1we deduce Theorem 1.1 from Theorem 1.4 in the following way. We first note that W ϕ (π) is a genuine representation of M γ , which by Theorem 1.4 has a finite-dimensional irreducible subrepresentation ρ. Then we construct from ρ a finite-dimensional genuine representation of G γ and extend it trivially to G ϕ . In Section 6.2 we state several geometric results from [44,45,47,50,51] and discuss the connection between the notions of special, admissible and quasi-admissible. We also deduce from Theorem 1.4 and from [39] that the Whittaker support of minimal representations is also minimal.
In Section 7 we formulate and prove Theorem 7.4 that provides information on WO(π) given WS(π). The proof is based on the method of Section 4, including the quasi-Whittaker quotients, and on Theorem 1.5. We deduce that for π ∈ Rep ∞ (GL n (F )), the set WO(π) is closed under the closure order. We also prove a partial analog for SL n (F ).
In Section 8 we formulate global analogs of Theorems 1.1-1.5 and explain how to adapt the proofs from Section 4-7 to the global case. For example, [6,Lemma 5.10] is replaced by the Fourier decomposition. Our exposition follows [27,Chapter 5].
We thank Joseph Bernstein, Guillaume Bossard, David Ginzburg, Maxim Gurevich, Henrik Gustafsson, Anthony Joseph, Dihua Jiang, Joseph Hundley, David Kazhdan, Axel Kleinschmidt, Erez Lapid, Baiying Liu, Daniel Persson, Gordan Savin, Eitan Sayag, and David Soudry for fruitful discussions, the anonymous referee for useful remarks, and Martin Andler for helping us with the French translation of the abstract.
Part of the work on this paper was done during the visit of the three authors to the Max Planck Institute fur Mathematik in Bonn. We thank the administration of the MPI for the invitation and for wonderful working conditions. Another part of this paper was written while two of us participated in the program "Automorphic forms, mock modular forms and string theory" at the Simons Center for Geometry and Physics, Stony Brook University. We are grateful to the administration of the SCGP and the organizers of the program for wonderful working conditions and for very interesting and helpful talks and discussions on the topic.
Let F be either R or a finite extension of Q p and let g be a reductive Lie algebra over F . We say that an element S ∈ g is rational semi-simple if its adjoint action on g is diagonalizable with eigenvalues in Q. For a rational semi-simple element S and a rational number r we denote by g S r the r-eigenspace of the adjoint action of S and by g S r the sum r r g S r . We will also use the notation (g * ) S r and (g * ) S r for the corresponding grading and filtration of the dual Lie algebra g * . For X ∈ g or X ∈ g * we denote by g X the centralizer of X in g, and by G X the centralizer of X in G.
If (f, h, e) is an sl 2 -triple in g, we will say that e is a nil-positive element for h, f is a nil-negative element for h, and h is a neutral element for e. For a representation V of (f, h, e) we denote by V e the space spanned by the highest-weight vectors and by V f the space spanned by the lowest-weight vectors.
From now on we fix a non-trivial unitary additive character (2.1)
such that if F = R we have χ(x) = exp(2πix) and if F is non-Archimedean the kernel of χ is the ring of integers.
We will need the following lemma which summarizes several well known facts about sl 2 -triples.
Lemma 2.1 ([7, Section 11] or [34]).
(i) Any nilpotent element is the nil-positive element of some sl
) is an sl 2 -triple and Z commutes with two of its elements then it commutes also with the third one.
It is easy to see that the Lemma 2.1 continues to hold true if we replace the nil-positive elements by nil-negative ones (and g h 2 by g h -2 ). Definition 2.2. -We will say that h ∈ g is a neutral element for ϕ ∈ g * if h has a nil-positive element in g, ϕ ∈ (g * ) h -2 , and the linear map
given by x → ad * (x)(ϕ) is an epimorphism. We also say that 0 ∈ g is a neutral element for 0 ∈ g * .
Note that if we identify g with g * in a G-equivariant way and assume ϕ = 0, this property becomes equivalent to ϕ being a nilnegative element for h, or -h being a neutral element for ϕ.
For non-Archimedean F we will work with l-groups, i.e. Hausdorff topological groups that have a basis for the topology at the identity consisting of open compact subgroups. This generality includes F -points of algebraic groups defined over F , and their finite covers (see [6,Section 1]).
For F = R we will work with affine Nash groups, i.e. groups that are given in R n by semi-algebraic equations, and so is the graph of the multiplication map. This generality includes R-points of algebraic groups defined over R, and their finite covers (see [9, Section 1.1], [2, Section 2,3], [13], [55, Section 3]).
Notation 2.3. -If G is an l-group, we denote by Rep ∞ (G) the category of smooth representations of G in complex vector spaces. For V, W ∈ Rep ∞ (G), V ⊗ W will denote the usual tensor product over C and V * will denote the linear dual.
If G is an affine Nash group, we denote by Rep ∞ (G) the category of smooth nuclear Fréchet representations of G of moderate growth. This is essentially the same definition as in [9,Section 1.4] with the additional assumption that the representation spaces are nuclear (see e.g. [56,Section 50]). For V, W ∈ Rep ∞ (G), V ⊗ W will denote the completed projective tensor product and V * will denote the continuous linear dual, endowed with the strong dual topology.
where the closure is needed only for F = R.
Note that if F = R and G is connected then π G = π/g C π which in turn is equal to the quotient of H 0 (g, π) by the closure of zero.
Proof. -It is easy to see (cf. [20, Appendix A]) that the integration map f → f gives the following natural isomorphism of representations of G, where G acts on functions by left shifts, and coinvariants are taken under the diagonal action on the representation and by right multiplication on the argument:
Substituting in this formula G itself as the subgroup H, we obtain S(G, π)
In the non-Archimedean case this is evident, and in the Archimedean case this is [9, Proposition 1.2.6]. We will also use the linear automorphism of S(G, π) given by
Note that T intertwines the action of G × G on S(G, π) given by f (g1,g2) (x) := π(g 2 )f (g -1 1 xg 2 ) with the action given by f (g1,g2) (x) := π(g 1 )f (g -1 1 xg 2 ). Altogether we have
Definition 2.9. -Let W n denote the 2n-dimensional F -vector space (F n ) * ⊕ F n and let ω be the standard symplectic form on W n . The Heisenberg group H n is the algebraic group with underlying algebraic variety W n × F and with the group law given by
Note that H 0 = F . Definition 2.10. -Let χ be the additive character of F , as in (2.1). Extend χ trivially to a character of the commutative subgroup 0⊕F n ⊕F ⊂ H n . The oscillator representation χ is the unitary induction of χ from 0 ⊕ F n ⊕ F to H n . Define the smooth oscillator representation σ χ to be the space of smooth vectors in χ .
Let G be a finite central extension of the group G alg of F -points of a reductive algebraic group defined over F . Let G ad denote the corresponding adjoint algebraic group.
). -Let U ⊂ G alg be a unipotent subgroup, and U be the preimage of U in G. Then there exists a unique open subgroup U ⊂ U that projects isomorphically onto U .
We will therefore identify the unipotent subgroups of G alg with their liftings in G. Definition 2.15.
(i) A Whittaker pair is an ordered pair (S, ϕ) such that S ∈ g is rational semi-simple, and ϕ ∈ (g * ) S -2 . Given such a Whittaker pair, we define the space of degenerate Whittaker models W S,ϕ in the following way: let u := g S 1 . Define an anti-symmetric form ω ϕ on g by ω ϕ (X, Y ) := ϕ([X, Y ]). Let n be the radical of ω ϕ | u . Note that u, n are nilpotent subalgebras of g, and [u, u] ⊂ g S 2 ⊂ n. Let U := Exp(u) and N := Exp(n) be the corresponding nilpotent subgroups of G. Let n := n∩Ker(ϕ), N := Exp(n ). If ϕ = 0 we define
Assume now that ϕ is non-zero. Then U/N has a natural structure of a Heisenberg group, and its center is N/N . Let χ ϕ denote the unitary character of N/N given by χ ϕ (exp(X)) := χ(ϕ(X)).
Let σ ϕ denote the oscillator representation of U/N with central character χ ϕ , and σ ϕ denote its trivial lifting to U . Define
(ii) For a nilpotent element ϕ ∈ g * , define the generalized Whittaker model W ϕ corresponding to ϕ to be W h,ϕ , where h is a neutral element for ϕ if ϕ = 0 and h = 0 if ϕ = 0. We will also call W h,ϕ neutral Whittaker model. By Lemma 2.1 W ϕ depends only on the coadjoint orbit of ϕ, and does not depend on the choice of h. Thus we will also use the notation W O for a nilpotent coadjoint orbit O ⊂ g * . See [20,Section 5] for a formulation of this definition without choosing h. (iii) To π ∈ Rep ∞ (G) associate the degenerate and generalized Whittaker quotients by
Lemma 2.16. -Let l ⊂ g S 1 be a maximal isotropic subalgebra and
Proof. -By Corollary 2.13 and Lemma 2.5 we have
Slightly different degenerate Whittaker models are considered in [20] and denoted W S,ϕ (π). By Lemma 2.16 and [20, Lemma 2.5.2] they relate to π S,ϕ by W S,ϕ (π) = π * S,ϕ . We changed the notion in this paper since for p-adic F , π S,ϕ are the models considered in [42,57] and for F = R, π S,ϕ are (nuclear) Fréchet spaces.
Remark 2.17. -If F = R, one can define W S,ϕ for any semi-simple S with real eigenvalues in the same way, and all our proofs will be valid for this case without changes.
Lemma 2.18. -Assume that G is an adjoint group, and let S ∈ g be semi-simple. Then there exists an algebraic group morphism ν : F × → G (defined over F ) with dν(1) = S if and only if all the eigenvalues of S in the adjoint action on g( F ) are integers, where F denotes the algebraic closure of F .
Proof. -Embed G into G := GL(g) using the adjoint action. Then any ν : F × → G defines ν : F × → G . Note that a semi-simple S ∈ g equals dν (1) for some ν : F × → G if and only if all the eigenvalues of S on g( F ) are integers. The "only if" part follows. For the "if" part, note that if S = dν (1) then Im(dν ) ⊂ g, hence Im(ν ) ∈ G and thus ν defines ν : F × → G with the required property.
Corollary 2.19. -Let S ∈ g be rational semi-simple. Then there exists an algebraic group morphism ν :
Proof. -Replacing S by an integer multiple we can assume that all the eigenvalues of S in the adjoint action on g( F ) are integers. Thus there exists an algebraic group morphism from F × to the adjoint group of G which includes the projection of S in its image.
Let γ = (e, h, f ) be an sl 2 -triple in g. Let G γ denote the joint centralizer of the three elements of γ. It is well known that G γ is a Levi subgroup of G f . Let ϕ ∈ g * be given by the Killing form pairing with f . Recall that it induces a nondegenerate symplectic form ω ϕ on g h 1 and note that G γ acts on g h 1 preserving the symplectic form. That is, there is a natural map
be the metaplectic double covering, and set
Observe that the natural map G γ → G γ defines a double cover of G γ . We denote by M γ the subgroup of G γ generated by the unipotent elements. Let M γ denote the preimage of M γ under the projection G γ → G γ . Note that different choices of γ with the same f lead to conjugate groups G γ and M γ .
One can also define a covering G ϕ of the group G ϕ = G f , using the symplectic form defined by ϕ on g/g ϕ . It is easy to see that this cover splits over the unipotent radical of G ϕ , and that the preimage of G γ in G ϕ is isomorphic to G γ , see e. Note that H 0 does not depend on the embedding of H into GL n . Note also that if H is semi-simple and connected then
As observed in [44], this definition of admissibility is compatible with Duflo's original definition for the Archimedean case, given in [12, Section II.2].
Definition 2.22. -We say that a nilpotent orbit O ⊂ g * is quasiadmissible if for some (equivalently, for any) ϕ ∈ O, the covering G ϕ → G ϕ admits a finite dimensional genuine representation, that is, a finite dimensional representation on which the non-trivial element ε in the preimage of 1 ∈ G ϕ acts by -Id. Definition 2.23. -Let us define the action of G γ on π ϕ . Since the adjoint action of G γ preserves g h 1 and the symplectic form on it, it preserves U/N . Since σ ϕ is the unique smooth irreducible representation of U/N with central character χ ϕ , we have a projective action of G γ on σ ϕ . By [60] this action lifts to a genuine representation of G γ . This gives rise to an action of G γ on W ϕ by ( gf )(x) = g(f (xg)). This action commutes with the action of G and thus defines an action of
More generally, for a Whittaker pair (S, ϕ) let G(S, ϕ) denote the subgroup of G corresponding to the Lie algebra (g S 0 ∩ g ϕ ) ⊕ g S >1 . Then G(S, ϕ) normalizes the groups U and N corresponding to the pair (S, ϕ) and acts on W S,ϕ and on π S,ϕ as above.
It is well known that the metaplectic cover splits over the Siegel parabolic corresponding to any Lagrangian. Indeed, the Weil representation on the Siegel parabolic is given by the action on the Heisenberg group by automorphisms (in the realization of the representation functions on the Heisenberg group as in Lemma 2.11). These considerations imply the following lemma.
Lemma 2.24. -Let (S, ϕ) be a Whittaker pair, let l ⊂ g S 1 be a maximal isotropic subalgebra and L := Exp(l). Let G denote the normalizer of L in G(S, ϕ). Then (i) The metaplectic cover of G(S, ϕ) splits over G .
(ii) The action of G on W S,ϕ defined by splitting the metaplectic cover corresponds under the isomorphism
, the action of G on π S,ϕ defined by splitting the metaplectic cover corresponds under the isomorphism
from Lemma 2.16 to the action on (π ⊗ χ ϕ ) L given by action on representatives in π.
Let P 2 (R) denote the group of affine transformations of the line. Let N denote the unipotent radical of P 2 (R) and n denote the Lie algebra of N . Let B 0 denote the connected component of the identity in P 2 (R)
. Suppose that V is not generic, i.e. (V * ) N,ψ = 0 for any non-trivial unitary character ψ of N . Then n acts locally nilpotently on V * .
In the p-adic case, an analogous lemma is proven by Bernstein and Zelevinsky using l-sheaves. We will prove this proposition in Section 3.1 using [9, Section 2]. Let us now derive some corollaries.
for some (possibly trivial) unitary character ψ of N .
Let M p denote the metaplectic group SL 2 (R), and s := Lie(M p) = sl 2 (R) denote its Lie algebra.
) be non-zero and non-generic. Then any λ ∈ V * generates a finite-dimensional subrepresentation, and V has a (non-zero) finite-dimensional quotient. In particular, every irreducible genuine V ∈ Rep ∞ (M p) is generic.
Proof. -Choose a standard basis e, h, f for s. Let B denote a Borel subgroup of SL 2 (R) and B ⊂ M p denote its preimage. Note that the connected component of B is isomorphic to B 0 as a Nash group. Thus Proposition 3.1 implies that both e and f act locally nilpotently on V * . By e.g. [3,Lemma C.0.3] this implies that any λ ∈ V * generates a finite-dimensional subrepresentation W . This W non-degenerately pairs with a quotient of V , and thus this quotient is finite-dimensional.
Remark 3.4. -Under the assumption of the corollary we cannot in general claim that the action on V is locally finite. For example, V could be the direct product of all irreducible finite-dimensional representations of SL 2 (R), with the topology given by projections.
In order to apply Corollary 3.3 we will need the following lemmas. Note that if m is semi-simple then ad-nilpotent is the same as nilpotent.
Lemma 3.6. -Let m be a Lie algebra and V be an m-module. Suppose that m has a basis X 1 , . . . X n such that each X i acts locally finitely on V . Then m acts locally finitely on V .
Proof. -Let U denote the universal enveloping algebra of m. For each i, let p i denote the subspace of U spanned by all the powers of X i . By the Poincaré-Birkoff-Witt Theorem, U = p n p n-1 . . . p 1 . By the assumption of the lemma, for any finite-dimensional subspace W ⊂ V , and any i, p i W is finite-dimensional. By induction we obtain that for any v ∈ V , and k n, Proof. -Any λ ∈ V * and v ∈ V define a distribution on R supported at 0 by ξ λ,v (f ) := λ( f v), where f denotes the Taylor series of f at 0. Then, for any λ ∈ V * there exists k such that t k ξ λ,v = 0, and thus λ(t k v) = 0. Thus V = ∪ Ker(t k λ). Since V is metrizable, the Baire category theorem implies that Ker(t k λ) = V for some k. Using Fourier transform we obtain the following corollary.
Suppose that AV = 0. Then d dt acts locally nilpotently on V * . Definition 3.10 ([9, Section 2]). -Let G be an affine Nash group and X be an affine Nash G-space. A (G, S(X))-module is a representation E ∈ Rep ∞ (G) with a continuous action π of the Fréchet algebra S(X) satisfying gπ(f )g -1 = π(f g )v, where f g (x) := f (g -1 x). We say that E is non-degenerate if the action map S(X) ⊗ E → E has dense image, and for every v = 0 ∈ E, there exists f ∈ S(X) with π(f )v = 0.
We denote the category of (G, S(X))-modules by Rep G,X and the subcategory of non-degenerate modules by Rep nd G,X .
Theorem 3.11 ([9, Lemma 2.5.7 and Theorem 2.5.8]). -Let G be an affine Nash group and X be a transitive Nash G-space. Define S(X) := C ⊕ S(X). Fix x 0 ∈ X and let m x0 ⊂ S(X) denote the maximal ideal consisting of functions vanishing at
Proof of Proposition 3.1. -Let the algebra A from (3.1) act on V using the identification N R. By Corollary 3.9 it is enough to show that AV = 0. Suppose by way of contradiction that AV = 0 and let E denote the closure of AV . Note that Fourier transform defines an isomorphism A S(R \ {0}). We further identify R \ {0} with the Nash manifold X of non-trivial unitary characters of N . The action of P 2 (R) on X is by multiplication by the reductive part, and the action of B 0 has two orbits: X + and X -.
We let S(X) act on E through A, and note that this is compatible with the action of the group B 0 . Theorem 3.11 implies that either (E * ) N,ψ = 0 for any ψ ∈ X + or (E * ) N,ψ = 0 for any ψ ∈ X -. Fix a ψ with (E * ) N,ψ = 0. From the short exact sequence
we obtain the exact sequence
Note that n acts locally nilpotently on (V /E) * by Corollary 3.9. Thus the action of any generator of n on (V /E) * ⊗ ψ is invertible and hence H 1 (n, (V /E) * ⊗ ψ) = 0. Since (E * ) N,ψ = 0 we obtain that (V * ) (N,ψ) = 0, which contradicts the conditions of the proposition. Thus AV = 0 and thus n acts locally nilpotently on V * .
Fix a Whittaker pair (S, ϕ). Let G • ϕ denote the coadjoint orbit of ϕ and G • ϕ denote its closure. Fix Z as in the lemma and let h := S -Z. For any rational number t 0 define (4.1) S t := S t,Z := S + (t -1)Z, u t := g St 1 , v t := g St >1 and w t := g St 1 .
Definition 4.2. -We call t regular if u t = u t+ε for any small enough ε ∈ Q. Observe that this is equivalent to w t ⊂ g Z . If t is not regular we call it critical. For convenience, we will say that 0 is critical.
Note that for any Z and ϕ there are only finitely many critical numbers. Recall the anti-symmetric form ω on g given by ω(X, Y ) = ϕ([X, Y ]).
Choose a Lagrangian m ⊂ g Z 0 ∩ g S 1 and let (4.2) l t := m+(u t ∩g Z <0 )+Rad(ω| ut ) and r t := m+(u t ∩g Z >0 )+Rad(ω| ut ). Note that these are maximally isotropic subspaces.
Lemma 4.4. -Let 0 t < T and suppose that there are no critical numbers in (t, T ). Then
Moreover, r t is an ideal in l T with commutative quotient and v T is an ideal in r t with commutative quotient.
Proof. -Decomposing to the eigenspaces of ad(Z) we obtain
Since there are no critical numbers in (t, T ) we have
This implies l T = r t ⊕(w T ∩g ϕ ) and v T ⊂ r t . The rest is straightforward. Note that for ϕ = 0 we obtain the degenerate Whittaker model W S,ϕ . Note also that ϕ vanishes on [u, u] and thus ϕ + ϕ defines the same antisymmetric form on u as ϕ.
Remark 4.7. -Let a be the first eigenvalue of S bigger than 1. Then
Recall that if F = R then ⊗ denotes the completed tensor product. We will say that π is (S, ϕ, ϕ )-distinguished if π S,ϕ,ϕ = 0. We will denote by QWO(π) the set of all orbits O for which there exists a Whittaker triple (S, ϕ, ϕ ) such that ϕ ∈ O and π is (S, ϕ, ϕ )-distinguished. The set of maximal orbits in QWO(π) will be denoted QWS(π).
Till the end of the subsection we let T > t 0 be such that there are no critical numbers in (t, T ). We also fix ϕ ∈
and let χ := χ ϕ+ϕ +ψ be the character of these groups given by ϕ + ϕ + ψ.
Similarly to Lemma 2.16 we have
Let f ∈ g be the unique nilpotent element corresponding to ϕ by the Killing form. Let h := S -Z and let γ = (e, h, f ) be an sl 2 -triple. Lemma 4.10. -Assume that π is (S t , ϕ, ϕ + ψ)-distinguished. Then for some (possibly zero) ψ ∈ (g
The restrictions of this form to r t and to v T are trivial. Thus there exists a maximal totally isotropic subspace l ⊂ u T with r t ⊂ l. Let L := Exp(l). Since v T ⊂ r t , the characters of A := L/R t are given by a quotient of (g * ) S T -1 . Thus, by Lemma 4.5 and Corollary 3.2, for some ψ ∈ (g * ) S T -1 we have 0 = Hom A (π Rt,χ , χ ψ ) = (π S T ,ϕ+ψ,ϕ +ψ ) * .
Let S, ϕ, h, f, e, Z, S t be as before.
. Then we have an epimorphism
Moreover, if π is not (S s , ϕ, ϕ + ψ )-distinguished for any s ∈ (t, T ) and any non-zero ψ ∈ (g
Proof. -Let t 0 := t, t 1 , . . . , t n-1 be all the critical values between t and T and t n := T . By Lemmas 4.4, 2.8, and 4.8 we have
The "moreover" part follows from Lemma 4.9.
Proposition 4.12. -Let t 0 and let
Suppose that π is (S t , ϕ, η)-distinguished. Then there exist T > t, Φ ∈ (g * ) S T -2 , and Φ ∈ (g * ) S T >-2 such that ϕ ∈ GΦ \ GΦ and π is (S T , Φ, Φ )distinguished.
Proof. -Since η ∈ (g * ) e ⊂ g h 0 , we have η ∈ g Z <0 . Thus for some s > t there exist ψ ∈ (g * ) Ss -2 ∩ (g * ) e and ϕ ∈ (g * ) Ss >-2 ∩ (g * ) e such that ψ = 0 and η = ϕ + ψ. Note that ϕ ∈ (g * )
Let a 0 := t, let a 1 , . . . , a m-1 be the critical values between t and s and a m := s. We prove the statement by induction on m.
The base case is m = 1, i.e. there are no critical values between t and s. Take T := s. Then Lemma 4.10 implies that π S T ,ϕ+ψ,ϕ +ψ = 0 for some ψ ∈ (g * ) S T -1 . Denote Φ := ϕ + ψ and Φ := ϕ + ψ . Note that ϕ ∈ GΦ. Indeed, by Corollary 2.19, there exists an algebraic group morphism ν : F × → G ad and a central element Z ∈ g such that Z -Z ∈ Im(dν). Let λ ∈ F × be small and g := ν(λ). Then Ad * (g)ϕ = ϕ and Ad * (g n )ψ → 0. Note also that GΦ = G ad Φ.
Note that Φ belongs to the Slodowy slice to Gϕ at ϕ and thus ϕ / ∈ GΦ. For the induction step, note that by Lemma 4.9, π is (S a1 , ϕ, η + ψ )distinguished for some (possibly zero) ψ ∈ (g * ) Sa 1 -1 ∩ (g * ) e . The Proposition 4.12 follows now from the induction hypothesis.
Note that it is possible that Φ = 0. Example 4.13. -Let G := GL 4 (F ), h := diag(1, -1, 1, -1), Z := diag(0, 0, 1, 1), t := 3. Identify g with g * using the trace form and let ϕ := f := E 21 +E 43 , η := E 14 , where E ij are elementary matrices. Then e = E 12 +E 34 and η ∈ g St -1 ∩ g e . We have s = 4, m = 1, ϕ = 0, ψ = η, Φ = ϕ + ψ. Then Φ is regular nilpotent and ϕ ∈ GΦ \ GΦ. Since g S4 1 = 0, we have Φ = 0.
For the next proposition we will need a couple of geometric lemmas, and a definition.
Proof. -By Corollary 2.19, there exists an algebraic group morphism ν : F × → G ad and a central element C ∈ g such that Z -C ∈ Im(dν). Since Z commutes with ϕ and ψ ∈ (g * ) Z >0 , this implies that there exists a sequence t n → 0 ∈ F with ϕ + t n ψ ∈ G S (ϕ + ψ) for every n. Thus ϕ + t n ψ ∈ Gϕ for every n. Consider the decomposition g * = (g * ) e ⊕ ad * (g)(ϕ). Since ad(S) preserves all these spaces we have
given by µ(X, g) := X + Ad * (g)ϕ is a submersion. Hence its image contains an open neighborhood of ϕ. Thus Im µ contains ϕ + t n ψ for some n. Since ϕ + t n ψ ∈ Gϕ, and the Slodowy slice ϕ + (g * ) e is strongly transversal to Gϕ, we obtain ϕ
We denote by in(S, ϕ) the number of all quasicritical t > 1.
Let us now show that in(S, ϕ) does not depend on the decomposition S = h + Z. Lemma 4.16 ([7, Section 11]). -Let h ∈ g S be a neutral element for f . (i) Im(ad(f )) ∩ Ker(ad(f )) is a subalgebra in g, which includes h -h and lies in g h <0 . (ii) Let n ⊂ g be a subalgebra such that all Y ∈ n are nilpotent and [h, n] = n. Then exp(ad(n))h = h + n.
Lemma 4.17. -Let h ∈ g S be a neutral element for f . Then there exists a nilpotent element X ∈ g S such that exp(ad(X))(h) = h . Proof. -Since ϕ ∈ QWS(π), π is (S , ϕ, ϕ )-distinguished for some S , ϕ . Without loss of generality we can assume S = S . Suppose first that Z = 0. In this case we can assume ϕ ∈ (g * ) h -1 . Also, in this case the form on u = g h 1 given by ω(X, Y ) = ϕ([X, Y ]) has no radical. Thus we can choose a Lagrangian subspace of u t on which ψ vanishes. Thus π h,ϕ,ϕ = 0 implies π h,ϕ = 0. Now we assume Z = 0 and prove the Proposition 4.19 by induction on in(S, ϕ). For the base assume that in(S, ϕ) = 0, and let t be such that all the positive eigenvalues of tZ are bigger than all the eigenvalues of h by at least 2. Then we have (g * ) Z >0 ⊂ (g * ) St 2 , and
and by Lemma 4.10, π is (S t , ϕ, ψ )-distinguished for some ψ ∈ (g * ) St >-2 . By (4.8) we have
Then η 1 vanishes on u t , and η 2 vanishes on the radical of the form ω ϕ on u t . Thus we can choose a maximal isotropic subspace of u t on which ψ vanishes. Thus π St,ϕ,ψ = π St,ϕ . By Proposition 4.11, π ϕ maps onto π St,ϕ . Since π St,ϕ,ψ = 0 we obtain π ϕ = 0.
For the induction step let t > 1 be the smallest quasi-critical number. By Lemma 4.10, π is (S t , ϕ + ψ, η )-distinguished for some η ∈ (g * ) St >-2 and some ψ ∈ (g * )
Then ϕ ∈ G(ϕ + ψ). Since ϕ ∈ QWS(π), we have ϕ ∈ G(ϕ + ψ), and by Lemma 4.14 ϕ = g(ϕ + ψ) for some g ∈ G St . Conjugating by g we get in(S t , ϕ+ψ) = in(S t , ϕ) < in(S, ϕ). The induction hypothesis implies now that π ϕ+ψ = 0. Thus π ϕ = 0.
Identify g with g * using the trace form and let
Let π be (S, ϕ, ϕ )-distinguished and let us show that ϕ / ∈ QWS(π). Then the first quasicritical value of t is t = 4/3. We have S 4/3 = diag(1, -1, 5, 3, 4 1 3 , 2
3 ), ϕ = E 21 + E 43 + E 65 + E 14 , ϕ = E 45 , h = diag(-1, -3, 3, 1, 1, -1), Z = diag(0, 0, 0, 0, 4/3, 4/3).
The first quasicritical value of t is 3/2. Now, Proposition 4.11 implies that any (S, ϕ, ϕ )-distinguished representation π is (S 3/2 , E 21 + E 43 + E 65 + E 14 + E 45 )-distinguished, and thus has π E21+E43+E65+E14+E45 = 0.
Proof. -First let us show that for all critical t > 0 and all non-zero ϕ ∈ (g * ) St -1 ∩ (g * ) e we have π S,ϕ,ϕ = 0. Suppose the contrary. Then by Proposition 4.12 for some T > 0 there exist Φ ∈ (g * ) S T -2 and Φ ∈ (g * ) S T >-2
such that ϕ ∈ GΦ \ GΦ and π is (S T , Φ, Φ )-distinguished. Thus there exists O ∈ QWS(π) that includes Φ in its closure. By Corollary 4.20 we have O ∈ WS(π), which contradicts the assumption Gϕ ∈ WS(π).
. By Proposition 4.11, we have a sequence of ( G γ ) Z -equivariant epimorphisms (4.9)
that in the p-adic case are isomorphisms, and in the real case are nonzero.
Remark 4.23. -Under the assumption that π is unitary one might be able to construct an invariant scalar product on π ϕ and deduce that the epimorphism of π ϕ onto π S,ϕ is an isomorphism also for F = R.
Example 4.24. -Let G := GL(4, F ) and let S be the diagonal matrix diag(3, 1, -1, -3). Identify g with g * using the trace form and let f := ϕ := E 21 + E 43 , where E ij are elementary matrices. Then we have
) and the weights of S t are as follows:
The critical numbers are 1/4 and 3/4. For t 3/4, the degenerate Whittaker model W St,ϕ is the induction ind G N χ ϕ , where N is the group of upperunitriangular matrices.
The sequence of inclusions r 0 ⊂ l 1/4 ∼ r 1/4 ⊂ l 3/4 = r 3/4 is:
Here, both * and -denote arbitrary elements. -denotes the entries in v t and * those in
The letter a denotes an arbitrary element, but the two appearances of a denote the same numbers. The passage from l 1/4 to r 1/4 is denoted by ∼. At 3/4 we have l 3/4 = r 3/4 .
Let π ∈ Rep ∞ (G) with Gϕ ∈ WS(π). The sequence of epimorphisms (4.9) is given by the sequence of inclusions (4.10). To see that these epimorphisms are non-zero (and are isomorphisms for F = R) we need to analyze the dual spaces to w f 1/4 and w f 3/4 . These spaces are spanned by E 13 + E 24 and by E 23 respectively. Thus, the dual spaces are spanned by E 31 + E 42 and by E 32 respectively. Note that the joint centralizer of h, Z and ϕ in G acts on these spaces by scalar multiplications, identifying all non-trivial elements. By Proposition 4.11 it is enough to show that π S 1/4 ,ϕ,E31+E42 = 0 and π S 3/4 ,ϕ,E32 = 0. This is guaranteed by Propositions 4.12 and 4.19, but for the sake of the example let us show this more directly.
First assume by way of contradiction that π S 3/4 ,ϕ,E32 = 0. Note that E 32 ∈ g S1 -2 and that w 1 = 0. Thus u 1 = l 1 = r 3/4 and π S1,ϕ+E32 π S 3/4 ,ϕ,E32 = 0.
Note that Φ := ϕ + E 32 = E 21 + E 43 + E 32 is a regular nilpotent element, and
) is a neutral element for it. Thus π Φ = 0, contradicting the assumption that Φ is maximal in WS(π). Now assume by way of contradiction that π S 1/4 ,ϕ,E31+E42 = 0. Note that Thus π Ψ = 0, contradicting the assumption that G•ϕ is maximal in WO(π).
In the non-Archimedean case, it is enough to prove that for any homomorphism ν : SL 2 (F ) → M γ , the image acts on π ϕ by ± Id. In the Archimedean case, by definition of M γ its Lie algebra m is generated by nilpotent elements. Thus, by Lemmas 3.5 and 3.6, it is enough to prove that any nilpotent element of m acts locally finitely on π * ϕ . Since any such nilpotent lies in the image of the differential of a homomorphism of the form ν : SL 2 (F ) → M γ , it suffices to show that the restriction of π * ϕ to the image of any ν as above is locally finite. By Lemma 4.5 and Corollary 3.3, in both cases it is enough to show that the restriction of π ϕ to the image of ν is non-generic.
Fix such a morphism ν and let (e , h , f ) be the corresponding sl 2 -triple in g γ , and let ϕ ∈ g * denote the nilpotent element given by the Killing form pairing with f . Let
. By Corollary 4.20, WS(π) = QWS(π). Thus ϕ ∈ QWS(π), and Proposition 4.12 implies that (5.1) π St,ϕ,ψ = 0 for any t ∈ (0, 1/2) and any non-zero ψ ∈ (g * ) St -1 ∩ (g * ) e By Proposition 4.11, this implies that we have a sequence of epimorphisms (5.2)
that in the p-adic case are isomorphisms, and in the real case are nonzero. By Lemma 2.24, these epimorphisms commute with the action of Exp(e ). Let χ denote the character of Exp(e ) given by ϕ , and denote (π St i ,ϕ ) e ,ϕ := (π St i ,ϕ ) Exp(e ),χ . Then (5.2) induces a sequence of epimorphisms
We note that the last element of the sequence is zero, since e ∈ v tn+1 and ϕ(e ) = 0, and thus Exp(e ) acts trivially on π St n+1 ,ϕ . In order to show that the restriction of π ϕ to the image of ν is non-generic it is enough to show that all the spaces in (5.3) vanish.
If F is p-adic then this is straightforward, since in this case all the maps in (5.2) are isomorphisms, and thus so are the maps in (5.3). Let us show that in the Archimedean case (π St i ,ϕ ) e ,ϕ = 0, by backwards induction on i. The base case is i = n + 1. For the induction step, recall that the map
) is given by
Consider the dual map (π Lt i ,ϕ ) * → (π Rt i-1 ,ϕ ) * . Its image is the space of invariants under the commutative Lie algebra a := l ti /r ti-1 . By Lemma 4.9 and (5.1), a acts locally nilpotently on (π Rt i-1 ,ϕ ) * . Since [e , l ti ] ⊂ r ti-1 , the actions of e and of a commute. Thus a preserves the space of (e , ϕ )-semiinvariants ((π Rt i-1 ,ϕ ) * ) e ,ϕ , and acts on it locally nilpotently. But a has no invariants on this space by the induction hypothesis, since this space of invariants is dual to (π St i ,ϕ ) e ,ϕ = 0. Thus ((π Rt i-1 ,ϕ ) * ) e ,ϕ = 0, and thus (π St i-1 ,ϕ ) e ,ϕ = 0.
Remark 5.1. -Let (S, ϕ) be a Whittaker pair with ϕ ∈ WS(π), and let G(S, ϕ) ⊂ G be the subgroup defined in Definition 2.23. The same argument shows that the cover of the subgroup of G(S, ϕ) generated by unipotent elements acts locally finitely on π * S,ϕ if F is Archimedean, and acts on π ϕ by ±1 if F is non-Archimedean.
Proof. -By a quasi-cuspidal π we mean a smooth (not necessarily admissible or finitely-generated) representation such that the Jacquet module r P (π) vanishes for any proper parabolic subgroup P ⊂ G.
Let π be quasicuspidal and let O ∈ WS(π). Suppose by way of contradiction that O is not F -distinguished. Thus there exists a proper parabolic subgroup P ⊂ G, a Levi subgroup L ⊂ P and a nilpotent f ∈ l such that ϕ ∈ O, where ϕ ∈ g * is given by the Killing form pairing with f . Let h be a neutral element for f in l. Choose a rational-semisimple element Z ∈ g such that L is the centralizer of Z, p := g Z 0 is the Lie algebra of P , and all the positive eigenvalues of Z are bigger than all the eigenvalues of h by at least 2. Note that n := g Z >0 is the nilradical of p. Let S := h + Z. By construction we have n ⊂ g S >2 and thus the degenerate Whittaker quotient π S,ϕ is a quotient of r P π. By Theorem 1.5, the maximality of O implies π S,ϕ π ϕ . Thus r P π does not vanish, in contradiction with the condition that π is quasi-cuspidal. Fix a nilpotent ϕ ∈ g * . Let γ = (e, h, f ) be an sl 2 -triple such that ϕ is given by pairing with f under the Cartan-Killing form on g.
Let (G γ ) ss be the subgroup of G γ generated by the exponents of the derived algebra for g γ , and (G γ ) ss be the corresponding subgroup of G γ .
Let K γ ⊂ (G γ ) ss be the anisotropic (and hence compact) part.
Proof. -For the Lie algebras we have
Thus, in order to prove (6.1) it is enough to show that M γ and K γ commute. Fix k ∈ K γ . By (6.2), the commutator map m → m k m -1 k -1 maps M γ to {1, ε}. Suppose, by way of contradiction, that the image is non-trivial. Then M γ is disconnected and thus M γ Z 2 ×M γ . Thus for some m ∈ M γ ⊂ M γ , the commutator map k → m km -1 k -1 is non-trivial. Thus K γ also splits, which implies that it commutes with M γ .
By the Peter-Weyl theorem, this implies that the induction has a finitedimensional K γ -isotypic component ρ. By Proposition 6.1, M γ preserves ρ and thus ρ is a genuine finite-dimensional representation of (G γ ) ss .
Proposition 6.3. -If M γ has a genuine finite-dimensional representation then so does G ϕ .
Proof. -Let Z denote the center of G γ and let H := Z (G γ ) ss . Let us show that H has a finite-dimensional genuine representation. By Proposition 6.2, (G γ ) ss has an irreducible genuine finite-dimensional representation (ρ 1 , V 1 ).
Notice that Z ∩ (G γ ) ss acts on (ρ 1 , V 1 ) by a character, that we will denote by χ 1 . By the classical theory of Pontryagin duality for locally compact abelian groups, we can extend the character χ 1 to a character of Z, see [11,Theorem 5]. This defines a genuine action of H on ρ 1 . Let ρ 2 be the induction of this representation to G γ . Since H has finite index in G γ , ρ 2 is finite-dimensional. By composing ρ 2 with the epimorphism G ϕ → G γ , we obtain a genuine finite-dimensional representation of G ϕ . Theorem 1.1 follows now from Theorem 1.4 and Proposition 6.3.
Proposition 6.4. -Assume that F is non-Archimedean. Then ϕ is quasi-admissible if and only if the cover (G ϕ ) 0 (see Section 2.6) splits over an open normal subgroup of finite index.
Proof. -First, if the cover (G ϕ ) 0 (see Section 2.6) splits over an open normal subgroup H ⊂ (G ϕ ) 0 of finite index then the cover H has a onedimensional genuine representation. The induction of this representation to (G ϕ ) 0 is still genuine and finite-dimensional. Now assume that G ϕ has a genuine finite-dimensional representation ρ.
Restrict ρ to (G ϕ ) 0 and let C denote the kernel of the restriction. Then C is an open normal subgroup. Let us show that it has finite index. Indeed, since ρ is finite-dimensional, Lemma 4.5 implies that C includes all the unipotent elements of (G ϕ ) 0 . Thus, C is cocompact and open and hence has finite index. Since ρ is genuine, the restriction of the covering map to C is one-to-one. Thus, the cover splits over the image of C in (G ϕ ) 0 .
Proposition 6.5. -All admissible orbits are quasi-admissible.
Proof. -Let O be an admissible orbit, let ϕ ∈ O and let γ be a corresponding sl 2 -triple. Then, by definition, the cover splits over the group (G ϕ ) 0 generated by exponents of g ϕ . This group includes M γ and thus M γ splits, and has a genuine character χ. By Proposition 6.3 this implies that O is quasi-admissible. Remark 6.6. -Any F -distinguished orbit is quasi-admissible, since for such orbits M γ is trivial. Over non-Archimedean F , F orbits are admissible since the metaplectic cover splits over compact subgroups, see [38,Theorem 4.6.1]. Over F = R, the minimal orbit in U (2, 1) is Rdistinguished but not admissible. In general, the R-distinguished orbits for semi-simple groups are classified in [51, (under the name compact orbits), and comparing this classification with the classification of admissible orbits given in [50,Theorem 3] for classical groups, and [47,48] for exceptional groups, we see that for the groups (6.3) SU(p, q)(with p, q 1), EII, EV, EVI, EVIII, EIX there exist R-distinguished non-admissible orbits (1) . On the other hand, for other real simple groups, all R-distinguished orbits are admissible. Thus it is possible that for simple groups not appearing in the list (6.3) admissibility is equivalent to quasi-admissibility. We conjecture that quasi-admissibility is equivalent to the splitting of M γ for all groups.
Let us now discuss the relation to special orbits.
Theorem 6.7 ([44, Corollaries 5.9 and 6.3], [45,Main Theorem]).
Let F be non-Archimedean. If G is classical then the set of admissible orbits coincides with the set of special orbits. If G is split exceptional different from E 8 then the set of admissible orbits includes the set of special orbits.
It is conjectured in [45] that the same holds for E 8 . For F = R, the sets of special and admissible orbits coincide for orthogonal, symplectic and general linear groups. However, for unitary groups all orbits are special but most orbits are not admissible. See Theorem 6.10 below for these facts. Also, for several exceptional groups, some split and some non-split, there are special non-admissible orbits and admissible nonspecial orbits (see [47,48]). (1) For SU(m, n), the R-distinguished orbits are the ones described by partitions in which all rows of the same size have also the same signs, while admissible orbits are described in Theorem 6.10 (ii) below.
It is conjectured that in the non-Archimedean case the Whittaker support consists of special orbits. By [31,41] this holds for classical (p-adic) groups.
The analogous conjecture cannot hold for exceptional G if F = R. Namely, for the minimal representation π min of G 2 (R) constructed in [58], WS(π) consists of the minimal orbit O min of G 2 , which is admissible but not special. Let us explain the notion of minimal representation and the relation to Whittaker support. Definition 6.8. -We call a smooth representation π of a real reductive group if its annihilator variety is the closure of the minimal orbit in g * (C). The annihilator variety is defined to be the set of common zeros of the symbols of the elements of the annihilator ideal of π in the universal enveloping algebra of g. Proposition 6.9. -The Whittaker supports of minimal representations consist of minimal orbits.
Proof. -Let π be a minimal representation. By [39,Corollary 4]
The conjecture on speciality of WS(π) also cannot be extended to complex reductive groups.
Let us now prove Proposition 1.2 that states that admissibility, quasiadmissibility and speciality are equivalent for the groups O(p, q), SO(p, q) and Sp 2n (R). Our proof is based on Theorem 6.7 and the following theorem from [50]. For the formulation, recall that the nilpotent orbits in real classical groups are given by signed partitions satisfying certain conditions. Fortunately, the signs have no effect on the admissibility and speciality. Theorem 6.10 ([50, Theorem 3]). -Let O ⊂ g * be a nilpotent orbit and λ be the corresponding partition.
(i) Let G is one of the groups O(p, q), SO(p, q), U (p, q) or Sp 2n (R).
Then O is admissible if and only if for each even row (i.e., row with even length) in λ, the number of odd rows in λ, which are shorter than the even row, is even and for each odd row in λ, the number of even rows in λ, which are longer than the odd row, is even. (ii) Let G = SU(p, q). Then O is admissible if and only if for each even row (i.e., row with even length) in λ, the number of odd rows in λ, which are shorter than the even row, is even and for each odd row in λ, the number of even rows in λ, which are longer than the odd row, is even.
(iii) For all other real classical groups, all nilpotent orbits are admissible.
Proof of Proposition 1.2. -First of all, comparing Theorem 6.10 (i) to the description of special orbits in [10, Section 6.3], and using Theorem 6.7 (for the case F = R) we see that for the groups O(V ), SO(V ), and Sp 2n (F ) the set of admissible orbits coincides with the set of special orbits. Next, by Proposition 6.5, this set is included in the set of quasi-admissible orbits. It is left to show that non-admissible orbits are not quasi-admissible either.
We will do it for the symplectic group, since the construction for the orthogonal case is very similar. Let O be a non-admissible orbit and λ be the corresponding partition. Since every odd part in λ appears with even multiplicity, Theorems 6.10 and 6.7 imply that there exists an odd part λ i in λ such that the number of even parts bigger than λ i (counted with multiplicity) is odd. Let 2m be the multiplicity of λ i in λ. By [44,Section 5.3], the centralizer G ϕ includes a group H isomorphic to Sp 2m (F ), over which the cover does not split. Since Sp 2m (R) is simple and has no non-trivial algebraic covers, if F = R then H cannot have genuine finitedimensional representations, and thus O is not quasi-admissible. If F = R then O is not quasi-admissible by Proposition 6.4.
To complete the picture for real classical groups we will need the following lemma. Lemma 6.11 ([33, Section 3, Lemma 7]). -Let V be a Hermitian space (of arbitrary signature) and SU(V ) be the corresponding special unitary group. Consider V as a real vector space and define a symplectic form on V to be the real part of the hermitian form. Then the corresponding metaplectic cover of SU(V ) splits. Corollary 6.12. -All nilpotent orbits in SU(p, q) and in U (p, q) are quasi-admissible.
Proof. -By Proposition 6.3, it is enough to show that M γ splits for any sl 2 -triple γ = (e, h, f ) in su(p, q). Let W = C p+q denote the standard representation of g and • , • denote the fixed hermitian form on W of signature (p, q). Note that γ defines a decomposition W = r 0 W (r), where W (r) is the direct sum of all simple γ-submodules of highest weight r. For each r, let H(r) denote the highest weight subspace of W (r), and define a sesquilinear form on H(r) by v, w r := v, f r w if r is even and v, w r := i v, f r w if r is odd. Since f is skew-hermitian, the form • , • r is hermitian. Note that v, w r is non-degenerate for all r and that M γ is isomorphic to r SU(H(r)).
By [44,Section 5.3] (which is written uniformly for all fields F ), the splitting on M γ is implied by the splitting of the metaplectic cover of SU(H(r))×SU(H(r )) inside Sp(H(r)⊗H(r )) for all pairs (r, r ) of different parity. However, SU(H(r)) × SU(H(r )) is a subgroup of SU(H(r) ⊗ H(r )) and by Lemma 6.11 the metaplectic cover splits on the latter group. Theorems 6.7 and 6.10 and Corollary 6.12 imply the following corollary. Corollary 6.13. -For classical groups, all special orbits are quasiadmissible.
It is possible that all special orbits are quasi-admissible for all groups.
The notion of quasi-Whittaker model and the method of Section 4 allow us to relate degenerate Whittaker models corresponding to different nilpotent orbits. Let (h, ϕ) be a neutral pair, let a rational semi-simple Z ∈ g commute with h and with ϕ and let S := h + Z.
Then we have an epimorphism W h,ϕ,ψ W S,ϕ+ψ .
Proof. -The proof is similar to that of Proposition 4.11. Let S t := h+tZ and let t 0 := 0, t 1 , . . . , t n-1 < 1 be all the critical values between 0 and 1 and t n := 1. By Lemmas 2.8, 4.4, we have
ϕ+ψ . This proposition is strengthened by the following lemma.
Proof. -Since the form ω ϕ on g h 1 is non-degenerate, we can choose a Lagrangian subspace l ⊂ g h 1 on which ψ vanishes. Let l := l ⊕ g h >1 . Then l is a maximal coisotropic subspace of g h 1 and ψ vanishes on l. Let L := Exp(l).
This theorem is strongest for the group GL n . In order to apply it to this case we will need the following proposition from linear algebra, that we will prove in the next subsection, following [20, For admissible π this corollary is [20,Theorem D]. In Section 7.2 below we formulate and prove a certain analog of this corollary for SL n (F ).
Let us first introduce some notation. A composition η of n is a sequence of natural (positive) numbers η 1 , . . . , η k with
For a composition η we denote by η the corresponding partition. A partial order on partitions of n is defined by
µ i for any 1 j length(λ), length(µ).
We will use the notation diag(x 1 , . . . , x k ) for diagonal and block-diagonal matrices. For a natural number k we denote by J k ∈ g k the lower-triangular λ i and λ i+1 by a single part p. It follows from (7.4) that λ i > p > λ i+1 , and thus λ is a partition of n . (7.4) also implies λ > µ . Let α be the reordering of λ obtained by putting the part p on the first place.
Choose a neutral pair (h , ϕ ) in g n with ϕ ∈ O µ . By the induction hypothesis, there exist a rational semi-simple Z ∈ g n and ψ ∈
such that ϕ +ψ ∈ O λ . Conjugating by GL n (Q) we may assume that ϕ +ψ = J α . This implies that Z is diagonal and that the first p entries are equal. By subtracting a scalar matrix, we may assume that the first p diagonal entries of Z are zeroes. Now let (h, ϕ) be a neutral pair with ϕ ∈ O µ . Conjugating by GL n (Q) we may assume that (7.5) ϕ = diag(J µi , ϕ ) and h = diag(h µi , h ). Let (7.6)
Let us show that Z and ψ satisfy the requirements of the theorem. Indeed, we have ψ
by construction and Y ∈ g Z <0 ∩ g h+Z -2 by Lemma 7.7. To see that ϕ + ψ ∈ O note that ϕ + ψ = Y + J β , where β 1 = µ i and β j = α j-1 for any j > 1. Decompose Y + J β = diag(A, J λ ), where A ∈ g λi+λi+1 , and λ is obtained from λ by omitting the part p. By Lemma 7.7 we have A ∈ O λi,λi+1 and therefore
First of all, let us fix a set of representatives for nilpotent orbits, for an arbitrary field L of characteristic zero, after introducing some notation.
For a composition η = (η 1 , . . . , η k ) we denote d(η) := gcd(η 1 , . . . , η k ). For a ∈ L × we denote D a := diag(a, 1, . . . , 1)J η and J a Identify sl n (L) with its dual space using the trace form. such that ϕ + ψ ∈ O a λ . Proof. -Since we can multiply a by (L × ) d(λ) and b by (L × ) d(µ) without changing the orbits, we can assume a = b. Then, applying the automorphism of sl n (L) given by conjugation by diag(a, 1, . . . , 1), we can assume a = b = 1. Next, note that, in the notation of Lemma 7.7, the matrix g in (7.3) that conjugates X to J p+q,r lies in SL n (Q). Now, the proposition follows by induction in the same way as Proposition 7.5. By Theorem 7.4 we obtain the following corollary.
Let K be a number field and let A = A K be its ring of adeles. In this section we let χ be a unitary character of A, which is trivial on Kand such that for any Archimedean place ν the restriction χ| Kν of χ to K ν is exp(2πi|x|), and for any non-Archimedean place ν, the kernel of χ| Kν is the ring of integers. Then χ defines an isomorphism between A and A via the map a → χ a , where χ a (b) = χ(ab) for all b ∈ A. This isomorphism restricts to an isomorphism (8.1)
Given an algebraic group G defined over K we will denote its Lie algebra by g and we will denote the group of its adelic (resp. K-rational) points by G(A) (resp. G(K)). We will also define the Lie algebras g(A) and g(K) in a similar way. Given a Whittaker pair (S, ϕ) on g(K), we set u = g S 1 and n to be the radical of the form ω ϕ | u , where ω ϕ (X, Y ) = ϕ([X, Y ]), as before. Let l ⊂ u be any choice of a maximal isotropic Lie algebra with respect to this form, and let U = exp u, N = exp n and L = exp l. Observe that we can extend ϕ to a linear functional on g(A) by linearity and, furthermore, the character χ L ϕ (exp X) = χ(ϕ(X)) defined on L(A) is automorphic, that is, it is trivial on L(K). We will denote its restriction to N(A) simply by χ ϕ .
Let G be a finite central extension of G(A), such that the cover G G(A) splits over G(K). Fix a discrete subgroup Γ ⊂ G that projects isomorphically onto G(K). Note that U(A) has a canonical lifting into G, see e. -Let (S, ϕ) be a Whittaker pair for g(K) and let U, L, N, χ ϕ and χ L ϕ be as above. For an automorphic function f , we define its (S, ϕ)-Fourier coefficient to be
We also define its (S, ϕ, L)-Fourier coefficient to be
Observe that F S,ϕ and F L S,ϕ define linear functionals on the space of automorphic forms. For a subrepresentation π of the space of automorphic forms on G, we will denote their restrictions to π by F S,ϕ (π) and F L S,ϕ (π) respectively. (2) Example 8. (2) To forestall confusion, we emphasize that L here stays for "Lagrangian" (actually, maximal isotropic), and not Levi.
One defines quasi-Fourier coefficients in a similar way. In order to adapt our arguments to the global setting we will have to replace Lemma 4.8 by the following one. Lemma 8.3 ([20, Lemma 6.0.2]). -Let π be subrepresentation of the space of automorphic forms on G. Let (S, ϕ, ϕ ) be a Whittaker triple. Then F S,ϕ,ϕ (π) = 0 if and only if F L S,ϕ,ϕ (π) = 0. More specifically, if F S,ϕ,ϕ (f ) = 0 for some f ∈ π then F L S,ϕ,ϕ (π(u)f ) = 0 for some u ∈ U (K). Also, note that the global analog of Lemma 4.5 is proven by decomposition to Fourier series. For more details see [27,Section 5.2].
For two nilpotent G(K)-orbits O, O ∈ g(K) we will say O O if for any completion F of K, the closure of O in g(F ) includes O . For a subrepresentation π of the space of automorphic forms on G, we denote by WO(π) the collection of all nilpotent G(K)-orbits in g * (K) such that F ϕ (π) = 0 for any ϕ ∈ O. We denote the set of maximal orbits in WO(π) by WS(π).
Repeating the arguments in Section 4-5 we obtain the following theorem. In order to formulate a global analog of Theorem 1.4 and deduce an analog of Theorem 1.1, we will introduce the global Weil representation ( [60]) and Fourier-Jacobi coefficients, following [27,Section 5.2].
For a symplectic space V over K, V is the only irreducible unitarizable representation of the double cover Jacobi group J(V ) := Sp(V (A)) H(V (A)) with central character χ, where H(V ) is the Heisenberg group of V . It has an automorphic realization given by theta functions
where f ∈ S(E(A)), E is a Lagrangian subspace of V , and g ∈ J(V ).
Fix an sl 2 -triple γ = (x, h, y) in g(K) and let ϕ ∈ g * be given by the Killing form pairing with y. Let V := g h 1 , with the symplectic form ω ϕ (A, B) := ϕ([A, B]). Then we have a natural map γ : U G γ → J(V ). We define a map F J :
Then, arguing as in Section 5.1 we obtain from Theorem 8.4 (i) the following corollary. -If G is quasi-split over K and semi-simple, and f is not constant then there exists a neutral Whittaker pair (h, ϕ) with ϕ = 0 such that F h,ϕ (f ) = 0.
Since the Weil representation V is genuine, the subgroup of M γ that acts trivially on the image of F J projects isomorphically on M γ . This implies the following corollary. Arguing as in the proof of Proposition 1.2 we deduce from this corollary that if G is classical and linear then all the orbits in WF(π) are special. This was already shown in [18,Theorem 2.1] and [31,Theorem 11.2].
Finally, the following analog of Theorem 7.4 holds, with an analogous proof.
Theorem 8.9. -Let (h, ϕ) ∈ g(K) × g * (K) be a neutral Whittaker pair. If there exists a Whittaker pair (S, Φ) ∈ g(K) × g * (K) such that G(K) • Φ ∈ WS(π), ϕ ∈ (g * ) S -2 , [h, S] = 0, and Φ -ϕ ∈ (g * ) S-h <0 then G(K) • ϕ ∈ WO(π).
As in Section 7, this theorem together with Proposition 7.5 implies Corollary 1.6. Furthermore, Theorem 8.9 and Proposition 7.10 imply the following version of Corollary 1.6 for SL n . none of the forms ω i is symplectic. This implies that λ is totally even in case (ii) and totaly odd in case (iii).
To finish the proof, assume by way of contradiction that ε = 1, λ is totally odd and m i > (l + 1)/2 for some i. Let (H, Q) denote the 2-dimensional quadratic space with Q(x, y) := xy and (K, q) denote the one-dimensional quadratic space with q(x) = x 2 . Let (W, ω ) := (W i , ω i ).Then dim W = l. Let n = 2k + j, where j ∈ {0, 1}. Then
By Witt's cancelation theorem, this implies H (l-j)/2 ⊕ K j (W, ω ). Let U := W i ∩ H (l-j)/2 . Then dim U m i + (l -j) -l = m i -j > (l + 1)/2 -j (l -j)/2, and thus U includes an isotropic vector. This contradicts the condition that W i is anisotropic.
For GL n this is a classical result of Piatetski-Shapiro, and the case of Sp 2n was shown in [18,54].
Remark 8.12. -Corollary 8.11 implies that the smallest possible partition in the Whittaker support of a cuspidal automorphic representation π of Sp 2n is 2 n . This bound is sharp for even n by [28]. For SO n,n and O n,n we obtain the lower bound 3 n/2 1 n/2 if n is even and 5 1 3 (n-3)/2 1 (n-1)/2 if n is odd. For O n+1,n and SO n+1,n obtain the lower bound 3 n/2 1 n/2+1 if n is even and 3 (n+1)/2 1 (n-1)/2 if n is odd. These bounds are conjectured to be sharp in [30,Conjecture 2.14].
Corollary 8.13. -Let G be a split classical group of rank at least 3. Let π be an irreducible automorphic representation of G, and let π = ν π ν be its decomposition to local factors. Suppose that for some place ν, every orbit in WS(π ν ) lies inside the Zariski closure of a complex next-to-minimal orbit. Then π cannot be realized in the cuspidal spectrum.
Proof. -It is easy to see that any orbit in WO(π) lies inside some orbit in WO(π ν ). Indeed, for any neutral Whittaker pair (h, ϕ) and any maximal isotropic subspace l ⊂ g, the Fourier coefficient F L h,ϕ is an L(K)-equivariant functional on π. Its existence implies the existence of an L(K ν )-equivariant functional on π ν .
Thus, any orbit in WS(π) has a minimal or a next-to-minimal partition. For GL n , SL n and Sp n , with n 4 the minimal partition is 2 1 1 n-2 and the next-to-minimal one is 2 2 1 n-4 . Both are different from (n), and include odd parts if n 6. For SO k,k+i with i ∈ {0, 1} and k 2, the minimal partition is 2 2 1 2k+i-4 and thus includes even parts. There are at most
ANNALES DE L'INSTITUT FOURIER
TOME 71 (2021), FASCICULE 1