Face synthesis is an important task with applications in digital content creation, film-making and Virtual Reality (VR). Generative Adversarial Nets (GANs), as a powerful generative model, have demonstrated remarkable success in generating high-quality faces. However, despite the impressive performance of GANs in both 2D [15,22,2,24,25,23] and 3D image synthesis [35,46,7,16], its extension to both 3D-aware and motion-controllable image synthesis has not been fully explored. This is largely due to the complexity of physical motion and the challenging yet consistent appearance changes of an identity. For example, dynamic head synthesis requires maintaining some 3D consistency over time to preserve facial identity while permitting other 3D deformations due to expression changes. It is even more challenging to enable interactive manipulation of identity, motion, and viewing directions.
Addressing this task of generalizable, 3D-aware, and motion-controllable synthesis of face dynamics enables to generate never-before-seen faces including their dynamic motion, while controlling the viewpoint, as shown in Fig. 1.
Most related to this task are motion transfer methods such as face reenactment [69,64,47,61,70,67,49,48]. Specifically, prior work animates a subject shown in a source image given target motion from a driving video. For this, prior methods commonly use 2D GANs to produce dynamic subject images with additional guidance such as reference keypoints [69,64,47,61,70,67] and self-learned motion representations [48,49]. However, due to intrinsic limitations of 2D convolutions, these approaches lack 3D-awareness. This leads to two main failure modes: 1) spurious identity changes -the subject, head pose or facial expression of the source image is distinct from the target when rotating a source head by a large angle; 2) serious distortions when transferring motion across identity based on keypoints which contain identity-specific information such as face shapes. To address these failure modes, recent face reenactment works [40,13] estimate parameters of 3D morphable face models, e.g., poses and expression, as additional guidance. Notably, Guy et al. [13] explicitly represent heads in a 3D radiance field showing impressive 3Dconsistency. However, one model per face identity needs to be trained [13].
Different from this direction, we develop a generalizable method of motion-controllable synthesis for novel, neverbefore-seen identity generation. This differs from prior works which either consider motion control but drop 3Dawareness [43,56,44,63,69,64,48,47,61,70,67], or are 3D-aware but don't consider dynamics [46,7,16] or can't generate never-before-seen identities [40,13].
To achieve our goal, we propose controllable radiance fields (CoRFs). We learn CoRFs from RGB images/videos with unknown camera poses. This requires to address two main challenges: 1) how to effectively represent and control identity and motion in 3D; and 2) how to ensure spatiotemporal consistency across views and time.
To control identity and motion, CoRFs use a stylebased radiance field generator which takes low-dimensional identity and motion representations as input, as shown in Fig. 1. Unlike prior head reconstruction work [40,13] that uses ground-truth images for self-supervision, there is no ground-truth target image for a generated image of a neverbefore-seen person. Thus, we propose an additional motion reconstruction loss to supervise motion control.
To ensure spatio-temporal consistency, we propose three consistency constraints on face attributes, identities, and background. Specifically, at each training step, we generate two images given the same identity representation yet different motion representations. We encourage that the two images share identical environment and subject-specific attributes. For this, we apply a head regressor and an identity encoder. A head regressor decomposes the images into representations of a statistic face model, including lighting, shape, texture and albedo. Then, we compute a consistency loss which compares the predicted attribute parameters of the paired synthetic images. Moreover, we use an identity encoder to ensure the paired synthetic images share the same identity. Further, to encourage a consistent background across time, we recognize background regions using a face parsing net and employ a background consistency loss between paired synthetic images that share the same identity representation.
As there is no direct baseline, we compare CoRFs to two types of work that are most related: 1) face reenactment work [48,49] that can control facial expression transfer; and 2) video synthesis work [56,44,63,55] that can produce videos for novel identities. We evaluate the improvements of our method using the Fréchet Inception Distance (FID) [19] for videos, motion control scores, and 3Daware consistency metrics on FFHQ [22] and two face video benchmarks [42,9] at a resolution of 256 × 256. Moreover, we show additional applications such as novel view synthesis and motion interpolation in the latent motion space. Contributions. 1) We study the new task of generalizable, 3D-aware, and motion-controllable face generation. 2) We develop CoRFs which enable editing of motion, and viewpoints of never-before-seen subjects during synthesis. For this, we study techniques that aid motion control and spatiotemporal consistency. 3) CoRFs improve visual quality and spatio-temporal consistency of free-view motion synthesis compared to multiple baselines [56,44,63,48,69,49,55] on popular face benchmarks [22,42,9] using multiple metrics for image quality, temporal consistency, identity preservation, and expression transfer.
3D-aware image synthesis. GANs [15] have significantly advanced 2D image synthesis capabilities in recent years, addressing early concerns regarding diversity, resolution, and photo-realism [22,2,24,25,23,17]. More recent GANs aim to extend 2D image synthesis to 3D-aware image generation [37,53,34,35,11], while permitting explicit camera control. For this, methods [37,53,34,35,11] use an implicit neural representation.
Implicit neural representations [32,8,31,66,28,51,50,38,5] have been introduced to encode 3D objects (i.e., its shape and/or appearance) via a parameterized neural net. Compared to voxel-based [21,41,65] and meshbased [36,59] methods, implicit neural functions represent 3D objects in continuous space and are not restricted to a particular object topology. A recent variation, Neural Radiance Fields (NeRFs) [32], represents the appearance and geometry of a static real-world scene with a multi-layer perceptron (MLP), enabling impressive novel-view synthesis with multi-view consistency via volume rendering. Different from the studied method, those classical approaches can't generate novel scenes.
To address this, recent work [46,7,16,72,71] studies 3D-aware generative models using NeRFs as a gener- ator. For instance, GRAF [46] and π-GAN [7] operate on a randomly sampled camera pose and latent vectors from prior distributions to produce a radiance field via an MLP. StyleNeRF and CIPS-3D [16,72] combine a shallow NeRF network to provide low-resolution radiance fields and a 2D rendering network to produce high-resolution images with fine details. LolNeRF [39] learns 3D objects by optimizing foreground and background NeRFs together with a learnable per-image table of latent codes. Zhao et al. [71] develop a generative multi-plane image (GMPI) representation to ensure view-consistency. These works only generate multi-view images for static scenes. In contrast, the proposed CoRF targets generation of face dynamics while enabling to control motion. We also notice concurrent NeRFbased generative models [52,20] aiming for 3D-aware semantic appearance or expression editing.
Dynamic object synthesis. Dynamic object synthesis has been studied using 2D-and 3D-based methods. Some 2Dbased methods unconditionally generate dynamic instances via convolutional neural nets such as GANs [58,43,44,56,63], but struggle with motion control. Follow-up work incorporates additional reference information during synthesis [69,64,48,47,61,70,62,40], e.g., for tasks like face reenactment. Our method is related but differs in that CoRF synthesizes images and controls motion for never-beforeseen identities from free viewpoints. 3D-based methods recover face geometry and control the expression by changing geometry parameters [54,26,14,33]. While they can control facial motion effectively, they struggle to produce real-istic hair, teeth, and accessories [14,33]. Recent work [13] adapts neural implicit methods, e.g., NeRFs to learn head reconstruction from images. However, one model is trained per identity, i.e., these methods don't generalize across different identities. Unlike prior work [13], CoRF generalizes across identities. Motion conditioning. We draw inspiration from GANbased image inversion and editing work [1,73]. To be concrete, Abdal et al. [1] proposed to embed images back to an extended latent identity space of a StyleGAN [24], i.e., the W + space. This extended latent identity space was shown to be remarkably useful for image editing and expression transfer in follow-up work [73]. In our case, we embed a motion representation into a layered latent motion space, which we then broadcast to every synthesis layer in the generator. In Section 4, we illustrate that such a motion embedding strategy benefits motion control. 3D consistency preservation. To ensure 3D consistency over time, a 3D convolutional discriminator is commonly used to distinguish the source of input video clips [55]. Recent work [68] finds that when employing an implicit neural rendering net as a generator, a 2D discriminator for 2 synthetic frames in a video is sufficient. Inspired by this result, we propose consistency losses on paired synthetic frames without using a 3D convolutional discriminator.
We aim for generalizable, 3D-aware and motioncontrollable synthesis of face dynamics. Concretely, we seek to generate never-before-seen faces while controlling dynamic motions and viewpoint arbitrarily. For this, we propose controllable radiance fields (CoRF), illustrated in Fig. 2, for which we provide an overview next. Overview. CoRFs are composed of a generator G, a discriminator D, a regressor R and an identity encoder E. The CoRF generator G renders an RGB image I based on a noise vector z and a motion representation m. Concretely, the generator G first applies two mapping modules f z and f m , before rendering the image via a style-based synthesis module G s , i.e., G = {f z , f m , G s }. Specifically, to ensure effective motion control, we embed motion representations m into a layered latent motion space using a mapping net f m , which is then combined with a style vector w := f z (z) via a summation. The discriminator D is applied to ensure photo-realism. The regressor R and the encoder E ensure spatio-temporal consistency of synthetic faces over dynamic changes.
During training, different from prior work, we generate two images given the same identity yet different motion representations. To preserve consistency over views and poses, the regressor R and the identity encoder E extract features for the paired images which are encouraged to be similar via a consistency loss L consist and an identity loss L id . To supervise motion control, we estimate the motion representation from images and incorporate a motion reconstruction loss L motion , to minimize the distance between the estimated motion and the conditioned one. Finally, we apply a background loss, L bg , to preserve background consistency over dynamic changes.
In the following, we briefly describe the generator G and the discriminator D (Section 3.1). We then describe our contributions: 1) how to control motion (Section 3.2) and 2) how to preserve spatio-temporal consistency (Section 3.3).
Generative radiance fields. We adopt the generative strategy of StyleNeRF [16] where a style-based radiance field generator G renders a synthetic image I . Specifically, we render colors for each pixel from an integral feature F r . To obtain an integral feature F r , we integrate features F of 3D points along a camera ray r. We present the generator G and its workflow in Fig. 2.
Formally, we cast rays from the camera origin o through the pixels of the image plane, and accumulate the sampled density σ and feature values F along the rays with near and far planes t n and t f . The aggregated feature F r is computed using the rendering equation [29] as follows:
where
Here, d refers to the ray direction and T (t) corresponds to the accumulated transmittance along the ray r from t n to t. The functions σ(r(t), m, z) and F (r(t), d, m, z) are the density and the feature at a 3D location, respectively.
Different from StyleNeRF [16], the generator G in CoRF is conditioned on a motion representation m ∈ R 50 for motion control. To this end, a ReLU mapping network f m embeds the motion representation m into a latent space, which is then injected to the synthesis module G s via weight modulation [24]. We discuss the details in Section 3.2. Also different from StyleNeRF [16], the generator G does not contain upsampling blocks. Instead, we increase the image resolution by sampling more rays. Discriminator. We use the estimated camera pose ξ and the motion feature m of a training image I ∼ p real to control generation of new images I ∼ p syn . The discriminator D is conditioned on a camera pose ξ and a motion feature m, following the conditional strategy of StyleGAN2-ADA [23,6]. We use a non-saturating adversarial loss with a gradient regularizer [30] applied on the discriminator D. We refer to this loss as L adv , i.e.,
where f (u)
-log(1 + exp(-u)). Further, p m and p ξ refer to distributions over motion vectors and pose respectively. In practice, we extract motion features and pose from training data samples using [12]. During training, we randomly sample a training image with its corresponding estimated motion feature and pose to control image synthesis.
A key goal of CoRF is to control synthesis of 3D-aware face dynamics. For this, we condition the generator G on low-dimensional motion representations m. In this section, we discuss our conditioning method, motion extraction from images, and the supervision which we use during training to encourage motion control. Motion conditioning. We embed the motion representation in a layered latent motion space using the mapping module f m . The layered latent motion space is a concatenation of N latent motion vectors, denoted as [d 1 , . . . , d N ] T := f m (m), where d i ∈ R 512 for i ∈ {1, . . . , N }. Here, N corresponds to the number of layers in the synthesis module G s , which we find to lead to feature embeddings with superior performance. The latent motion vectors are then combined with the latent style vector w. Formally, we combine style vector and motion representation via a summation, i.e.,
The conditioned latent vectors are then given to each layer of the synthesis module G s via modulation [24]. Instead of summation, other conditioning strategies for control exist. However, we find that not every strategy works equally well in our task. For example, recent head reconstruction work [13] concatenates a motion representation with a noise vector z which is then provided as input to the first layer of the model. We compare our conditioning strategy with this prior method [13] in Section 4 and find that our motion conditioning strategy accelerates convergence of the motion reconstruction loss L motion . Efficient motion extraction. To obtain a low-dimensional motion representation m for face dynamics, we extract coefficients of a blendshape basis of a statistical head model using a pre-trained regressor R with an accurate statistical head model [27]. For this we use a publicly available package. * Note that our method is also compatible with other pretrained head models. Compared with existing work [13] that uses an optimization-based approach for expression estimation, use of a pre-trained regressor R enjoys two advantages. First, using the regressor R is computationally more efficient as coefficient prediction requires only a single forward-pass. Second, we can use the regressor R as a differentiable plug-in to regress and supervise the expression representation of synthetically generated frames. Motion learning. A motion reconstruction loss L motion is applied to supervise motion control. It assesses whether the synthetically generated image exhibits the motion for which the representation was provided as input. For this we use a simple quadratic loss, i.e.,
where m is estimated by the regressor R, i.e., m = R(I ).
The aforementioned losses are insufficient to ensure spatio-temporal consistency. To address this, we propose * Code available from https://github.com/YadiraF/DECA two losses which focus on the generated face and one loss which focuses on background regions. All three losses are simple yet effective at preserving spatio-temporal consistency. In summary, to optimize CoRF we minimize the weighted objective
where the hyper-parameters λ 1 , λ 2 , . . . , λ 5 adjust the strength of the different loss terms. In practice, the regressor R and the identity encoder E are pre-trained offline on a large-scale dataset [12,4]. We describe each of the three additional losses L consist , L id and L bg next. Consistency of attributes. The motion conditioning occasionally changes lighting and person-specific attributes such as albedo and shape, resulting in flickering textures in rendered images. To alleviate this issue, we propose a consistency loss to supervise attributes independent from motion (see Fig. 3). Specifically, we extract lighting, texture, shape, and albedo representations for paired artificially generated images which have an identical noise vector z, yet a different motion representation m. These extracted attributes are independent from motion and thus should be identical for paired images. To encourage their consistency via a loss, we subsume representations of these attributes (lighting, albedo, texture, shape) for both paired images in the set A = {(l 1 , l 2 ), (a 1 , a 2 ), (t 1 , t 2 ), (s 1 , s 2 )}. We then compute the consistency loss
by comparing pairs of estimated attributes x 1 and x 2 . Here, l, a, t, and s are the attributes of lighting, albedo, texture, and shape, respectively. β x is the loss weight. In practice, these attributes are predicted by the regressor R. Consistency of identity. Beyond attribute consistency, we use an identity loss on paired images, following prior face reconstruction work [12] (see Fig. 3). To be concrete, an encoder E outputs identity embeddings of paired generated images which have an identical noise vector z yet a different motion representation. Let I 1 and I 2 refer to the two generated images. The identity loss is proportional to the negative cosine similarity between embeddings of both images, i.e., it is computed via
Consistency of background. Beyond the constraints on faces, we also want to encourage that the background remains mostly static over time. For this, we recognize background regions of paired synthetic images I 1 and I 2 , using Source Driving FOMM [48] MRAA [49] Ours Source Driving FOMM [48] MRAA [49] Ours Figure 4: Visual comparison of face reenactment on FFHQ. We transfer expression and pose from driving images to the source images. We compare our results with results of FOMM [48] and MRAA [49] at a resolution of 256 × 256 in challenging cases. We show face reenactment across identity, gender, pose, or with a partially occluded driving image.
a pre-trained face parsing net. † The corresponding binary background masks, which we refer to via s 1 and s 2 , use a value of 1 to indicate the background regions and a value of 0 to indicate the face regions. Using both masks, we define a background consistency loss on the overlapping background regions of paired images via
where • denotes an element-wise product. Due to the limited space, we illustrate details of the background loss L bg in the supplementary material.
Datasets and preprocessing. We evaluate the proposed approach on FFHQ [22] and two face video datasets, including Faceforensics++ [42] and VoxCeleb2 [9]. (i) FFHQ [22]: We show CoRF learning motion dynamics on FFHQ, which is a single image dataset. (ii) The Faceforen-sics++ dataset [42] contains 1,000 raw video sequences.
(iii) The VoxCeleb2 dataset [9] contains over 1 million videos for 6,112 identities. As videos in these two face datasets [42,9] contain low-quality frames with small or occluded faces, a pre-processing step is applied to clean the data. Specifically, we detect the faces in the videos using a pretrained landmark detection model [3], sharpen faces, filter out frames with small, blurry, or occluded faces, and align the remaining faces. We finally use 990 and 4,804 videos in each dataset [42,9] for CoRF training.
Baselines and evaluation metrics. We evaluate CoRF from two perspectives. First, we quantitatively compare our CoRF with face reenactment work [48,55]. To this end, we use the Fréchet Inception Distance (FID) [19] and 3 cosine similarity scores for identity preservation and expression transfer, namely ID, EXP † and EXP ‡ . Regarding identity † Code available from https://github.com/zllrunning/face-parsing.PyTorch Source Driving Bi-layer [69] Ours Figure 5: Visual comparison to Bi-layer [69].
preservation, we extract face features using a popular image identity recognition mode pretrained on VGGface2 [4]. Regarding expression transfer, for a fair evaluation, we use another pretrained facial expression estimation network [10] (EXP ‡ ), which differs from the encoder [12] (EXP † ) used in our model. Second, as CoRF generates never-before-seen dynamic faces from a prior noise distribution, similar to unconditional video synthesis approaches [56,44,63,55], we compare CoRF with these baselines using the Fréchet Video Distance (FVD) [57] score. The FVD score is a video-level FID score commonly used to examine both visual quality and temporal coherence of synthesized dynamics [63,18].
Comparison to face reenactment methods. We present a visual comparison to FOMM [48], MRAA [49], and Bilayer [69] in Fig. 4 and Fig. 5. For this, we transfer expressions and poses from real 'driving' images to synthetic source images in four challenging cases, including crossidentity, cross-gender, cross-pose, and driving image occlusion. We observe that the baseline methods struggle in these cases, while CoRF better preserves face geometry and identity. Additionally, we show targeted video generation on the VoxCeleb2 dataset [9] at a 256 × 256 resolution in Fig. 6. Different from the face reenactment baselines [48,49], CoRF is capable of controlling more factors such as novel views and field-of-view (FOV) without the requirement of a reference image, e.g., in Fig. 7. We show additional results in the supplementary material. We generate head videos (row 2 and 4) conditioned on facial expression information of the driving frames (row 1 and 3) at a 256 × 256 resolution. Note, the synthetic portraits have a similar expression as the driving frame (see highlighted eyes in row 2, and the mouth in row 4).
In Tab. 1, we quantitatively compare face reenactment methods on 10,000 pairs of source and driving images (variances in the supplementary material). We find CoRF to significantly improve upon the baselines [48,49] with regards to identity preservation and expression transformation. Complex expression animation. We show complex expression animation in Fig. 8. Among all randomly generated images, we observe the results of the proposed method to follow the expressions in the driving image, highlighting the method's capability. Motion interpolation in the latent space. To analyze the latent motion space, we render from multiple views a trajectory of synthetic images conditioned on linear interpolations between two motion representations, as shown in Fig. 9. The smooth interpolation between two expressions over viewpoint changes demonstrates that the latent motion space captures semantics and CoRF enables to generate dynamic faces from multiple views with 3D-awareness. Comparison to unconditional video synthesis methods. We present a quantitative temporal coherence comparison in Tab. 2 where we use 10,000 images/videos for numerical evaluation. Tab. 2 shows that CoRF significantly improves video-level (FVD) quality compared to the baselines [63,FaceF [42] Vox2 We compare our method with FOMM [48] and MRAA [47] at a 256 × 256 resolution. We compute the FID score [19], the face identity preservation score (ID), and expression similarities (two expression extractors [12] and [10] are used, denoted as EXP † and EXP ‡ , respectively).
Driving Result Driving Result Driving Result 55]. In Fig. 10, we also show a comparison of unconditional video synthesis methods [63,55] at a 64 × 64 resolution (size is chosen to be compatible with the baselines [63,55]). We observe that CoRF produces sharper faces and persistent head geometry over time and space (row 3). Ablation study. In Fig. 11 we show the motion reconstruction loss curves during training with different motion conditioning strategies. The baseline method concatenates a motion representation m with a noise vector z, as used in prior work [13] (blue curve). Compared to the baseline method, our motion embedding strategy (green and red curves) accelerates convergence of the motion reconstruction loss L motion . Moreover, we evaluate the effectiveness Table 2: Quantitative evaluation for video synthesis. We evaluate our method using FVD [57] scores (↓) and compare to MoCoGAN [56], G 3 AN [63], TGANv2 [44] at the 64 × 64 resolution, and with MOCOGAN-HD [55] at the 256 × 256 resolution.
[63]
[55]
Ours Figure 10: Visual comparison of video synthesis. We compare our method with popular unconditional video synthesis approaches G 3 AN [63] and MoCoGAN-HD [55] on VoxCeleb2 [9]. The appearance of our method's results remains consistent over view and expression changes, while baselines struggle (see frames highlighted with red border).
of conditioning the discriminator on motion. We find that motion conditioning of the discriminator (green curve) is slightly better than omission of motion conditioning (red curve). Thus, we use a conditional discriminator in all the experiments. Fig. 12 provides a visual comparison for the ablation settings of the proposed consistency losses, illustrated below the figure (column 1-3). For each setting, we render two synthetic images applying an identical identity representation yet a different motion representation (top and bottom). We find that the identity loss L id , the background loss L bg , and the consistency loss L consist help to preserve the face identity and a stable background over expression and pose changes. The table on the right shows the averaged loss values with or without optimizing the corresponding loss terms, which supports the visual observation.
We propose controllable radiance fields (CoRFs) to address the task of generalizable, 3D-aware, and motioncontrollable synthesis of face dynamics. CoRFs allow not only explicit 3D-aware motion control and arbitrary view- Figure 12: Ablation study for losses. We render paired synthetic images conditioned on the same identity representation but different motion representations (row 1-2) using different ablation settings for identity, background, and consistency losses. In the table, we show averaged values of the losses L id and L bg (column 2-3), without (or with) optimizing them.
points but also diverse identity generation in photo-realistic quality. Extensive experiments highlight that CoRFs improve performance w.r.t. 3D consistency and identity preservation over pose and motion changes, compared to state-of-the-art methods on benchmark video datasets. We hope this work inspires future work on the challenging task of 3D-aware dynamic face synthesis.