Human faces arguably represent the most important class of stimulus in social interaction. Any normal adult can quickly make approximated judgments about the gender, age, and race of a person even though the face might be unfamiliar [29]. For people with certain pathological conditions (e.g., autism), they might have difficulty with extracting face-related information [32]. Face perception is a problem of fundamental importance not only to computer vision but also psychology and neuronscience. Computational studies of face images have attracted increasingly more attention in recent years especially due to rapid advances in deep learning -from facial landmark detection [56] and face recognition (e.g., DeepFace [37]) to face superresolution [3] and face synthesis (e.g., StyleGAN [17]). Most recently, there are a flurry of works on face-related novel applications such as beautification [23] and makeup editing [4], facial gesture synthesis [6], face aging studies [48,45] and face gender classification (e.g., [21,30]).
Among them face-related synthesis is particularly interesting thanks to the advanced GANs (Generative Adversarial Networks) [10] and has a wide range of applica-tions in graphics, human computer interaction and social computing. Novel network architectures such as Cycleconsistent Generative Adversarial Networks (CycleGAN) [57] and Multimodal Unsupervised Image-to-Image Translation (MUNIT) [14] have been widely studied for style transfer or translation. However, none of the existing unsupervised learning approaches are capable of delivering satisfactory synthesis results for face gender transfer. How to separate gender (style) from identity (content) for face images turns out to be nontrivial and has been under-explored in the open literature (content-style separation problem has only been studied for textual images recently in [55]). In this paper, we propose to cast face gender transfer as a special class of style transfer problems with the additional constraint of identity preservation. With the help of the proposed gender synthesis framework, we can address the problem of gender bias present in many facial image datasets.
Inspired by the Learned Perceptual Image Patch Similarity (LPIPS) [54], we propose to tackle the problem of face gender transfer in the space of deep feature representation rather than face images [4] or latent representations [14]. More specifically, we have adopted the lightCNN [46] as the pre-trained network to transform any face image to a 256-dimensional (256D) deep feature representation. Based on the observation that perceptual similarity is an emergent property shared across deep visual representations, we introduce a novel probabilistic gender mask to softly separate the gender from the identity information in the feature space. Accordingly, we have designed a whole class of new loss functions that jointly achieves the objectives of gender transfer and identity preservation. It is worth mentioning that unlike face makeup editing [4], we target at learning a pair of symmetric nonlinear mapping functions between the space of male and female faces.
Our main contributions are summarized as follows:
• We have developed a novel approach in a light-CNN pre-trained 256 dimensional deep feature space for face gender transfer that preserves the identity information; we have also designed a class of new loss functions based on a probabilistic mask in (0, 1) 256 separating the gender from the identity information. Gender mask learning has led to the discovery of a collection of sparse features (≈ 20 out of 256) uniquely responsible for face gender perception;
• Our experimental results have demonstrated the superiority both on visual quality and representation interpretability to other competing methods including Cy-cleGAN [57], MUNIT [14], DRIT [20] and StarGAN [5].
• We have also empirically verified the effectiveness of deep gender feature representation learning by demon-strating a high correlation between the learned sparse features and the gender information. Our results corroborate a hypothesis about the independence between face recognizability and gender classifiability [29] in the literature of psychology.
The capability of deep generative models has dramatically improved thanks to the invention of generative adversarial networks (GAN) [10]. By concatenating a generator with a discriminator and training them by playing a zerosum game, GAN has achieved impressive results in various image synthesis tasks [7,57,16,17,2,6,18]. In particular, virtual generation of face images has been studied for photo-sketch synthesis [44], face image alignment [46], face aging studies [1] and facial gesture transfer [6]. GAN and its variants (e.g., conditional GAN [26], contextual GAN [22], progressive GAN [16], styleGAN [17,18]) have been among the most popular and successful approaches toward face image synthesis. For a recent survey on face image synthesis, please refer to [42] and its references.
Among various synthesis tasks, the class of unpaired image-to-image translation is particularly interesting and has many practical applications. The definition of a source and a target domain might include face photo and sketch [31], faces with different ages (e.g., [1]), faces with and without makeup [4] and faces with varying expressions [34]. Various GAN-based architectures have been adapted and extended for face image-to-image translation -e.g., conditional GAN [15], StarGAN [5], DualGAN [50], StackGAN [51], pairedCycleGAN [4], pyramid GAN for age progression [49], and ExprGAN for expression editing [8].
Computational modeling of face perception has always been at the intersection of basic (e.g., to explain how we perceive human faces) and applied (e.g., to improve the performance of face recognition) sciences. Early works based on the principal component analysis (PCA) have been studied for both face recognizability [39] and gender classifiability [29]). Recent models developed based on deep neural networks have shown a unified solution to both problems of face recognition and gender classification (e.g., [30]).
However, there still remains a significant gap between the computational modeling (computer vision community) and biological modeling (neuroscience and psychology) of face perception. Taking face gender as an example, both computers and humans can effortlessly recognize the gender from a face image; but their underlying computational model or sensory processing mechanism might be different. Recent works on face image reconstruction from fMRI data [41] and face space representation in deep CNN [28] can be viewed as early attempts to bridge the gap between these communities. In this work, we also attempt to shed light on both the problems of face gender transfer and perception.
We propose to formulate the problem of face gender transfer by generating a face image with the opposite gender without changing the identity. It is conceptually similar to the unsupervised learning for image-to-image translation [15] and style transfer [57]. However, the constraint of preserving the face identity (content) while transferring the gender (style) distinguishes this work from other existing works in the literature. Note that someone might argue that gender is part of identity (e.g., soft biometrics); we opt for a narrow-sense definition of identity here (i.e., twins would be treated the same class due to their almost identical visual appearance). Additionally, a secondary objective of this study is to obtain an interpretable computational model for face gender perception.
Let us introduce some notation first. We will denote the two domains by M (Male) and F (Female) respectively and training samples by
where f i ⊂ F . As shown in Fig. 2, our model consists of two encoder-decoders as cross-domain generators and two discriminators for each domain:
or cross-domain translation
where we have used F (M ), M (F ) as the short hand for gender transfer M → F, F → M and ŝf , ŝm to denote the random perturbation of swapped style codes (for gender transfer) as shown in Fig. 2.
The introduction of discriminators D m , D f is to ensure that the translated images F (M ), M (F ) satisfy the gender constraint (i.e., F (M ), M (F ) do appear like real F/M ). Similar to GAN [10], we have used the following adversarial losses:
and the corresponding reconstruction losses are defined by
where || • || 1 denotes the L 1 -norm.
In MUNIT [14], one of the key insight is brought by latent reconstruction (as shown by the rightmost four boxes in Fig. 2) -i.e., we should be able to reconstruct the latent (style or content) code sampled from the latent distribution after decoding and encoding. Following [14], the latent reconstruction losses associated with F (M ) are given by
and similarly we can define L c f REC , L ŝm REC associated with M (F ). However, our empirical studies have shown that such latent reconstruction is not powerful enough for the challenging task of face gender transfer. Inspired by the success of Learned Perceptual Image Patch Similarity (LPIPS) [54], we propose to design novel loss functions in the space of feature representation (FR) instead of latent representation such as adopted by MUNIT [14].
In this work, we have adopted a recently developed lightCNN [46] as the pre-trained network to extract a 256dimensional feature representation from a given image (as shown in Fig. 3). A key motivation behind the adoption of lightCNN lies in its capability of learning a compact embedding from original face space (M, F ) ∈ R H×W to x ∈ R 256 even in the presence of massive noisy labels. Such a compact representation of large-scale face data enables us to enforce more powerful constraints in the feature (instead of image or latent) space. Note that in MUNIT [14], latent reconstruction loss function is intrinsically coupled with encode-decoder pairs, which limited its role of regularization. By contrast, using a pre-trained network as a tool of nonlinear dimensionality reduction greatly facilitates the task of network regularization.
The other important observation is that no content/style encoder is known for extracting the identity/gender information from a face image. The architecture of content/style encoder in MUNIT [14] is simply too ad-hoc for the task of face gender transfer. Therefore, we propose to learn a probabilistic gender mask 0 ≤ w i ≤ 1(i = 1, 2, ..., 256) to dynamically filter gender representation in our 256D feature representation w. More specifically, w i = 1 -p i where p i denotes the probability of the i-th dimension is genderrelevant (a smaller w i implies a higher influence on gender performance). Note that the update of mask can be conveniently implemented by the ReLU operator during back propagation.
The definition of gender mask w allows us to simultaneously achieve the objectives of gender transfer and identify preservation as shown in Fig. 3. By dividing the 256D feature representation into gender-relevant (w i → 0) and identity-relevant (w i → 1) components, we can conquer them separately by designing a pair of loss functions in the masked feature space: one has to assure that the transferred face image has the opposite gender (i.e., maximally separated from the original); and the other is to assure that the transferred face image is still visually similar to the original (i.e., to minimize the perceptual distortion or maximize the perceptual similarity [54]). Based on the above motivations, we proceed with the design of novel loss functions next. With the pre-trained network for feature extraction, the perceptual loss in the image space can be redefined in the feature space as follows:
and similarly, we can redefine the similarity between the original and transferred faces in the feature space by:
3.3.2 Classification Loss Function: Separating Male from Female.
The objective is to assure the success of gender transferi.e., among the 256D feature representation, those relevant to gender will be maximally separated after the transfer. Toward this objective, we have adopted a three-layers Perceptron for the task of gender classification (due to their simplicity). As we mentioned above, gender-relevant feature information is distilled when w i → 0. Therefore, it is natural to work with 1 -w when feeding the three-layer linear regression. More specifically, the classification loss function (CLF) for gender transfer is defined as following:
where • denotes the element-wise multiplication and the Binary Cross Entropy (BCE) function is defined by
Figure 3: The two pillars of our approach: feature representation and gender mask. A 256D deep feature representation is extracted from a given face image by light CNN; gender mask plays the role of killing two birds (gender transfer L mf/fm CLF and identity preservation L x,mf/fm mask ) using one stone ( w).
Now the dual objective is to preserve the identity (perceptual similarity) as much as possible regardless of gender transfer. Based on similar reasoning as before, we conclude that w i → 1 indicates the gender-irrelevant entries in the feature representation. As shown in Fig. 3, we simply work with w instead of 1-w for this dual task. More specifically, the perceptual similarity loss function with masked entries will be defined in a similar manner to Eqs. ( 6) and (7) as follows
where • again denotes the element-wise multiplication. Note that Eq. ( 10) will be optimized if w = [0, 0, 0...] (a pathological case). This is an unfortunate consequence of ignoring important a priori knowledge about the sparse distribution of gender in the feature space -i.e., the L 1 norm of w cannot be too small because we know in advance that the feature dimensions associated with gender are much smaller than 256 (actually our experiments will show later that the dimensionality of gender subspace is around 20 << 256).
To avoid this pitfall, we propose to introduce the following regularization/prior term as following:
Joint optimization of Eqs. ( 10) and ( 11) leads to a sparse Bayesian learning of w, which is also coupled with the dual CLF defined above.
Putting things together, we have the total loss function given by:
where λ mask ,λ clf , λ rec are the weights balancing the importance of different loss terms.
CelebA [24] is widely used in various face-related research due to its large number of facial attributes. There are 202,599 images (totaling 10,177 identities) each of which contains 40 face attribute labels including gender information. Different from their protocol [24], we opt to adopt their validation set as our training set and use the original testing set for evaluation in our experiments.
As shown in Fig. 2, content encoder consists of two stride-2 convolutions and one residual block [11] and all convolutional layers are followed by an Instance Normalization (IN) [40] module; style encoder E s contains several stride convolutional layers with a global average pooling layer and a fully connected layer. Regarding the generator, Adaptive Instance Normalization [13] is employed with residual blocks. Also, VGG [33] feature is extracted to keep perceptual invariance (similar to [14]). Additionally, Least-Square GAN (LSGAN) [25] and multi-scale discriminators [43] techniques are used for discriminator training. The popular Adam algorithm [19] is used as the training optimization method with a learning rate of 0.001 and a batch size of 2.
In order to evaluate the proposed method of face gender transfer, we have conducted the following two experiments: 1) user study on Amazon Mechanical Turk providing subjective quality measurements and 2) calculate Fréchet Inception Distance (FID) [12] as an objective image quality metric. We have compared our approach against three existing methods: CycleGAN [57]: A cycle consistency loss is proposed to enforce image style transfer between a source domain and a target domain, which lays a solid framework for image-toimage translation using unpaired training data. MUNIT [14]: A framework for multimodal unsupervised image-to-image translation. It assumes that image representation can be decomposed into a content code that is domain-invariant and a style code that captures domainspecific properties. By combining content code with a random style code, MUINT can generate a variety of outputs from a single input. DRIT [20]: A network architecture decomposes image representation into two subspaces: a domain-invariant content subspace capturing shared information across domains and a domain-specific attribute subspace. By swapping domainspecific representations, the DRIT model is capable of generating diverse outputs and implementing flexible image style transfer in an unsupervised manner using a cross-cycle consistency loss.
In our user study, 100 originally-male and 100 originallyfemale identities were used. We have conducted two sets of surveys focusing on Translation and Similarity respectively. In the Translation surveys, participants were presented with two images of the same identity and asked to choose the one which "looks more like a male/female". One of the two images was generated by our method and the other was generated by one of the three competing methods (CylceGAN, MUNIT, and DRIT). In the Similarity surveys, participants were also presented with two images of the same identity, but were asked to rate "from 0 (extremely different) to 10 (extremely similar), how similar are the two faces". One of the two images was the original and the other was generated rating mean std. dev. The order of presentation and the left/right location of images were fully randomized. In total, there were 600 pairs of images in the Translation surveys and also 600 pairs of images in the Similarity surveys. 25 responses were collected for each pair of images. The results show that our method outperforms the three existing methods in terms of preference ratio calculated from the Translation survey. As to the preservation of identity, our method is better than DRIT and MUNIT, but not as good as CycleGAN. Such experimental findings seem to suggest that cycle-consistency loss is beneficial to the task of identity preservation.
FID [12] has been widely used for measuring the subjective quality of synthetic images such as [2]. FID metric is calculated over features extracted from an intermediate layer in the Inception network [36]. We have conducted an evaluation with FID between the original images and the end images after gender transfer. The feature data are modelled by a multivariate Gaussian distribution with mean µ and covariance Σ. The FID value between the real image x and the synthetic image y is given by the formula below:
Where T r(A) denotes the trace of square matrix A. Lower FID values imply better image quality. Our approach has achieved the best performance in terms of FID as shown in Table 3.
To better quantitatively evaluate the gender transfer performance, we have designed an experiment to fool the classifier using translated images. First we train a gender classi-
CycleGAN MUNIT DRIT Ours fooling rate 72.40% 65.55% 30.08% 78.67% 4. It can be observed that our approach achieves the highest fooling rate, which justifies its superiority to other competing approaches.
As mentioned before, our architecture learns an adaptive mask to separate gender relevant information from identity representation of face images. Our objective is to not only show the superiority in terms of gender style transfer but also provide an explainable solution (so-called interpretable machine learning [27] or explainable AI) to learn genderrelated representations from face images.
To gain deeper insight into the learned gender-related representations, we have designed the following experiment with a linear SVM classifier to demonstrate that deep facial features selected by the probabilistic gender mask w in Sec.
3.2 has a strong correlation with the actual gender attribute. In our experiment, we have compared with two different schemes of feature selection: one is based on the learned probabilistic gender mask and the other random sampling (i.e., randomly select from the 256-dimensional feature). It can be seen from Figure 5, when the number of selected features is within the range of [5,75], the classification accuracy of learned gender mask is much higher than randomly selected features. Such experimental result provides strong supporting evidence about the high correlation between the learned deep facial features and gender facial attribute. On the other hand, the interpretability of convolutional neural networks (CNNs) [52], has only recently received some attention from the computer vision community. We argue that making the learned representation interpretable is not only for the purpose of breaking the bottlenecks of deep learning [53] but also to facilitate the communication between computer vision and cognitive science communities. Our result supports a well-known hypothesis in psychology -the independence between face recognizability and gender classifiability [29].
Though our model is able to capture the key characteristics about gender information and achieves appealing results in face gender transfer, some failure translation does exist. From our experiments, wearing eyeglasses, large pose variation, and extreme age groups (refer to Fig. 6) are typical failure cases that our generator cannot perform a quality transfer. The failure example can be classified into two categories: one is image quality based, such as occlusion including eyeglasses, hair occlusion, and large pose variation. These challenges are still problematic in computer vision community, no matter for image generation or recognition task, which is also our next step work. The other is age-related (e.g., for people who are too old or too young), we argue it is because they have less gender related features compared to normal adults. Furthermore, we also observe that our model is not robust enough to generate some marginal details of face -e.g. eyebrows and face symmetry as shown in Fig. 7. This is also the most challenging task for current GAN model to generate realistic images including state-of-the-art generation models [17,18].
In this paper, we present a novel GAN-based face gender translation architecture with a sparse representation learning. Our model not only generates high quality facial synthesis on gender transfer, but learns a gender related compact representation on the deep facial features space. It is a first experiment attempting at the problem of gender representation interpretation from a GAN-based model. We believe the proposed method can serve as a practical solu- tion to address the gender bias issue, commonly present in many public facial image datasets for various face recognition tasks [38,35,47,9].