With the huge success of deep learning techniques, current state-of-the-art computer vision algorithms have approached or exceeded human ability in recognizing a face (Taigman, Yang, Ranzato, & Wolf, 2014;Stewart, Andriluka, & Ng, 2016) and identifying the objective properties of a face, such as age and gender estimation, (Guo, Fu, Dyer, & Huang, 2008). However, humans not only read objective properties from a face, like expression, age, and identity, but also form subjective impressions of social aspects of a face (Todorov, Olivola, Dotsch, & Mende-Siedlecki, 2015) at first sight, such as facial attractiveness (Thornhill & Gangestad, 1999), friendliness, trustworthiness (Todorov, Baron, & Oosterhof, 2008), sociability, dominance (Mignault & Chaudhuri, 2003), and typicality. In spite of the subjective nature of social perceptions, there is often a consensus among human in how they perceive attractiveness, trustworthiness, and dominance † These authors contributed equally. in faces (Falvello, Vinson, Ferrari, & Todorov, 2015;Eisenthal, Dror, & Ruppin, 2006). This indicates that faces contain high-level visual cues for social inferences, therefore making it possible to model the inference process computationally. Social judgments, as an important part of people's daily interactions, have a significant impact on social outcomes, ranging from electoral success to sentencing decisions (Oosterhof & Todorov, 2008;Willis & Todorov, 2006).
Are deep learning models, which are successful in various visual tasks, also capable of predicting subjective social impressions of faces? Even before the advent of deep learning, there have been models using traditional computer vision algorithms and simulated faces to model the perception of facial attractiveness (Thornhill & Gangestad, 1999;Eisenthal et al., 2006;Kagian et al., 2008;Gray, Yu, Xu, & Gong, 2010), trustworthiness (Falvello et al., 2015;Todorov, Baron, & Oosterhof, 2008), sociability, aggressiveness (Mignault & Chaudhuri, 2003), familiarity (Peskin & Newell, 2004), and memorability (Bainbridge, Isola, & Oliva, 2013;Khosla, Bainbridge, Torralba, & Oliva, 2013). Recently, there has been work on modeling the "big five " personality traits perceived by humans when viewing another person in video clips (Escalera et al., 2016).
In this paper, we examine human social perceptions of faces in 40 dimensions extensively and systematically. We evaluate the human consistency and correlation in 40 social features (20 relevant pairs) that are typically studied by social psychologists (Todorov, Said, Engell, & Oosterhof, 2008), and relevant to social interactions (Todorov et al., 2015;Oosterhof & Todorov, 2008), and use state-of-the-art deep learning algorithms to model all 40 of them. Using the internal representations learned from the deep learning models, our model can successfully predict human social perception whenever human have a consensus. We further visualize the key features defining different social attributes to facilitate a understanding of what makes a face salient in a certain social dimension.
To predict human social impressions of faces, we use a public dataset (Bainbridge et al., 2013) consisting of 2,222 face images and annotations for 40 social attributes. Each attribute is rated on a scale of 1-9 by 15 subjects. We take the average rating from all raters as a collective estimation of human judgment for the social features of each face.
The 40 social attributes consist of 20 pairs of related traits: (attractive, unattractive), (happy, unhappy), (friendly, unfriendly), etc. Some of these traits are highly correlated and predictable from others, especially within the trait pairs. To understand the human-perceived correlations between these traits, we compute the Spearman's rank correlation between the average human ratings of every pair of social features and show their correlations in a heatmap (Figure 1(a)). We order traits in the map based on similarity and positive or negative connotation. From the figure, we see that negative social features such as untrustworthy, aggressive, cold, introverted, and irresponsible form a correlated block. Likewise, the most positive features such as attractive, sociable, caring, friendly, happy, intelligent, interesting, and confident are highly correlated with each other. Although we choose 20 pairs of opposite features, they are not completely complementary and redundant. Principal Component Analysis of the covariance matrix shows that it takes 24 principal components to cover 95% of the variance.
After averaging human ratings, each face receives a continuous score from 1 to 9 in all social dimensions. We model these social scores with a regression model. We propose a ridge regression model on either features from deep convolutional neural networks (CNN) or traditional face geometry based features, and present results from both feature sets. Such visual features are usually high-dimensional, so we first perform Principal Component Analysis (PCA) on the extracted features of the training set to reduce dimensionality. The PCA dimensionality is chosen by cross-validation on a validation set, separately for each trait. The PCA weights are saved and further used in fine-tuning our CNN-regression model.
Past studies have found that facial attractiveness can be inferred from the geometric ratios and configurations of a face (Eisenthal et al., 2006;Kagian et al., 2008). We suggest that other social attributes can also be inferred from geometric features. We compute 29 geometric features based on definitions described in (Ma, Correll, & Wittenbrink, 2015), and further extract a 'smoothness' feature and 'skin color' feature according to the procedure in (Eisenthal et al., 2006;Kagian et al., 2008). The smoothness of a face was evaluated by applying a Canny edge detector to regions from the cheek and forehead areas (Eisenthal et al., 2006). The more edges detected, the less smooth the skin is. The regions we chose to compute smoothness and skin color are highlighted in the right subplot of Figure 2. The skin color feature is extracted from the same region as smoothness, converted from RGB to HSV. However, regressing on these handcrafted features alone is not enough to capture the richness of geometric details in a face. We therefore use a computer vision library (dlib, C++) to automatically label 68 face landmarks (see Figure 2) for each face, and then compute distances and slopes between any two landmarks. Combining 29 handcrafted geometric features, smoothness, color and the distance-slope features, we obtain 4592 features in total. Since the features are highly correlated, we apply PCA to reduce dimensionality. Again, the PCA dimensionality is chosen by cross-validating on the hold out set separately for each facial attribute. Then a ridge regression model is applied to predict social attribute ratings of a face. The hyper-parameter of ridge regression is selected by leave-one-out validation within the training set.
Previous studies have shown that pretrained deep learning models can provide feature representations versatile for related tasks. We therefore extract image features from pretrained neural networks, choosing from six architectures with different original training goals: (1) VGG16, trained for object recognition (Simonyan & Zisserman, 2014), (2) VGG-Face, trained for face identification (Simonyan & Zisserman, 2014), (3) AlexNet, trained for object classification (Krizhevsky, Sutskever, & Hinton, 2012), (4) Inception from Google, trained for object recognition (Szegedy et al., 2015), (5) a shallow Siamese neural network that we train from scratch to cluster faces by identity, (6) a state of the art VGG-derived network (Face-LandmarkNN) trained for the face landmark localization task.
To find the best CNN features among the six networks, we first find the best-performing feature layers of each network in the ridge regression prediction task. Before the ridge regression, we perform PCA and pick the PCA dimensionality that gives best results on the validation set. Then, we compare the results among networks to select the best features overall.
After comparing all 6 networks, we find that the conv5 2 layer of VGG16 (trained for object classification) lead to the best results. This set of features significantly outperforms the three networks trained solely on faces, while also slightly outperforming AlexNet and Inception networks. These bestperforming CNN features also exceed the prediction correlation of the geometric features in most attributes. Figure 3 compares prediction performance of the CNN model and the geometric feature model.
We speculate that the poor performance from the face recognition networks can be attributed to their optimization for specific facial tasks. Learning face landmark configurations and differences between faces that define identity may not correlate well with the task at hand, which looks for commonalities behind certain social features beyond identity. The consider them in a decision. A robot need not treat an attractive or unattractive person differently for its own purposes, but this knowledge could affect how interactions are made for the sake of the human, knowing in advance how that person may feel that they fit into the social landscape.
Expansions on this work may include investigating the image properties that determine high level social features, beyond the attractiveness features we display in Figure 5. Additionally, social trait prediction may benefit from a single model with a shared representation, while this paper approaches each attribute as a separate regression task.
For future work, we aim to develop a generative model which can automatically modify a face's attributes (either objective or subjective) while preserving its realism and identity. Practically speaking, such a model could improve a face's perceived social features in positive ways (e.g. make a face look more sociable, trustworthy). More importantly, it would enable psychologists to quantify human biases during the formation of social impression in a precise and systematic manner. Psychologists could generate variants of a real face differing in age, gender, race, and explore how various factors separately and jointly affect the social impressions of faces.