In this paper, we seek to draw connections between the
frontal and profile face images in an abstract embedding
space. We exploit this connection using a coupled-encoder
network to project frontal/profile face images into a common
latent embedding space. The proposed model forces
the similarity of representations in the embedding space by
maximizing the mutual information between two views of
the face. The proposed coupled-encoder benefits from three
contributions for matching faces with extreme pose disparities.
First, we leverage our pose-aware contrastive learning
to maximize the mutual information between frontal
and profile representations of identities. Second, a memory
buffer, which consists of latent representations accumulated
over past iterations, is integrated into the model so it
can refer to relatively much more instances than the minibatch
size. Third, a novel pose-aware adversarial domain
adaptation method forces the model to learn an asymmetric
mapping from profile to frontal representation. In our
framework, the coupled-encoder learns to enlarge the margin
between the distribution of genuine and imposter faces,
which results in high mutual information between different
views of the same identity. The effectiveness of the proposed
model is investigated through extensive experiments, evaluations,
and ablation studies on four benchmark datasets,
and comparison with the compelling state-of-the-art algorithms.
more »
« less
Pose Attention-Guided Profile-to-Frontal Face Recognition
In recent years, face recognition systems have achieved
exceptional success due to promising advances in deep
learning architectures. However, they still fail to achieve the expected accuracy when matching profile images against a
gallery of frontal images. Current approaches either perform
pose normalization (i.e., frontalization) or disentangle
pose information for face recognition. We instead propose a
new approach to utilize pose as auxiliary information via
an attention mechanism. In this paper, we hypothesize that
pose-attended information using an attention mechanism
can guide contextual and distinctive feature extraction from
profile faces, which further benefits better representation
learning in an embedded domain. To achieve this, first, we
design a unified coupled profile-to-frontal face recognition
network. It learns the mapping from faces to a compact embedding
subspace via a class-specific contrastive loss. Second,
we develop a novel pose attention block (PAB) to specially
guide the pose-agnostic feature extraction from profile
faces. To be more specific, PAB is designed to explicitly
help the network to focus on important features along
both “channel” and “spatial” dimensions while learning
discriminative yet pose-invariant features in an embedding
subspace. To validate the effectiveness of our proposed
method, we conduct experiments on both controlled and in the-
wild benchmarks including Multi-PIE, CFP, and IJB-C, and
show superiority over the state-of-the-art.
more »
« less
- Award ID(s):
- 1650474
- NSF-PAR ID:
- 10401301
- Date Published:
- Journal Name:
- IEEE Int. Joint Conference on Biometrics (IJCB'22)
- Page Range / eLocation ID:
- 1 to 10
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
We introduce caption-guided face recognition (CGFR) as a new framework to improve the performance of commercial-off-the-shelf (COTS) face recognition (FR) systems. In contrast to combining soft biometrics (e.g., facial marks, gender, and age) with face images, in this work, we use facial descriptions provided by face examiners as a piece of auxiliary information. However, due to the heterogeneity of the modalities, improving the performance by directly fusing the textual and facial features is very challenging, as both lie in different embedding spaces. In this paper, we propose a contextual feature aggregation module (CFAM) that addresses this issue by effectively exploiting the fine-grained word-region interaction and global image-caption association. Specifically, CFAM adopts a self-attention and a cross-attention scheme for improving the intra-modality and inter-modality relationship between the image and textual features. Additionally, we design a textual feature refinement module (TFRM) that refines the textual features of the pre-trained BERT encoder by updating the contextual embeddings. This module enhances the discriminative power of textual features with a crossmodal projection loss and realigns the word and caption embeddings with visual features by incorporating a visualsemantic alignment loss. We implemented the proposed CGFR framework on two face recognition models (Arc- Face and AdaFace) and evaluated its performance on the Multimodal CelebA-HQ dataset. Our framework improves the performance of ArcFace from 16.75% to 66.83% on TPR@FPR=1e-4 in the 1:1 verification protocol.more » « less
-
Although face recognition (FR) has achieved great success in recent years, it is still challenging to accurately recognize faces in low-quality images due to the obscured facial details. Nevertheless, it is often feasible to make predictions about specific soft biometric (SB) attributes, such as gender, age, and baldness even in dealing with low-quality images. In this paper, we propose a novel multi-branch neural network that leverages SB attribute information to boost the performance of FR. To this ed, we propose a cross-attribute-guided transformer fusion (CATF) module that effectively captures the long-range dependencies and relationships between FR and SB feature representations. The synergy created by the reciprocal flow of information in the dual cross-attention operations of the proposed CATF module enhances the performance of FR. Furthermore, we introduce a novel self-attention distillation framework that effectively highlights crucial facial regions, such as landmarks by aligning low-quality images with those of their high-quality counterparts in the feature space. The proposed self-attention distillation regularizes our network. to learn a unified quality-invariant feature representation in unconstrained environments. We conduct extensive experiments on various real-world FR benchmarks varying in quality. Experimental results demonstrate the superiority of our FR method compared to state-of-the-art FR studies.more » « less
-
We introduce caption-guided face recognition (CGFR) as a new framework to improve the performance of commercial-off-the-shelf (COTS) face recognition (FR) systems. In contrast to combining soft biometrics (e.g., facial marks, gender, and age) with face images, in this work, we use facial descriptions provided by face examiners as a piece of auxiliary information. However, due to the heterogeneity of the modalities, improving the performance by directly fusing the textual and facial features is very challenging, as both lie in different embedding spaces. In this paper, we propose a contextual feature aggregation module (CFAM) that addresses this issue by effectively exploiting the fine-grained word-region interaction and global image-caption association. Specifically, CFAM adopts a self-attention and a cross-attention scheme for improving the intra-modality and inter-modality relationship between the image and textual features, respectively. Additionally, we design a textual feature refinement module (TFRM) that refines the textual features of the pre-trained BERT encoder by updating the contextual embeddings. This module enhances the discriminative power of textual features with a cross-modal projection loss and realigns the word and caption embeddings with visual features by incorporating a visual-semantic alignment loss. We implemented the proposed CGFR framework on two face recognition models (ArcFace and AdaFace) and evaluated its performance on the Multi-Modal CelebA-HQ dataset. Our framework significantly improves the performance of ArcFace in both 1:1 verification and 1:N identification protocol.more » « less
-
Despite the advances in the field of Face Recognition (FR), the precision of these methods is not yet sufficient. To improve the FR performance, this paper proposes a technique to aggregate the outputs of two state-of-the-art (SOTA) deep FR models, namely ArcFace and AdaFace. In our approach, we leverage the transformer attention mechanism to exploit the relationship between different parts of two feature maps. By doing so, we aim to enhance the overall discriminative power of the FR system. One of the challenges in feature aggregation is the effective modeling of both local and global dependencies. Conventional transformers are known for their ability to capture long-range dependencies, but they often struggle with modeling local dependencies accurately. To address this limitation, we augment the self-attention mechanism to capture both local and global dependencies effectively. This allows our model to take advantage of the overlapping receptive fields present in corresponding locations of the feature maps. However, fusing two feature maps from different FR models might introduce redundancies to the face embedding. Since these models often share identical backbone architectures, the resulting feature maps may contain overlapping information, which can mislead the training process. To overcome this problem, we leverage the principle of Information Bottleneck to obtain a maximally informative facial representation. This ensures that the aggregated features retain the most relevant and discriminative information while minimizing redundant or misleading details. To evaluate the effectiveness of our proposed method, we conducted experiments on popular benchmarks and compared our results with state-of-the- art algorithms. The consistent improvement we observed in these benchmarks demonstrate the efficacy of our approach in enhancing FR performance. Moreover, our model aggregation framework offers a novel perspective on model fusion and establishes a powerful paradigm for feature aggregation using transformer-based attention mechanisms.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1109/IJCB54206.2022.10007935
