MMGInpainting: Multi-Modality Guided Image Inpainting Based On Diffusion Models

Cong Zhang İD , Wenxia Yang İD , Xin Li İD , Fellow, IEEE, Huan Han

Abstract-Proper inference of semantics is necessary for realistic image inpainting. Most image inpainting methods use deep generative models, which require large image datasets to predict and generate content. However, limited control makes predicting the missing regions and generating coherent content difficult. Existing approaches include image-guided or text-guided image inpainting, but none of them has taken both image and text as the guidance signals, as far as we know. We propose a multi-modality guided (MMG) image inpainting approach based on the diffusion model to fill this gap. This MMGInpainting method uses image and text as guidance for generating content within the target area for inpainting, effectively integrating the semantic information conveyed by the guiding image or text into the content of the inpainted region. To construct MMGInpainting, we start by enhancing the U-Net backbone with a customized Nonlinear Activation Free Network (NAFNet). This adapted NAFNet incorporates an Anchored Stripe Attention mechanism, which utilizes anchor points to model global contextual dependencies effectively. To regulate inpainting, we use a Semantic Fusion Encoder to guide the inverse process of the diffusion model. The process is iteratively executed to denoise and generate the desired inpainting result. Additionally, we explore how different modes of meaning interact and coordinate to offer users helpful guidance for a more manageable inpainting procedure. Experimental results demonstrate that our approach produces faithful results adhering to the guiding information while significantly improving computational efficiency. Github Repository: https://github.com/skipper-zc/MMGInpainting/ Index Terms-Image inpainting, multi-modality guidance, diffusion models, NAFNet, controllable inpainting.

I. INTRODUCTION

I MAGE inpainting is a computer vision technique that aims to intelligently fill in the missing parts of an image seamlessly and coherently. Its primary objective is to generate visually plausible content with reasonable semantics that harmonizes with the surrounding regions, which not only focuses on the image's appearance but also considers the semantic content to ensure that the inpainted areas make sense within the overall image. At present, the most advanced methods are mainly based on Generative Adversarial Networks(GANS) [1] [2] [3], or diffusion models [4] [5] [6] [7] [8]. Among them, GAN-based inpainting models typically involve the construction of a generator that takes incomplete images as input.

Manuscript received October 18, 2023; This work is partially supported by the National Natural Science Foundation of China under Grants 11971024 and 11901443. (Corresponding author:Wenxia Yang).

Cong Zhang, Wenxia Yang, and Huan Han are with the Department of Mathematics, the School of Science, Wuhan University of Technology, Wuhan, Hubei 430070, China (e-mail:zhangcong , wenxiayang, han-huan11@whut.edu.cn).

Xin Li is with the Department of Computer Science at the University at Albany, State University of New York, Albany, New York 12222, USA (email:xli48@albany.edu).

The generator is then trained to produce reasonable content for the missing regions, while concurrently, the discriminator is trained to differentiate between the generated images and the ground truth images. This approach guides the generator toward generating progressively more realistic and persuasive content.

In contrast, inpainting methods based on diffusion models operate by generating images from noise and iteratively removing noise to approximate the data distribution of natural images. These models seek to predict the content of missing regions and generate completed images through the assimilation of knowledge from an extensive dataset [5] [9] [10]. Pioneering diffusion-based generative models, such as Denoising Diffusion Probabilistic Models (DDPM) [9], Denoising Diffusion Implicit Models (DDIM) [11], and Latent Diffusion Models (LDM) [6], have demonstrated their superiority over state-of-the-art GAN-based methods in image synthesis [4]. Consequently, they have found widespread application in various vision tasks, including image editing [12] [13], image translation [14], and text-to-image synthesis [15]. DDPM progressively introduces noise into an image until it reaches a state of pure noise. Subsequently, it samples from this stochastic noise distribution and applies a predetermined number of denoising steps iteratively, culminating in the final image sample [16]. DDPM employs the U-Net architecture to predict the noise added in the forward process, allowing the model to learn the potential distribution of target image data. By gradual denoising, it generates image samples that conform to the distribution of the target image [17] [18].

As a classical inverse problem, one challenge in image inpainting is that the outcomes often lack diversity, and visual quality can significantly degrade when missing regions are large [19]. LaMa [20] perceives that both the inpainting network and the loss function lack effective receptive fields; it introduces a novel inpainting network architecture that employs Fast Fourier Convolution (FFC) with a fully optimized graph receptive field, guided by a high-receptive field-aware loss. FT-TRD [21] aims to detect corrupted areas in face images and generate visually reasonable content within the masked regions. For the joint inpainting task with other tasks (such as mosaic removal), the staged training often performs better [22] [23].

Another major challenge is the lack of guidance and controllability over results, which restricts the practicality of image inpainting in achieving desired outcomes [24] [25]. To address this, several methods attempt to introduce additional prompts [26] [6] for fine-grained control over generated images. Previous work typically incorporated text [27] [28] [29],

scene graphs [30] [31], sketches [32] or doodles [33] [34]. The fundamental idea involves extracting semantic information from this guidance and embedding it into diffusion generation with reference information [35].

Despite the above improvements, most existing methods are often confined to a single modality. When dealing with complex semantics, such as images containing multiple objects or scenes, depending solely on image or text information may limit the feasibility of leveraging multiple modalities for more robust completions. The advantages of image-based guidance are evident, as images provide intuitive reference information such as human facial features, postures, expressions, and makeup styles, to name a few. Meanwhile, text-based guidance offers greater flexibility, given its convenience in providing concise and precise textual descriptions. When used together, the interaction between text and image guidance can supply richer prompt clues, helping better understand the missing regions' semantics and structures. In particular, cross-domain guidance enriches the diversity and enhances the consistency and controllability of the inpainting outcome.

To fully harness the advantages of both text and image guidance and their interaction, we present MMGInpainting, a multi-modality guided image inpainting model. MMGInpainting supports image-guided or text-guided inpainting independently and facilitates a hybrid guidance approach that combines both modalities for improved robustness and finegranularity control. We use the diffusion model to fill in the missing regions and integrate the inverse diffusion process with CLIP [36] to encode semantic information. Additionally, to make the results more realistic, we design a cyclic iterative denoising strategy to incorporate reference information effectively. Furthermore, to enhance the generation of realistic inpainting results while incorporating reference semantic information, we modify the original U-Net architecture [37] with an improved NAFNet [38]. This modification streamlines the model parameters and significantly improves the inpainting outcomes. The new architecture enhances model performance by introducing the Anchor Strip Attention mechanism(ASA), which leverages anchor points to model global image structural features, leading to a higher quality of image inpainting. To our knowledge, MMGInpainting is the first inpainting model capable of simultaneously integrating image and text guidance during the inpainting process. Our extensive qualitative and quantitative experiments demonstrate that, compared to singlemodality guidance or unconditional inpainting approaches, MMGInpainting excels in achieving fine texture details while providing desirable controllability.

The contributions of this paper are summarized as follows.

• We propose MMGInpainting, a comprehensive framework for guided image inpainting that accommodates the guidance of text, image, or a combination of both modalities. This unified approach provides versatility in inpainting tasks, especially for large regions. • We enhance the baseline U-Net architecture by replacing it with a modified NAFNet within the diffusion model. We further incorporate a stripe self-attention mechanism centered on anchor points. This addition facilitates global dependency modeling, improving inpainting results and computational efficiency. • We thoroughly analyze the semantic relationship between images and texts used in inpainting guidance. We conclude with a general recommendation on effective strategies for combining these modalities for multi-modality guidance.

• The experimental results conducted on CelebA-HQ [39] and Places2 [40] datasets demonstrate the remarkable guidance performance of our proposed model MMGInpainting. A closed-loop process with image captioning was designed to validate the semantic consistency of inpainting results. The remaining part of this article is organized as follows. Section II introduces the related work, including image inpainting based on GANs, the Diffusion Model, and guided image synthesis. Section III provides a detailed description of the proposed method. Section IV presents the experimental results and analysis. Finally, Section V concludes with the results and discussions.

II. RELATED WORK

Image Inpainting based on GANs. GAN-based image inpainting approaches have demonstrated significant advancements in generating meaningful content within masked holes. A typical pipeline involves employing deep neural networks to create semantically reasonable content and subsequently using this content to fill the corrupted regions. Context Encoder(CE) [3] first adopts an encoder-decoder architecture with adversarial learning to generate new content for masked areas. Early improvements focus on refining network designs and learning strategies, including Partial Convolutions [2], Gated Convolutions [41], and Contextual Attention [42]. Liu et al. [43] focuses on improving the quality of image inpainting by detecting and preserving dominant linear structures, defined as a set of lines. To generate finer details of the structure within the filled regions, EdgeConnect [44] and CTSDG [45] use edge information to guide the reconstruction of image structures. However, these methods require additional networks to extract the corresponding edge for each dataset image, and the performance is sensitive to the quality of the edge images. Prior-guided GAN [46] introduces a data-driven parametric network to predict the matching prior directly for an occluded image. To enable scene inpainting, Zhang et al. [47] proposed a general framework for facade inpainting and automatic object removal using object detection and image inpainting. Multi-GAN [48] uses a LBP(Local-binary-patterns [49])-based loss function to minimize differences between generated and natural textures. BSN [50] introduces an improved objective function for deep style transfer (DST) and an enhanced Shift-Net with multiscale feature connectivity and depthwise separable convolution (DSC) to capture local details and global semantics effectively. With the increasing applications of transformer models [51], MAT [52] tailors a transformer block for inpainting, where the attention module aggregates non-local information exclusively from partially valid tokens indicated by dynamic masks. These methods achieve visually realistic results. However, their inpainting capabilities are limited in generating specific targets or desired facial attributes, primarily due to the absence of guidance during the inpainting process.

Image Inpainting based on Diffusion Models. Diffusion models have shown remarkable performance across diverse domains [53], including image generation, image inpainting, image-to-image translation and image editing, etc. [54] [55]. For inpainting tasks, addressing issues like edge artifacts and the lack of complete contextual information provided to the model has been a research focus. Palette [14] trains the inpainting model on freely generated masks and enhances it using simple rectangular masks. Repaint [5] uses a pre-trained unconditional Denoising Diffusion Probabilistic Model (DDPM) as the generative prior. Unmasked regions are extracted from the forward records during the backward denoising process, while masked areas are filled with noise. Repaint also incorporates resampling, which involves adding noise to the denoised spliced result during each denoising step. By repeating this process multiple times, a semantically consistent output is obtained. DiffEdit [7] utilizes a textconditioned diffusion model for semantic image editing tasks, enabling image edits based on textual queries. To address fidelity and realism concerns, SDEdit [8] introduces a novel image synthesis and editing method based on the diffusion model. It synthesizes realistic images by iterative denoising using a stochastic differential equation. By fine-tuning Imagen [56] into a text-conditioned image editor on a base image, UniTune [57] synthesizes images using simple prompts while maintaining fidelity to the input image. While these methods for editing images can be adapted for inpainting tasks, they often rely on random generation for filling or are constrained to single-modality guidance. Unlike previous approaches, we propose the adoption of image-text blended guidance to produce results that are more controllable with the desired outcomes.

Image Synthesis with Guidance. Image synthesis aims to create content that is not only realistic but also diverse and visually distinctive. Several innovative approaches have emerged to enhance the quality and details of synthetic content. Ren et al. [58] propose a mask embedding mechanism to facilitate efficient initial feature projection in the generator. TediGAN [59] propose a novel framework for multimodality image generation and manipulation based on text descriptions. Dhariwal et al. [4] demonstrate that diffusion models outperform current state-of-the-art generative models in the quality of generated images. By incorporating a crossattention layer into the model architecture, the Latent Diffusion Model (LDM) [6] transforms the diffusion model into a flexible generator capable of handling various conditional inputs, such as text or bounding boxes, enabling guided synthesis through convolution. SDG [35] introduces a novel unified framework for semantic diffusion guidance, allowing for text or image guidance, or both. The pretraining-based image-toimage translation (PITI) [60] framework adjusts a pre-trained diffusion model to accommodate various types of image-toimage translation by using a pre-trained neural network to capture the natural image manifold. Singh et al. [34] propose a novel guided image synthesis framework that models the generated image to solve a constrained optimization problem, aiming to generate an image with more vivid details. It is important to note that these methods are primarily geared to enhance the overall fineness and naturalness of synthesized images, which often involve global changes in the synthesized images. This differs from the purpose of the proposed MMGInpainting, which aims to introduce guided semantics exclusively within specific regions designated for inpainting.

III. PROPOSED METHOD

The pipeline of MMGInpainting is shown in Figure 1. This section begins with a concise overview of guided imagegeneration methods based on explicit classifiers. Since this paper will use some notation, we will also introduce the guided image synthesis method based on a fine-tuned CLIP model. Then, we will present the enhanced Nonlinear Activation-Free Blocks (NAFBlocks) bolstered with attention mechanisms. Finally, we elaborate on the MMGInpainting model that can accommodate mixed guidance from textual descriptions and images.

A. Class-guided synthesis

In diffusion models, the forward process is to add noise to the initial input image x 0 via a Markov Chain over T time steps, ultimately resulting in pure noise,

where {β t=1:T } denotes a fixed or learned variance schedule that regulates the size of the noise step. The reparameterization trick is then used to sample x t directly,

where α t = 1 -β t and ᾱt = Π t s=1 α s . The reverse process of diffusion models is designed to reconstruct the original data from Gaussian noise. It is reasonable to assume that the reverse process follows a series of Gaussian distributions. However, it is not feasible to gradually fit the distribution. Consequently, a parametric distribution must be constructed for estimation [61],

To learn the reverse process, neural networks are trained to predict µ θ and Σ θ can be fixed or trained as a neural network.

If an extra guidance signal y is introduced, the conditioned sampling distribution becomes:

where Z is a normalization constant. It has been proven [4] that after incorporating the guidance, the new sampling distribution in the reverse process can be approximated by a Gaussian distribution with shifted mean [62]:

where

B. Improved NAFBlocks based Anchored Stripe Attention

The predominant architecture framework in diffusion models is U-Net based on residual blocks and attention mechanisms. While this architecture performs well on numerous downstream tasks, fine-tuning the U-Net architecture is often required. For image inpainting tasks, efficient modeling of global contextual dependencies in high-dimensional images is crucial due to the anisotropy of the regions to be inpainted [63]. To achieve this goal, we first introduce an Anchorbased Stripe self-attention mechanism (ASA) [64] to capture global-scale dependencies. Currently, to reduce parameters and enhance model performance, we replace U-Net with NAFNet and incorporate ASA into modified nonlinear activation-free blocks (NAFBlocks), as illustrated in Figure 2. ASA first Fig. 2: Modified NAFBlock. The NAFBlock has an additional attention module. Here, "simple channel attention" aggregates global information and channel-wise interaction, while "Anchored Stripe Attention" models global image structural features.

introduces Anchors as an intermediary to reduce the number of tokens. When the image is summarized in a low-dimensional space using anchors, the overall image structure is preserved. The specific operations of ASA can be expressed as [64]

where A ∈ R M ×d denotes the anchor, M e ∈ R N ×M and M d ∈ R M ×N denote the attention maps of query-anchor and anchor-key, respectively. By introducing anchors to capture the spatial relation of image features, ASA can adaptively model features separately within vertical and horizontal stripes and automatically adjust attention weights based on the content of images, thus capturing the local structure and contextual information in images. Since the masks in inpainting are typically anisotropic because they vary significantly in different directions, the anisotropic image features offered by ASA make it more efficient to exploit the anisotropic property of the inpainting domain.

To reduce the parameter count, we replace all nonlinear activation functions with "SimpleGate" in NAFNet, which involves splitting the feature map into two parts along the channel dimension and applying multiplication. LayerNorm is used instead of BatchNorm because small batches may introduce unstable statistical data that hamper image details inpainting [38]. By introducing ASA, our network can capture the image structural features from local and global regions while effectively modeling the local features using a simplified channel attention mechanism. As shown later in our ablation study, the introduction of ASA to NAFNet can improve both image fidelity and computational efficiency of MMGInpainting.

C. Multi-modality Guided Inpainting

In this paper, the ground truth image is denoted by x, and the mask is a binary matrix m where 0 indicates the known regions and 1 indicates the missing pixels to be inpainted. Thus, (1 -m) ⊙ x denotes the known pixels, m ⊙ x is the masked unknown pixels, and ⊙ is the Hadamard product. The inpainted result at the t -1 step is obtained by

Our inpainting method is built on a diffusion model. Specifically, the diffusion model generates the missing part and seamlessly combines it with the noisy original image. Then, iterative denoising is performed to ultimately inpaint the complete result. The primary objective is to control the reverse diffusion process to achieve multimodality-guided inpainting.

The optimization goal of diffusion models is essentially to fit an optimal gradient direction ∇ log P (x t ) towards the target data distribution in the data space [65] [18]. To facilitate the guided generation, the Bayesian theorem can be applied to decompose the gradient for conditional generation into two components, including a regular, unconditional generation gradient and a gradient based on an explicit classifier [54] [66]:

Among them, y can be text, image, or multi-modality guidance. For clarity, we refer to the classifier as:

Next, we utilize the alignment representation between texts and images in the Contrastive Language-Image Pre-Training (CLIP) model [36] to compute specific loss values. During each step of text-guided image generation, the distance between the current image representation and the text representation is calculated, typically using the inner product distance [57] or cosine similarity (a.k.a. CLIPScore [67]):

where E ′ I denotes the image encoder trained on noisy images with an additional time step input [35], and E L denotes the text encoder. In the case of image-guided image generation, inner distance does not account for spatial information. To incorporate spatial context information, we consider the L2 norm difference at the corresponding positions in the feature maps. By considering the spatial layout, we obtain the following accumulation for an entire image:

Algorithm 1 Multi-Modality Guided Inpainting Input : Guidance y, Editing strength t edit , scaling factor s Given: diffusion model(µ θ (x t , t), Σ θ (x t , t)), Guidance function F ϕ (x t , y, t) x T ∼ N (0, I) for t = T, ..., 1 do for u = 1, ..., U do

where E ′ I () j ∈ R Cj ×Hj ×Wj denotes the spatial feature maps of E ′ I . As for multi-modality guided synthesis, we assign specific weights s 1 and s 2 (s 1 + s 2 = 1) to the image-guided and text-guided functions, respectively.

where F ϕ1 , F ϕ2 denote the loss functions corresponding to image-guided and text-guided inpainting, respectively.

To integrate the high-level semantic information of the reference image (or text) into the reverse generation process, in each denoising step, we extract the noisy image from the forward process and concatenate it with the missing region generated with reference information guidance, followed by a reverse denoising step. To prolong the sampling process to generate overall semantically consistent content, the denoised image is subjected to noise addition repeatedly. Inpainting results can be obtained with reference semantics when this process is repeated multiple times. The Semantic Fusion Encoder is a fine-tuned CLIP model that can accept noisy images as input, which encodes supplementary reference information into the latent semantic space and iteratively integrates the reference semantics into every inpainting process.

Inspired by Wang et al. [60], we adopt a pre-trained image-to-image translation framework with an encoder that transforms the input into a task-independent latent space and a decoder that performs diffusion modeling. The main idea is to fix the pretrained decoder, update only the encoder, and then jointly fine-tune the entire network. This staged training approach maximizes the utilization of pre-trained knowledge while ensuring practical guidance for inpainting. Unlike [60], our pretext condition involves text and image guidance. The two hyperparameters in Eq. ( 14) control the weight adjustment between text and image guidance. This way, our conditional fine-tuning works in the pre-trained semantic space of multi-modality. Algorithm 1 summarizes the proposed multi-modality guided inpainting model. Our model can also operate without guidance(guidance-free case), with the signal y being empty in such cases.

IV. EXPERIMENTAL RESULTS

A. Dataset and Implementation Details

We conducted guided image inpainting experiments on CelebA-HQ [39] and Places2 [40] datasets, using the same experimental setup as NAFNet [38] with the image size 256×256, and the batch size 8. We use the AdamW [68] optimizer, with β 1 = 0.9 and β 2 = 0.9. The initial learning rate was set to 3 × 10 -3 , which was decayed to 1e-7 using a cosine scheduler. The weight parameters s 1 and s 2 in Eq.( 14) were set at 0.4 and 0.6 by the experiment discussed in section IV, part D. The Semantic Fusion Encoder is CLIP ResNet50x16 fine-tuned on the noisy images of each dataset, with the initial learning rate set to 10 -4 , weight decay to 10 -3 , and the batch size to 64. The denoising steps of the diffusion model were set to T = 250, with the resampling setting consistent with Repaint [5]. It was trained on a single 3090 GPU for approximately five days. To ensure semantic consistency throughout the image, the injection procedure of guiding semantics was confined within a predefined range of steps, denoted as [T, t edit ], without being introduced into the resampling loop. The hyperparameter t edit was set to 75 by experiment.

B. Experimental Result Analysis 1) Experimental settings and Quantitative Comparisons:

To evaluate the performance of our model, we first conducted quantitative experiments on CelebA-HQ and Places2. We compared the proposed MMGInpainting with five state-ofthe-art image-inpainting models from the literature, including LAMA [20], LDM [6], RePaint [5], MAT [52], and BLD [69]. Among them, LDM can accept either image or text guidance, BLD only supports text-driven, while LAMA, Repaint, and MAT are guidance-free. For fairness, we use the official pretrained models provided by the corresponding authors.

The experiment was conducted on 100 images selected from the test sets of CelebA-HQ and Places2, respectively. We masked the most critical components for each image using a 64×64 center mask. Because the text or image used for inpainting guidance can be diverse, the experimental setup was as follows: for CelebA-HQ, if the model is image-guided, the ground-truth image is used as guidance; if it is text-guided, the text is generated using the labels corresponding to the groundtruth image. For example, if the attribute labels of the original image indicate "Smiling = 1" and "Male = -1", the text used for guidance would be "a smiling woman". In Places2, we use additional object cues to guide different scenes, such as outdoors, buildings, and vegetation, as labeled in the dataset. In this way, the semantic information of the text and image is consistent with the original image. This setup was designed to ensure uniformity in the guidance semantics for different models when conducting experiments on many samples, thereby making the comparison of experimental results meaningful.

Since our proposed MMGInpainting supports text guidance, image guidance, and simultaneous guidance from both, we conducted three sets of experiments, obtaining results with no guidance, guided solely by text, solely by images, and by both simultaneously.

Six evaluation metrics, namely Fréchet Inception Distance (FID) [70], Learned Perceptual Image Patch Similarity (LPIPS) [71], Peak Signal to Noise Ratio (PSNR), Paired/Unpaired Inception Discriminative Score (P/U-IDS) [72], and CLIP-based scoring function(PickScore) [73] are calculated. Among them, FID serves as a quantitative metric to assess the similarity between the statistical distributions of real images and generated images, and P/U-IDS reflects the fidelity of the generated images by calculating the linear separability between the generated images and the real images in the feature space of the perceptron. The P/U-IDS is given by

where x denotes the real image and x ′ denotes the corresponding generated fake image. I(•) is the pre-trained Inception v3 model that maps the input images to the output features of 2048 dimensions. And f (•) denotes the (linear) decision function of the SVM. Compared to CLIPScore [67], PickScore is a text-to-image metric that measures the fidelity of the generated content based on learned human preferences. The matching score between text x and image y is calculated as:

where T denotes the learned scalar temperature parameter of CLIP. The objective function L pref aims to optimize the parameters of the scoring function by minimizing the KLdivergence between the preference distribution p and the softmax-normalized scores of y 1 and y 2 :

Table I presents a quantitative comparison of MMGInpainting with other methods on Celeb-HQ and Place2 test sets containing 100 images, respectively. For CelebA-HQ, our primary focus is on the fidelity of the inpainted facial results and the presence of additional semantic information. We use the FID, LPIPS, PSNR and PickScore metrics to assess this. On the other hand, for Place2, given the complexity of scene data, our emphasis lies on the overall cohesiveness of the images. Therefore, we use the FID, LPIPS, PSNR, and P/U-IDS metrics for evaluation. As shown in Table I, our model performs favorably regarding image fidelity compared to the unconditional inpainting model. The fidelity of our model's inpainting results guided by the original image exhibits the lowest FID, even slightly surpassing the state-of-the-art LaMa. LDM achieves the optimal LPIPS, but its PSNR is somewhat lower than our model. Furthermore, our model, guided by a combination of image and text, generates inpainting results with a higher PickScore than the guided inpainting model. This indicates that our inpainting results share more semantic similarities with the reference information. Regarding model parameters, our parameter count is lower than other guided methods and slightly higher than unguided methods. This difference is attributed to incorporating an additional feature extraction module and an attention mechanism to enhance the final guided inpainting effect.

2) Qualitative results and Comparisons: We further experimented with various texts and images to assess different models' guiding effects and inpainting quality. In this experiment, the guiding image is entirely different from the ground truth, and the semantics of the text used for guidance are set to be consistent with the semantics of the guiding image. Figure 3 illustrates some examples of CelebA-HQ. We purposely masked some critical facial features such as eyes, overall facial features, hair, or chins to verify whether the inpainted images contained additional guidance information.

Visual inspection of Figure 3(a)-(h) reveals that all unconditional inpainting methods like RePaint, LaMa, MAT, and our guidance-free case successfully obtain good texture details with natural facial expressions, despite there may be some color discrepancies around the borders in MAT. Nevertheless, there are comparatively noticeable flaws and blurred details in inpainting the hair details in Figure3(f). In Figure 3(c)-(e), MAT may generate shadows when there are occlusions of prominent facial features. LaMa generates content with overall semantic consistency but lacks guidance during the generation process. LDM and BLD exhibit inadequate coherence in overall semantics for guided inpainting models. In contrast, our model efficiently models global feature information in the images to be inpainted, considering both the unmasked regions and the guidance information. Specifically, our model excels in inpainting texture details of hair and beard in Figure 3(f) and (b), highlighting its advantages in preserving texture details and conducting guided region inpainting.

We observed that LDM performs well in image-guided generation, while our model achieves the best results with mixed guidance. Therefore, for the Place2 dataset, we use image guidance for LDM and composite guidance for our model. As shown in Figure 4, for object-guided generation, our model effectively inpaints the details of the reference object and integrates it into the target image without producing artifacts like BLD. LDM successfully incorporates the reference object but loses some details. Regarding unrestricted inpainting, the focus is on maintaining overall semantic consistency, where LaMa performs the best.

To further evaluate the effectiveness of text guidance, we conducted a closed-loop experiment. Specifically, we used an off-the-shelf image captioning API [74] to generate the descriptions for the inpainted results. In Figure 3 and Figure 4, we have highlighted the identical words found in both the guiding texts and the generated captions. The semantic information from the text guidance is well embedded into the inpainted regions. Additionally, we observed that the surplus text obtained after image captioning captures semantic details related to the guiding image. This closed-loop experiment shows the effectiveness of semantic guidance through text and image elements.

C. Analysis of the relationship of the semantics between text and image guidance

In the preceding subsections, we demonstrate that our model can achieve richer details and more desirable visual outcomes Fig. 6: Inpainting Results with contradictory semantics of guiding image and text . When there is a semantic conflict, it may cause the inpainting results to uncertain directions. when the semantics of the text and image used for inpainting guidance are consistent. However, what happens when the two semantics are contradictory or complementary? In this section, we conduct experiments to explore these scenarios. Semantic complementary guidance involves using an image while providing textual descriptions of significant attributes absent in the guiding image. Figure 5 illustrates some results for each image; it can be observed that although the inpainted images are diverse, the texts of image captioning have subtle differences. Furthermore, we make the following observations:

(I) For face inpainting, when guided by semantic complementary information, the overall appearance is determined by the guiding image. In contrast, the complementary text introduces diversity and finer details into the inpainting results, such as attributes like "short hair" or "glasses." (II) For Scene Inpainting, image guidance ensures that the inpainting result contains objects guided by the input image. In contrast, text guidance allows for specifying object layouts and the number of objects, adding extra content, and introducing variations.

We argue that complementary semantics enhance diversity and boost finer details in inpainting results. When the semantics are identical, more precise guidance is provided, emphasizing specific semantics through repetition. However, when semantics are opposed or in apparent contradiction with the original image, conflicting information can lead to randomly generated results, some of which are visually unreasonable. This may produce misleading and even ethically concerning results, as shown in Figure 6. Therefore, we conclude with the following recommendations. For face inpainting, prioritize image guidance with text as a supplementary source. For Place2 inpainting, consider using a specific object as the guiding image, ideally with a structure similar to the original image. Text guidance can be employed for object layout and specific morphological changes.

D. Analysis of the weights s 1 and s 2

In Eq.( 14), the ratio of weights s 1 and s 2 signifies the proportion of semantics embedded by the guiding image and text during the inpainting, respectively. When the guiding image and text semantics are consistent enough, their ratio has little impact on the results. However, when the semantics of the image and text complement each other, different settings of s 1 and s 2 significantly influence the results. In this section, we experimented to explore the impact of varying weight settings. As shown in Figure 7, when the text weight is smaller, although the results still contain the semantics guided by the text, the probability is much lower. In contrast, facial appearances or animal postures in the results are more similar to those of the guiding images, as illustrated in Figure 7(b), (d), and (f). On the contrary, when the text weight is increased, the probability of combining complementary semantics increases significantly, obtaining results with more semantics indicated by the text, such as "eyeglasses" in Figure 7(a), "smiling" Fig. 8: Impact of different t edit on the inpainting results. When t edit is too small, the "excessive guidance" problem will occur with excessive guiding semantics embedded, leading to inconsistent skin color. Conversely, when t edit is too large, it can cause "insufficient guidance". Setting t edit to be 75 achieves a relatively proper balance. in Figure7(c), and "Two horses" in Figure7(e). Based on the experiment, we set s 1 = 0.4 and s 2 = 0.6 in this paper.

E. Discussion on the setting of t edit

The editing strength t edit is a user-controllable hyperparameter that regulates guidance strength. If the guidance steps are excessively long, it may disrupt the global semantics of the image, leading to an inconsistent result. Conversely, additional prompt injection may be insufficient if the steps are too short. In Figure 8, we investigate the impact of varying t edit on the inpainting results. We used the weighted combination of FID and PickScore, PS-FID (PickScore-based FID), to strike a balance between the global semantic coherence of the inpainted image and its similarity to the guiding semantics. PS-FID is calculated by : PS-FID = αFID -βPickScore (19) In this case, a smaller PS-FID represents a better inpainting result. We select images from Figure 3 in the CelebA-HQ dataset with α = 5, β = 1 and vary t edit with an interval length of 25. Then, we obtain the polyline graph of the mean PS-FID as a function of t edit , as shown in Figure 9. Notably, when t edit is too large, it leads to "insufficient guidance", which means that the guiding semantics are subtly embedded into the inpainting procedure, and when t edit is too small, the "excessive guidance" effect will cause overall semantic incoherence such as inconsistent skin color. Both scenarios increase the PS-FID. Therefore, we recommend that a t edit = 75 strikes an acceptable balance.

F. Ablation Study

To assess the effectiveness of our proposed modified NAFNet and ASA module, we conducted ablation studies on the CelebA-HQ dataset. In comparison, we evaluated our approach against the diffusion model based on U-Net. We used an extra evaluation metric termed Runtime [63] to quantify the time required for computing the inpainting of a single Fig. 9: Changing trend of PS-FID with respect to t edit . The relatively optimal value of t edit is around 75.

image. Runtime measures the computational efficiency gains achieved by our method relative to the baseline. The experimental results are presented in Table II and Figure 10. The results show that replacing the U-Net backbone with NAFNet significantly reduces the inference time and slightly improves image quality. Our MMGInpainting, combined with ASA, showcases exceptional performance in terms of image fidelity and guiding content measurement, surpassing the baseline in computational efficiency.

V. CONCLUSION AND DISCUSSIONS

In this paper, we present a multi-modality guided image inpainting method aimed at enhancing the controllability of the inpainting process. Specifically, we replace the U-Net backbone with a modified NAFNet to achieve superior inpainting results and boost computational efficiency. This modified architecture incorporates an Anchored Stripe Attention mechanism, leveraging anchor points for comprehensive global contextual modeling. Then, we design a novel inpainting method capable of intelligently filling in image regions through a hybrid-guided approach. Experimental results demonstrate that our method significantly improves the controllability of inpainting, resulting in higher-quality inpainting results and faster computational efficiency. Our model seamlessly integrates text and image guidance within a unified framework, offering flexibility for diverse applications. While the long sampling steps are designed to generate detailed and high-quality inpainting results, their time-consuming nature hinders practical applications, especially for large-scale image inpainting tasks or real-time scenarios. In future work, we plan to explore methods to divide large-scale image inpainting tasks into multiple sub-tasks, enabling parallel processing and acceleration of the overall speed. Cross-modal alignment [75] is an underresearched topic in MMGInpainting, especially when text and image guidances are inconsistent. How to control the multi-modality guided diffusion in a collaborative manner (e.g., using bilateral connections [76]) deserves a more systematic study.