<?xml-model href='http://www.tei-c.org/release/xml/tei/custom/schema/relaxng/tei_all.rng' schematypens='http://relaxng.org/ns/structure/1.0'?><TEI xmlns="http://www.tei-c.org/ns/1.0">
	<teiHeader>
		<fileDesc>
			<titleStmt><title level='a'>Integrating Vision-Language Semantic Graphs in Multi-View Clustering</title></titleStmt>
			<publicationStmt>
				<publisher>IJCAI</publisher>
				<date>08/03/2024</date>
			</publicationStmt>
			<sourceDesc>
				<bibl> 
					<idno type="par_id">10545727</idno>
					<idno type="doi"></idno>
					
					<author>J Ke</author><author>Z Wen</author><author>Y Yang</author><author>C Cui</author><author>Y Ren</author><author>X Pu</author><author>L He</author>
				</bibl>
			</sourceDesc>
		</fileDesc>
		<profileDesc>
			<abstract><ab><![CDATA[In recent years, a variety of graph learningbased multi-view clustering (MVC) methods have emerged. However, these methods continue to face challenges in extracting latent features from real-world data, particularly in scenarios involving high-resolution color images and high-dimensional features. This task is notably difficult in cases where images are visually similar yet semantically diverse. To address this issue, we present a novel large-scale pre-trained model for multiview clustering, named Integrate Vision-Language Semantic Graphs in Multi-View Clustering (IVS-GMV), which harnesses the capabilities of visuallanguage pre-training models to enhance clustering performance and confronts issues in the unsupervised tuning of pre-trained models for multiview data. We introduce an effective unsupervised approach for creating semantic graphs from image multi-view datasets using pre-trained encoders. Our method addresses the inherent spatial noise and imbalance in these encoders by employing graph filters and a joint process that integrates both image node and edge features. Additionally, we demonstrate the application of our approach to multi-view image clustering on extensive datasets, notably the high-resolution MVImgNet, achieving an impressive 82% accuracy. Furthermore, our method extends the zero-shot capabilities of large-scale pretrained models, resulting in good performance in clustering tasks on untrained multi-view datasets.]]></ab></abstract>
		</profileDesc>
	</teiHeader>
	<text><body xmlns="http://www.tei-c.org/ns/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="1">Introduction</head><p>In recent years, multi-view clustering (MVC) has emerged as a pivotal component in the realms of cross-modal representation learning and data-driven decision-making. This technique has been shown to be highly effective in various domains, such as image <ref type="bibr">[Guan et al., 2024b;</ref><ref type="bibr">Guan et al., 2024a]</ref> and video analysis <ref type="bibr">[Xu and Wei, 2021]</ref>. MVC methods capitalize on exploiting the consistency and complementary information of multiple views to group samples into distinct * Corresponding author.</p><p>clusters <ref type="bibr">[Ren et al., 2022;</ref><ref type="bibr">Yan et al., 2024;</ref><ref type="bibr">Wang et al., 2021;</ref><ref type="bibr">Wang et al., 2023;</ref><ref type="bibr">Zhang et al., 2018a;</ref><ref type="bibr">Ling et al., 2023]</ref>. Currently, self-supervised learning has achieved significant improvements in multi-view clustering <ref type="bibr">[Zhou et al., 2023;</ref><ref type="bibr">Zhou et al., 2024b]</ref>. At the core of this approach lies the extraction and utilization of the intrinsic representational attributes of the data to further enhance clustering performance.</p><p>Nevertheless, previous MVC methods still encounter challenges when it comes to extracting latent features from realworld data, especially in scenarios involving high-resolution color images and high-dimensional features.</p><p>One potential resolution to these problems involves the incorporation of pre-trained models within the encoding process. Many approaches have used pre-trained models to explore the limits of clustering in single-view data <ref type="bibr">[Adaloglou et al., 2023;</ref><ref type="bibr">Shen et al., 2023;</ref><ref type="bibr">Cai et al., 2023]</ref>. However, there still exist challenges in the unsupervised tuning of pretrained models for multi-view data. First, for k-means, it usually leads to unbalanced clustering <ref type="bibr">[Yang et al., 2017;</ref><ref type="bibr">Cui et al., 2023]</ref> and is mainly applicable to data samples that are uniformly dispersed around the center <ref type="bibr">[Van Gansbeke et al., 2020]</ref>. On the other hand, when it comes to multi-view image data, that are similar in feature space do not always have the same semantic category <ref type="bibr">[Van Gansbeke et al., 2020]</ref> and therefore must be treated as noisy pairs, which will result in noisy spatial relations in images embedded.</p><p>In this paper, combining the classic image-text dual encoders model CLIP (Contrastive Language-Image Pretraining <ref type="bibr">[Radford et al., 2021]</ref>), we propose a novel multiview clustering method using pre-trained models. In the face of the two previous limitations, we introduce the filter-based graph to utilize filters compatible with homogeneous and heterogeneous graphs to combat embedding noise and spatial imbalance, resulting in representations that balance spatially homophilous and heterophilous relationships. In addition, we use a joint graph to combine node features and edge relationships, where embeddings can be used to efficiently combine global feature relationships among multi-view data. To the best of our knowledge, our method is the first to apply largescale pre-trained models multi-view data clustering.</p><p>Our main contributions can be summarized as follows:</p><p>&#8226; We introduce large-scale vision-language pre-trained models to multi-view clustering by proposing a method to counteract the embedded spatial noise and imbalance of the pre-trained encoder with graph filters and a graph joint process. &#8226; Our method efficiently applies multi-view image clustering to large-scale multi-view image datasets, including the high-resolution multi-channel multi-view image dataset MVImgNet, achieving an accuracy of 82%. &#8226; Our method achieves good performance for the clustering task on top of its untrained dataset, extending the zero-shot performance of CLIP.</p><p>2 Related Work</p><p>2.1 Multi-View Clustering The collection of data from multiple perspectives and sources has become commonplace with the development of multimedia technology. Multi-view clustering (MVC) methods leverage complementary information from different views of the same instance to address the limitations of traditional clustering methods [Wu et al., 2024; Ren et al., 2024]. Several algorithms and techniques have been proposed for MVC, including classic methods like spectral clustering, as well as more recent developments in deep learning. Recently, deep learning has emerged as a powerful tool in the field of MVC, with numerous proposed architectures of deep neural networks tailored for this endeavor, including the introduction of new constraints, feature learning techniques, and graph filtering frameworks. LT-MSC <ref type="bibr">[Zhang et al., 2015]</ref> introduces a low-rank tensor constraint to explore the complementary information from multiple views. <ref type="bibr">[Xu et al., 2021b]</ref> proposes a framework for contrastive multi-view clustering (MFLVC) that utilizes multi-level feature learning to improve clustering effectiveness. <ref type="bibr">[Wen et al., 2024]</ref> suggests an adaptive hybrid graph filter based on homophily degree, which dynamically captures both low and high-frequency information to improve graph clustering. These methods have advanced the field of multi-view clustering significantly. However, there has been no exploration of incorporating the recently popular visionlanguage pre-training models into MVC, which has the potential to elevate multi-view clustering to unprecedented heights.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2.2">Vision-Language Pre-training Models</head><p>Vision-Language Pre-training (VLP) models aligning multimodal data in a common feature space have been applied in various areas such as large vision language model <ref type="bibr">[Zhou et al., 2024a]</ref> and adversarial attack <ref type="bibr">[Dong et al., 2023]</ref>.</p><p>They can be categorized into two main groups: those that use language-based training strategies, including Mask Language Modeling, such as mask language/region modeling Vi-sualBert <ref type="bibr">[Li et al., 2019a]</ref>, or autoregressive language modeling, such as image captioning and text-based image generation DALL-E <ref type="bibr">[Ramesh et al., 2021]</ref>. The other category is to utilize cross-modal contrastive learning to align the visual and textual information into a unified semantic space, e.g. <ref type="bibr">CLIP [Radford et al., 2021]</ref>. VLP aims to model the interaction between images and texts. Unicoder-VL <ref type="bibr">[Li et al., 2020]</ref> combines visual and textual embeddings, feeding them into a single encoder. In contrast, CLIP <ref type="bibr">[Radford et al., 2021]</ref> obtain visual and textual embeddings with separate encoders.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3">Method</head><p>As shown in Figures <ref type="figure">1</ref> and <ref type="figure">2</ref>, the proposed method consists of two steps: 1) Semantic Graph Construction: Based on nodes characterized by images, we propose a method for constructing graph data, incorporating prior knowledge from the design of the CLIP model, we further construct edge relationship. This includes an unsupervised step of meaningful noun selection we refer to as word filtering, aiming to identify better nouns that accurately convey the overall details depicted in the image; 2) Adaptive Hybrid Graph Filtering: Herein, we design a filtering method that further processes based on the previously presented images and node-edge relationships. This method accounts for the heterogeneity arising from inconsistencies in their representations, resulting in an adaptive filter of Nodes and Edges that learns both consistency and complementarity, containing a graph joint process that combines features of nodes and edges, maximizes the use of consistency and complementarity across different views, while also leveraging that of node features and edge features as much as possible.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.1">Semantic Graph Construction</head><p>Since the node feature of our graph to process is the embeddings of raw feature, it would be less meaningful when we directly use the similarity matrix between CLIP encoding results or original images as the adjacency matrix: the former does not take full advantage of the core of CLIP dual encoder design and training for computing image-text similarity, and the latter introduces noise in the edge relations because the original images are not encoded.</p><p>Given the application scenario of CLIP's design, a natural approach is to unsupervisedly select some words and then refer to the way CLIP predicts between them. By adding prefix words, we would calculate the cosine similarity between images and words to form the image-text bipartite graph.</p><p>In terms of word selection, to maximize the effectiveness of the final matrix representation, the meanings of the words should represent and distinguish the embeddings of the image dataset as much as possible without being overly generalized. In other words, this requires the identification of nouns that accurately convey the overall details depicted in the image. So first and foremost, we design a noun filter for selecting nouns unsupervisedly in the WordNet dataset <ref type="bibr">[Miller, 1995]</ref> as raw noun set N r to select a suitable noun set N .</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Noun Filter</head><p>Firstly, to exclude some nouns overly generalized, i.e., Thing, Object, Item, Matter, Entity, we initially exclude nouns from the WordNet that are close to the centers of almost all words. More precisely, we use cosine similarity between embeddings to measure the similarity of objects, as used in CLIP training. This selection can be formulated as selecting set N d follows:</p><p>Let t be an element of the set N d . This membership is defined such that t &#8712; N d if and only if the Cosine Distance CD between the text encoder output f (t, &#952; t ) and the centroid C N is greater than or equal to a threshold &#949;. Formally, we first select nouns t satisfied:</p><p>we use set N d to mark the selected nouns t. Where, f t (&#8226;) refers to the text encoder parameterized by &#952; t , Cosine Distance between any two vectors a and b is defined as CD(a, b) = 1 -a&#8226;b &#8741;a&#8741;&#8741;b&#8741; , &#949; refers to the threshold hyperparameter, in the following process and experimental setup, we maintain the threshold hyperparameter &#949; at 0.05. The centroid C N is calculated as the mean of normalized text embeddings U m for each noun m in the raw WordNet noun set N r , expressed as:</p><p>where k is the number of nouns in N r .</p><p>To further select an appropriate subset of nouns, considering other priors that can be integrated into the noun filter process that is merely the number of categories and the image embeddings themselves, it is noteworthy that experiments demonstrate over 70% accuracy in direct clustering of the embeddings of images from common datasets. Under these circumstances, for the given multi-view image datasets I subjected to clustering, we cluster the directly concatenated encoded results. Based on the clustering result, we select the final nouns N that satisfied:</p><p>) where normalized centroids C Ii are computed based on the embeddings of each dataset and their respective clustering labels as:</p><p>where for any specific sample m, CL m denotes the label assigned to the sample by the k-means clustering algorithm.</p><p>Here, w i represents the number of samples in I with CL m = i, m j refers to the j-th view of the sample m, and v is the number of views in I, f i (&#8226;) refers to image encoder parameterized by &#952; i . Additionally, &#949; th is defined as a dynamic threshold that is set such that the number of nouns meeting the criterion outlined in Eq. ( <ref type="formula">3</ref>) is equal to the hyperparameter &#946; (i.e. the total number of nouns selected equal to &#946;N Class , where N Class refers to the classes number of image dataset) for each cluster label i.</p><p>In the following content, we call the selected nouns set as filtered nouns set N .</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Graph Construction with Filtered Nouns</head><p>Drawing upon the methodology of CLIP, particularly its approach to calculating result probabilities, we proceed to construct a comprehensive graph dataset utilizing the Filtered nouns obtained after the initial step. This is achieved by first creating a cosine similarity matrix derived from the embeddings of both the images and the Filtered nouns. Notably, diverging from the standard application of CLIP, in our approach, the constructed graph-text bipartite graph utilizes the cosine similarity matrix directly as the adjacency matrix A, as opposed to the softmax operation typically employed in CLIP. This process is described by the following equation:</p><p>where B v represents the adjacency matrix of image-nouns bipartite graph based on cosine similarity in the v-th view. Z v I j refers to the j-th row of Z I in v-th view, Z N k refers to the k-th row of Z N , m&#1013;I v , refer to the j-th image of dataset I 's v-th view, t&#1013;I v , refer to the k-th nouns of filtered nouns set N , f i (&#8226;) refers to image encoder parameterized by &#952; i , f t (&#8226;) refers to text encoder parameterized by &#952; t . N refers to row normalization.</p><p>Conclusively, within the computed image-text bipartite graph, since the image is our only clustering target, we have chosen an approach that involves multiplying the image-text similarity matrix with its transpose. This operation produces the final adjacency matrix A v of v-th view, that is,</p><p>A v would then be utilized for graph clustering. This strategy allows the clustering to also take into account the intrinsic connection between the image and its corresponding text.</p><p>For the above process, in the case of using CLIP, we use the pre-trained image encoder as f i (&#8226;) and text encoder with the prompt template A photo of a nouns as f t (&#8226;). We construct the adjacency matrix of the semantic graph to characterize edge relationships based on the multi-view dataset and unsupervised selection of nouns from WordNet. </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.2">Spectral Self-Supervised Learning with</head><p>Adaptive Hybrid Graph Filter</p><p>Graph Joint Process Considering that the semantic graph constructed in Section 3.1 may have some noisy edges, i.e., there may be inconsistencies between image embedding and word. It is feasible to use it directly for filtering, but its dominance in the filtering operation may lead to excessive loss of node feature information. Thus we try to utilize image embedding to further correct the semantic graph. Specifically, we implement a graph joint process, which injects the nodes' original image features into the semantic graph. We first explore A from Eq. ( <ref type="formula">6</ref>) as follows:</p><p>where f a (&#8226;) represents deep auto-encoder, &#952; a are learning parameters of the autoencoder. Z A are the encoded outputs of A. Since the graph data A v of each view is a semantic graph generated by the same method, in order to motivate further extraction of inter-view consistency of the model, we adopt the practice of using a shared deep auto-encoder parameter &#952; a for the semantic graphs of all views. We actually consider here the adjacency matrix A of the semantic graph being constructed as a kind of feature information, i.e., an edge feature, and thus we adopt a similar encoding operation as for image features.</p><p>Next, the image feature information is injected into the semantic graph. Consensus information between the image embedding and the edge features is then utilized to correct the semantic graph. To be more specific, we perform the following:</p><p>where f i (&#8226;) represents pre-trained image encoder with parameters &#952; i , the dimension of S is the same as the original adjacency matrix A of semantic graph, we regard graph joint matrix S as the modified graph. Z I is obtained through f i&#952; , f i&#952; is the pre-trained image encoder. As S joins the image feature information of the nodes and the edge feature information of the semantic graph, it is more reliable than the adjacency matrix of the original semantic graph. Additionally, we propose discretizing and sparsifying S. This approach can save memory and accelerate computation, while also removing weakly correlated edge noise. We perform the following operation to remove the weakly correlated edge noise:</p><p>where N is the number of nodes and S dis is the discretized S.</p><p>All S below denote the discretized S, i.e., S dis if not otherwise specified.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Adaptive Hybrid Graph Filter</head><p>After completing the graph joint process, a natural idea is to leverage popular graph neural networks <ref type="bibr">[Zhou et al., 2020]</ref> to explore structural information and feature information to enhance clustering performance.</p><p>In terms of filter selection for graph filtering of the resulting semantic graph, the most common choice is to use the widely used Graph Convolutional Neural Network (GCN) to achieve this goal. Previous studies have pointed out that GCN is essentially a low-pass filter [Nt and Maehara, 2019]. From a spectral perspective, GCN only captures low-frequency information on the graph, completely losing high-frequency information. However, Bo et al. point out that both low-frequency and high-frequency information are equally crucial for learning representations of nodes on the graph [Bo et al., 2021]. Completely discarding high-frequency information would lead to significant information loss, as in the constructed semantic graph, information is present both in the homogeneous and heterogeneous components.</p><p>A better idea is to use weighted high-pass and low-pass filters for obtaining high and low frequencies on the graph respectively to retain as much information as possible, and we design an adaptive hybrid graph filter as follows:</p><p>where H hybrid represents the output of the adaptive hybrid graph filter. hr is a learnable parameter that measures the homophily degree and is used to control the adaptive process of the hybrid graph filter, which will be calculated in Eq. ( <ref type="formula">11</ref>). The diagonal matrix D ii = j a ij represents the degree matrix. L is the normalized Laplace matrix. k is the order of the filter.</p><p>In Eq. ( <ref type="formula">10</ref>), ( A) k Z I represents the low-pass filter and ( L) k Z I represents the high-pass filter.</p><p>Instead of manually setting the weights for the low-pass and high-pass filters, to further enhance the universality of the filtering mechanism, we have implemented an adaptive mechanism for the hybrid graph filter based on the homophily ratio. Homophily edges are edges on a graph that connect two similar nodes, and homophily ratio is a measure of the proportion of homophily edges on a graph. If a graph has a high homophily ratio, there will be more homophily edges, and from a graph signal processing perspective, low-frequency signals dominate the graph, and conversely, high-frequency signals dominate the graph. In other words, the homophily ratio of a graph can reflect the frequency distribution and the proportion of signals on the graph. Therefore, we considered assigning weights to hybrid graph filters using the homophily ratio and designing adaptive mechanisms. As the real label information is unavailable in the unsupervised setting, we estimate the homophily ratio (hr) using pseudo-labels:</p><p>where, SUM(&#8226;) denotes the summation operation, &#8857; denotes the Hadamard product, and P &#8712; {0, 1} n&#215;c is the one-hot encoding of the pseudo label. Specifically, for the obtained semantic graph, if it has a high homophily ratio, then the low-pass filter in the adaptive hybrid filter will play a major role, and conversely, the high-pass filter will play a major role, which just matches the frequency distribution on the graph, thus extracting of information effectively.</p><p>The final learned consensus embedding of n views is set as an adaptive weighted sum of the individual view embeddings after the graph filter: H = n v=1 &#969; v h H v &#119970; &#120783; (&#119920; &#120783; , &#119808; &#120783; ) Input &#119943; &#119938; (&#119808; &#120783; , &#120637; &#119938; ) &#120027; &#119955;&#119942;&#119940; &#120783; (&#119808; &#119953;&#119955;&#119942;&#119941; &#120783; , &#119808; &#120783; ) Graph Joint Progress &#119808; &#120783; &#119943; &#119946;&#120637; &#120783; &#8230; &#119943; &#119938;&#120637; &#120783; High-pass Filter &#119945;&#119955; &#119930; &#119833; &#119920; Low-pass Filter Adaptive Hybrid Graph Filter &#119920; &#120783; &#120027; &#119955;&#119942;&#119940; &#119951; (&#119808; &#119953;&#119955;&#119942;&#119941; &#119951; , &#119808; &#119951; ) Graph Joint Progress &#119808; &#119951; Feature-combined Graph S &#119943; &#119946;&#120637; &#119951; &#119943; &#119938;&#120637; &#119951; &#119920; &#119951; &#8230; &#61524; &#61524; &#119970; &#119951; (&#119920; &#119951; , &#119808; &#119951; ) Input Feature-combined Graph S Homophily Degree Measure Adaptive Mechanism Related hr &#119945;&#119955; &#120783; &#119943; &#119946; (&#119920; &#120783; , &#120637; &#119938; ) &#119815; &#120783; &#3365; &#119815; &#120654; &#119945; &#120783; &#120654; &#119945; &#119951; &#8230; &#120027; &#119948;&#119949; &#119815; &#119847; &#8853; &#119833; &#119920; &#120783; &#8855; &#119833; &#120783; &#215; &#119833; &#120783; &#119931; &#119833; &#119912; &#120783; &#119833; &#120783; &#119826; &#120783; &#119833; &#119920; &#119951; &#8855; &#119833; &#119951; &#215; &#119833; &#119951; &#119931; &#119833; &#119912; &#119951; &#119833; &#119951; &#119826; &#119951; &#8230; &#119912;&#119958;&#119957;&#119952;&#119916;&#119951;&#119940;&#119952;&#119941;&#119942;&#119955; &#119943; &#119938;&#120637; &#119951; &#119927;&#119955;&#119942;&#119957;&#119955;&#119938;&#119946;&#119951;&#119942;&#119941; &#119916;&#119951;&#119940;&#119952;&#119941;&#119942;&#119955; &#119943; &#119946;&#120637; &#119951; &#61524; &#119924;&#119938;&#119957;&#119955;&#119946;&#119961; &#119924;&#119958;&#119949;&#119957;&#119946;&#119953;&#119949;&#119946;&#119940;&#119938;&#119957;&#119946;&#119952;&#119951; &#8855; +(&#120783; -&#119945;&#119955;)(&#119816; -&#119826;)&#119833; &#119920; &#119943; &#119938; (&#119808; &#119951; , &#120637; &#119938; ) &#119943; &#119946; (&#119920; &#119951; , &#120637; &#119938; ) Homophily Degree Measure Adaptive Mechanism Related hr &#119945;&#119955; &#119951; Consensus Embedding &#3365; &#119815; Output </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.3">Model Optimization</head><p>In IVSGMV, learning from previous multi-view clustering studies <ref type="bibr">[Zhao et al., 2021;</ref><ref type="bibr">Ren et al., 2022]</ref>, in order to guide deep auto-encoder f a (&#8226;) and learn consistency and complementarity between views, we apply reconstruction loss on A as follows:</p><p>where L CE (&#8226;; &#8226;) denotes cross-entropy loss. We obtain the L KL from the soft clustering distribution Q and the target distribution P as follows:</p><p>where L KL (&#8226;; &#8226;) denotes Kullback-Leibler (KL) divergence loss <ref type="bibr">[Kullback and Leibler, 1951]</ref>, q v ij &#8712; Q v describes the probability that node i in the v-th view belongs to the center of cluster j. P v represents the target distribution of nodes embedding H v in the v-th view. Q and P denote the soft and target distributions, respectively, of the consensus embedding H. L KL encourages the soft distribution of each view to match the target distribution of the final consensus embedding H. Additionally, it enhances the consistency between the soft distribution and the target distribution of the consensus embedding.</p><p>Eventually, the loss of IVSGMV is defined as:</p><p>4 Experiments</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.1">Datasets</head><p>As shown in Table <ref type="table">1</ref>, we use the following four real-world multi-view datasets in our study. <ref type="bibr">MNIST [LeCun et al., 1998</ref>] is a widely used dataset of handwritten digits from 0 to 9. of various fashion items, including T-shirts, dresses, coats, etc. The COIL dataset <ref type="bibr">[Nene et al., 1996]</ref> contains images of various objects, such as cups, ducks, and blocks, shown in different poses. We use multi-view datasets derived from origin datasets: Multi-COIL-10, Multi-COIL-20, Multi-MNIST, and Multi-Fashion. Each dataset includes multiple views of each example, all randomly sampled from the same category.</p><p>In Multi-COIL-10 (K = 10) and Multi-COIL-20 (K = 20), different views of an object correspond to various poses, but retain the same label. In Multi-MNIST, different views of a digit represent the same digit written by different individuals. In Multi-Fashion, different views of a product category signify different fashionable designs for that category.</p><p>MVImgNet <ref type="bibr">[Yu et al., 2023]</ref> is a multi-view image dataset presented with a large scale, high accuracy, and large diversity, providing an average of 30 image views for each sample, based on its "MVlmgNet by categories" subset that is available in 2023, we select the most differentiated 3 views to build a 3-views images dataset in 14 categories.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.2">Comparison with State-of-the-Art Methods</head><p>Comparison Methods The comparison methods include three single-view clustering methods: K-means <ref type="bibr">[MacQueen, 1967]</ref>, &#946;-VAE (&#946;-VAE: learning basic visual concepts with a constrained variational framework <ref type="bibr">[Higgins et al., 2017]</ref>), and VaDE (variational deep embedding: an unsupervised and generative approach to clustering <ref type="bibr">[Jiang et al., 2017]</ref>), the input of which is the concatenation of all views, and five stateof-the-art MVC methods: BMVC (binary multi-view clustering <ref type="bibr">[Zhang et al., 2018b]</ref>), SAMVC (self-paced and autoweighted multi-view clustering <ref type="bibr">[Ren et al., 2020]</ref>), RMSL (reciprocal multi-layer subspace learning for multi-view clustering <ref type="bibr">[Li et al., 2019b]</ref>), DEMVC (deep embedded multiview clustering with collaborative training <ref type="bibr">[Xu et al., 2021a]</ref>), FMVACC (fast multi-view anchor-correspondence clustering <ref type="bibr">[Wang et al., 2022]</ref>), GCFAggMVC (Global and Cross-view Feature Aggregation for <ref type="bibr">Multi-view Clustering [Yan et al., 2023]</ref>), MFLVC (Multi-level feature learning for contrastive multi-view clustering <ref type="bibr">[Xu et al., 2022c]</ref>), DIMVC (Deep incomplete multi-view clustering via mining cluster complementarity <ref type="bibr">[Xu et al., 2022a]</ref>), SDMVC (Self-supervised discriminative feature learning for deep multi-view clustering <ref type="bibr">[Xu et al., 2022b]</ref>)</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Evaluation Metrics</head><p>We evaluate the effectiveness of clustering by three commonly used metrics, i.e., clustering accuracy (ACC), normalized mutual information (NMI), and adjusted rand index (ARI). A higher value of each evaluation metric indicates a better clustering performance. Table 3: Clustering results of various methods on MVImgNet dataset.</p><p>Where MVImgNet (VIT/L) refers to using the image embeddings of CLIP ViT-L/14@336px model encoder as input, MVImgNet (DS) refers to using the downsampling of the dataset at the highest possible resolution under the same 48GB memory limit as input. The best results for each row are shown in bold and the second-best results are underlined.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Results</head><p>Tables <ref type="table">2</ref> and <ref type="table">3</ref> shows the quantitative comparison between the proposed method and baseline models for several datasets. IVSGMV achieved superior performance compared to baselines on all datasets, and the state-of-theart comparison method still mighty underperforms our proposed method on the multichannel high-resolution multiview dataset MVImgNet even when CLIP encoding embeddings are used as inputs. This reflects our full application of large-scale pre-trained model CLIP.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Vision-Language Model Implementation</head><p>For the pre-trained encoder, we used the CLIP ViT-L/14@336px model, whose visual and text backbones are <ref type="bibr">ViT [Dosovitskiy et al., 2020]</ref>. </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.3">Ablation Studies Effect of Semantic Graph Construction and Pre-trained Encoder</head><p>To understand the importance of our graph construction method and pre-trained encoder. We conducted three ablation experiments on both to analyze their impact on the performance of the IVSGMV in the Multi-Fashion dataset, where f i&#952; represents the use of pre-trained self-encoder as node feature and w/of i&#952; represents the direct use of the original image features without encoder, A represents the use of the proposed adjacency matrix construction method. As alternative constructions of A, A e and A r refer to graph construction method accord to the spectral clustering, representing the use of affinity matrix based on cosine similarity constructed using the raw features (A r ) and image embeddings (A e ) of the pre-trained encoder, respectively.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Effect of Graph Process Components</head><p>Graph joint aggregation matrix S and adaptive hybrid graph filter are important components of the IVSGMV. We conducted three more ablation experiments. w/o S represents the alternative implementation of graph joint aggregation matrix S with adjacency matrix A, w/o H hybrid represents the alternative implementation of adaptive hybrid graph filter with a common GCN low-pass filter H l p and w/o S&amp;H hybrid represents the combination of the above two. As Table <ref type="table">4</ref> demonstrates, the performance of the model is greatly and adversely affected whether the graph joint aggregation matrix or adaptive hybrid graph filter, or both are eliminated.</p><p>The results of ablation Table <ref type="table">4</ref> show that compared to A e and A r , the semantic graph construction method of A leads to 1.5% and 3.8% ACC improvement, and the pre-trained encoder f i&#952; brings about 18.7% accuracy improvement. Also, the graph joint aggregation and hybrid graph filter lead to 0.8% and 2.5% ACC improvement.</p><p>Notably, the semantic graph construction method of A also delivers 1.3% and 7.3% ACC improvement compared to A e and A r without the use of f i&#952; , respectively, which demonstrates the methodological superiority of semantic graph construction. </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Parameter Sensitivity Analysis</head><p>The sensitivity analysis for order is on Figure <ref type="figure">3</ref>. From the spatial perspective, order controls the aggregation order of the graph filter. The higher order enables nodes to aggregate information from more distant ones, while nodes can only access feature information of closer nodes in lower order. </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Visualization of Consensus Embedding</head></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5">Conclusion</head><p>In this study, we introduce large-scale pre-trained models into the field of multi-view clustering. We analyze the challenges of unsupervised fine-tuning pre-trained models and combat imbalance and noise relations in the embedding space with the help of building semantic graphs and graph filters. For semantic graph building, we propose an unsupervised construction process based on the CLIP model priori, which builds a graph-word bipartite graph by adaptively selecting words from the WordNet, and further obtains the adjacency matrices that imply semantic relations. On the graph clustering model, we apply an adaptive hybrid graph filter for multi-view clustering to adaptively mine the low and highfrequency information in the graph to learn distinguishable node embeddings, which in turn mines the homogeneous and heterogeneous information among embedding relations of the pre-trained model. In addition, the joint graph process is used to construct filters to enhance the distinguishability between low and high-frequency signals, where threshold discretization is applied to combat noise. Our IVSGMV has good performance on several multi-view datasets, extending the application of multi-view clustering to realistic image datasets while maintaining competitive performance on other datasets.</p></div><note xmlns="http://www.tei-c.org/ns/1.0" place="foot" xml:id="foot_0"><p>Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence </p></note>
		</body>
		</text>
</TEI>
