<?xml-model href='http://www.tei-c.org/release/xml/tei/custom/schema/relaxng/tei_all.rng' schematypens='http://relaxng.org/ns/structure/1.0'?><TEI xmlns="http://www.tei-c.org/ns/1.0">
	<teiHeader>
		<fileDesc>
			<titleStmt><title level='a'>HDMI: High-order Deep Multiplex Infomax</title></titleStmt>
			<publicationStmt>
				<publisher></publisher>
				<date>2021 2nd Quarter (CY)</date>
			</publicationStmt>
			<sourceDesc>
				<bibl> 
					<idno type="par_id">10232525</idno>
					<idno type="doi"></idno>
					<title level='j'>TheWebConf</title>
<idno></idno>
<biblScope unit="volume"></biblScope>
<biblScope unit="issue"></biblScope>					

					<author>Chanyoung Park Baoyu Jing</author>
				</bibl>
			</sourceDesc>
		</fileDesc>
		<profileDesc>
			<abstract><ab><![CDATA[Networks have been widely used to represent the relations between objects such as academic networks and social networks, and learning embedding for networks has thus garnered plenty of research attention. Self-supervised network representation learning aims at extracting node embedding without external supervision. Recently, maximizing the mutual information between the local node embedding and the global summary (e.g. Deep Graph Infomax, or DGI for short) has shown promising results on many downstream tasks such as node classification. However, there are two major limitations of DGI. Firstly, DGI merely considers the extrinsic supervision signal (i.e., the mutual information between node embedding and global summary) while ignores the intrinsic signal (i.e., the mutual dependence between node embedding and node attributes). Secondly, nodes in a real-world network are usually connected by multiple edges with different relations, while DGI does not fully explore the various relations among nodes. To address the above-mentioned problems, we propose a novel framework, called High-order Deep Multiplex Infomax (HDMI), for learning node embedding on multiplex networks in a self-supervised way. To be more specific, we first design a joint supervision signal containing both extrinsic and intrinsic mutual information by high-order mutual information, and we propose a High- order Deep Infomax (HDI) to optimize the proposed supervision signal. Then we propose an attention based fusion module to combine node embedding from different layers of the multiplex network. Finally, we evaluate the proposed HDMI on various downstream tasks such as unsupervised clustering and supervised classification. The experimental results show that HDMI achieves state-of-the-art performance on these tasks.]]></ab></abstract>
		</profileDesc>
	</teiHeader>
	<text><body xmlns="http://www.tei-c.org/ns/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="1">INTRODUCTION</head><p>Network or graph structures have been widely used to represent the relations among objects, such as the citation relation among papers, the co-author relation among researchers, as well as the friendship relation among people. Mining and discovering useful knowledge from networks has been an active research area, within which network representation learning has garnered substantial research attention and it has been demonstrated to be effective for various tasks on networks <ref type="bibr">[6,</ref><ref type="bibr">45,</ref><ref type="bibr">48]</ref>, such as node classification, node clustering and link prediction.</p><p>Self-supervised network representation learning aims at extracting node embedding without introducing external supervision signals. The dominant strategy of the self-supervised network representation learning is to design a signal based on the proximity of the nodes in the network, such that the node embedding will retain the proximity. For example, given a node, DeepWalk <ref type="bibr">[30]</ref> and node2vec <ref type="bibr">[9]</ref> maximize the probabilities of its neighbors sampled by random walks. LINE <ref type="bibr">[34]</ref> maximizes the probabilities between a node and its first or second-order proximate neighbors. Albeit the effectiveness of these methods, the proximity based supervision signals only capture the local characteristic of networks. With the introduction of the powerful network encoders, such as Graph Convolutional Network (GCN) <ref type="bibr">[16]</ref> which can naturally capture such local proximity based on graph convolution <ref type="bibr">[3]</ref>, the improvement brought by the traditional proximity based supervision signals is limited <ref type="bibr">[37]</ref>. To further improve the quality of node embedding, Deep Graph Infomax (DGI) <ref type="bibr">[37]</ref> uses GCN as the network encoder and trains it by maximizing the mutual information between the node embedding and the global summary of the network. However, there are two major limitations of DGI.</p><p>One limitation of DGI is that it focuses on the extrinsic supervision signal (i.e., whether the node embedding and the global summary come from the same network), and it does not fully explore the intrinsic signal (i.e., the mutual dependence between the embedding vector and the attribute vector of a node). Node attributes themselves often contain discriminative information about the nodes, and capturing the mutual dependence between them helps the embedding obtain more discriminative information and thus discover the potential relations among nodes. For example, in the citation network, papers always contain textual attributes such as abstracts and keywords. Two papers focusing on different sub-areas (e.g. dynamic graph algorithms and sub-graph mining) might not have a direct citation relation, but they might share the same or similar keywords (e.g. social network). In such a case, they are likely to belong to the same area, e.g., social analysis or web mining. The extrinsic signal only models the global property of the citation network and ignores the intrinsic mutual dependence, and thus it might be unable to discover the hidden relation between the two papers. The existing dominant strategy for capturing the mutual dependence between embedding and attributes is using reconstruction error of attributes given node embedding <ref type="bibr">[8,</ref><ref type="bibr">24,</ref><ref type="bibr">46]</ref>. However, as pointed out by <ref type="bibr">[41]</ref>, there is no substantial connection between the discriminative ability (or quality) of embedding with the reconstruction loss. In this paper, we propose to use mutual information between node embedding and node attributes as the intrinsic supervision signal. Moreover, to further capture the synergy between the extrinsic and the intrinsic signals, we propose to use high-order mutual information among the node embedding, global summary vector, and node attributes, and we propose a novel High-order Deep Infomax (HDI) to maximize the high-order mutual information.</p><p>Another limitation of DGI is that it assumes a single type of relation among the nodes in a network, but nodes in a network are usually connected via multiple edges with different relation types. For example, in an academic network, two papers can be connected via shared authors or citation relation. In a product network, products can be linked by relations such as AlsoView, AlsoBought, and BoughtTogether. In a social network, two people can be connected by many relations such as direct friendship, the same school, and the same employer. A network with multiple types of relations is referred to as a multiplex network, and a multiplex network can be decomposed into multiple layers of networks, where each layer only has one type of relation. Figure <ref type="figure">1</ref> provides an example of a multiplex network. The simplest way to learn node embedding for a multiplex network is to first extract node embedding independently from different layers and then use average pooling over the node embedding from different layers to obtain final embedding for each node. However, as observed by many recent studies <ref type="bibr">[4,</ref><ref type="bibr">28,</ref><ref type="bibr">33,</ref><ref type="bibr">39,</ref><ref type="bibr">44]</ref>, layers are related with each other and they can mutually help each other for downstream tasks. For example, in the academic multiplex network mentioned above, the citation network layer and the shared author network layer usually provide different aspects of the papers' subject. To be more specific, the citation layer usually links a paper (e.g. about the attributed network) with related papers from other areas (e.g. computer vision), while a specific author usually focuses on a specific area (e.g. network representation learning). To capture such a mutual relation, the attention mechanism <ref type="bibr">[1]</ref> is mostly adopted to calculate the weights for different layers <ref type="bibr">[31,</ref><ref type="bibr">39]</ref>. However, these methods require external supervision (e.g. labels of nodes) to train the attention module. Recently, DMGI <ref type="bibr">[28]</ref> proposes a complex combination module, which first uses an attention mechanism to obtain reference node embedding and then uses a consensus regularization to obtain final node embedding. In this paper, we propose an alternative semantic attention <ref type="bibr">[43]</ref> based method as the fusion module to combine node embedding from different layers. More importantly, different from existing works, the proposed fusion module is trained via the proposed high-order mutual information. Our main contributions are summarized as follows:</p><p>&#8226; We propose a novel supervision signal based on the highorder mutual information, which combines extrinsic and intrinsic mutual information, for learning network embedding on both attributed networks and attributed multiplex networks. &#8226; We introduce a novel High-order Deep Infomax (HDI) to optimize the proposed high-order mutual information based supervision signal. &#8226; We propose an attention based fusion module to combine node embedding from different layers of a multiplex network, which is trained via the high-order mutual information based supervision signal. &#8226; We evaluate the proposed methods on a variety of real-world datasets with various evaluation metrics to demonstrate the effectiveness of the proposed methods. The rest of the paper is organized as follows. We briefly introduce the attributed multiplex network, mutual information and DGI in Section 2. We introduce the high-order mutual information, as well as the proposed HDI and HDMI in Section 3. The experimental results are presented in Section 4. We provide a brief review of the most relevant works in Section 5. Finally, we conclude the paper in Section 6.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2">PRELIMINARIES</head><p>In this section, we first introduce relevant concepts for attributed multiplex networks. Then we introduce mutual information related concepts as well as Deep Graph Infomax (DGI) <ref type="bibr">[37]</ref>. We also summarize the notations used in this paper in Table <ref type="table">1</ref>.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2.1">Attributed Multiplex Networks</head><p>An attributed multiplex network (Definition 2.2) is comprised of multiple layers of attributed networks (Definition 2.1).</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Definition 2.1 (Attributed Network</head><p>). An attributed network is represented by G(&#119860;, &#119865; ), where &#119860; &#8712; R &#119873; &#215;&#119873; denotes adjacency matrix and &#119865; &#8712; R &#119873; &#215;&#119889; &#119865; denotes the attribute matrix, &#119873; and &#119889; &#119865; denote the number of nodes and the dimension of attributes respectively.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Definition 2.2 (Attributed Multiplex Network). An attributed multiplex network</head><p>denotes the &#119903; -th layer. Note that different layers capture different types of relations between nodes, and all of the layers share the same node attribute matrix.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2.2">Mutual Information</head><p>Mutual information measures the mutual dependence between two random variables, which is based on Shannon entropy, and its formal definition is given in Definition 2.3. In order to measure the mutual dependence between multiple random variables, the concept of high-order/multivariate mutual information is introduced in the information theory <ref type="bibr">[23]</ref>.</p><p>Definition 2.3 (Mutual Information). Given two random variables &#119883; and &#119884; , the mutual information between them is defined by:</p><p>where &#119867; (&#119883; ) and &#119867; (&#119883; |&#119884; ) denote entropy and conditional entropy respectively, &#119867; (&#119883;, &#119884; ) denotes the entropy for the joint distribution of &#119883; and &#119884; .</p><p>Definition 2.4 (High-order Mutual Information <ref type="bibr">[23]</ref>). High-order mutual information is a generalization of mutual information on &#119873; &#8805; 3 random variables. Given a set of &#119873; random variables &#119883; 1 , &#8226; &#8226; &#8226; , &#119883; &#119873; , the high-order mutual information is defined analogous to the definition of mutual information (Definition 2.3):</p><p>where &#119867; (&#119883; &#119894;  </p><p>High-order mutual information not only captures the mutual information between each pair of two random variables but also the synergy among multiple random variables. Equation ( <ref type="formula">4</ref>)-(6) provide an example when &#119873; = 3.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2.3">Deep Graph Infomax</head><p>DGI is a self-supervised learning or an unsupervised learning method for learning node embedding on attributed networks (Definition 2.1), the main idea behind which is to maximize the mutual information between node embedding h and the global summary vector s of the entire network G: &#119868; (h, s).</p><p>When maximizing &#119868; (h, s), DGI leverages negative sampling strategy. Specifically, it first generates a negative network G via a corruption function G = C(G). Then it uses the same encoder E (e.g. GCN <ref type="bibr">[16]</ref>) to obtain the node embedding for the positive network {h 1 , &#8226; &#8226; &#8226; , h &#119873; } as well as the embedding for the negative network { h1 , &#8226; &#8226; &#8226; , h&#119873; }. Here, &#119873; is the number of nodes in both of the networks. The summary vector of the positive network G is obtained by a readout function s = R ({h 1 , &#8226; &#8226; &#8226; , h &#119873; }) (e.g. average pooling). Finally, given s, a discriminator D is used to distinguish the node embedding of the positive network h &#119899; with the one from the negative network h&#119899; . For more details, please refer to <ref type="bibr">[37]</ref>. Maximizing the objective function given in the following definition can effectively maximize &#119868; (h &#119899; , s).</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Definition 2.5 (Deep Graph Infomax</head><p>). Given a node embedding h &#119899; and a summary vector s of the attributed network and a negative node embedding h&#119899; , the mutual information between h &#119899; and s can be maximized by maximizing the following objective function:</p><p>where D denotes the discriminator aiming at distinguish the negative node embedding h&#119899; with the real embedding h &#119899; , E denotes the expectation.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3">METHODOLOGY</head><p>In this section, we first introduce High-order Deep Infomax (HDI) for optimizing high-order mutual information on attributed networks in Section 3.1 and then extend the proposed HDI to attributed multiplex networks by introducing High-order Deep Multiplex Infomax (HDMI) in Section 3.2.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.1">High-order Deep Infomax on Attributed Networks</head><p>In this section, we propose a novel High-order Deep Infomax (HDI) for self-supervised learning on attributed networks based on DGI <ref type="bibr">[37]</ref> and high-order mutual information (Definition 2.4). As mentioned in the introduction and preliminary (Section 2.3), DGI merely considers the extrinsic supervision signal: the mutual information between the node embedding h &#119899; and the global summary vector s of a network. Intrinsic signal, i.e., the mutual dependence between node embedding h &#119899; and attributes f &#119899; has not been fully explored.</p><p>We introduce high-order mutual information to simultaneously capture the extrinsic and intrinsic supervision signals as well as the synergy between them in Section 3.1.1. We describe the details of HDI in Section 3.1.2.  3.1.1 High-order Mutual Information Estimation. For the &#119899;-th node in the attributed network G, we propose to jointly maximize the mutual information for three random variables, i.e., node embedding h &#119899; , the summary vector s and node attributes f &#119899; , via the high-order mutual information.</p><p>According to the definition of the high-order mutual information (Definition 2.4), when &#119873; = 3, we will have the following equation:</p><p>The above equation can be further re-written as the following equation:</p><p>where &#119868; (&#119883; 1 ; &#119883; 2 , &#119883; 3 ) denotes the mutual information between the distribution of &#119883; 1 with the joint distribution of &#119883; 2 and &#119883; 3 .</p><p>Therefore, by replacing the random variables &#119883; 1 , &#119883; 2 and &#119883; 3 with h &#119899; , s and f &#119899; we will have:</p><p>In the above equation, &#119868; (h &#119899; ; s) captures the extrinsic supervision signal: the mutual dependence between node embedding h &#119899; and the global summary s. &#119868; (h &#119899; ; f &#119899; ) captures the intrinsic supervision signal: the mutual dependence between node embedding h &#119899; and attributes h &#119899; . &#119868; (h &#119899; ; s, f &#119899; ) captures the interaction between the extrinsic and intrinsic signal.</p><p>Maximizing the high-order mutual information in Equation (6) will result in the following equation:</p><p>As can be noted in Equation <ref type="bibr">(7)</ref>, maximizing the high-order mutual information is equivalent to maximize the three mutual information: &#119868; (h &#119899; ; s), &#119868; (h &#119899; ; f &#119899; ) and &#119868; (h &#119899; ; s, f &#119899; ).</p><p>Considering the fact that, during training, different mutual information on the right-hand side of Equation ( <ref type="formula">7</ref>) might have different magnitudes, therefore, we use different coefficients for different mutual information. The the final objective function is given by:</p><p>where &#120582; &#119864; , &#120582; &#119868; and &#120582; &#119869; are tunable coefficients.</p><p>Extrinsic Signal. For maximizing the extrinsic mutual dependence &#119868; (h &#119899; ; s), we follow <ref type="bibr">[37]</ref> and we will have:</p><p>where h &#119899; and h&#119899; are the node embedding from the positive (i.e., original) G and the negative network G respectively; the negative network is obtained by corrupting the positive network via the corruption function C: G = C(G); D &#119864; denotes the discriminator for distinguishing h &#119899; and h&#119899; . The green part in Figure <ref type="figure">2a</ref> provides an illustration for the extrinsic signal.</p><p>Intrinsic Signal. For the maximization of the intrinsic mutual dependence &#119868; (h &#119899; ; f &#119899; ), we replace s with f &#119899; in Equation ( <ref type="formula">9</ref>) and we will have:</p><p>where f &#119899; denotes the attribute vector of the &#119899;-th node and D &#119868; denotes the discriminator. The orange part in Figure <ref type="figure">2a</ref> provides an illustration for the intrinsic signal.</p><p>Joint Signal. The joint signal &#119868; (h &#119899; ; s, f &#119899; ) is comprised of three random variables: h &#119899; , s and f &#119899; . Rather than substituting s with (s, f &#119899; ) in Equation 9 and using the negative node embedding h&#119899; to build the negative sample pairs, we propose to use the negative node attributes f&#119899; to construct the negative samples. This is because the extrinsic signal has already captured the mutual dependence between h &#119899; and s, as well as the independence between h&#119899; and s by maximizing Equation <ref type="formula">9</ref>. The intrinsic signal has captured the mutual dependence between h &#119899; and f &#119899; , as well as the independence between h&#119899; and f &#119899; via Equation <ref type="formula">10</ref>. If we add a new discriminator to distinguish h &#119899; from h&#119899; given (s, f &#119899; ), it will not bring substantial new information. Instead, we propose to let the discriminator to distinguish (h &#119899; , s, f &#119899; ) with (h &#119899; , s, f&#119899; ). By introducing negative samples of attributes f&#119899; , the encoder E can better capture the joint mutual dependence among h &#119899; , s and f &#119899; , especially the mutual dependence between s and f &#119899; , which is not captured by either extrinsic signal or intrinsic signal. Therefore, we have the following objective:</p><p>where D &#119869; denotes the discriminator for the joint signal. The blue part in Figure <ref type="figure">2b</ref> provides an illustration for the joint signal.</p><p>Training Objective. The final supervision signal is to maximize the following objective:</p><p>where &#120582; &#119864; , &#120582; &#119868; and &#120582; &#119869; are tunable coefficients.</p><p>3.1.2 Model Architecture. We elaborates the details of the model architecture in this section.</p><p>Network Encoder E. We use a single layer GCN <ref type="bibr">[16]</ref> as E:</p><p>where Readout Function R. We use average pooling as R:</p><p>where h &#119899; &#8712; R &#119889; is the &#119899;-th row of &#119867; , s &#8712; R &#119889; is the global summary vector.</p><p>Dicriminator D. For D &#119864; and D &#119868; , we follow DGI <ref type="bibr">[37]</ref>:</p><p>where &#119872; &#119864; &#8712; R &#119889;&#215;&#119889; and &#119872; &#119868; &#8712; R &#119889;&#215;&#119889; &#119865; are parameter matrices, &#120590; is the sigmoid activation function.</p><p>As for the discriminator for the joint signal D &#119869; , since f &#119899; and s are not in the same space, we first project them into the same hidden space, and then use a bi-linear function to obtain the final scores.</p><p>where &#119882; &#119891; &#8712; R &#119889;&#215;&#119889; &#119865; , &#119882; &#119904; &#8712; R &#119889;&#215;&#119889; , &#119882; &#119911; &#8712; R &#119889;&#215;2&#119889; and &#119872; &#119869; &#8712; R &#119889;&#215;&#119889; are parameter matrices, &#120590; denotes the sigmoid activation function and [; ] denotes concatenation operation.</p><p>Corruption Function C. We follow DGI <ref type="bibr">[37]</ref> and use the random permutation of nodes as the corruption function. Specifically, we randomly shuffle the rows of attribute matrix &#119865; .</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.2">High-order Deep Multiplex Infomax</head><p>In this section, we extend HDI to the multiplex network and propose a High-order Deep Multiplex Infomax (HDMI) model. An attributed multiplex network is comprised of multiple layers of attributed networks. In order to learn node embedding for the multiplex networks, we first learn node embedding on each of its layers separately and then combine them via a fusion module (Section 3.2.1). We leverage the high-order mutual information to train the fusion module (Section 3.2.2).</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.2.1">Fusion of Embedding.</head><p>The simplest way of combining node embedding from different layers is average pooling. However, different layers are related to each other. Therefore, we use the semantic attention <ref type="bibr">[43]</ref> based method to fuse the node embedding, as shown in Figure <ref type="figure">3</ref>. Given a node embedding from the &#119903; -th layer h &#119903; &#119899; &#8712; R &#119889; , we use layer-dependent semantic attention to obtain its score &#120572; &#119903; &#119899; by:</p><p>where &#119881; &#119903; &#8712; R &#119889; &#8242; &#215;&#119889; is the parameter matrix, y &#119903; denotes the hidden representation vector of the &#119903; -th layer, tanh denotes the tangent activation function.</p><p>Then the weights of node embedding from different layers are given by:</p><p>where &#119877; is the number of layers in the multiplex network. The final embedding of the &#119899;-th node can be obtained by:</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.2.2">Training.</head><p>To train the fusion module, we propose to use highorder mutual information introduced in Section 3.1. Given the fused embedding h &#119899; , we maximize the mutual information between h &#119899; with its attributes f &#119899; and global summary vector s of the multiplex network. The training objective and model architecture are the same as those introduced in Section 3.1, but there are two differences: the global summary vector s and negative node embedding h&#119899; . For s of the multiplex network, we use average pooling over the fused embedding h &#119899; to obtain it. Additionally, we obtain the negative node embedding h&#119899; of the multiplex network by combining the negative node embedding h&#119903; &#119899; from different layers via the fusion module.</p><p>The fusion module can be trained jointly with HDI on different layers, and the final objective of HDMI is:</p><p>where L &#119872; and L &#119903; denote the objective for the fusion module and the &#119903; -th layer respectively, &#120582; &#119872; and &#120582; &#119903; are tunable coefficients. Note that both L &#119872; and L &#119903; are given by Equation ( <ref type="formula">12</ref>).</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4">EXPERIMENTS</head><p>We presents the experiments to answer the following questions: Q1 How will the HDI and HDMI improve the quality of the learned node embedding? Q2 Will the fusion module assign proper attention scores to different layers? Amazon. The Amazon<ref type="foot">foot_3</ref> dataset <ref type="bibr">[11]</ref> contains 7,621 items from four categories (Beauty, Automotive, Patio Lawn and Garden, and Baby) with three types of relations (Also View, Also Bought and Bought Together). The attribute of each item is a 2,000-dimensional bag-of-words representation of its description.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.1.2">Comparison</head><p>Methods. We compare our proposed method with two sets of baseline methods: network embedding methods and multiplex network embedding methods.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Network Embedding.</head><p>&#8226; Methods disregarding attributes: DeepWalk <ref type="bibr">[30]</ref> and node2vec <ref type="bibr">[9]</ref>. These two methods are random-walk and skip-gram based embedding models. &#8226; Methods considering attributes: GCN <ref type="bibr">[16]</ref> and GAT <ref type="bibr">[36]</ref>.</p><p>These two methods learn node embedding based on the local structure of the nodes, and the best performance of the two methods are reported. DGI <ref type="bibr">[37]</ref> maximizes the mutual information between node embedding and the global summary vector. ANRL <ref type="bibr">[46]</ref> uses skip-gram to model the local contextual topology and uses an auto-encoder to capture attribute information. CAN <ref type="bibr">[24]</ref> learns node embedding and attribute embedding in the same semantic space. DGCN <ref type="bibr">[49]</ref> considers both local and global consistency. Multiplex Network Embedding.</p><p>&#8226; Methods disregarding attributes: CMNA <ref type="bibr">[5]</ref> uses the crossnetwork information to refine both of the inter-network and intra-network node embedding. MNE <ref type="bibr">[44]</ref> uses a common embedding with multiple additional embedding from different layers for each node. &#8226; Methods considering attributes: mGCN <ref type="bibr">[21]</ref> and HAN <ref type="bibr">[39]</ref> use GCN/GAT to extract node embedding for each layer of the multiplex networks and then combine them via attention mechanism. DMGI and DMGI attn <ref type="bibr">[28]</ref> extend DGI onto multiplex networks and uses consensus regularization to combine node embedding from different layers, where DMGI attn leverages attention to obtain the reference node embedding.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.1.3">Evaluation Metrics.</head><p>We evaluate our proposed HDMI and comparison methods on both of the unsupervised tasks (i.e., clustering and similarity search <ref type="bibr">[28,</ref><ref type="bibr">39]</ref>) and a supervised task (i.e., node classification) with the following evaluation metrics.</p><p>&#8226; Macro-F1 and Micro-F1 are used to evaluate models on the node classification task. &#8226; Normalized Mutual Information (NMI) is adopted for the node clustering task. &#8226; Sim@5 evaluates how well does a model project nodes with the same label to nearby positions in the embedding space.</p><p>For the clustering and classification, we first train models with their self-supervision signals and then run the clustering methods (i.e., K-means) and classification methods (i.e., logistic regression) to obtain the NMI and Macro-F1/Micro-F1 scores. For similarity search, we follow <ref type="bibr">[28,</ref><ref type="bibr">39]</ref> to first compute the cosine similarity between each pair of nodes based on their embedding. Then for each node, the top-5 most similar nodes are selected to calculate the ratio of nodes sharing the same label (Sim@5).</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.1.4">Implementation Details.</head><p>We set the dimension of node embedding as 128, and use Adam optimizer <ref type="bibr">[14]</ref> with the learning rate of 0.001 to optimize the models. The weight of the self-connection for GCN is fixed as 3. Following <ref type="bibr">[28]</ref>, for HDMI, we use the same discriminator for different layers. We use grid search to tune the coefficients and report the best results. Early stopping with a patience of 100 is adopted to prevent overfitting.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.2">Quantitative Evaluation</head><p>In this section, we present the experimental results on both supervised and unsupervised downstream tasks to quantitatively demonstrate the effectiveness of the proposed supervision signals as well as the proposed models HDI and HDMI.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.2.1">Overall</head><p>Performance. We present the experimental results of node classification, node clustering and similarity search on the multiplex networks in Table <ref type="table">3</ref> and Table <ref type="table">4</ref>. HDI separately learns embedding on different layers and uses average pooling to combine node embedding from different layers. Table <ref type="table">3</ref> shows that HDMI outperforms the state-of-the-art models for all of the supervised tasks, and Table <ref type="table">4</ref> shows that HDMI achieves higher scores for most of the cases on the unsupervised tasks. These results demonstrate that the embedding extracted by HDMI is more discriminative. Additionally, the scores of HDI are generally competitive to the state-of-the-art methods, which demonstrate the effectiveness of the proposed HDI to a certain degree.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.2.2">Ablation Study.</head><p>Performance of the fusion module. To evaluate the proposed fusion module, we compare it with average pooling. As shown in Table <ref type="table">3</ref> and Table <ref type="table">4</ref>, HDMI outperforms HDI on all metrics for all datasets, except for the Sim@5 on the ACM dataset. However, it can be noted that the gap is very tiny.</p><p>Performance of different supervision signals. We compare the performance of the Extrinsic (E.), Intrinsic (I.), and Joint (J) signal as well as the Reconstruction (R.) error on attributed networks (different layers of the multiplex networks) in the datasets, and present experimental results in Table <ref type="table">5</ref> and Table <ref type="table">6</ref>. Firstly, incorporating the mutual dependence between embedding and attributes could improve the model's performance on both supervised and unsupervised tasks, which can be observed by comparing E. with E.+R. and E.+I. Secondly, maximizing mutual information between node embedding and attributes (E.+I.) is better than minimizing the reconstruction error of attributes given node embedding (E.+R.). Thirdly, the joint signal (E.+I.+J.) can further improve the discriminative ability of node embedding for the node classification task. Finally, combining node embedding from different layers (lower parts of Table <ref type="table">5</ref>-6) will result in better results, indicating that different layers can mutually help each other. </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.3">Qualitative Evaluation</head><p>In this section, we present the qualitative evaluation results.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.3.1">Visualization of node embedding.</head><p>Node embedding of an attributed network. We show the t-sne <ref type="bibr">[22]</ref> visualization of the node embedding of the IOI layer in the Amazon dataset learned by the Extrinsic (E.), the Extrinsic + Reconstruction (E. + R.), the Extrinsic + Intrinsic (E. + I.) and the Extrinsic + Intrinsic + Joint (E. + I. + J.) signals in Figure <ref type="figure">4</ref>. Different colors in the figure represent different classes. It is obvious that the intrinsic signal and the joint signal significantly improve the discriminative ability of the node embedding, and the joint signal can further improve the quality of the node embedding Additionally, Figure <ref type="figure">4b</ref> shows that the reconstruction error does not substantially connect with the discriminative ability of the node embedding.</p><p>Node embedding of a multiplex network. The t-sne visualization for the node embedding learned by HDI on the PSP and the PAP layer of the ACM multiplex network are presented in Figure <ref type="figure">5a-5b</ref>. Figure <ref type="figure">5c-5d</ref> show the combined embedding obtained by average pooling and fusion module. As can be noted in the red boxes of Figure <ref type="figure">5b-5c</ref>, average pooling for the embedding of different layers better separates nodes than the embedding learned from PAP layer alone. Additionally, Figure <ref type="figure">5c</ref>-5d show that the fusion module separates the nodes even better than the average pooling.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.3.2">Attention scores.</head><p>We present the visualization of attention scores and the Micro-F1 scores for each layer of the multiplex networks in Figure <ref type="figure">6</ref>. Generally, for the layers that have higher Micro-F1 scores, their attention scores are also higher, demonstrating the effectiveness of the proposed fusion module.       </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5">RELATED WORKS</head><p>In this section, we briefly review the most relevant works, including the self-supervised graph representation learning (Section 5.1), the graph neural networks (Section 5.2), as well as the mutual information methods (Section 5.3).</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.1">Self-supervised Network Representation Learning</head><p>In this section, we mainly focus on self-supervised network embedding methods. For a comprehensive review, please refer to <ref type="bibr">[6,</ref><ref type="bibr">45,</ref><ref type="bibr">48]</ref>. We leave the mutual information related methods to Section 5.2, and graph neural networks with other signals to Section 5.3. Inspired by word2vec <ref type="bibr">[25]</ref>, deepwalk <ref type="bibr">[30]</ref> and node2vec <ref type="bibr">[9]</ref> leverage random walks to sample context of a node, and then learn node embedding by maximizing the probabilities of the sampled neighbors. SDNE <ref type="bibr">[38]</ref> and LINE <ref type="bibr">[34]</ref> propose to learn node embedding based on the first-order and the second-order proximities between nodes. Struct2vec <ref type="bibr">[38]</ref> further designs a training signal based on the local structural similarity of the nodes. However, these methods ignore node attributes. Zhang et al. <ref type="bibr">[46]</ref> uses a neighbor enhancement auto-encoder to encoder attributes with the re-construction error. Meng et al. <ref type="bibr">[24]</ref> propose to co-embed attributes and nodes into the same semantic space via Variational Auto-Encoder (VAE) <ref type="bibr">[15]</ref>, but VAE tends to minimize the mutual information between inputs and hidden features <ref type="bibr">[47]</ref>.</p><p>The multiplex network is also known as multi-dimensional network <ref type="bibr">[20]</ref>. Liu et al. <ref type="bibr">[19]</ref> propose a random walk based method to map multiplex networks into a vector space. Zhang et al. <ref type="bibr">[44]</ref> propose a scalable multiplex network embedding method trained via negative sampling. Shi et al. <ref type="bibr">[33]</ref> leverage preservation and collaboration to learn node embedding. Ma et al. <ref type="bibr">[21]</ref> propose mGCN for multiplex networks trained via negative sampling. Cen et al. <ref type="bibr">[4]</ref> use the random walk based method to train a unified network for attributed multiplex networks. Ban et al. <ref type="bibr">[42]</ref> introduce a density metric as the training signal to improve the performance of node clustering. Li et al. <ref type="bibr">[17]</ref> introduce MANE considering both within-layer and cross-layer dependency.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.2">Graph Neural Networks</head><p>In this section, we briefly review the graph neural networks with external training signals (e.g. cross-entropy between predicted labels and real labels). For a comprehensive review of the graph neural networks, please refer to <ref type="bibr">[45,</ref><ref type="bibr">48]</ref>. Kipf et al. <ref type="bibr">[16]</ref> propose Graph Convolutional Network (GCN) based on <ref type="bibr">[7]</ref>. Hamilton et al. <ref type="bibr">[10]</ref> propose an aggregation-based model called GraphSAGE. Graph Attention Network (GAT) <ref type="bibr">[36]</ref> learns different weights for a node's neighbors by attention mechanism. Zhuang et al. <ref type="bibr">[49]</ref> propose DGCN which considers both local and global consistency.</p><p>Qu et al. <ref type="bibr">[31]</ref> use an attention mechanism to embed multiplex networks into a single collaborated embedding. Wang et al. <ref type="bibr">[39]</ref> propose an attention based method called HAN for merging node embedding from different layers. Chu et al. <ref type="bibr">[5]</ref> propose CrossMNA which leverages cross-network information for network alignment. Yan et al. <ref type="bibr">[40]</ref> introduce DINGAL for dynamic knowledge graph alignment. Jing et al. <ref type="bibr">[13]</ref> propose TGCN for modeling high-order tensor graphs.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.3">Mutual Information Methods</head><p>Belghazi et al. <ref type="bibr">[2]</ref> propose MINE to estimate mutual information of two random variables by neural networks. Mukherjee et al. <ref type="bibr">[26]</ref> propose a classifier based neural estimator for conditional mutual information. Recently, the infomax principle <ref type="bibr">[18]</ref> has been used for self-supervised representation learning in computer vision <ref type="bibr">[12]</ref> and speech recognition <ref type="bibr">[32]</ref> by maximizing the mutual information between different hidden features. In the field of network representation learning, DGI <ref type="bibr">[37]</ref> maximizes the mutual information of node embedding with the global summary vector. Peng et al. <ref type="bibr">[29]</ref> propose GMI for homogeneous networks. Park et al. <ref type="bibr">[27,</ref><ref type="bibr">28]</ref> extend DGI onto multiplex networks and propose a consensus regularization to combine embedding of different layers.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="6">CONCLUSION</head><p>In this paper, we introduce a novel High-order Deep Multiplex Infomax (HDMI) to learn network embedding for multiplex networks via self-supervised learning. Firstly, we propose a High-order Deep Infomax (HDI) and use high-order mutual information as the training signal for attributed networks. The proposed signal simultaneously captures the extrinsic signal (i.e., the mutual dependence between node embedding and the global summary), the intrinsic signal (i.e., the mutual dependence between node embedding and attributes), and the interaction between these two signals. Secondly, we propose a fusion module based on the attention mechanism to combine node embedding from different layers. We evaluate the proposed HDMI on four real-world datasets for both unsupervised and supervised downstream tasks. The results demonstrate the effectiveness of the proposed HDMI, HDI, and the fusion module.</p></div><note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="1" xml:id="foot_0"><p>https://www.acm.org/</p></note>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="2" xml:id="foot_1"><p>https://www.imdb.com/</p></note>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="3" xml:id="foot_2"><p>https://aminer.org/AMinerNetwork</p></note>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="4" xml:id="foot_3"><p>https://www.amazon.com/</p></note>
		</body>
		</text>
</TEI>
