necessary due to their pervasiveness in web directories 1 , business category listsfoot_1 and many other domains (Figure 1).

Although recent studies on hierarchy-aware neural models [24,44,55] and pre-trained language models [5,38] greatly improve the performance of (hierarchical) text categorization, they are less concerned with two challenges in real applications:

Limited Training Data. Neural classifiers are data hungry and require a significant amount of manually labeled training documents to achieve good performance. In some scientific domains (e.g., arXiv papers and GitHub repositories), to obtain a large training set is expensive because annotations have to be acquired from domain experts. In these scenarios, it would be favorable to perform classification without much annotation effort. To be specific, one may only expect very few (e.g., 5) training examples for each category.

Heterogeneous Signals. Documents on the Web are often accompanied by metadata information. We raise three examples in Figure 1: each GitHub repository has its creator and several topic tags; each arXiv paper has its author(s); each Amazon review is associated with a user and a product. These signals, as potential category indicators, should be considered together with the plain text sequence. Moreover, the label hierarchy, which reflects category dependencies and correlation, should also be viewed as one type of signal. In recognition of these heterogeneous signals, how can we jointly leverage them during category inference?

In this paper, to tackle the two challenges, we develop HiMe-Cat, an embedding-based generative framework for Hierarchical Metadata-aware document Categorization under weak supervision. HiMeCat features a hierarchical generative process in the embedding space to simultaneously model (1) category dependencies via a top-down generative assumption along the label hierarchy, (2) metadata information via a joint generative assumption that documents are dependent on both their categories and metadata, and (3) textual semantics via a document-word generative assumption. This hierarchical generative process guides the two important steps of our method: (1) representation learning: Maximizing the likelihood of the hierarchical generative process yields an embedding learning objective which jointly optimizes the representations of the label hierarchy, metadata and texts to effectively exploit the heterogeneous signals;

(2) data augmentation: Following the hierarchical generative process allows us to synthesize training documents that augment the original, small-scale training set, which overcomes the limited training data challenge. Combining real and synthesized training data, we train a neural classifier for hierarchical text categorization by taking word representations learned from the previous step as embedding initialization.

Inspired by related studies on spherical hierarchical clustering [13] and spherical text embedding [29] which better capture directional similarities and outperform Euclidean space models, we propose to define our hierarchical generative process in the spherical space, where conditional probabilities are described by the von Mises-Fisher (vMF) distribution [8] and Riemannian optimization [2] is adopted for embedding learning.

To summarize, our contributions are as follows:

• We explore the task of jointly leveraging limited supervision and metadata information for hierarchical text classification. • We develop HiMeCat with a representation learning module and a data augmentation module guided by a novel hierarchical generative process in the spherical space. Our representation learning module jointly learns embeddings from heterogeneous signals, and our data augmentation module synthesizes training documents to address the supervision scarcity bottleneck. • We conduct comprehensive experiments on three datasets from different domains and observe a consistent improvement of HiMe-Cat over competitive baselines. We also show that leveraging the hierarchy, leveraging metadata, and generating training samples are all beneficial to the categorization performance.

The inputs of our hierarchical metadata-aware text categorization task are a collection of documents D = {𝑑 1 , ..., 𝑑 | D | } and a label hierarchy T . Each document 𝑑 𝑖 ∈ D has both text and metadata information. We explain these concepts one by one.

Text. In this paper, text refers to all free-text fields of a document. For example, in Figure 1, a GitHub repository has its description and README file as text information; an arXiv paper has its title and abstract. To simplify our discussion, we concatenate all these fields into one word sequence, denoted as 𝑤 1 𝑤 2 ...𝑤 𝑛 .

Metadata. Each document can have multiple types of metadata (e.g., an Amazon review has both user and product information). Given a specific metadata type, there may be an arbitrary number (could be zero) of metadata instances associated with a document (e.g., a repository may have no tags, and a paper can have more than one author). For a document 𝑑, we represent its metadata as a set of metadata instances

Label Hierarchy. We assume the label hierarchy T has a tree structure. Each node 𝑙 ∈ T represents a category. If, in some cases, T is a directed acyclic graph (DAG), we follow [55] and convert T to a tree by distinguishing each label node as a single-path node. Formally, we define our task as follows:

Problem Definition. Given a collection of documents D and a label hierarchy T , where each leaf category 𝑙 ∈ T is characterized by a small set of training documents D 𝑙 ⊂ D, the task is to assign appropriate category labels to the remaining documents D\(∪ 𝑙 D 𝑙 ).

The labels of each document should form a path in T .

The von Mises-Fisher (vMF) distribution [8,26] will be extensively used in our proposed framework. It defines a probability density over a unit sphere and is parameterized by a mean direction vector 𝝁 and a concentration parameter 𝜅. Formally, let S 𝑝-1 = {𝒙 ∈ R 𝑝 : ||𝒙 || 2 = 1} denote the 𝑝-dimensional unit sphere. The probability density function for 𝒙 ∈ S 𝑝-1 , ||𝝁 || 2 = 1, 𝜅 > 0 is given by

Here, the normalization constant 𝑐 𝑝 (𝜅) is

where 𝐼 𝑟 (•) represents the modified Bessel function of the first kind at order 𝑟 [8,26]. Intuitively, the vMF distribution is an analogue of the Gaussian distribution on a sphere. The distribution concentrates around the mean direction 𝝁, and is more concentrated if 𝜅 is large.

Motivation. In recognition of various types of information (i.e., the label hierarchy, weak supervision, metadata, and text), we propose a probabilistic generative process to characterize these heterogeneous signals. By adopting the vMF distribution, our generative process is defined in a spherical space (i.e., S 𝑝-1 ) instead of a Euclidean one (i.e., R 𝑝 ). The motivation of doing so is inspired by two lines of related studies. First, the vMF distribution has demonstrated its effectiveness in several tasks including hierarchical clustering [13] and text sequence generation [19], which bears some similarities with our hierarchical representation learning and data augmentation modules, respectively. Second, normalizing word embedding vectors onto a sphere [20,49] or directly training embeddings in a spherical space [29] can better capture directional similarities, which is proved to be beneficial to tasks like semantic similarity search and document classification [29]. As shown in Figure 2(a), the process can be divided into the following steps.

Parent Label → Child Label. Given the label hierarchy T , the semantics of each child category 𝑙 𝑐 should largely depend on the semantics of its parent category 𝑙 𝑝 . In other words, if we use an embedding vector to represent each category, the child embedding 𝒍 𝑐 should be close to its parent embedding 𝒍 𝑝 . Inspired by the softmax function used in word embedding [32] and network embedding [40], we define the generation probability as

Here, one needs to notice that 𝑝 (•|𝑙 𝑝 ) describes a probability distribution of an embedding vector 𝒍 𝑐 . Since 𝒍 𝑐 can be any point in a continuous embedding space, 𝑝 (•|𝑙 𝑝 ) should also be a continuous distribution. Therefore, we cannot fully borrow the softmax function which defines a discrete choice from finite candidates. Instead, the vMF distribution introduced in Section 2.2 is a proper tool here. Assume all category embedding vectors reside on a sphere S 𝑝-1 . Recall Eq. ( 1). If we set 𝜅 = 1, then 𝑓 vMF (𝒙 |𝝁, 1) = 𝑐 𝑝 (1) exp(cos(𝒙, 𝝁)) = 𝑐 𝑝 (1) exp(𝒙 𝑇 𝝁). (4) By comparing Eq. ( 4) with Eq. (3), we know that the vMF distribution meets our requirement of the generation probability (because 𝑐 𝑝 (1) is a constant given the dimension 𝑝). Thus, we rewrite Eq. (3) as

(5) Given the conditional probability, the category embedding can be generated in a top-down manner: we first determine the embedding of T 's root, and then proceed to the next level to generate embeddings of categories at that level. The embeddings of child labels are supposed to be surrounding the embedding of their parent label, and, according to the property of the vMF distribution, we have E[𝒍 𝑐 |𝒍 𝑝 ] = 𝒍 𝑝 . This process can be repeated until we reach the leaf categories.

Label & Metadata → Document. Our second step generates document embeddings according to their metadata and label information (if applicable). Consider how we humans write a document given a topic category (or a tag/product). We need to first have a general idea of the article, which can be represented as the document embedding 𝒅. To impose the coherence between 𝒅 and its category/metadata information, we assume

where 𝑙 𝑑 is the category label of 𝑑 (here, we are referring to the leaf label of 𝑑); 𝑚 𝑑 is a metadata instance of 𝑑; the corresponding embedding variables are in bold. Following the derivation above, by assuming all embedding vectors reside on a sphere, we can rewrite Eq. ( 6) as

One unique problem we need to address here is that label and metadata information can be missing. For example, due to supervision scarcity, only a small proportion of documents in D have label information. Besides, a GitHub repository may have no tags. In general, suppose 𝑉 instances 𝑧 1 , ..., 𝑧 𝑉 are missing in {𝑙 𝑑 } ∪ M 𝑑 .

In this case, we assume their embeddings 𝒛 1 , ..., 𝒛 𝑉 can be any vector on the sphere with equal probability (i.e., 𝒛 1 , ...,

.., 𝑧 𝑉 } be the remaining observable instances. Therefore, Eq. ( 7) becomes

The second-to-last step holds because ∫ S 𝑝-1 𝑐 𝑝 (1) exp(𝒅 𝑇 𝒛 𝑖 ) d𝒛 𝑖 ≡ 1 (i.e., the probability density function of a vMF distribution integrates to 1 over the whole sphere). Eq. ( 8) tells us that, when some label/metadata information is missing, we only need to consider the remaining observable part (i.e., O 𝑑 ) in the conditional probability. Document → Word. Based on the document embedding 𝒅, a sequence of words 𝑤 1 , ..., 𝑤 𝑛 is then generated to describe the overall semantics of 𝑑. Similar to Eqs. ( 3) and ( 6), we assume the probability of word 𝑤 𝑖 appearing in document 𝑑 to be

Again, we rewrite the probability using a vMF distribution: 5), ( 8) and ( 10) together fully specify our proposed hierarchical generative process.

Guided by the hierarchical generative process, we aim to jointly learn the embeddings of all labels, metadata instances, documents and words. Our learning objective can be divided into two parts: one for modeling the label hierarchy information (corresponding to the first step in our generative process), and the other for modeling corpus statistics and metadata information (corresponding to the second and third steps).

Objective for Label Hierarchy. Given a parent-child label pair (𝑙 𝑝 , 𝑙 𝑐 ), our goal is to maximize the log-likelihood log 𝑝 (𝑙 𝑐 |𝑙 𝑝 ) during embedding learning. Inspired by studies on knowledge graph embedding [3,42], we adopt the following margin-based ranking objective:

where 𝛾 > 0 is a margin hyperparameter and (𝑙 ′ 𝑝 , 𝑙 ′ 𝑐 ) is a negative training sample. Given the positive pair (𝑙 𝑝 , 𝑙 𝑐 ), we generate the negative pair by replacing the parent label with any other label in T while keeping the child label. According to this corruption strategy, our objective can be defined as

Based on the definition of 𝑝 (𝑙 𝑐 |𝑙 𝑝 ) in Eq. ( 5), we have

Therefore,

Objective for Metadata and Corpus. Given the label and metadata information of a document 𝑑, the log-likelihood of observing the document and its word sequence is

According to Eqs. ( 8) and (10), it can be written as

Again, we adopt the margin-based ranking objective to push the negative pairs distant enough from the positive pairs (𝑧, 𝑑) and (𝑑, 𝑤 𝑖 ). Formally,

Following the derivation in Eq. ( 13), we have

To summarize, our joint representation learning step can be formulated as the following optimization problem:

Here, M is the set of metadata instances appearing in the dataset, and V is the vocabulary (i.e., the set of words).

Optimization. Given the constraint that all embedding vectors live on a sphere, directly performing stochastic gradient descent in the Euclidean space is not an effective way to optimize the objective. Instead, following recent studies on hyperbolic embedding [33,34] and spherical embedding [29,45], we apply the Riemannian gradient method [2] to learn the embeddings.

According to [29], on a unit sphere, the relationship between the Riemannian gradient ∇ 𝑅 and the Euclidean gradient ∇ 𝐸 is

Based on Eqs. ( 14) and ( 18), the Euclidean gradient of each embedding vector is easy to compute. For example, given a parent-child label pair (𝑙 𝑝 , 𝑙 𝑐 ), we have

where 1(•) is the indicator function. Then, following Eq. ( 20), we can obtain the Riemannian gradient ∇ 𝑅 J (𝒍 𝑐 ), ∇ 𝑅 J (𝒍 𝑝 ) and ∇ 𝑅 J (𝒍 ′ 𝑝 ). The gradient of other embedding vectors can be calculated in a similar way. After knowing the Riemannian gradient, we update the parameters in the following form [2,29]:

where 𝛼 𝑡 is the learning rate at step 𝑡. Intuitively, this update can be viewed as an addition along the Riemannian gradient direction followed by a projection onto the unit sphere. Since computing the gradient requires enumerating all labels or documents, in our actual computation, we adopt the negative sampling strategy [3,32] to accelerate this process. To be specific, for each positive pair (𝑙 𝑝 , 𝑙 𝑐 ), (𝑧, 𝑑) or (𝑑, 𝑤 𝑖 ), we sample 5 negative pairs to estimate the gradient.

Now we proceed to tackle the limited training data challenge. After representation learning, we have obtained the embedding vectors of all observed labels, metadata instances, documents and words. Using these embeddings, we can generate synthesized training documents for each category. Formally, given a category 𝑙, we aim to create a document set D * 𝑙 to augment its original training set Algorithm 1 Hierarchical Data Augmentation if 𝑙 is a leaf category then 5:

for 𝑖 = 1 to 𝛽 do 7:

Generate 𝒅 * according to Eq. ( 24)

8:

Generate 𝒘 * 1 𝒘 * 2 ...𝒘 * 𝑛 according to Eq. ( 25) for 𝑥 ∈ Child(𝑙) do Data Augmentation for a Leaf Category. Recall Figure 2(a). If 𝑙 is a leaf category, it will directly participate in the generation of document embeddings according to Eq. ( 8). If we denote a synthesized document as 𝑑 * (to distinguish it from a "real" document 𝑑), the generation probability will become

Note that we do not have any metadata information of the synthesized document. Therefore, the only observable variable here in {𝑙 } ∪ M 𝑑 * is the label 𝑙. Also, it is not necessary to generate documents associated with a certain metadata instance because our goal is to predict the label, instead of the metadata information, of a document. Substituting O 𝑑 * = {𝑙 } into Eq. ( 23), we have

After getting the document embedding 𝒅 * , we continue our generative process to create a sequence of words describing 𝑑 * . In principle, we should follow Eq. ( 10) and sample words from 𝑓 vMF (•|𝒅 * , 1). However, in practice, we expect that the generated word embedding 𝒘 * can be mapped back to a concrete word that has occurred in our vocabulary V. In [19], Kumar and Tsvetkov propose a way that first samples an vector 𝒆 from a continuous distribution, and then find the nearest neighbor of 𝒆 in the embedding space among all 𝑤 ∈ V. In contrast, we directly require the generated word to reside in the neighborhood of 𝒅 * in our joint embedding space. In this way, Eq. ( 10) degenerates to a discrete softmax function:

where V 𝑁 (𝑑) is the set of top-𝑁 nearest neighbors of 𝒅 * in the embedding space among all 𝑤 ∈ V. Using Eq. ( 25) repeatedly, we can obtain a sequence of words 𝑤 * 1 𝑤 * 2 ...𝑤 * 𝑛 as the content of 𝑑 * . Data Augmentation for a Non-Leaf Category. In our hierarchical generative process, a non-leaf label does not directly generate documents. Instead, it determines the embeddings of its children, and the child categories further determine its lower-level descendants. Formally, given a non-leaf label 𝑙, let T 𝑙 ⊆ T be the subtree with root 𝑙 and Leaf(T 𝑙 ) be the leaf descendants of 𝑙. Then,

After the joint representation learning step, all label embeddings have been fixed.

If we adopt a bottom-up order to generate D * 𝑙 , this can also be written in a recursive way:

In practice, some non-leaf categories have more children than their siblings have. Thus, if we directly apply Eq. ( 29 𝑙 has a sequence of words, whose embedding vectors are already learned in joint representation learning. Now, based on the augmented training data, we need to train a hierarchical text classifier. Hierarchical text classification has been extensively studied for decades with many effective models proposed (see Section 5 for a detailed discussion). The main goal of this paper is to tackle signal heterogeneity and supervision scarcity instead of proposing a novel neural architecture for hierarchical classification. Therefore, we simply adopt the top-down training strategy [7,18,22] that classifies documents at the top layer and then propagates the results to the next layer until the leaves. For each non-leaf category 𝑙 ∈ T , we need to train a flat classifier that assigns documents to its child classes Child(𝑙) for more fine-grained predictions. We adopt Kim-CNN [17], a popular convolutional architecture, as our flat text classifier. The embeddings {𝒘 : 𝑤 ∈ V} trained in the joint representation learning step are used as initialization embeddings of the input layer. The model adopted here can be easily improved by replacing Kim-CNN with a more advanced flat classifier (e.g., [41,51]) or by adding a hierarchical regularization (e.g., [12,14,35]). However, since we would like to demonstrate the contribution of our embedding and augmentation modules when comparing with baselines, we keep the classifier as simple as possible.

Datasets. We use three datasets from different domains. 3 The statistics of the datasets can be found in Table 1.

• GitHub [54]. This dataset is collected by paperswithcode.com.

It contains 1,596 GitHub repositories implementing state-of-theart machine learning algorithms for different tasks (e.g., object detection, question answering, speech synthesis, etc.). The tasks form a taxonomy which is viewed as the label hierarchy. • ArXiv. This dataset is crawled from arXiv.org. There are 5 level-1 categories (i.e., cs, math, physics, q-bio, and q-fin) and 88 level-2 categories. For each category, we randomly sample 300 papers to constitute the dataset. • Amazon [27]. This dataset is a large collection of Amazon product reviews. The label of each review is its product category (e.g., automotive, car care, beauty, skin care, etc.). These categories are organized into a catalog taxonomy [25]. We select 147 large leaf categories and sample 1,000 reviews for each of them. According to our weakly supervised setting, we use 5 documents per leaf category for training and all the other documents for testing.

Baseline Methods. We evaluate the performance of HiMeCat against the following classification/embedding algorithms:

• HierSVM [7] is a supervised hierarchical classification method.

It adopts a top-down classification strategy and trains an SVM for each non-leaf node to distinguish its categories. • WeSHClass [30] is a weakly supervised hierarchical classification method. It adopts LSTM to generate data for pre-training and iteratively refines results based on the hierarchy. • PCEM [48] is a weakly supervised hierarchical classification method. It proposes a path-cost sensitive hierarchical classifier and applies an EM technique to utilize the unlabeled data. • HiGitClass [54] is a weakly supervised hierarchical classification method. It uses heterogeneous network embedding to encode relationships between labels, text and metadata and then leverages WeSHClass to train a hierarchical classifier. • MetaCat [52] is a weakly supervised flat classification method.

It leverages metadata information but cannot utilize the label hierarchy. We use it to directly classify documents to the leaf layer and then infer higher-level labels using the hierarchy. • Metapath2vec [6] is a metadata-aware embedding method. It models the proximity between heterogeneous elements through meta-path guided random walks.

• Poincaré [33] is a hierarchy-aware embedding method. It preserves the tree structure of embedded elements by putting them into a hyperbolic space. • Pretrained BERT [5] is a benchmark language model that provides contextualized word representations. For text classification, following [5], we add a classification layer upon the [CLS] token of the last transformer layer and fine-tune the model using labeled documents. After predicting the leaf category of a document, we infer its higher-level labels using the hierarchy.

For Metapath2vec and Poincaré, inspired by our hierarchical generative process, we construct a graph with edges (𝑙 𝑝 , 𝑙 𝑐 ), (𝑙, 𝑑), (𝑚, 𝑑) and (𝑑, 𝑤) and perform node embedding on the graph. (We use four meta-paths 𝑙-𝑙, 𝑑-𝑙-𝑑, 𝑑-𝑚-𝑑 and 𝑑-𝑤-𝑑 for Metapath2vec.) Then, to apply these two baselines in weakly supervised hierarchical classification, we use the learned embeddings to train a WeSHClass classifier (by replacing its word2vec embedding module). Since some baselines only utilize text information, for each document, we append its metadata instances to the end of its text sequence so that they can exploit these signals. 𝑇 𝑃 𝑙 +𝐹 𝑁 𝑙 . Following previous works [30,48], we also calculate the Overall Micro/Macro F1, which is defined accordingly on T \{ROOT} instead of Leaf(T ).

Table 2 shows the categorization performance of compared algorithms on the three datasets. We repeat each experiment five times and report the average F1 scores. To measure statistical significance, we conduct a two-tailed paired t-test to compare HiMeCat and each baseline approach. The significance level ("p-value < 0.01" or "p-value < 0.05") of each result is also marked in Table 2.

We observe that: (1) HiMeCat consistently outperforms the compared baseline methods on all three datasets. In most cases, the performance gap is statistically significant. The improvement of HiMeCat over the best baseline is 8.3%, 13.0% and 47.5% on GitHub, ArXiv and Amazon, respectively. (2) For the three metadata-aware text categorization methods, HiMeCat performs better than HiGit-Class and MetaCat, indicating the importance of considering label hierarchy and incorporating it into representation learning. (3) Although Metapath2vec and Poincaré are assumed to be strong tools for metadata and hierarchy embedding respectively, they do not achieve competitive performance when we need to jointly embed these signals. This observation also emphasizes the contribution of our joint representation learning module. (4) Despite the great success of BERT in many supervised NLP tasks, it does not perform well in our problem settings. There may be two possible reasons. First, directly using BERT for text classification does not consider metadata and hierarchy information. Second, the small set of training data may not be enough to fine-tune BERT.

Two key steps, representation learning and data augmentation, are proposed in our HiMeCat framework. Now we validate their contribution to the categorization performance through ablation analysis. We start from the representation learning module.

Hierarchy and Metadata. In our joint hierarchy-metadata-text representation learning module, to check whether the label hierarchy and various types of metadata are useful, we create the following ablation versions of HiMeCat.

• No-Hierarchy does not consider parent-child relationships in representation learning. Formally, it ignores J 𝐿 in Eq. ( 19) and only maximizes J 𝑀𝐶 . • No-Metadata does not consider metadata-document relationships in representation learning. Formally, in Eq. ( 18), we will have

Moreover, instead of ignoring all metadata instances, we can overlook one specific type of metadata. This will yield ablations including No-User/Author, No-Tag and No-Product. Table 3 depicts the comparison between the full HiMeCat model and its ablations on the three datasets. (Due to space limitations, we only show Overall F1 scores. Similar observations can be drawn from Leaf F1 scores.) We find that: (1) The full model outperforms No-Hierarchy, which validates our claim that modeling label dependencies and correlation is beneficial to hierarchical categorization. Moreover, when the dataset has a larger label hierarchy, the gap between HiMeCat and No-Hierarchy is more statistically significant. The average improvement of HiMeCat over No-Hierarchy on the six metrics is 5.0%. (2) The full model outperforms No-Metadata, which validates our claim that incorporating metadata signals is

When studying different types of metadata separately, we find all of them play a positive role on our datasets. The contribution of metadata is larger than that of label hierarchy, possibly because metadata information is denser than hierarchy information. That being said, each document has its own metadata, while the whole corpus only has one label hierarchy.

Dimension. The embedding dimension 𝑝 is an important parameter of the representation learning module. To check the sensitivity of 𝑝, we plot Overall F1 scores of HiMeCat with 𝑝 = 10, 50, 100, 200 and 300 in Figure 3. We can observe that: if 𝑝 is too small, the embedding vectors cannot sufficiently capture heterogeneous signals in the dataset; if 𝑝 is too large, we may face overfitting problems, especially under weak supervision. According to the results, setting 𝑝 between 100 and 200 is reasonable on our datasets.

Now we proceed to the hierarchical data augmentation module.

We need to answer two questions here: (1) Can data augmentation mitigate the label scarcity problem? (2) How many synthesized training samples should we create for each category? To investigate these two questions, we show the performance of HiMeCat with 𝛽 = 0, 50, 100, 500, 1000 and 2000 in Figure 4. We observe that, when comparing 𝛽 = 500 with 𝛽 = 0 (i.e., we do not generate any synthesized training data and use real training data only), the performance boost is quite evident. For example, the average absolute improvement of Overall F1 scores on the three datasets is 21.4%. Thus, we validate our claim that the hierarchical data augmentation module can help improve the classification accuracy under weak supervision. Meanwhile, the performance gap between 𝛽 = 2000 with 𝛽 = 500 is quite subtle. In fact, the average absolute gap of Overall F1 scores on the three datasets becomes 0.1%. Also, too many synthesized training samples will make the training process inefficient. Seeking a balance, we believe setting 𝛽 = 500 is an appropriate choice (and we do set 𝛽 = 500 in previous experiments).

In order to show how the label hierarchy is modeled and how the labels are distributed in our joint embedding space, we plot label embeddings in Figure 5 after reducing their dimensions using t-SNE [23]. Labels are denoted as stars, and names of all level-1 categories and the root are shown. We have two observations: (1) The tree structure of the label hierarchy is well preserved in our embedding space. Child categories center around their parent categories. (2) In the ArXiv embedding space, "q-fin" (Quantitative Finance) is embedded near the root, and its children are embedded closer to "math" and "physics" than they are to "cs" and "q-bio" (Quantitative Biology). This is reasonable because quantitative finance is a quite interdisciplinary area and has many overlap with mathematics and physics. This observation shows that our joint representation learning module not only considers the label hierarchy but also captures semantics from text and/or metadata information.

Hierarchical Text Classification. Many efforts have been put on how to utilizing the label hierarchy to improve text classification. For example, Dumais and Chen [7] and Liu et al. [22] adopt a top-down training strategy and use SVMs to distinguish child categories of the same parent. In contrast, bottom-up classification [1] backpropagates the labels from the leaves to the top layer. Gopal and Yang [12,14] propose a recursive regularization framework encouraging the similarity between the child classifiers and the parent classifier. Peng et al. [35] extend this regularization to deep graph-CNN models. Wehrmann et al. [44] and Huang et al. [15] combine the ideas of training a local classifier per level and optimizing the global classification results to mitigate exposure bias. The global structure of hierarchies is also used in many other models, such as meta-learning [46], reinforcement learning [24] and tree/graph based neural networks [55]. However, these fully supervised methods rely heavily on the amount of human-labeled training data. Under weakly supervised settings, hierarchical dataless classification [36] jointly embeds class labels and documents using Explicit Semantic Analysis; WeSHClass [30] models topic semantics in the word2vec embedding space and applies a self-training scheme; PCEM [48] introduces a path-cost sensitive classifier for semi-supervised hierarchical classification; HierCon [21] projects documents and taxonomy categories into a common concept space and calculates their fine-grained similarity. However, these studies are not concerned with metadata information.

Metadata-Aware Text Classification. Several previous studies try to incorporate metadata information into a text classifier. For example, Ghani et al. [11] use HTML meta tags to help hypertext classification; Steyvers et al. [37] leverage author information in topic classification; Tang et al. [39] learn user and product representations for sentiment analysis; Zhang et al. [53] employ user biography data for tweet localization. Chen et al. [4] encode user mobility data and social network information for joint time and location prediction; Kim et al. [16] propose a general framework to inject categorical metadata signal into a deep text classifier. However, these models assume a fully supervised setting. Zhang et al. [54] present a weakly supervised hierarchical classification approach to classify GitHub repositories. Later, they also propose a flat metadata-aware text categorization framework [52]. Mekala et al. [28] explore to incorporate metadata as additional supervision for text classification with seed words only. However, in these studies, the label hierarchy is not leveraged in the embedding space.

Tree and Metadata Embedding. Recent studies on non-Euclidean embedding models, such as Poincaré [33], Lorentz [34], hyperbolic cones [10] and spherical tree embedding [31], consider to preserve a tree structure in the embedding space. Along another line of work, heterogeneous network embedding algorithms [6,9,50] are widely used to learn representations of metadata. HHNE [43] further considers to perform heterogeneous network embedding in a hyperbolic space. Despite their respective success in capturing hierarchy and metadata information, to the best of our knowledge, there lacks a framework which allows simultaneous modeling of tree-structure dependencies, text semantics and metadata signals.

We present HiMeCat, an embedding-based generative framework for hierarchical metadata-aware document categorization under weak supervision. The framework is featured by a joint hierarchymetadata-text representation learning module and a hierarchical data augmentation module. We propose a generative process in the spherical space to guide the design of both modules. Through experiments on three datasets from different domains, we show the superiority of HiMeCat towards competitive baselines in our task. We also conduct ablation studies to validate the contribution of our proposed embedding and augmentation modules. Interesting future work include: (1) discovering new categories from the unlabeled dataset and put them into the existing hierarchy and (2) integrating different forms of weak supervision (e.g., annotated documents and class-related keywords) in hierarchical text classification.