Recent research has made rapid progress on extremely weak-supervised text classification (XWS-TC) (Wang et al., 2023), limiting the supervision to a brief natural-language description without any annotated samples. For example, text mining-based XWS-TC (Meng et al., 2020;Wang et al., 2021;Shen et al., 2021;Mekala et al., 2022;Zhao et al., 2023;Dong et al., 2023a) takes only class names or seed words from humans and discovers potential in-class texts following designated heuristics. Minority classes are arguably the most challenging part of XWS-TC. The class distribution I believe in luck, and when luck is not on my side, I feel beaten and sometimes upset.
_ believe __________ when luck _______________ feel _____________________ Minority class "Surprised"
into "Surprised"
I can't believe it when luck suddenly changes, and I feel completely astonished.
Stage 1: Potential Text Mining Gather texts with beneficial components to appear in the grafted results.
Stage 2: Template Creation Mask the components that do not contribute to grafting. Framework Mining? Train Data Data Quality In-Distribution Text Mining Text Raw Noisy Yes Data Synthesis None Generated Clean Hardly Text Grafting (ours) Template Grafted Clean Mostly
Table 1: High-level comparison among three discussed XWS-TC frameworks.
in real-world datasets is often a long-tailed distribution (Zhang et al., 2023), with a non-trivial number of minority classes. These minority classes have a very small number of documents in the raw corpus, therefore, it is difficult to locate the right documents by mining-based methods, leading to noisy pseudo-labels. Under extreme circumstances, the mining-based methods may end up with no sample for minority classes.
A potential way to address this issue is data synthesis-based XWS-TC (Ye et al., 2022a,b;Peng and Shang, 2024), which hopes to generate inclass texts by prompting large language models (LLM) (Brown et al., 2020;OpenAI, 2023;Touvron et al., 2023a,b;Meta, 2024;Mesnard et al., 2024;OpenAI, 2024) with class names or defini-tions. However, such synthesized texts may follow a distribution different from the corpus where the text classifier will be later applied (Mitchell et al., 2023), which makes the learned text classifier outof-distribution, leading to poor performance.
This paper combines the advantages of miningbased and synthesis-based frameworks to propose a new framework, text grafting, which aims to obtain clean and near-distribution weak supervision for minority classes. As specified in Figure 1, text grafting incorporates three stages: (1) Potential Text Mining gathers raw texts with beneficial components to synthesize in-class texts for the target minority class. (2) Template Creation forms templates by masking the components that do not contribute to the in-class text synthesis. (3) Template Filling synthesizes in-class texts by filling in the blanks. Table 1 systematically compares the weak supervision obtained by different frameworks.
To identify the words not contributing to the classification, we borrow the marginalization idea from LLM reasoning (Holtzman et al., 2021). We get the probability logit of each word in the raw text by instructing LLMs (relatively small, specifically Gemma (Mesnard et al., 2024)) to generate with or without the in-class as a requirement. The difference between the two logits represents the potential of each word to appear in the grafted text. As only words with high potential will be left, we use the average potential of top-K% words to represent the text potential score. The bottom-(100 → K)% words will be masked to form the template for data synthesis. We rank the templates by their potential scores and select top-T % templates for the last template-filling stage. Finally, these selected templates are filled by prompting a state-of-the-art LLM, GPT-4o (OpenAI, 2024).
We compare the three mentioned frameworks on various raw corpora to classify different minority classes. The experiment results show text grafting can outperform state-of-the-art text mining and dataset synthesis methods. The ablation study verifies that all stages and the intermediate template contribute to the success of our proposed text grafting. The mask-and-filling scenario also shows its advantage over simple in-context generation, since it forces the LLM to incorporate components from the raw texts. We also involve an extreme situation where the target class does not appear in the raw corpus completely. Remarkably, text grafting shows its robustness to this extreme situation, indicating its applicability does not require the target class to appear in the raw corpus. This enables text grafting to work on a very small corpus which boosts efficiency.
Furthermore, we analyze and discuss the property of text grafting. We apply principal component analysis to visualize that the drafted texts are indeed near in-distribution. We also find the grafted texts are near-distribution enough that we do not need to synthesize negative samples as in traditional data synthesis, which reduces the cost. We also conduct a comprehensive hyperparameter analysis of our method. Interestingly, we found that The mask ratio is searched to be better set to a high value like 0.75 and the mined template number can be as small as 200. These case studies explore the advantages of text grafting in distribution approximation and its failure when the raw texts are near the distribution of LLM generation.
We summarize our contributions as follows, • We propose a novel XWS-TC framework for minority classes, text grafting, combining the indistribution advantage of text mining and the inclass advantage of data synthesis. • We implement text grafting following the marginalization idea from LLM reasoning, utilizing the probability logits for template mining and masking. • We provide comprehensive analysis and case studies to show the strength, property, and possible failure of text grafting.foot_0
Extremely Weak-Supervised Text Classification (XWS-TC) needs only minimal human guidance to label the text, such as a few rules by human experts that match the text to the labels (Wang et al., 2023).
Mainstream XWS-TC methods can be divided into two categories: Text Mining and Data Synthesis.
Text Mining is a fundamentak task (Han and Kamber, 2000) for natural language processing. In XWS-TC, the text miner follows high-level rules from humans to annotate raw texts, which are used to train the text classifier. A mainstream rule is whether a seed word appears in the raw text (Mekala and Shang, 2020;Meng et al., 2020;Wang et al., 2021), categorized as seed methods.
Another mining way is to prompt language models for logits that reflect the probability of texts falling in classes (Brown et al., 2020), which can be calibrated by several techniques (Holtzman et al., 2021;Zhao et al., 2021;Han et al., 2023). The strong performance of existing text mining methods is highly dependent on the precision of the class-indicative rules (Dong et al., 2023a), which is hard to maintain for minority classes.
Data Synthesis (He et al., 2022) addresses the precision degradation in text mining by directly prompting LLMs with the label names to generate in-class texts (Ye et al., 2022a;Peng and Shang, 2024). With the powerful generative ability of LLMs, the synthesized texts are generally clean (in-class) for training strong classifiers. However, synthesized texts hold LLM-specific patterns, discovered by LLM-generated text detectors (Mitchell et al., 2023;Wu et al., 2023). This pattern is hard to be eliminated even with in-context learning (Koike et al., 2024). Thus, synthesized texts are generally out-of-domain and consequently fine-tune a weaker classifier on the test set.
Minority Classes widely appear in classification datasets as a result of long-tailed distribution (Zhang et al., 2023;Henning et al., 2023). For minority classes with supervised annotations, techniques like re-sampling (Shen et al., 2016;Pouyanfar et al., 2018;Tepper et al., 2020) and data augmentation (Wei and Zou, 2019;Juuti et al., 2020;Tian et al., 2021;Chen et al., 2021). However, these methods are applied to unbalanced annotations, which are unavailable under XWS.
Counterfactual Augmentation refers to generating annotated data out of the dataset or raw corpus. Different from regular augmentation, counterfactual augmentation changes the reference, e.g., label flipping (Zhou et al., 2022;Peng et al., 2023).
Counterfactual augmentation is also applied for text-to-text tasks like translation (Liu et al., 2021) or summarization (Rajagopal et al., 2022). Counterfactual augmentation shares the same requirement for known reference as regular augmentation. This paper explores a counterfactual augmentation method for unannotated raw text under XWS.
3 Text Grafting f (X) that discerns a text falling in c or not. We denote the j-th word in the i-th text of the raw corpus as x (i,j) .
Text Mining gathers in-class texts with highlevel rules g(X) that can precisely assign X to target class c. Example rules include whether X contains words indicating c (seed words) (Dong et al., 2023a) or X has top confidence to be in c by prompting LLMs (Brown et al., 2020) among
|D| is combined with some randomly sampled negative texts (due to the scarcity of c) to train f (•).
However, text miners fail in minority classes due to their low proportion in the raw corpus. By running a state-of-the-art text mining method (Dong et al., 2023a) on AG-News (Zhang et al., 2015) with class name proportion modified by sampling, we observe the mining precision drops sharply with the decrease of proportion, presented in Figure 2. Another concern is the class might be too minor that even no ground truth can be mined from the raw corpus, limiting the precision to 0% no matter how intuitive the mining rule is.
Data Synthesis does not annotate raw texts for classifier fine-tuning but directly prompts LLMs to generate in-class texts (X → ↑ LLM(I c )), where I c is an instruction to write a text in class c. With the strong capability of state-of-the-art LLMs (OpenAI, 2024;Meta, 2024), the generated X → are highly confident to fall in class. Another advantage of data synthesis is the ability of LLMs to generate negative samples (Ye et al., 2022a;Peng and Shang, 2024). However, synthesized texts consist of patterns different from other sources (Mitchell et al., 2023), which indicates classifiers f (•) fine-tuned by synthesized texts are out-of-domain, consequently weaker in the classification task.
As depicted in Figure 3, our text grafting is a hybrid method that combines the strengths of text mining and data synthesis. The core observation is that out-of-class texts can contain useful components for writing in-class texts. The text mining stage of text grafting aims to discover these potential components and formalize them as templates. In the data synthesis stage, the templates are filled by LLMs to produce in-class texts. With components from both raw texts and synthesis, the grafted texts are both in-class and near-distribution, which are supposed to fine-tune a better classifier than only text mining or data synthesis.
In detail, the text mining stage includes Potential Text Mining and Template Creation, while in the data synthesis stage we conduct Template Filling. The text mining stage requires relatively small open-source LLMs with higher efficiency and accessible logits. Template Filling can utilize stateof-the-art LLMs even with API accessibility.
Potential Text Mining discovers texts with potential components to appear in the grafted texts. We evaluate the potential of each word x (i,j) in the raw text X (i) with regularized logits prompted from LLMs following the regularization idea in DC-PMI (Holtzman et al., 2021). The potential !p (i,j) for x (i,j) is defined as the difference between the probability logit of x (i,j) prompted by an instruction with the class name (I c ) and an instruction for regularization (I r ). The difference can also be viewed as the probability of x (i,j) raised by incorporating the class name c into the instruction.
The words with top-K% !p among the words in text X i will remain in the template. Thus, the average of their !p represents the potential (!P i ) of the template created based on X i . As we are mining potential templates rather than directly inclass texts, the mining rate K% can be much larger than text mining.
Then the texts are ranked by their grafting potential !P and texts with top-N % potential are mined to create the templates. Template Creation simply masks the words with bottom-(100 → K)% potential !p by blank tokens "_" and uses the top-K% as template part. Text X i is thus converted to template T i , which is prepared for LLMs to fill in during the data synthesis stage. As the example in Figure 3, the components with the top potential to be in a grafted "Surprised" remain in the template such as "believe", "when luck", "feel". These components support the data synthesis to better write an in-class text while keeping the style in distribution with the writing structure from the raw corpus.
Template Filling prompts an LLM to fill in the blanks in T , which produces a grafted text that generally falls in the target class c. Referring to the example in Figure 3, the LLM well utilizes the writing structure in the template and fills in the blanks to produce the in-class text. As the template keeps the writing structure of the raw corpus, the grafted text is quite similar to the original one but flipped into the target minority class. Specific prompts in these stages are shown in Table 2, where the label and distribution information is filled to support the text grafting.
Datasets We take several minority classes from popular text classification datasets to evaluate the performance of different XWS-TC methods on minority classes. We include 1) TweetEval (Barbieri et al., 2020) and Emotion (Saravia et al., 2018), which contain minority emotion classes "Optimism" (8.9%) and "Surprised" (3.6%); 2) 20 News (Lang, 1995), which contains minority news topic "Religion" (3.3%) and "Politics" (4.1%); 3) BigPatent (Sharma et al., 2019), which contains minority patent class "Mechanical Engineering" (7.0%). The raw corpus is down-sampled to 10, 000 samples to improve experiment efficiency and save budget costs. We use the F1 score as the metric for evaluation.
I believe in luck , and when luck is not on my side , I feel beaten and sometimes upset . Instruction for Regularization (I r ): Write a sentence. Instruction with Class Name (I c ): Write a **surprised ** sentence. P|I c P|I r 4.0 6.5 9.0 7.5 7.0 4.5 8.0 9.5 9.0 4.5 5.5 6.5 9.0 9.5 8.0 8.0 2.0 6.5 5.0 3.5 9.0 ΔP T 3.5 2.5 9.0 6.0 7.5 5.5 5.0 6.5 9.0 5.0 6.5 7.0 9.0 9.5 9.0 5.0 5.0 6.0 5.5 7.5 9.5 -0.5 4.0 0.0 1.5 -0.5 -1.0 3.0 3.0 0.0 -0.5 -1.0 -0.5 0.0 0.0 -1.0 3.0 -3.0 0.5 -0.5 -4.0 -0.5 _ believe _______________________________ when luck _______________________________________________________________ feel ________________________________________________ 4.0 3.0 3.0 3.0 S Average 3.75
Raw Text: I believe in luck, and when luck is not on my side, I feel beaten and sometimes upset. Template: _ believe _________ when luck _______________ feel _____________ Score: 3.75
Raw Text: I never had that sense of belonging anywhere and where if anywhere is anyone supposed to belong and feel accepted. Template: _ never had that ____ of ________________________________________________________________ Score: 3.50
Raw Text: I really remember is feeling wonderful in the oatmeal bath. Template: _ really remember_______________________ Score: 3.00
Top ΔP
_ believe _________ when luck _______________ feel _____________ I can't believe it when luck suddenly changes, and I feel completely astonished.
Figure 3: The overview of text grafting with the minority class "Surprised" in the Emotion dataset as an example. Text grafting includes two stages: 1) Text (Template) Mining: Create scored templates and select the ones with the top scores. 2) Data Synthesis: Prompt the LLM to fill in the templates to synthesize in-class texts.
Baselines We include various text mining and data synthesis methods as the baselines for comparison to illustrate the advantage of our text grafting.
Text mining methods include, • Prompting Confidence (Brown et al., 2020), which is a prompting method that directly queries an LLM whether the text falls in the target minority class, and uses the probability logit of answering "yes" for ranking. Considering the class minority, the mining rate is set to 1%. • Debiased Seed Word (Dong et al., 2023a), which is the current state-of-the-art XWS-TC method. This method uses a seed word (the same as the label name) to match the target minority class and then drops the seed word from the context to eliminate spurious correlation. Then the texts are filtered by text selection (Mekala et al., 2022) to produce the final mined texts. Data synthesis methods include, • ZeroGen (Ye et al., 2022a), which directly prompts the LLM to synthesize texts in or out of the target minority class. • In-Context Generation (Dong et al., 2023b), which uses raw texts as the in-context examples to generate texts with a similar writing style as the raw corpus. • Incubator (Peng and Shang, 2024), which uses instruction-tuned LLMs and in-context learning based on annotated instruction-to-dataset samples to generate data points for fine-tuning.
All text synthesis methods synthesize 1000 texts as positive (in the target minority class) or negative samples (out of the target minority class, 2000 in total).
The LLM used for text mining is a popular and advanced open-source LLM, Gemma (Mesnard et al., 2024) (gemma-1.1-7b-it) with accessible possibility logits. The LLM used for data synthesis is the state-of-the-art LLM, GPT-4o (OpenAI, 2024).
Grafting Hyperparameters The mining rates of our text grafter are set to 25% (K%) for potential components in templates and 10% (N %) for potential templates. Thus, the synthesized data number is less than 1000, not more than the data number from pure data synthesis.
We fine-tune a RoBERTa-Large (Liu et al., 2019) as the classifier with the AdamW (Loshchilov and Hutter, 2019) as the optimizer whose learning rate is initialized to 1 ↓ 10 ↑5 . The classifier is fine-tuned by 10 epochs with batch size 8 and 20% training data are split for validation to select the best-performing checkpoint. All the experiment results are achieved by an average of 5 runs. The two stages in text grafting apply the same LLM as text mining and data synthesis.
Dataset TWEET PATENT EMOTION 20NEWS Average Distribution Tweet Patent Tweet News Minority Class Optimism Mechanical Surprised Religion Politics Class Proportion 8.9% 7.0% 3.6% 3.3% 4.1% Supervised 45.88 34.30 32.28 24.10 32.27 35.14 Text Mining (TM) Prompting Confidence 17.93 14.59 7.00 6.50 15.77 12.81 Debaised Seed Word 19.15 20.46 8.78 11.47 19.53 15.88 Data Synthesis (DS) ZeroGen 10.82 24.17 7.19 6.97 17.60 13.35 Incubator 22.46 20.86 7.44 23.96 24.48 19.84 In-Context Generation 16.24 24.53 22.24 21.98 24.13 21.83 TM+DS Text Grafting (Ours) 32.70 25.42 27.46 25.32 27.32 27.64 Ablation w/o Mining 26.54 16.74 24.32 17.69 15.16 20.09 w/o Synthesis (DC-PMI) 17.86 11.34 7.34 4.33 4.28 9.03 w/ Random Masking 30.11 19.07 23.37 23.57 26.65 24.55 w/ MF ↑ ICG 21.31 20.58 15.33 23.60 25.06 21.18 Zero-Occur Debaised Seed Word 0.00 17.66 5.88 8.79 20.73 10.61 In-Context Generation 18.84 23.15 19.50 20.63 24.11 21.25 Text Grafting (Ours) 30.61 25.27 31.08 26.15 25.54 27.73 Table 4: Results (Macro F1 Score) on end-to-end XWS-TC for different languages. Emotion (English) contains minority classes "Surprised" and "Love" while TNEWS (Chinese) has a minority class "Stock".
The main results from our experiments are presented in Table 3. The comparison inside text mining methods shows the advantage of the seed method over the prompt method, consistent with the findings of Wang et al.. The comparison among text synthesis methods reflects the importance of knowledge about the distribution of the corpus, as in-context generation outperforms other baselines with raw texts as an example for synthesis. Finally, text grafting outperforms all the baselines, which verifies the benefit of text grafting to produce inclass and near-distribution texts. However, there is still a significant gap between the performance of supervised classification and XWS-TC even with text grafting. This indicates the grafted texts still have differences with the raw corpus distribution for further improvement.
Table 3 also includes the ablation study results for text grafting in the Ablation columns. The first comparison focuses on the necessity of text mining and data grafting in the pipelines of text grafting. Without Mining removes the template score-based sorting and lets the LLM fill in randomly selected templates, which significantly underperforms the initial grafting. Without Synthesis does not create templates for data synthesis, but directly uses the !p averaged over all words to mine texts for finetuning, equal to DC-PMI (Holtzman et al., 2021). The result is similar to the Prompting Confidence method, which shows the limitation of text mining for minority classes. Then we emphasize the necessity of intermediate templates. With Random Masking randomly masks the mined texts instead of following the word-level potential !p, which also results in a performance drop. With Mask Filling ↔ In-Context Generation takes the mined texts as the in-context examples, which result in a similar performance as the one without mining, indicating the importance of template creation and filling. Based on these ablation results, our grafting framework is shown to be essential for achieving optimal performance by effectively combining data synthesis, text mining, and templates.
Q1: How does Text Grafting Benefit End-to-End XWS-TC? Table 4 shows how text grafting can be integrated into end-to-end XWS-TC pipelines for different languages. We include the English Emotion dataset with and "Love" as the minority classes and the Chinese TNEWS dataset (Xu et al., 2020) a minority class "Stock". For the minority classes, texts are synthesized by grafting while other classes apply the traditional debiased seed word method. The result shows text improves end-to-end XWS-TC on languages, which verifies the cross-lingual benefit of integrating text grafting into XWS-TC pipelines to handle minority classes.
Q2: What if the class proportion is 0%? In the Zero-Occur part of Table 3, we also include the discussed extreme situation when the raw corpus does not contain any text falling in the target minority class. A dramatic drop appears in the performance of text mining as there is no ground truth that any miner can get. The data synthesis and text grafting methods are robust to this change as they do not require the existence of ground truth examples. Thus, text grafting is verified to be applicable to raw corpus without the target minority class. Thus, text grafting can be based on a small subset of the corpus which might not contain the target minority class to boost efficiency.
Q3: How are grafted texts "near-distribution"?
In Figure 4, we apply semantic text embeddings (Gao et al., 2021) to represent the texts mined or synthesized by different methods. These embeddings are then reduced to 2-dimension by principal component analysis (F.R.S., 1901) for visualization. We use the "Optimism" class of the TweetEval benchmark and compare the most competitive methods (Debiased Seed Word, Incubator, Text Grafting) of different frameworks. We can observe that text mining only discovers a limited proportion of in-class texts. The synthesized texts fall into a very different domain from the raw corpus, which fine-tunes an out-of-domain classifier with limited generalizability. In contrast, the grafted texts are much more near-distribution, contributing to the performance of the fine-tuned classifier.
Q4: Is Negative Data Synthesis Necessary? For data synthesis-based methods, the synthesis of negative data is an essential stage in the pipeline, which doubles the calls for LLM to synthesize texts. In text grafting, we efficiently use the raw texts as the negative examples. Thus, we explore the necessity of negative synthesis by evaluating the performance of data synthesis (In-Context Generation) and text grafting with or without negative data synthesis with the results presented in Figure 5.
Based on the results, we observe negative data synthesis is very necessary to pure data synthesis as the performance drops dramatically by removing
________weight 221___old___________please__ senator johnson's stance on the new policy carries significant weight. the 221-page document, though old, still holds relevance in today's political climate. please stay informed and engaged. i'm looking for a singer featherweight 221 sewing machine (old, black sewing machine in black case). please contact: senator maria hernandez, a strong advocate for environmental policies, will address the upcoming climate summit in geneva. attendees can expect: Template Creation In-Context Generation Template Filling ____ shot__ 1923_ which famous wolf was shot in 1923? what mexican leader was shot dead in 1923? what bird species was declared extinct in 1914? Template Creation In-Context Generation Template Filling Strength: Able to adapt hard templates to in-class texts (Raw News Paragraph → Religion News Paragraph) Failure: Not necessary when text distribution is not special (Raw Question → Animal Question) this stage. In contrast, text grafting without negative data synthesis works even better, indicating that our text grafting can work more efficiently by reducing the effort to call LLM at double times.
We attribute this efficiency to the near-distribution property of the grafted texts, which makes the discrimination between them and the original raw texts no longer degrade to the classifying of text sources (Mitchell et al., 2023).
Q5: What mask ratio to choose? In Figure 6, we analyze the mask ratio used in text grafting. Within the considered set of mask ratios, {0.5, 0.625, 0.75, 0.875, 1.0}, the best-performing ratio is 0.75 among different datasets, the same as the setup in our experiments. We can also observe a trend of performance decrease when the mask ratio becomes away from 0.75. This indicates a too-high masking ratio will make the synthesized text deviate from the domain of raw corpus (100% leads to in-context generation). On the other hand, a too-low mask ratio will limit the synthesizer to generate in-class texts, which might cause more severe performance drops.
Q6: How many templates to mine? In Figure 7, we further analyze the necessary number of templates to train a strong classifier, which can guide the efficient application of text grafting. The result of the "surprised" class shows about 200 samples can reach the best performance, which results in about $0.2 budget for each class (OpenAI, 2024). We also present how the efficiency of text mining (Debiased Seed Word) and data synthesis (In-Context Generation) is affected by sample numbers. Text mining cannot fine-tune a well-performing classifier due to severe noise in minority class mining. Data synthesis shows a similar scaling trend as text grafting but generally underperforms text grafting.
Function Prompt TM-V1 (Ic) "Please write a with attribute <label>." TM-V1 (Ir) "Please write a <style>." TM-V2 (Ic) "Provide me with a <label> <style>." TM-V2 (Ir) "Provide me with a <style>." DS-V1 "Complete the empty fields in the template to generate a <label> <style>." DS-V2 "Complete the template by filling in the blanks to create a <label> <style>." Table 6: Results on prompt variants.
Q7: Is grafting robust to prompt templates?
We rerun the experiments in Table 3 with prompt variants to evaluate the robustness of text grafting. We apply the prompt variants in Table 5 and illustrate the averaged F1 score over the 5 minority categories in Table 6. The result verifies our text grafting framework robust to specific prompt design.
In Figure 8, we depict workflows of text grafting in comparison with in-context generation to illustrate the strength of grafting and possible failure.
Strength of text grafting is the ability of stateof-the-art LLMs to fill in hard templates as shown in the first case. While the template is not easy to be grafted into the target "Politics" class, the LLM comes up with the methodology to synthesize such a text. The text is also more similar in writing style to the original text than the in-context generation, which depicts the benefit from text grafting.
Failure of text grafting can happen when the corpus does not have a writing style very far from the way that LLMs can imitate. As shown in the second case, the LLM can synthesize the animal question without the intermediate template on the TREC corpus (Li and Roth, 2002), which reduces the necessity of text grafting. The XWS-TC of the minority class "Animal" on this corpus also shows a similar performance between data synthesis (F1 Score = 53.88) and text grafting (F1 Score = 53.46), which again emphasizes "near-distribution" to be an essential motivation to use text grafting.
We introduced text grafting, a technique to generate in-distribution texts for minority classes using LLMs. By mining high-potential masked templates from the raw corpus and filling them with state-ofthe-art LLMs, we achieve significant improvements in classifier performance on minority classes. Our analysis and case studies demonstrate the effectiveness of text grafting in enhancing text synthesis for minority classes. Future work will concentrate on improving the precision of template mining and the extension of text grafting to other tasks like information extraction.
Despite the presented strengths in the paper, there are still several limitations in the text grafting pipeline. As a hybrid method, text grafting requires a large raw corpus more than data synthesis and LLM calls more than text mining. Other limitations of text grafting also succeed from text mining and data synthesis, such as the dependency on LLM ability (for mining and synthesis). Thus, the application scope for text grafting depends on how LLM comprehends the class name semantics. The performance of different classes might also be biased to the LLM ability in different classes.
The datasets and models used in the experiments are released in github.com/KomeijiForce/TextGrafting