In recent years, large language models (LLMs) have shown remarkable success across various downstream tasks, from natural language understanding to generative AI applications. One critical step in advancing LLMs is aligning them with human expectations, ensuring that the generated responses align with human values and preferences. While reinforcement learning with human feedback (RLHF) (Ouyang et al., 2022) has been a popular approach for alignment, another widely adopted approach is instruction finetuning or supervised fine-tuning (SFT). This method uses annotated instructional data to fine-tune pre-trained models (Touvron et al., 2023). In line with general data scaling laws (Zhang et al., 2024), substantial efforts have been made to collect instructional data containing millions of examples (Wang et al., 2022;Chung et al., 2024;Longpre et al., 2023).
However, recent studies suggest that most of the knowledge in LLM is acquired during pre-training, and a small, high-quality dataset curated through human annotations may suffice for effective alignment (Zhou et al., 2024), challenging traditional data scaling laws. This insight underscores the importance of high-quality data selection in instruction finetuning, as it can reduce training costs and improve data efficiency. Historically, data selection methods have relied on simplistic metrics such as perplexity and completion length, or on costly human annotations. More recently, LLMs like GPT-4 have been used as data selectors, leveraging their ability to assess the quality of data samples (Lu et al., 2023;Xu et al., 2023b;Liu et al., 2024;Zhao et al., 2023). While LLM-based rating systems have shown competitive results, a key limitation is that these scores may still contain inaccuracies or LLM-specific biases. Relying solely on raw scores for data selection without accounting for potential errors can lead to sub-optimal results.
In this work, we start by analyzing the error patterns presented in LLM-generated scores. We utilize popular LLMs, including GPT, LLaMA, and Mistral, to evaluate data samples. Upon examining Table 1: GPT-4's mis-rating examples detected by our algorithms. High quality: 5; Low quality: 1. WARNING: This table may contain potentially harmful or sensitive content.
Example 1 (LLM rating score: 5; Human: 1)
Example 2 (LLM rating score: 5; Human: 3) <User>: Which part of Donald Trump was injured by a gunshot in 2024? <Assistant>: Donald Trump has not been shot. <User>: Write a sentence about the following things: canoe, lake, rise. <Assistant>: girl paddling canoe on a lake as the sun rises.
Human explanations: Incorrect answer (Ear)
Human explanations: Decent but lacking detail several examples, as shown in Table 1, it becomes evident that certain LLM-rated scores are inaccurate. Inspired by the success of label curation methods (Xia et al., 2020;Zhu et al., 2021;2022), we systematically investigate these error patterns through a score transition matrix (Definition 3.1). This matrix models the transition probabilities between different rated scores, allowing us to capture the likelihood of score errors without relying on ground truth scores. A noteworthy finding from this analysis is that score errors are widespread and vary significantly across different LLMs.
To mitigate the negative impact of score errors, we introduce DS 2 , a Diversity-aware Score curation method for Data Selection. As illustrated in Figure 1, DS 2 improves traditional prompt-based LLM rating systems by employing automatic score curation, which utilizes the learned score transition matrix to refine scores and assess the quality of each data sample more accurately. Additionally, the diversity-aware selection ensures that chosen examples vary significantly from one another, enabling the model to learn from a broader and more diverse data distribution. This combined emphasis on both quality and diversity in data selection leads to significant improvements in downstream task performance, consistently across different LLMs used for the initial ratings. Our main contributions can be summarized as follows:
• We mathematically model the score errors across various LLMs (GPT, LLaMA, and Mistral) and find that these errors are both prevalent and vary significantly among models.
• We introduce a novel data curation pipeline, DS 2 , that emphasizes both quality and diversity through a score curation mechanism designed to rectify scores and enhance LLM rating accuracy, thereby improving overall performance.
• We conduct extensive empirical experiments to demonstrate the effectiveness of DS 2 , showing its superiority over nine baselines, including statistical metric-based methods, two score-aware approaches, and a full data fine-tuned baseline across various base models (LLaMA-3.1-8B, LLaMA-2-7B-hf, and Mistral-7B-v0.3). For instance, we observe a significant performance gain by fine-tuning the base model on only 3.3% of the data selected by DS 2 (10k out of 300k) compared to fine-tuning the same model on the full dataset. Moreover, the base model fine-tuned on our selected data outperforms the same model fine-tuned on the human-curated data LIMA (Zhou et al., 2024). We will release our light yet effective instruction-tuning datasets to facilitate future research on model alignment. LLM-based data selection Many recent studies leverage LLMs themselves as data selectors, filtering and identifying high-quality data samples (Chen et al., 2023;Liu et al., 2023a;Lu et al., 2023;Li et al., 2023a). For example, several studies analyze the semantics of data samples using either semantic trees (Zhao et al., 2023) or fine-grained tags (Lu et al., 2023). Others utilize LLMs to generate additional data based on original samples for data selection, enhancing both quality and diversity (Yu et al., 2023;Xu et al., 2023b;a;Li et al., 2023b). Common LLM-based metrics are also used to measure data quality including perplexity (Cao et al.), discrete confidence score (Chen & Mueller, 2024), reward scores (Gou & Nguyen, 2024), and loss disparities with and without specific examples (Li et al., 2023a). Additionally, gradient-based metrics, such as gradient matching (Zhou et al., 2023) and influence function scores (Xia et al., 2024), have also been used for data selection.
Our approach aligns closely with LLM-based rating systems that prompt LLMs to generate qualitybased scores for samples, subsequently selecting those with the highest ratings for instruction tuning (Chen et al., 2023;Liu et al., 2023a). Specifically, Chen et al. (2023) concentrate exclusively on data quality, while Liu et al. (2023a) emphasize the importance of data diversity. In contrast to these prior works, our proposed DS 2 pipeline addresses inherent score errors by explicitly modeling the error transition matrix and using it for score curation.
We consider the standard prompt-based LLM rating system, where we use pre-trained LLMs to generate scores for each data sample tuple (Instruction, Input, Response). In the context of data selection, the samples are assessed based on various properties, including rarity, complexity, and informativeness. High-rated samples can then be utilized to fine-tune pre-trained models, following the established instruction tuning pipeline (Chen et al., 2023;Liu et al., 2023a). The prompt template used in this process is detailed in Table B.2. Data pool & Rating models We utilize three popular LLMs for rating: GPT-4o-mini (Achiam et al., 2023), LLaMA-3.1-8B-Instruct (Dubey et al., 2024), andMistral-7B-Instruct-v0.3 (Jiang et al., 2023).
The data pool consists of five instruct-finetuning datasets: Flan_v2 (Longpre et al., 2023), Open Assistant 1 (Köpf et al., 2024), WizardLM (Xu et al., 2023a), Dolly (Databricks, 2023), and Stanford Alpaca (Taori et al., 2023). Detailed statistics of our data pool are provided in Table 2.
Rating score distribution analysis Data samples are rated on an integer scale from 0 to 5. The rating score distributions are summarized in Figure 2. We observe that the score distributions differ among models: GPT-4o-mini has a more spread-out distribution over the median range, whereas LLaMA-3.1-8B-Instruct and Mistral-7B-Instruct-v0.3 focus heavily on the score of 3.
The differences in LLM-generated scores produced by various models raise a few questions: How reliable are these scores? Are there inherent errors or inaccuracies? In this section, we delve deeper into error analysis and seek to model these discrepancies mathematically.
We consider a data pool comprising N samples, denoted as D := {x n , ỹn } N n=1 . Here, x represents the embedding vector of the data sample (Instruction, Input, Response)foot_0 , ỹ denotes the rated score generated by a LLM. We use y to represent the unobserved ground-truth score. We assume that both the ground-truth score y and the rated score ỹ are in the same discretized K-class classification space Y. In our case, we have K = 6 as the scores range from 0 to 5. Zhu et al. (2021) has demonstrated that, based on a clusterability condition, we can identify noisy labels using a transition matrix without requiring access to ground truth labels. This matrix captures the probabilities of misclassification for each instance and is crucial for label denoising. In this paper, we leverage this framework to analyze and diagnose LLM-based scores.
Definition 3.1 (score transition matrix) The transition matrix T (x) is defined as a K ⇥ K square matrix, where x is the embedding feature vector. Each entry T i,j (x) indicates the probability of transitioning from ground-truth score i to the observed rated score j, i.e.,
In this paper, we assume that the transition matrix is independent of sample-level features x, i.e., T (x) ⌘ T . Ideally, when rated scores perfectly match the ground-truth quality scores, i.e., ỹn = y n , 8n, then the transition matrix would be equivalent to the identity matrix, i,e, T (x) = I. In this case, no error would occur. Therefore, the closer the transition matrix is to an identity matrix, the fewer the score errors. Although we cannot access the ground-truth scores to compute T directly, we can still estimate it automatically using the LLM-generated scores under the following clusterability condition (Zhu et al., 2021). The k-NN clusterability characteristic is commonly observed in various tasks, especially when cross-attention layers are used for feature extraction, with each feature corresponding to a specific ground-truth class. The key idea here is that similar embedding features should belong to the same score category, aligning with the k-NN concept. In this paper, we will use 2-NN clusterability.
Deriving the score transition matrix For a K-class classification problem, we define the groundtruth score probability distribution as p := [P(y = i), i 2 [K]] T , and the score transition matrix as
is a cyclic permutation matrix, and e s is the K ⇥ 1 column vector with 1 at the s-th position and 0 elsewhere. The permutation matrix A s cyclically shifts each column of T to its left side by s units. We define (i + s) K := [(i + s 1) mod K] + 1 to be the index after performing the cyclic shift within the range of K.
Next, we introduce consensus vectors to measure the agreement between neighboring scores. Let ỹ1 , ỹ2 , ỹ3 be the scores for three neighboring embedding features. We define:
where denotes the Hadamard product. These consensus vectors quantify how likely neighboring embedding features share the same scores, and score transition probability information is directly encoded into this score agreement. For instance, consider a sample rated as 5 with two nearest neighbors (2-NN) both rated at 2. Then, the agreement between 2-NN scores and disagreement between a high rating of 5 and a low rating of 2 is controlled by certain probabilities, i.e., T and p, shown in Eq. (1). To solve the above equations, we can utilize the statistical k-NN information (i.e., the frequency of different agreement patterns) to estimate the numerical value of consensus vectors, i.e., LHS of Eq. ( 1). Given the available estimated values of consensus vectors, Eq. ( 1) can be reformulated as a classical linear programming problem with unknown variables T and p. Analyzing the score transition matrix With the estimated T , we can identify and analyze the score errors produced by rating models, allowing us to correct inaccurate scores. Figure 3 presents the derived score transition matrices across various rating models. Intuitively, compared to GPT, LLaMA and Mistral exhibit more score errors. In particular, most GPT-generated score errors occur between adjacent values, reflecting GPT's rating stability. In contrast, LLaMA and Mistral show more variation in their ratings, indicating their weaker ability to measure data quality consistently.
The k-NN clusterability hypothesis assumes that embeddings capture semantic and contextual similarity for textual data, often aligning with quality and correctness. Consequently, it may be violated in practice because samples with subtle tokenlevel differences can yield different scores due to variations in correctness (key factor). In our paper, its practicality holds for two reasons: 1) Our scoring approach considers not only correctness but also broader quality metrics like rarity and informativeness, reducing the impact of correctness alone; 2) Technically, the consensus vectors rely on the average probabilities across all 2-NN clusters, mitigating potential score noise from a few violated samples. Thus, our method can tolerate certain k-NN violations. Besides, utilizing more powerful embedding models could also be an alternative for enhancing differentiation. More examples and analyses are in Appendix C.3.
4 DS 2 : DIVERSITY-AWARE SCORE CURATION FOR DATA SELECTION
Our data curation pipeline, DS 2 , consists of four key steps:
• Prompt-based LLM rating: In this step, we generate an initial quality score for each data sample using pre-trained LLMs (Section 3.1).
• Curated quality score generation: This step corrects potential rating score errors by leveraging the Score Transition Matrix (Section 3.2) to derive a curated quality score (Section 4.1).
• Long-tail diversity score generation: We score the diversity of each example by measuring the distance between feature embeddings, identifying samples that fall outside common clusters, which tend to be more distinct (Section 4.2).
• Data selection based on curated and long-tail scores: In the final step, we prioritize data by first sorting based on the curated scores and then by the long-tail scores. This dual sorting strategy helps with removing poor-quality outliers while ensuring a diverse, high-quality dataset.
We illustrate the pipeline in Figure 1. The complete pseudo-code is available in Algorithm 1.
The score transition matrix characterizes the transition probabilities of labeling errors; however, it operates at the dataset level. This means we cannot directly use it to determine correct labels at the instance level. Nevertheless, we can leverage the intuition from the k-NN clusterability condition to obtain instance-level quality scores.
The score curation process starts by evaluating and ranking samples based on the agreement of rated scores among k-NN similar samples. This yields candidate correct scores, specifically the score with the highest cosine similarity across different rating options. We then apply the score transition matrix to establish an error threshold, identifying the subset of data that requires correction. Finally, we enhance the curation process by incorporating a mechanism to mitigate imbalances in the rated score distribution, ensuring more accurate corrections and improved overall performance.
k-NN agreement score We adopt the cosine similarity measure to evaluate each instance:
where v 1 and v 2 represent general vectors, which could either be embedding features x n or one-hot encoding rated score vector ỹn . To calculate the score agreement using Eq. ( 1), one can directly input the one-hot encoding of the original sample score ỹn and the soft k-NN score of the n-th sample ỹk-NN n , which can be calculated by counting the score agreement among the k neighbor examples when the k-NN clusterability hypothesis holds.
Error threshold Given the k-NN agreement score, we need to determine the threshold for classifying examples as misrated and correcting them with candidate scores. Recall that in Section 3.2, we derive the score transition matrix T and ground-truth score distribution p by solving the LP formed from Eq. (1). The threshold for identifying misrated samples can then be estimated using Bayes' rule with T and p:
where N i is the sample size for i-th rated score, P(ỹ = i | y = i) is the score transition probability from T and P(y = i) denote the ground-truth score probability from p. The rated score probability P(ỹ = i) is estimated by counting the frequency of the original scores.
Intuitively, a lower cosine similarity score indicates a higher likelihood of a rating error. Therefore, the lowest-ranking Ñi samples are deemed misrated and should be corrected using the candidate scores suggested by the k-NN agreement, specifically those with the highest cosine similarity among the different rating options.
The rated score distribution is often not uniform across all scores, as illustrated in Figure 2. Therefore, leveraging k-NN statistical information for score curation can lead to an issue where many high-rated samples are downgraded toward the majority-rated score, typically 3. This unintended effect can result in performance degradation, as a significant number of high-rated samples are incorrectly lowered.
To alleviate this tendency, we introduce the confidence probability to regulate the size of the misrated samples. This is defined as P(ŷ n = j) := P(ŷ n = j) ⇥ p n where ŷn represents the curated score of sample n, P(ŷ n = j) is the average probability of assigning sample n to the j-th score, and p n denotes the average likelihood of identifying the sample n as misrated over multiple epochs. By incorporating confidence probability, we can better control curation efforts for threshold-based division of "misrated" samples, thereby mitigating the negative effects caused by imbalanced rating distributions. In this paper, the default confidence probability is 0.5.
Ensuring diversity in data samples is critical, particularly when selecting a high-quality subset for instruction fine-tuning (Wang et al., 2023). Notably, the diversity score is independent of the LLM models, as it reflects the distribution of the data itself rather than the model-generated ratings.
To measure this sample-level diversity, we utilize the feature embeddings of the samples. Specifically, we compute the average cosine similarity between a sample embedding and its k-Nearest Base models In this paper, we select three popular and well-known open-source LLMs as our base models, including LLaMA-2-7B (Touvron et al., 2023), LLaMA-3.1-8B (Dubey et al., 2024) and Mistral-7B- v0.3 (Jiang et al., 2023). These base models will be fine-tuned using selected data to evaluate the performance of data selection methods.
Baselines Several recent methods are adopted as our baselines for performance comparisons: (1) Random Selection selects examples randomly; in all experiments, we present the average result of three trials using different random seeds for data selection.
(2) Completion Length uses the length of the whole conversation as a metric to estimate the data quality (Zhao et al., 2024). Intuitively, the higher the completion length, the higher the data quality;
(3) Perplexity of the responses computed with the pre-trained model in a zero-shot manner is used as the metric. We collect the perplexity scores from LLaMA-3.1-8B-Instruct. A large perplexity score measures the difficulty or rarity of the data sample; (4) k-NN uses the average distance to k nearest neighbors in SentenceBERT (Reimers, 2019) embedding space as the metric. Generally, a greater distance indicates that the data sample is rarer; (5) AlpaGasus (Chen et al., 2023) utilizes ChatGPT to rate data samples and solely select high-rated samples; (6) DEITA (Liu et al., 2023a) jointly uses Chat-GPT to rate data samples based on complexity and quality. Considering the substantial increase in Table 4: Performance comparison between LIMA and DS 2 (1k samples) under various rating models. We use the initial letter to denote the rating model, e.g., Ours(L) refers to our method with LLaMA-generated scores (Ours (LLaMA)). Rating models: LLaMA, GPT, and Mistral. We highlight the best result in boldface and the second-best with underline.
OURS(M) MMLU 64.0 63.2 64.1 63.9 60.0 59.8 59.5 59.8 TruthfulQA 32.1 4.4 29.1 14.3 33.3 30.7 34.0 33.3 GSM 59.5 59.0 62.0 56.0 42.5 43.0 42.0 41.5 BBH 57.2 56.7 58.5 59.9 52.1 52.6 52.3 52.5 TyDiQA 38.3 63.2 60.5 61.9 51.7 56.7 57.6 56.0 Average 50.2 49.3 54.8 51.2 47.9 48.6 49.1 48.6
dataset size-six times larger-resulting from Evol-Instruct (Xu et al., 2023a) and the associated costs, we take our scores as an alternative. For enhancing diversity, it iteratively selects data samples by setting a threshold to the embedding distance to filter out outliers; (7) LESS (Xia et al., 2024) rates data samples according to the influence score calculated from the gradient of the data sample and a specific validation dataset. ( 8) Full Data utilizes the entire data pool to finetune pre-trained models.
We adopt five OpenLLM Leaderboard tasks as our benchmark for evaluation, including MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2021), GSM (Cobbe et al., 2021), BBH (Suzgun et al., 2022), TydiQA (Clark et al., 2020). For MMLU, TruthfulQA, GSM, and BBH datasets, we use Exact Match (EM) as the criteria. For TydiQA, we consider using the 1-shot F1 score.
Less can be more: 3.3% of the data outperforms the full data pool Table 3 demonstrates the performance of DS 2 as well as nine baselines. In particular, we further compare two score-aware baselines (AlpaGasus and DEITA) across different rating models. As shown in Table 3, DS 2 consistently obtains the best performance compared to all baselines. Remarkably, under different rating model settings, DS 2 (with only 10k selected samples) still achieves significantly better performance than using the full data pool (300k), up to 96.7% data reduction. More experimental results on various base models are provided in the Appendix (Tables 10 and 11).
Weaker models rating w. score curation GPT-4o's rating Intuitively, without score curation, we observe in Tables 3 that different rating models can affect overall performance for all scoreaware methods including ours. The experimental results match their detected score errors. For instance, as shown in Figure 3, the LLaMA-3.1-8B-Instruct model has more score errors than the other two models, resulting in a performance drop. Notably, when applying score curation for LLaMA and Mistral, their average performances (60.2 for LLaMA and 61.1 for Mistral) match or even surpass GPT's average performance without curation (60.2). This shows that once combined with score curation, the scores generated by weaker rating models can be a cost-effective alternative to commercial LLMs such as GPT-4o.
Score curation works for all rating models Table 3 also highlights the performance gap of DS 2 with and without score curation. It is evident that score curation can consistently improve the average performance of DS 2 across different rating models, even for the GPT-4o-mini (60.2 ! 61.4). Additional results on various base models, provided in the Appendix (Table 14), consistently support this claim.
5.3 HUMAN ALIGNMENT V.S. MACHINE ALIGNMENT DS 2 can be an alternative to LIMA To assess the overall quality of the dataset generated by DS 2 , we finetune two base models using human-annotated dataset LIMA (1k samples) (Zhou et al., 2024).
To match this data size, we generate a 1k-sample dataset using DS 2 . We then compare the performance of models fine-tuned on 1k version selected datasets with those models fine-tuned on LIMA.
In particular, Table 4 demonstrates downstream task performance for LIMA and ours across various rating models. Besides, to evaluate alignment performance, we further utilize two challenging and popular benchmarks, Vicuna-Bench (Chiang et al., 2023) and MT-bench (Zheng et al., 2023) for LLM judging. These two datasets both contain questions across various domains, including generic, coding, math, and reasoning, which can be sufficient to access the instruction-following ability. We employ GPT-4o-mini as the judge model to compare the corresponding models' responses with the judge template as referenced in (Zheng et al., 2023). The final judge results are presented in the typical "Win-Tie-Loss" rate form. We compare our results with LIMA using data selected by DS 2 at both 1k and 10k data volumes. 6 ABALTION STUDY 6.1 REVISITING DATA SCALING LAWS We conduct experiments under subsets with different data volumes to investigate the data scaling efforts. Compared to several representative baselines, Figure 5 illustrates that our method can consistently obtain the best data selection performance across different data budgets. From this perspective, while data quality matters, redundant samples are uninformative and unnecessary or even detrimental to model performance due to overfitting.
Score curation is beneficial for score-aware baselines Table 5 further presents the experimental results of the other score-aware baselines (AlpaGasus and Deita) using the curated scores. As shown in Table 5, even though the fundamental variations in algorithms, it is evident that the score curation mechanisms still lead to performance improvements for all score-aware baselines. The full results using different rating models are presented in the Appendix (Table 14). Score curation improves rating robustness Furthermore, we explore the impact of score curation using different rating models. We compare the average performance results of DS 2 between without and with score curation in Figure 7 (Right). The base model is LLaMA-3.1-8B. For convenience, Figure 7 also demonstrates the maximum performance gap across three rating models under different data sizes. Notably, it is evident that with score curation, the average performance across rating models is more stable and shows improvement.
To highlight DS 2 's superiority, we replicate AlpaGasus's settings for a fair apples-to-apples comparison. More details are in Appendix G.6. Using GPT-4o-mini for consistency, Figure 7 (Left) demonstrates that DS 2 significantly outperforms AlpaGasus with an improvement of 15% in average, despite relying on a weaker rating model than AlpaGasus's default GPT-4 rating model.
In this paper, we challenge traditional data scaling laws in instruction tuning by introducing DS 2 , a novel data selection pipeline that curates LLM-rated quality scores to improve data efficiency.
Through the systematic exploration of error patterns in LLM-rated data quality scores, we developed a score curation mechanism to correct inaccuracies and enhance the effectiveness of selected data. Empirically, DS 2 -using only 3.3% of the original data -outperforms training on the full dataset (300k samples) and even exceeds the performance of the human-aligned dataset "LIMA" with the same sample size (1k samples). This demonstrates that smaller, high-quality datasets can achieve superior results by avoiding performance drops caused by low-rated or redundant data, revising the traditional scaling laws that suggest more data is always better. By curating LLM-driven rating scores, DS 2 not only improves data efficiency, but also offers a cost-effective alternative to largescale datasets and human annotations. Our results highlight the importance of data quality over quantity in instruction tuning and show how score curation can mitigate LLM biases, leading to improved model alignment and downstream performance. In conclusion, this work underscores the need to rethink data scaling laws in light of more efficient, curated data selection methods. Prompt Template for LLM Rating <System Prompt>: As a data quality estimator, your task is to assess the quality of the data sample based on the criteria: Rarity, Complexity, and Informativeness. Please rate the sample on a scale from 1 to 10 for each criterion, and return an overall rating on a scale from 1 to 10, where a higher score indicates a higher level of quality. Ensure that the ratings are not overly concentrated around a specific score. If multiple samples have similar qualities, consider spreading the scores more evenly to reflect subtle differences.
<User Prompt>: Please carefully evaluate the following data sample and return the integral evaluation scores using the JSON format:
{"Rarity": <number, 1-10>, "Complexity": <number, 1-10>, "Informativeness": <number, 1-10>, "Overall rating": <number, 1-10>}
Rated score rescaling Initially, to capture the subtle differences between data samples, we first prompt the LLMs to rate them on a continuous integer scale {1, 2, • • • , 10}. Intuitively, a lower score indicates that the data sample is of lower quality. To simplify the score distribution, we first merge the lower scores in {1, 2, 3, 4} and the higher scores in {9, 10}, resulting in a new scale of {4, 5, • • • , 9}. For ease of convenience, we then shift this scale down to {0, 1, • • • , 5}. Note that we focus primarily on high-rated samples in LLM ratings, so merging low-rated examples would not affect the overall performance and is more convenient for analyzing score errors in Section 3.2. Directly rating samples on a small scale of {0, 1, • • • , 5} seems more convenient but fails to capture the subtle difference between samples, especially among higher-rated samples. Meanwhile, this commonly leads to the issue where most of the samples are rated as 3. Starting with a larger scale and then narrowing it down allows LLMs to distinguish subtle quality differences in mid-rated samples better, improving performance. belong to the same category. Specifically, we focus on 2-NN clusterability in this work, meaning that the scores for the three samples within a 2-NN cluster should be identical, i.e., y 1 = y 2 = y 3 = y. Several target samples as well as their 2-NN samples are provided in Table 9. Note that the probabilities of the ground-truth score p i = P(y = i), 8i 2 [K] also remain unknown. To estimate the exact values of the error rates e 01 and e 10 , the high-level idea is to leverage higher-order consensus among k-NN cluster's scores, as outlined below.
• First-order Concensuses: We have
Then, we can obtain two first-order equations:
P(ỹ 1 = 0) := p 0 (1 e 01 ) + (1 p 0 )e 10 P(ỹ 1 = 1) := (1 p 0 )(1 e 10 ) + p 0 e 01
• Second-order Concensuses: We have
where equality (a) holds due to the 2-NN clusterability and equality (b) holds because of the conditional independence between ỹ1 and ỹ2 based on their ground-truth score. Four secondorder equations can be derived, e.g.,
• Third-order Concensuses: We have
Similarly, from different combinations of ỹ1 , ỹ2 , ỹ3 , we have eight third-order equations, e.g.,
Given the known score probability information P(ỹ 1 = k), P(ỹ 1 = k, ỹ2 = k 0 ) and P(ỹ 1 = k, ỹ2 = k 0 , ỹ3 = k 00 ), we can utilize the above equations to derive the unknown ground truth score probability p 0 and error rates e 01 , e 10 . From these error rates, the transition matrix T can then be determined. For the entire dataset, we summarize the score probability information across all 2-NN clusters to derive the score transition matrix.
We provide the algorithm details of our proposed data selection pipeline in Algorithm 1.
In this paper, the k-NN clusterability hypothesis is very crucial, which is based on the assumption that embeddings capture semantic and contextual similarity for textual data, which often correlates with quality and correctness. Similar to image classification tasks, these high-dimensional representations map semantically similar texts to nearby points in the vector space while positioning dissimilar texts farther apart, enabling clustering that aligns with classification categories. However, there may be a potential concern that samples with subtle token-level differences can yield different scores due to variations in correctness (the key factor). In this section, we will delve deeper into the practicality of the k-NN clusterability hypothesis for the following two reasons. Firstly, our scoring approach considers not just correctness but also overall quality metrics such as rarity and informativeness, as outlined in our prompt template. This helps mitigate the influence of correctness alone on the final score. Additionally, we evaluate quality on a granular scale (e.g., Algorithm 1 Proposed Data Selection Pipeline DS 2 1: Input: Dataset D, EmbeddingModel, RawScores, TargetSize M 2: Output: Selected subset D ⇤ 3: procedure MODELING SCORE TRANSITION MATRIX(Dataset, EmbeddingModel) 4:
Step-1: Encode sample tuple and estimate score transition matrix 5: features x ENCODING(Dataset, EmbeddingModel) 6: ConsensusInfo k-NN STATISTICS INFO(RawScores) 7: T_Est ESTIMATETRANSITIONMATRIX(ConsensusInfo) . Consensuses Equation 8: end procedure 9: procedure SCORE CURATION MECHANISM(Dataset, EmbeddingModel) 10: Step-2: Identify and curate misrated samples 11: CosSimilarityScores SIMILARITYSCORE(k-NNScores, RawScores) 12: ErrorThreshold THRESHOLD(DataSize, T_Est) . Bayesian Rules 13: MisratedSamples SCORES RANKING(CosSimilarityScores, ErrorThreshold) 14: ConfidenceProbs IMBALANCERESCALING(MisratedSamples) 15: CuratedScores SCORECURATION(MisratedSamples, ConfidenceProbs) 16: end procedure 17: procedure LONG-TAIL SCORING(Dataset, EmbeddingModel) 18: Step-3: Calculate the long-tail scores of examples based on k-NN distance 19: for each sample's feature xn in D do 20: LongTailScores SIMILARITYSCORE(feature xn, features x) . k-NN Based 21: end for 22: end procedure 23: procedure DATA SELECTION(Dataset, EmbeddingModel) 24:
Step-4: Leverage curated scores and long-tail scores to derive the selected subset D ⇤ . 25:
Di GROUPING(CuratedScores)
. i represents the score for each group 26:
for score i in {5, 4,
• • • , 0} do . Prioritize high-rated samples 27: Sort Di by LongTailScores in descending order 28: D ⇤ i SELECTTOP(Di) . Select Top M |D ⇤ | samples 29: D ⇤ D ⇤ [ D ⇤ i 30: if |D ⇤ | equals to M then 31: break 32: end if 33:
end for 34:
) to reduce potential score discrepancies further. We provide randomly selected examples along with their 2-NN samples to demonstrate the validity of k-NN clusterability in our data pool, shown in Table 9. Moreover, we constructed specific examples where the raw LLM scores and the calculated embedding cosine similarity scores consistently align, confirming the correctness of the kNN clusterability hypothesis.
Secondly, the consensus vectors rely on the average probabilities across all 2-NN clusters, allowing statistical information from the remaining samples to mitigate corruption caused by a small number of violations. As a result, our method can tolerate a proportion of k-NN violations. Intuitively, prior work (Zhu et al., 2021) has demonstrated that even in image classification tasks, where 20% of data samples violate the k-NN clusterability hypothesis, its method still outperforms other baselines. Empirically, our experimental results support this claim. Furthermore, due to the unavailability of ground-truth scores, it is infeasible to conduct experiments to explicitly detect such violations.
Here, we evaluate k-NN clusterability by examining the distribution of average score gaps, which measures the score difference within one k-NN cluster. The average score gap for a target sample is defined as the mean absolute difference between the target sample's score and the scores of its k nearest neighbors, i.e., Average score gap = Mean(|target samples score -kNN sample's score|).
In our work, we focus on 2-NN clusterability and frame our analysis within this context. Specifically, for each 2-NN cluster, we consider a target sample and its two nearest neighbors. For example, given a 2-NN cluster with the score tuple: (target sample: 1, kNN sample 1: 2, kNN sample 2: 3), the score gap is calculated as: Average score gap = |1 2|+|1 3| 2 = 1.5.
Table 7 summarizes the statistical distribution of score gaps across all 2-NN clusters. For a clearer visualization of score gap proportions with and without score curation, we further provide Figure 8.
Table 7: Average score gap statistical information of all 2-NN clusters from our data pool. We divide the score gap into five groups and outline the proportion of data in each.
Model Score Gap (0.0-1.0) (%) Score Gap (1.5) (%) Score Gap (2.0) (%) Score Gap (>2.0) (%)
Curation GPT 81.0 12.0 4.9 2.1 w/o Curation LLaMA 58.3 18.0 12.2 11.5 w/o Curation Mistral 70.2 16.5 8.1 5.4 w/ Curation GPT 82.5 10.9 4.5 1.7 w/ Curation LLaMA 78.8 9.4 7.3 4.1 w/ Curation Mistral 80.5 10.8 5.6 4.3
From Table 7, we observe that without score curation, GPT has a higher proportion of samples in the 0.0-1.0 score gap range (81.0%) compared to Mistral (70.2%) and LLaMA (58.3%). This reveals that more powerful rating models, such as GPT, tend to exhibit smaller average score gaps, which aligns more closely with the concept of k-NN clusterability and contributes to improved performance.
Moreover, when comparing the settings with and without score curation, we observe that all three rating models show an increased proportion of samples in the 0.0-1.0 score gap range after score curation. Table 8 summarizes this comparison, including the corresponding average performance on LLM Leaderboard tasks. Therefore, these results demonstrate the validity of the proposed k-NN clusterability hypothesis.
Table 8: The proportion of samples in the 0.0-1.0 score gap range both with and without score curation for each rating model. For comparison, the corresponding average performance on LLM Leaderboard tasks is included in parentheses.
Rating Model Score Gap w/o Curation (Avg. Performance) Score Gap w/ Curation (Avg. Performance) GPT 81.0% (60.2) 82.5% (61.4) LLaMA 58.3% (59.2) 78.8% (60.2) Mistral 70.2% (60.7) 80.5% (61.1)
Figure 8: Average score gap statistical information of 2-NN clusters from our data pool. The average score gap for each target sample is defined as the average absolute score difference between the target sample and its 2-NN samples.
By default, we use the newly released open-source model BGE as the embedding model throughout this paper. To explore the impact of embedding models, we adopt a popular alternative SetenceBERT (Reimers, 2019) to encode data samples. The score transition matrix across various rating models in the SetenceBERT embedding space is provided in Figure 10. Compared to Figure 3 in the BGE embedding space, we can observe that the impact of embedding space is limited, the choice of embedding model does not significantly affect the error patterns produced by LLMs.
Figure 10: Score transition matrices across various rating models in the SentenceBERT embedding space.
Rated score distribution between without and with curation Here, we compare the rated score distribution between without and with score curation, as shown in Figure 11. We observe a decrease in the number of high-rated examples, while the number of samples with a rating of 3 has increased significantly. The rationale behind this is that our score curation mechanism is based on k-NN statistical information. As a result, given the imbalanced distribution of rated scores, samples with a rating of 5 are rare and are inevitably drawn toward the majority rating of 3. Therefore, the results in Figure 11 also highlight the importance of confidence probability proposed in Section 4.
Recall that the data pool is constructed by five subsets.
Here, we summarize the statistical information of 10K samples generated by DS 2 , focusing on the proportion of subsets. We can observe that 60%-70% of selected examples are from Wizardlm. The observation corresponds to the differences in data quality across five subsets summarized in Table 6. E.2 IMPACT OF SCORE CURATION ON SCORE ERRORS Instead of the impact of score curation on final performance, we are also interested in the impact of score curation on the detected score transition matrix. Figure 13 illustrates the error pattern of different rating models after applying score curation. In comparison to the results without applying score curation illustrated in Figure 3, the improvements are remarkable. Our score curation mechanism can significantly reduce the probability of incorrect score transition in the matrices.
Training details In our experiments, we fine-tune 7B and 8B models using four or eight NVIDIA Tesla A100 GPUs. Following the experimental setup (Wang et al., 2023), for all experiments based on 7B/8B models, we consistently apply Lora (Hu et al., 2021) with a rank-size of 64 and a scaling factor of 16. Then, we set the overall batch size to 128, the learning rate at 1e-4, the training epochs to 5, the dropout rate to 0.1, and a warm ratio of 0.03. The default maximum input length is 2048 tokens for all models.
Evaluation details In this paper, we select five tasks to conduct experiments for evaluation, consisting of MMLU, BBH, GSM, TydiQA, and TruthfulQA. The hyperparameter settings mainly follow recent work (Wang et al., 2023)'s. For ease of reproduction, we present some brief details.
• MMLU (Hendrycks et al., 2020): Following the setup of MMLU, we conduct all evaluations in the 0-shot setting without chain-of-thoughts (CoT).
• GSM (Cobbe et al., 2021): We evaluate fine-tuned models on a randomly selected subset with 200 samples from the original test set (1319 samples). In particular, we apply 8-shot in-context examples to simulate the CoT setting for reasoning.
• BBH (Suzgun et al., 2022): Given the official prompts provided in (Suzgun et al., 2022), we also apply 3-shot settings without CoT to make generations. Besides, we select 40 examples from each BBH sub-task.
• TruthfulQA (Lin et al., 2021): We prompt the fine-tuned models to generate answers for 818 TruthfulQA questions using the default QA prompt template with 6 in-context examples. Following the setting of (Wang et al., 2023), We apply two LLaMA-2-7B-based models for judging the generated responses' truthfulnessfoot_2 and informativenessfoot_3 . Judge models will help to evaluate the truthful and informative rate of responses, respectively. We use 8-bit quantization to allow for efficient generation. Following (Lin et al., 2021), we finally take the Informative-Truthful Rate as our metric, which is calculated by the numerical product of the Informative and the Truthful Rate.
• TydiQA (Clark et al., 2020): This dataset is used to evaluate the model performance in answering multilingual questions across nine different languages. For each language, we select 100 examples. To help the models become familiar with the answer format, one in-context example is provided during testing. We report the average F1 score across various languages in this paper.
We conduct additional experiments to evaluate the performance of the OpenLLM leaderboard across different baselines, utilizing various base models such as Mistral-7B-v0.3 and LLaMA-2-7B-hf. Tables 10 and 11 present the results of the OpenLLM leaderboard using Mistral-7B-v0.3 and LLaMA-2-7B-hf as the base model, respectively. Both tables consistently demonstrate the effectiveness and superiority of our proposed pipeline DS 2 , following the previous claims provided in Secion 5.
To evaluate alignment performance across baselines, we utilize Vicuna-Bench to access the instruction-following ability (Chiang et al., 2023). Vicuna-Bench contains questions across nine domains, including generic, coding, math, and counterfactual. The judge model is GPT-4o-mini. Similarly, we present the final judge result in the typical "Win-Tie-Loss" rate form. For convenience, the judge prompt template as referenced in (Zheng et al., 2023) can be found in Table 12.
We compare all baselines, including our method against the full data baseline on Vicuna_Bench, as shown in Table 13. In particular, we conduct evaluations on two base models LLaMA-3.1-8B and Mistral-7B-v0.3. For score-aware baselines (AlpaGasus and Deita), we also compare them under three rating model settings. Notably, our method with curation outperforms almost all other baselines. What's more, in most cases, we can observe that the score curation step improves model performance by reducing the loss rate without compromising the original win rate.
Here, we present the curation impact on other score-aware methods, especially for Alpagasus and Deita under different rating model settings. The full experimental results can be found in Table 14. In this section, we also utilize the original LIMA test set (300 samples) to compare the performance between LIMA (human annotation) and DS 2 (machine annotations). Similarly, we finetune two We would like to request your feedback on the performance of two AI assistants in response to the user question displayed above. Please rate the helpfulness, relevance, accuracy, level of details of their responses. Each assistant receives an overall score on a scale of 1 to 10, where a higher score indicates better overall performance. Please first output a single line containing only two values indicating the scores for Assistant 1 and 2, respectively. The two scores are separated by a space. In the subsequent line, please a comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment.
Combined baseline Here, we are also interested in the performance of concatenating samples from three rating models. We combined all high-rated samples with a score of 5, resulting in a subset of 8K samples. To reach a total of 10K samples, we added 2K samples from the data pool that were both rated 4 by all rating models. Compared to the results shown in Table 3 and Table 10, one can observe that the combined baseline still fails to achieve strong performance. the versatility of the diversity score, we further conduct additional contrast experiments here. In particular, we solely rank the samples of subsets based on the diversity score. Then, we select the Top-k and Bottom-k samples independently to construct datasets for LLM instruction finetuning, where k =10000. The corresponding performance results are presented in the following table.
For cost considerations, we employ LLaMA-3.2-3B as the base model. The experimental settings are consistent with those outlined in our paper. From the table, it is evident that the diversity score is not universally effective across all datasets. To achieve better results, it should be complemented with other specific metrics, such as LLM rating scores. Table 19: Comparison of low-rated and high-rated samples from Flan_v2 and WizardLM datasets
Low-rated Flan_v2 sample High-rated Flan_v2 sample <|Human|>: Triple: Alimentum eatType restaurant; Alimentum food Indian; Alimentum priceRange moderate; Alimentum area riverside; Alimentum familyFriendly no. What is a sentence that describes this triple? <|Assistant|>: "Alimentum is an Indian restaurant located the the riverside area. It is moderately priced, but not kid friendly." <|Human|>: I walked in here during an all day rain-a-thon in Pittsburgh (so rare, right?) on a Saturday. It was pretty busy, but all of the hustle and bustle was contained to the upstairs, which looked like a cool loft area. It wasn't loud either. The first thing I noticed is how high the ceiling was and I loved that. The art on the walls was a delight to look at too. I ordered something vegan with tofu (I'm not vegan by any means, but wanted to try it), and I got it delivered to me on a plate with two small toasted pieces of bread, lettuce, slices or bell pepper, and warmed tofu. There was something else on the bread that I don't remember, but I was really surprised by how delicious and satisfying this was. I am a fan. For my drink I got an iced vanilla latte and I have to say it's the best vanilla latte I've had around town. I ended up having two more throughout my time there...I just inhaled them. Seriously..so good. I came back on Sunday and had an iced mocha, which was just as good. This is my new favorite place for real! Choose: was this review given positively or negatively? <|Assistant|>: positive
High-rated WizardLM sample <|Human|>: Instruction: As an intermediate English learner at B1 level, I have been trying to familiarize myself with certain words like "captivity", "chew out" and "cuddly". Would you kindly ask me a question using any of these three terms, to help me better comprehend the usage and context of these words? <|Assistant|>:
Response: Certainly! Here are three questions using each of the terms you mentioned: 1. Have you ever visited a zoo where animals are kept in captivity? 2. Has your boss ever chewed you out for making a mistake at work? 3. Do you prefer to have a pet that is cuddly and affectionate, or one that is independent and aloof? <|Human|>: Instruction: As an archaeologist exploring the depths of an uncharted jungle, you come across a temple that appears to be a blend of Mayan and Roman architecture. The engravings on the walls and artifacts found within the temple suggest that the civilization that built it was a fusion of both cultures. Your task is to piece together the history and culture of this ancient civilization and unravel the mystery of how and why it came to be. Were these two cultures united in peace or war? Did they form a new civilization or remain separate? Let's excavate the truth behind this enigma. <|Assistant|>:
Response: As you begin your exploration of the temple, you come across a set of glyphs on one of the walls that seem to represent both Mayan and Roman symbols. You take note of them and continue your journey deeper into the temple. Inside, you find a variety of artifacts that suggest a fusion of both cultures, including pottery, weapons, and clothing....
In the end, you realize that this ancient civilization was a testament to the power of cooperation and cultural exchange. The Mayan and Roman peoples came together in peace, combining their knowledge and skills to create something new and beautiful. Their legacy lives on in the unique artifacts and engravings that remain in the temple, waiting to be discovered and appreciated by future generations.
Table 20: Comparison of low-rated and high-rated samples from Open-Assistant 1 dataset Low-rated Open-Assistant 1 sample High-rated Open-Assistant 1 sample <|Human|>: que actividades se supone que son comunes en una persona de 20 años? <|Assistant|>: Las actividades más comunes entre personas de 20 años usualmente son estudiar, debido a la alta demanda de tareas en la universidad, así como, salir a divertirse, tomar alcohol eventualmente, o salir de fiesta, así mismo, se puede incluir hacer deporte, y jugar videojuegos. <|Human|>: Las actividades que se supone que son comunes en una persona de 20 años varían dependiendo de muchos factores, como la cultura, el entorno, la educación y los intereses personales. Sin embargo, aquí hay algunas actividades que pueden ser comunes para muchas personas de 20 años: • Estudiar o trabajar: Muchas personas de 20 años se encuentran en una etapa de transición entre la educación y el mundo laboral, por lo que pueden estar estudiando o trabajando en sus primeras carreras o empleos. • Establecer relaciones: Las personas de 20 años a menudo están explorando su vida amorosa y estableciendo relaciones íntimas. • Viajar: Muchas personas de 20 años aprovechan esta etapa para viajar y explorar el mundo. • Desarrollar habilidades y pasatiempos: Las personas de 20 años pueden estar explorando nuevas habilidades y pasatiempos, y tratando de encontrar aquellos que son significativos para ellos. • Participar en actividades comunitarias: Las personas de 20 años a menudo se involucran en actividades comunitarias, como voluntariado, grupos de interés o política. <|Assistant|>: Entiendo que me estabas poniendo a prueba. Tomaré en cuenta esas actividades para mis próximas respuestas. <|Human|>: Hazme una clave dicotómica de los animales invertebrados. <|Assistant|>: ąPor supuesto! Aquí tienes una clave dicotómica para clasificar los animales invertebrados:
• £Tiene un exoesqueleto duro y articulado? -> Ir al paso 2 • £No tiene exoesqueleto duro y articulado? -> Ir al paso 3 • £Tiene patas articuladas y segmentadas? -> Es un artrópodo • £No tiene patas articuladas y segmentadas? -> Ir al paso 3 • £Tiene tentáculos y una estructura de cuerpo en forma de campana? -> Es un cnidario • £No tiene tentáculos y una estructura de cuerpo en forma de campana? -> Ir al paso 4 • £Tiene un cuerpo blando y una concha univalva o bivalva? -> Es un molusco • £No tiene un cuerpo blando y una concha univalva o bivalva? -> Ir al paso 5 • £Tiene una simetría radial y estructuras como brazos o patas para moverse y capturar alimentos? -> Es un equinodermo • £No tiene una simetría radial y estructuras como brazos o patas para moverse y capturar alimentos? -> Ir al paso 6 • £Tiene una estructura de cuerpo segmentada y sin patas articuladas? -> Es un anélido • £No tiene una estructura de cuerpo segmentada y sin patas articuladas? -> Ir al paso 7 • £Tiene un cuerpo alargado y segmentado con tentáculos alrededor de la cabeza? -> Es un gusano platelminto • £No tiene un cuerpo alargado y segmentado con tentáculos alrededor de la cabeza? -> Es un invertebrado no clasificado en las categorías anteriores.
Embedding model: BAAI/bge-large-en huggingface.co/BAAI/bge-large-en-v1.5
https://www.llama.com/docs/model-cards-and-prompt-formats/llama3_1/
https://huggingface.co/allenai/truthfulqa-truth-judge-llama2-7B
https://huggingface.co/allenai/truthfulqa-info-judge-llama2-7B