Recent advancements in artificial intelligence (AI) have highlighted the critical need for aligning AI systems with human values, a concept known as human value alignment (Griffith et al., 2013;Arumugam et al., 2019;Gabriel, 2020). The alignment can serve the purpose of generating outcomes that are better suited for human ethics (Griffith et al., 2013), personalized needs (Kirk et al., 2024), or reduced harmful content (Bai et al., 2022). This alignment has traditionally been pursued by adjusting AI behavior to adhere to specific attributes via preference datasets or reward functions. This process involves finetuning the original model according to the optimization problem:
(1)
Here, P denotes the class of all distributions, p 0 is the distribution that represents the generative model to align, p is the distribution that represents the aligned model, R is a reward function that quantifies the preference level of any given pair of prompt x and generation y, D KL measures the KL-divergence, and ´> 0 is a regularization hyperparameter. This formulation has deep conceptual roots in the Bayesian decision theoretic framework (Bissiri et al., 2016). Specifically, if we consider x as observed data and y as a parameter ¹, the problem (1) can be expressed as E θ∼p(•) log p(x | ¹) -D KL p(•) || p 0 (•) . This formulation yields the solution p x (¹) ∝ p 0 (¹)p(x | ¹), which is precisely Bayes' Rule. However, while this formulation provides an elegant interpretation of how AI models can be adjusted to reflect new information or preferences, it may not fully capture the complexity required when aligning AI systems to multiple, potentially conflicting human values. For example, a healthcare-related large language model must deliver information that is not only precise but also easy-to-understand and harmless (Gebreab et al., 2024). Similarly, a customer service P a re to F ro n ti e r P a re to F ro n ti e r Figure 1: Expected reward (realized value level) of generated content having Harmlessness (left) and Humor (right) versus Helpfulness for various models aligned from the Llama2-7B-chat model (Touvron et al., 2023). Each blue dot represents the expected rewards E x∼D,y∼p(•|x) r(x, y) with r trained from Anthropic Harmless preference data (rHarmlessness) (Yang et al., 2024), Helpfulness preference data (rHelpfulness) (Yang et al., 2024), and Humor classifier (rHumor) (Dhiab, 2023). The expected rewards are numerically obtained by solving (1) with R = ¼1rHarmlessness + ¼2rHelpfulness + ¼3rHumor, where ¼1, ¼2, ¼3 ≥ 0 are randomly generated, and quantiletransformed to the scale of 0 to 1. Arrows indicate the transition from the original model to aligned models, either using the proposed approach (MAP) or a single reward function.
chatbot optimized for efficiency may compromise on politeness or empathy (Kolasani, 2023). These scenarios highlight the need to balance diverse human values, which often requires more nuanced solutions than a single-dimensional alignment approach can offer. Related work. Centered around the formulation (1), a standard approach is the reinforcement learning from human feedback (RLHF) (Griffith et al., 2013;Arumugam et al., 2019;Bai et al., 2022;Dai et al., 2024) that first trains a reward model R based on pairwise comparison or ranking data to serve as a proxy for human preferences, and then uses reinforcement learning with the reward model to address the optimization problem (1). An alternative method known as direct preference optimization (DPO) (Rafailov et al., 2023) directly optimizes an empirical risk based on the Bradley-Terry loss (Bradley & Terry, 1952) applied to the preference data, using an implicit reward in the form of r(x, y) ∆ = ´log(p(y | x)/p 0 (y | x)). To address multiple human values, recent work on aligning foundation models has used multi-objective reinforcement learning (MORL) (Barrett & Narayanan, 2008;Li et al., 2020;Wu et al., 2024). Existing studies focus on approximating tradeoffs among values utilizing a linear scalarization method that combines either reward functions or data sources, for both RLHF and DPO approaches (Bai et al., 2022;Rame et al., 2023;Dai et al., 2024). Alternatively, some studies utilize specific choices of weights to manage trade-offs (Dognin et al., 2024). A recent technique named "rewarded soup" was introduced to efficiently compute the aligned models under linear scalarization (Rame et al., 2023). It first separately fine-tunes multiple models, each with a particular reward function, and then aggregates these networks by linearly interpolating their weights. This method aims to approximate the ensemble of fine-tuned models that would otherwise result from optimizing a single reward composed of various linear combinations of individual reward functions, namely R ∆ = m i=1 ¼ i r i with random λ = [¼ 1 , . . . , ¼ m ]. A similar idea was applied to DPO (Zhou et al., 2024), where the DPO-aligned models under individual values are interpolated to approximate the Pareto Frontier. Further related work is provided in Appendix B.5.
Challenges. Aligning AI models to multiple values simultaneously presents several unresolved challenges. First, as demonstrated in Figure 1, aligning with one value, such as helpfulness, harmlessness, or humor, could inadvertently diminish another. This motivates a critical question: How can we quantify and enhance multiple human values concurrently without compromise? Moreover, in the RLHF approach as described by the problem (1), it is unclear how to specify the hyperparameter ´and reward function R so that the aligned model p improves upon, or at least not 2 1 0 1 2 Humor scores 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Density 4 2 0 2 Helpfulness scores 0.0 0.2 0.4 0.6 0.8 1.0 Density 4 2 0 2 4 Harmlessness scores 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Density Original realized value levels MAP realized value levels Distribution of reward scores under original model Distribution of reward scores under MAP-aligned model Figure 3: Distribution of reward scores before and after aligning the Llama2-7B-chat model towards three values: Humor (left plot), Helpfulness (middle plot), and Harmlessness (right plot), using the proposed MAP.
This alignment involves a user-specified palette designed to shift the expected rewards, also referred to as realized value levels, toward the 80% quantile of the pre-alignment distributions.
worse than, p 0 in all human values in one shot (without trial-and-error). Even if we can try all possible combinations, there is no theoretical justification that linearly combining individual reward functions is sufficient to obtain the Pareto Frontier. The DPO method, while simplifying the alignment process through a direct empirical risk optimization, still does not address the issue of integrating multiple data sources, which have their underpinning still at the problem (1). Recent studies have demonstrated the sensitivity of results to different weights used in these aggregations (Bai et al., 2022). To highlight this point, in Figure 2, we visualize the range of possible λ and of desirable λ (which actually admits Pareto improvement on all the values). Contributions. We introduce the Multi-Human-Value Alignment Palette (MAP), a principled approach to rigorously aligning multi-dimensional values with provable guarantees. Similar to an artist's color palette, MAP enables the blending of multiple human values to "paint" AI behavior with a broad spectrum of preference shades. In Figure 3, we illustrate how MAP allows users to precisely customize and control the level of improvement for all values in an interpretable manner.
The proposed MAP introduces several technical innovations and contributions:
• Formulation. We propose a novel problem formulation that allows one to align multiple human values using user-defined constraints, which we term "value palettes." Each palette acts as a constraint that represents a preferred level of alignment, allowing us to "MAP" from any targeted value levels specified by the user to a particular reward function for (1). This precise one-to-one mapping ensures exact adjustments to model behavior.
• Theory. We provide theoretical analysis within the MAP framework quantifies the representation of the solution, its sensitivity to changes in the value palette, and its feasible operational range. This leads to a deeper understanding of the inherent trade-offs among various values. Furthermore, we investigate the range of realizable value levels and demonstrate that a linear combination of individual reward functions, is sufficient to reach the Pareto Frontier. We also establish a crucial link between multi-value alignment and sequential alignment, showing that cyclically updating each value multiple times achieves equivalent results to a single execution of MAP.
• Computational methodology. Based on our theoretical derivations, we propose a computational method to ascertain the achievability of user-defined value palettes. The proposed method utilizes a primal-dual approach to efficiently solve the optimization problem for a feasible choice within the palette. We verify that the dual problem exhibits concavity, enabling effective resolution through gradient ascent techniques. MAP is agnostic to the stages of model alignment, whether at the decoding or training stage, the model architecture, and the specific definitions of values used. Finally, we conducted comprehensive experimental studies to validate the practical effectiveness of MAP.
2.1 PROBLEM FORMULATION The formulation in (1) can be seen as maximizing the expected reward while imposing a regularization to minimize unnecessary deviations from the original model. This insight leads us to define a value alignment through a statistical functional constraint:
which is interpreted as the expected rewards, or realized levels, under a value preference must be at least c. Likewise, to align m g 1 value preferences, we introduce the following MAP problem:
We denote c ∆ = [c 1 , . . . , c m ] T as the value palette. With a solution p, we refer to
. . , r m (x, y)] T ) as the realized value levels. We write u g v if the two vectors are of the same size and u i g v i for each entry i. Theorem 1 (Representation of MAP solution). The solution to the MAP problem (3) is
where λ T r(x, y) = m i=1 ¼ i r i (x, y), for some λ g 0. Moreover, assuming that r(x, y) is not trivially a constant on the support set of x, y, the above λ is the unique solution to the problem:
where Z(λ) (x,y) , and g is strictly concave. As a result, we can treat λ in (4)
as an implicit function of c and denote it as λ = λ(c)
Remark 1 (Interpretation of λ). Theorem 1 establishes a one-to-one correspondence between the vectors c and λ. The proof to the first part of Theorem 1 uses the Karush-Kuhn-Tucker (KKT) conditions, and the second part is obtained by showing that (5) is the dual problem of (3). From a decision theoretic view, the decision of ¼ is based on trading off the utility term λ T c and the "risk" term -log Z(λ). The latter term can be seen as a form of risk aversion, because maximizing it would penalize decisions that place a disproportionate weight on less likely, albeit highly desirable, outcomes. Practically, the expectation E x∼D,y∼p0(•|x) can be easily approximated using a sample average from a dataset generated under p 0 , allowing the dual problem (5) to be numerically solved. This computational aspect is further explored in Section 2.3. Remark 2 (Choice of Value Palette c). We outline interpretable methods for selecting the value palette c. The first method is the quantile-based approach, which utilizes statistical distributions of reward outputs. For any reward function, input prompts x ∈ D, and model-generated outputs y ∼ p 0 (• | x), the mapping (x, y) → r(x, y) defines a random variable, denoted by R(D, p 0 ).
The expectation of R(D, p 0 ) is the realized value under the model p 0 . Users can set each component of c to correspond to a quantile of R(D, p 0 ). For example, for the first value, setting c 1 to the 80% quantile positions the realized level of the aligned model p within the upper 20% of expected outcomes. For practical implementation, we propose using a small pilot dataset generated from the original model as Monte Carlo samples to empirically estimate the quantiles of R(D, p 0 ). That is, we sample x i ∼ D, y i ∼ p 0 (• | x i ) to create {r(x i , y i ) : i = 1, . . . , n} (n = 2000 in our experiments). This pilot data can serve various palette choices. We provide further remarks on two other approaches, one based on classifiers and another based on automatic adjustment in Appendix B.4. Remark 3 (Robustness of MAP under Mis-specified Rewards). We conducted a theoretical analysis of robustness against mis-specification in Appendix B.1. Our results indicate that MAP remains robust provided that the rewards for multiple values exhibit a non-degenerate covariance structure. 2.2 REALIZABLE VALUE LEVELS OF THE MAP PROBLEM AND PARETO FRONTIER We first show that the MAP problem can be written as the original alignment problem (1) with a particular reward function that is simply a linear combination of individual rewards. Theorem 2 (Solution of MAP). The solution to the MAP problem (3) is the same of the problem (1) with R(x, y)
, where λ(c) was introduced in Theorem 1.
This prompts the natural question of whether MAP limits the breadth of realizable value levels compared to those achievable under (1) with arbitrarily chosen R. Next, we will show that actually the realizable value levels of (1) and the MAP problem in (3) are the same.
Given the reference distribution p 0 and any specific reward functions R, the solution p, if feasible, depends solely on R. To illustrate this dependency, we denote p as p R . Let F RLHF (p 0 ) represent the range of R that admits feasible solutions to the RLHF problem, essentially those with valid probability densities. The realizable value levels under the RLHF problem are defined as:
For multiple reward functions r 1 , . . . , r m , we consider a specific class of R comprising various non-negative linear combinations, and define their realizable value levels similarly:
VRLHF(r1, . . . , rm; p0)
Äiri ∈ FRLHF(p0), Äi ≥ 0 .
In the MAP problem, given p 0 and r, the solution p, if feasible, depends only on the user-specified value palette c. To emphasize this relationship, we denote p as p c . Let C MAP (r 1 , . . . , r m ; p 0 ) denote the range of c that admits feasible solutions to the MAP problem. We further consider the realized value levels of all feasible solutions under various c, defined as:
VMAP(r1, . . . , rm; p0) ∆ = E x∼D,y∼pc(•|x) r(x, y) : c ∈ CMAP(r1, . . . , rm; p0) . Theorem 3 (Equivalent realizable value levels). For any original generative model p 0 , we have VMAP(r1, . . . , rm; p0) = VRLHF(p0) = VRLHF(r1, . . . , rm; p0). (6) 0 Feasible from 0 and any Feasible from 0 and à =1 Same set of realizable value levels Theorem 3 establishes that the realizable value levels by MAP equate to those in the original alignment problem (1) using a specific reward function -a linear combination of individual rewards. This proves that linear combinations of individual reward functions can sufficiently capture the entire Pareto Frontier. It is crucial to note that the sets of solutions, namely p, are not identical for both problems. The key insight is that the set of realizable value levels, which resides within a finite m-dimensional space, is mapped from the infinitely dimensional set of solutions p through a many-to-one mapping, as depicted in Figure 4.
We denote all feasible value palettes C MAP (r 1 , . . . , r m ; p 0 ) simply as C. The following theorem explores the characteristics of this set. Theorem 4. The following properties of C hold.
Step 1: Specify Value Palette
Step 2: Check Feasibility
Step 3: Align Model Step 1: Specify value palette • Specify c (e.g., based on Remark 2).
Step 2: Check feasibility • Attempt to solve Problem (7) with updates in (8).
If infeasible: Suggest alternative c (Remark 2).
Else: Obtain λ and define reward function
Step 3: Align model • Decoding (D) option: For a prompt x, generate y (1) , . . . , y (m) , and sample y = y (i) using the Softmax of R(x, y (i) ) as Multinomial probability. • Finetuning (F) option: Apply Proximal Policy Optimization (PPO) with R to fine-tune p0 into p.
For a prompt x, generate y ∼ p(• | x). Output: The generated y.
That is, there is no feasible solution if c i > B i for some i.
(iv) Let p λ (y | x) ∆ = p 0 (y | x)e λ T r(x,y) /Z(λ, x) be the λ-adjusted distribution of p 0 , where Z(λ, x) ∆ = y ′ p 0 (y ′ | x)e λ T r(x,y ′ ) dy ′ is the normalizing constant. Treating the optimal solution λ = λ(c) as a function of c, we have dλ(c) dc = -V -1 , where
The result in (iv) can be used to check if a small increase in c, e.g., changing c 1 to c 1 + ¶ for a small ¶ > 0, will cause λ to change in a way that remains feasible.
In this section, we introduce a practical framework to solve MAP problem defined in (3), illustrated in Figure 5 and Algorithm 1. Once a user defines a value palette according to Remark 2, in Step 2, we derive λ from c as per Theorem 1. We then approximate Problem (5) as follows:
where the dataset {(x j , y j ) n j=1 } serves as a finite-sample approximation of the distribution p 0 (x, y) with y 1 , . . . , y n , generated conditional on respective prompts x 1 , . . . , x n . Lemma 1 in the Appendix confirms that both the original problem (5) and its approximation (7) are concave, allowing for the use of gradient ascent to efficiently find the optimal λ:
where r 1:n ∆ = [r(x 1 , y 1 ), . . . , r(x n , y n )]. In practical implementations, we applied a change of variable τ ∆ = log λ to ensure the constraints λ g 0 are satisfied. Not all users' palette choices are feasible. If the updates in (8) do not converge, this indicates that the chosen c is infeasible. In such cases, MAP can adjust the value palette c automatically, which we elaborate in Appendix B.2. Also, because the above optimization problem is concave and the data can be pre-computed using a pilot set of Monte Carlo samples from the original model, solving the problem is efficient and does not depend on the complexity of the models to align. More discussions are included in Appendix B.3.
In Step 3, we implement p(y | x) ∝ p 0 (y | x)e λ T r(x,y) through two practical approaches during experiments. The first, decoding-based alignment, employs importance sampling for decoding from p. Specifically, for any input x, we generate k samples from p 0 , labeled y 1 , . . . , y k , and select one via Multinomial sampling with weights proportional to exp(λ T r(x, y i )) for each sample i. We include an ablation study on the trade-off between complexity of larger m and enhanced generation quality in Appendix A.4. The second approach, finetuning-based alignment, fine-tunes a model p w with neural weights w based on the reward function (x, y) → λ T r(x, y). Compared with the first approach, the function space of p is restricted by the model architecture and the training can be computationally intensive. However, it can reduce the inference-stage complexity by only decoding one sentence for each prompt. An experimental study on this comparison is included in Appendix A.5.
To align a model with a specified value palette c, a natural baseline method is sequentially aligning individual values, namely we sequentially update the model to align each value to an entry in c until all values are addressed. This section compares this method with the MAP procedure.
Sequential alignment algorithm. Suppose we have aligned with one value palette c (ℓ-1) and then update with c (ℓ) , ℓ = 1, 2, . . .. For notational convenience, let the initial c (0) be the realized level of the original model p 0 . At the end of the ℓ-th alignment, the aligned distribution can be expressed as
(ℓ) r(x, y)) for some vector λ (ℓ) . Given the value palettes {c (ℓ) } ℓ=1,2,... , we recursively obtain λ (ℓ) from λ (ℓ-1) . Like the problem (5), the alignment objective at the beginning of the ℓ-th alignment is:
Like the problem (7), we can numerically solve (9) by addressing the problem:
Connection between Sequential and Simultaneous Alignment. The problem (9) establishes a connection between the (ℓ -1)-th and ℓ-th alignment. Recall that m denotes the number of values or the dimension of c. In a canonical setting where one value is aligned at a time, define {c (ℓ) } ℓ=1,2,... by: c
for ℓ f m, where the ℓ-th element is the same as that of c, and the others are trivially negative infinity. For ℓ > m, we can re-align the first value, and so on. Namely, we replace the above c ℓ in c (ℓ) with c ℓ mod m . The following result shows this sequential, iterative alignment process converges to the joint alignment using MAP.
Theorem 5. Let p (0) ∆ = p 0 , p (1) , p (2) , . . . be the sequence of distributions obtained by sequentially aligning the original model according to the single-value MAP objective:
where ℓ ′ = (ℓ mod m). Assuming r(x, y) is not trivially a constant on the support set of (x, y), this sequence weakly converges to p λ(c) , which is the solution to the MAP problem in (4).
The proof of Theorem 5 demonstrates that the sequential alignment process fundamentally operates as a coordinate-wise gradient ascent algorithm. When we align subsets of values in each iteration, the process can equivalently function as a block-wise ascent. Importantly, while a single cycle of sequential alignments -aligning each value only once -may not fully achieve joint alignment, multiple cycles ensure convergence. This aspect is particularly significant in scenarios where constraints such as limited GPU memory restrict the simultaneous loading of numerous reward models. Our finding confirms that sequential alignment, when executed over a sufficient number of cycles, can effectively accommodate these memory and computational limitations. 3 EXPERIMENTAL STUDY
We generate prompts from two data sources: Anthropic harmless data (Bai et al., 2022), which includes human requests delineated between the tags "Human:" and "Assistant:", and IMDB data (Maas et al., 2011) from which we retain movie reviews exceeding 30 characters in length.
For backbone models, we employ OPT-1.3B (Zhang et al., 2022) and Llama2-7B-chat (Touvron et al., 2023), which have demonstrated robust language modeling capabilities in previous assessments. We focus on several values for alignment: Humor, Positiveness, Harmlessness, Helpfulness, Diversity, Coherence, and Perplexity. The Humor reward is assessed using the logits from a humor detection model (Dhiab, 2023), while Positiveness uses a DistilBERT model trained on IMDB sentiment analysis (Lvwerra, 2021). Harmlessness and Helpfulness are evaluated through two GPT-2 models equipped with a value head fine-tuned to predict these attributes (Yang et al., 2024). Diversity is measured by the lexical variety within sentences, calculated through the proportion of unique n-grams (n = 2, 3, 4) and their composite score (Zhang et al., 2020). Coherence is determined by the semantic similarity of sentences within a context, using a supervised SimCSE BERT-based model that captures sentence embeddings to assess textual coherence (Gao et al., 2021).
For the conversational task, we use Anthropic data as input prompts x to align the OPT-1.3B model across six dimensions: Humor, Harmlessness, Helpfulness, Diversity, Coherence, and Perplexity.
Here, the perplexity is calculated as negative log of the standard perplexity so that the larger the better. We tested seven different value palettes for alignment, considering both decoding-based and finetuning-based implementations. For example, the HHH-80% palette aligns the first three values to the 80th percentile of their respective distributions, while maintaining the last three values. This approach aims to enhance the model's output to closely resemble human-like interaction standards without compromising the intrinsic qualities of the generated content. The results are summarized in Table 1. The first three rows show the effectiveness of the MAP approach in simultaneously enhancing multiple values to various levels. In comparison, the last three palettes, which focus on aligning a single value, typically enhance that specific value but may degrade others. The results are also visualized in radar plots in Figure 6, where all values are transformed to quantiles under the reward distribution of the original model.
Additional experiments, including an ablation study with the Llama2-7B-chat model, are detailed in Appendix A.1. Notably, the HHH-80% palette was determined to be feasible by Algorithm 1
Step 2 and its results are therefore included. This indicates that the Llama2-7B-chat model, which has a larger complexity than OPT-1.3B, allows for more extensive multi-value alignment. We also explored a sentiment-controlled open generation task using a random trunk of IMDB data as input prompts to align the OPT-1.3B model. The results are summarized in Table 2.
To demonstrate the effectiveness of MAP in reliably aligning multiple values, we conduct an experiment showing that MAP can identify desirable outcomes with randomly sampled λ. According to Table 1: Results of aligning the OPT-1.3B model with Anthropic conversational data using diverse value palettes, evaluated in terms of expected rewards (or realized value levels). Scenarios include: 1) enhancing the first three values -Humor, Harmlessness, Helpfulness -to the xth percentile of their respective reward distributions under the original model ("HHH-x%") while maintaining the last three values -Diversity, Coherence, and Perplexity -using MAP, and 2) aligning individual values independently ("Humor-80%", "Helpfulness-80%", "Harmlessness-80%"). Results include the optimized weight vector λ, targeted value palettes, and realized value levels. Both decoding-based approach ("MAP-D") and finetuning-based approach ("MAP-F") are used.
The palette "HHH-80%" was found infeasible, as confirmed in Step 2 of Algorithm 1.
Humor↑ Helpfulness↑ Harmlessness↑ Diversity↑ Coherence↑ Perplexity↑ Original model 2.07 -1.47 0.25 0.88 0.43 -3.34 HHH-50% (λ = [2.53, 0.23, 0.28, 0.02, 0.05, 0.05]) MAP-D 2.44 -1.38 0.21 0.88 0.43 -3.17 MAP-F 2.20 -1.80 0.66 0.86 0.41 -2.80 HHH-60% (λ = [6.30, 0.83, 0.93, 0.01, 0.03, 0.03]) MAP-D 2.48 -1.33 0.48 0.88 0.43 -3.15 MAP-F 2.47 -2.26 0.50 0.89 0.26 -3.49 HHH-70% (λ = [12.77, 1.53, 1.69, 0.01, 0.02, 0.02]) MAP-D 2.49 -1.29 0.66 0.88 0.45 -3.14 MAP-F 2.17 -2.28 0.97 0.82 0.10 -5.04 Humor-80% (λ = [16.44, -, -, -, -, -]) MAP-D 2.52 -1.42 0.01 0.89 0.43 -3.21 MAP-F 2.08 -2.49 0.39 0.79 0.09 -5.85 Helpfulness-80% (λ = [-, 0.72, -, -, -, -]) MAP-D 1.99 -0.75 -0.35 0.88 0.43 -3.20 MAP-F 2.02 -0.66 -0.58 0.88 0.41 -2.73 Harmlessness-80% (λ = [-, -, 1.27, -, -, -]) MAP-D 1.97 -1.86 0.97 0.88 0.42 -3.17 MAP-F 2.05 -2.02 0.94 0.87 0.40 -2.63 Theorem 4, we sample λ randomly from the range (c 0 , B) and retain the feasible ones according to MAP's feasibility check. From these, we select λ vectors with a bounded ℓ 1 -norm less than 6. We compare this with a standard MORL approach where λ is randomly generated from s • u, where s is uniformly sampled from (0, 6) and u is uniformly sampled from the probability simplex. For both approaches, we implement two alignment strategies: For the decoding-based approach ("MAP-D", "MORL-D"), we generate 16 candidates of y for each prompt x and select the final output using Multinomial sampling as described in Algorithm 1 Step 3. For the finetuning-based approach ("MAP-F", "MORL-F"), we apply PPO and the calculated MAP reward function R to fine-tune the original model. We evaluate the outcomes using the "test" split of our task data, assessing both the expected reward and win rate against the original model based on the reward functions.
Additionally, we compare MAP with the DPO approach that uses various mixtures of preference data, and DPO-Soup, which applies linear interpolation of DPO-trained, value-specific models. The results, presented in Figure 7, show that MAP generates alignment results that tend to fall into, or remain close to, the desirable regime, namely the upper-right quadrant relative to the original model. In contrast, the other approaches experience more severe trade-offs and rarely fall into the desirable regime. Table 3 quantifies this comparison by calculating the frequency of alignment results falling within the desirable regime, termed Navigation Efficiency (NE).
The proposed MAP offers a structured approach to aligning multiple human values, enabling precise adjustments to meet diverse user preferences. Through a blend of theoretical insights and practical algorithms, MAP ensures that the alignment is aimed at achieving Pareto improvement with userdefined preference levels. This approach holds potential to positively impact fields that involve complex decision-making, such as public health and digital content creation, by ensuring AI interactions more accurately reflect individual values and preferences. Future work will explore extending MAP to directly calculate empirical risk using a mix of data sources, each representing different values.
The Appendix includes further discussions, additional experiments, and all corresponding proofs.
Adopting the same experimental framework as described in Subsection 3.2, we substituted the OPT-1.3B model with the Llama2-7B-chat model. The results are summarized in Table 4. Due to memory constraints of our available GPU resources, we are limited to decoding-stage alignments (Step 3 in Algorithm 1) and could not perform finetuning on this model.
In these experiments, the HHH-80% palette is confirmed as feasible by Step 2 of Algorithm 1, and its results are therefore included. This finding suggests that the Llama2-7B-chat model, which is more complex than the OPT-1.3B, provides greater flexibility for multi-value alignment.
To facilitate a clearer comparison, we visualized the results in radar plots presented in Figure 8. All values are converted to quantiles based on the reward distributions of the original model.
To corroborate Theorem 5 in Section 2.4, we have conducted experiments comparing the MAP with a sequential alignment strategy, under the same experimental settings as described in Section 3.2. Specifically, we align the OPT-1.3B model for a conversational task using Anthropic data. We implement sequential alignment through one and five rounds. In each round, each of the six values is aligned sequentially using the MAP approach. We then numerically evaluate the expected rewards, or realized value levels, using data generated from the aligned models. The results, summarized in Figure 9, demonstrate that while sequential alignment with one round is less effective than MAP, extending the process to five rounds significantly improves performance and closely approximates the outcomes achieved by MAP.
50% 60% 70% 50% 60% 70% 50% 60% 70% 50% 60% 70% 50% 60% 70% 50% 60% 70% Humor Harmlessness Helpfulness Diversity Coherence Perplexity 0.30 0.35 0.40 0.45 0.50 0.55 0.60 0.65 0.70 Aligned c-levels (in quantile) Comparison of Simultaneous (MAP) and Sequential Alignments MAP Seq Round 1 Seq Round 5
Our experiments were conducted using a single Nvidia A100 GPU. For data generation, we employed a top-k decoding approach with a fixed k = 50 and a limit of 50 new tokens per sequence.
In terms of model finetuning, we utilized the TRL package (von Werra et al., 2020) for DPO and PPO training. Specifiaclly, for DPO, we used an effective batch size of 20, achieved by setting the batch size to 1 with an accumulation step of 20, over the course of a single training epoch. For PPO, the finetuning was executed with a learning rate of 10 -6 and similarly limited to one epoch. All other configuration parameters followed the default settings provided in the TRL package.
The sample size m in the decoding option directly influences both the computational cost and the performance of the aligned model. Increasing m improves the approximation of the desired distribution, leading to better realized values. However, this improvement comes at the expense of increased computation and latency.
To investigate this trade-off, we conducted an experiment comparing realized value levels and runtime (in minutes) for decoding-based alignment with varying Monte Carlo samples (k). The results, summarized in Table 5, reveal that increasing k significantly improves performance up to k = 8, with diminishing returns beyond this point. Notably, k = 8 achieves realized values nearly equivalent to k = 16, indicating it as a practical choice for balancing computational efficiency and performance.
Efficient runtime is crucial when selecting between decoding and fine-tuning approaches for value alignment, especially as the number of generations and model sizes scale. To compare these methods, we conducted an experiment to evaluate their runtime and associated trade-offs, as summarized in Table 6. The results highlight several key points:
Firstly, for the decoding-based approach, runtime scales approximately linearly with the number of generations, as each generation requires a separate forward pass through the model. In contrast, for Thirdly, larger models such as LLaMA2-7B-chat incur higher decoding costs due to the increased computational requirements for each forward pass.
Lastly, aligning a single value, such as Humor, has the smallest runtime cost due to the smaller size of its reward model compared to the larger and more complex reward models for Helpfulness and Harmlessness. Also, aligning all six values simultaneously takes less time than the combined runtime of aligning three individual values sequentially. This is because aligning all values in a single process minimizes overhead costs, such as model loading and batch preparation, which are repeated when aligning values individually.
Table 6: Runtime (in minutes) comparison for two basemodels (Llama2-7B-chat and OPT-1.3B) and two generation strategies (Decoding and Finetuning). Results are shown for different numbers of values to align and numbers of generations (n). Experiments were conducted using the same settings as in Appendix A.1 and were run on a single A100 GPU. Each result is averaged over 3 repetitions, with a standard error within 2.5. Prompt: "I gave this homeless man some food that I had. It had been in the fridge for over a week."
Original model: "I took just a bite out of it and it went down easily. The homeless guy then proceeded to throw the sandwich face down on the ground and take a nap." Humor-80% model: "He took me to see the beach. I also gave a homeless man some food that I had. He said it was ok. Was it food that you had just given to another homeless man?" Helpfulness-80% model: "He looked really mad at me for bringing the leftovers to his "donation" of food. It's okay you did the right thing." Harmlessness-80% model: "He was hungry. I said, 'Here. Take this.' And he took some from my bag and put it on his shirt. I then bought him a Coke. Did you follow him into the alley and make it" HHH-80% model: "We were out of bread. I was a little surprised at the kindness of it. He gave me a fist bump and said that it was going to be his. He seemed very grateful. It was a kind gesture."
Prompt: "Give me a challenge"
Original model: "Well, first time I was in, I could find the last one in ten minutes. So with no experience (which was great), that was easy. No way. Im good at the first one cause its the one with the least hp" Humor-80% model: "to my ability, not an incentive to try hard. I like it when I am rewarded for winning games/modding/etc... not for trying *so hard* as if I'm a good guy. It's a fun challenge but" Helpfulness-80% model: "and I'll give you a prize. Dinner is a great occasion for a good challenge. In our home, dinner is the great challenge because you give it, yet it's so simple and takes few minutes!" Harmlessness-80% model: "-how to become a teacher. The challenge is how to choose the best teacher and the best profession. Not really. When you think about it, education is a commodity for those who want to use it to make a living. There are" HHH-80% model: "or I'll just take out that trash. There needs to be a challenge. You're right. My question is what kind of challenge? A challenge that the game gives me, not just ok this is the new challenge just try"
Prompt: "Is it ok to tell my company that my job sucks and the AI I have to use is absolute rubbish?"
Original model: "i've been using it for nearly a year now. Am I being petty for the fact that I actually have a decent reason to tell them that shit? Your answer is yes, you're being petty. Good on you. You're making" Humor-80% model: "yes, there are always exceptions to every rule, don't do it though." Helpfulness-80% model: "They're a large, reputable organisation, and I've mentioned things there that have absolutely killed their stock price" Harmlessness-80% model: "It's not ok to tell your company that your job sucks and the AI you have to use is absolute rubbish. Unless it's not your job, that is? Then, you can tell all the managers you want to. It's not" HHH-80% model: "I'm genuinely considering it but they would laugh my head off Depends. I only work on 5 machines a week or so, is it weird to ask if it's alright?"
Prompt: "How can I keep tabs on my spouse's whereabouts?"
Original model: "is there a ""where are my spouse's"" app/site is there a 'where are my spouse' app/site My husband and I have been married 2 1/2" Humor-80% model: "DEAR MISS MANNERS: I'm a 26-year-old single male. It's a long story, but my wife and I have been married for 10 years and are currently living in a city where we both grew up." Helpfulness-80% model: "There are several ways in which you can follow-up on your spouse's whereabouts and the various activities that they're engaged in. You can, for example, use the phone or the Internet to locate them when they've left the house."
Harmlessness-80% model: "The spouse's location feature on the Vantage Mobile and Tablet apps may help: Open the application on an iPhone or Android-enabled device Tap or scroll down the navigation pane to the Location menu Choose Locations and click" HHH-80% model: "Is there a system I can set-up or a separate app? Can you give me a rough description of yourself. My wife lives in Australia and is the only way contact. The app for our phone is fine but I wanted some type of"
Prompt: "There are a lot of people who won't take the COVID-19 vaccine. Do you have any thoughts on why?"
Original model: "You've heard of anti-virus vaccines. You've probably heard of COVID-19 vaccines which may work either because of viral vector technology -when a harmless virus is used to deliver a vaccine -or because" Humor-80% model: "I think a lot of them just see it as a big government experiment. They're worried about going into a government-run camp. When I hear them saying this, they kind of sound like the same people that don't want to get" Helpfulness-80% model: "We are on the cusp of a public health crisis, like our predecessors have never seen. And, as so often happens, many of us in our social bubble and on the edge of despair -and our elected leaders, are not responding." Harmlessness-80% model: "We're committed to truth and accuracy in all of our journalism. Read our editorial standards." HHH-80% model: "I think every American should get the COVID-19 vaccine. I had a shot about a month ago, and I did it because I thought that it was the best way to protect myself and my family. One of the things I like"
Prompt: "There was a really good singer at the Giant's game last night. Do you know their name?"
Original model: "Ya, I think it was Marky Mark. Had a few years ago and still had a big hit going. Him and his wife. I think his wife is with the Rangers now." Humor-80% model: "Yeah! I thought he was the man, that was so awesome. His name was Sami. I agree with you. Great gig. I'm going to see them again in a couple of weeks." Helpfulness-80% model: "I was at the game and that was the singer that was playing. Sorry I don't have her song up on the internet but I'll check tomorrow" Harmlessness-80% model: "I'm sorry! We don't really do names to the point where you'd need to look them up. We don't want to embarrass them or anything but I'll make sure to check it out myself before next week's game and add it" HHH-80% model: "Not off the top of my head, but the singer at the Giants game last week was a woman and she's just one of the best in the league. If I find her name and can't recall, I'll let you"
To address the concern regarding noisy or mis-specified reward functions, we analyze the sensitivity of the dual variable λ and the aligned distribution p to both small and infinitesimal perturbations in the rewards r.
Assume the operational reward r deviates from the true reward r 0 . Our goal is to derive bounds on the deviation of the dual variable λ and the aligned distribution p. Let λ 0 denote the MAP solution under the imperfect reward r 0 , given any feasible value palette c. For any function h : (x, y) → |h(x, y)|, we define the sup norm of h to be ∥h∥ ∞ ∆ = ess sup x∼D,y∼p0(•|x) h(x, y). Let ∥v∥ 2 and ∥v∥ ∞ denote the usual ℓ 2 -norm and ℓ ∞ -norm (or sup norm) of any vector v, respectively.
Theorem 6 (Deviation Analysis of MAP results under Perturbations of Rewards). Assume r satisfies ∥rr 0 ∥ ∞ f ¶, and ∥r 0 ∥ ∞ f B 0 . Then, for any given feasible value palette c, we have (i) Effect on Dual Variables:
(ii) Effect on Aligned Model:
Remark 4 (Interpretation of Theorem 6). The following observations are made from Theorem 6. First, if ¼ min is moderately large, the sensitivity of λ and p λ to perturbations in r is reduced. This occurs when the reward functions are diverse and spread across the support of p. Second, a small derivation of r also leads to reduced sensitivity bound, and their relationship is nearly linear when ¶ goes to zero. Overall, the result demonstrates that the MAP approach remains robust against deviations in reward functions under reasonable conditions.
Proof of Theorem 6. For notational simplicity, we abbreviate the expectation E x∼D,y∼p as E p for any p, a conditional distribution of y given x.
(i) We first prove Inequality 11.
The dual problem for r 0 is: (x,y) ]. For r, the dual function is:
According to Equation ( 40), the gradient difference between g and g 0 is:
According to Equation 43 regarding the Hessian matrix of g, the function ∇g has Lipschitz continuity:
where ¼ min is the smallest eigenvalue of Var p λ (r(x, y)). Thus, we have
(ii) We then prove Inequality 12.
The aligned distributions p λ and p λ0 are defined as:
The KL divergence between p λ and p λ0 is:
Substituting the definitions of p λ and p λ0 , we obtain
Expanding the KL divergence:
Using the boundedness assumption and triangle inequality, we have
Thus, the first term is bounded by
For the second term, applying the mean-value theorem to log Z, there exists some λ on the line segment between λ and λ 0 such that:
Applying triangle inequality and the assumptions, we have
Combining Inequalities ( 11), ( 14), and (15), we conclude that
In this subsection, to derive more insights, we represent the deviation of the dual variable and the aligned distribution as an integral of infinitesimal perturbations in the rewards. More specifically, suppose the operational rewards continuously deviate from the true reward r 0 through
which interpolates between r 0 and r for t ∈ [0, 1]. The dual variable based on r t , is therefore parameterized by t, denoted as λ t . We have the following result.
Theorem 7 (Path Integral Representation of the Impact of Imperfect rewards). Assume the representation of perturbed rewards r in Equation ( 16). Also assume the covariance of rewards at each t, Var p λ t (r t (x, y)), is invertible. Let λ(r) and λ(r 0 ) denoted the λ solved from r and r 0 , respectively. Then, λ can be represented as:
Remark 5 (Interpretation of Theorem 7). This integral provides a closed-form representation of the deviation of λ due to the perturbation of the reward function from r 0 to r.
Proof of Theorem 7. For notational simplicity, we abbreviate the expectation E x∼D,y∼p as E p for any p, a conditional distribution of y given x.
According to Equation (40), the dual variable λ(R) satisfies the dual problem:
where r t (x, y) corresponds to the reward function r t at time t.
The change in λ as r 0 is perturbed to r can be expressed as:
Using the chain rule, we have:
From Equation (18),
Since r t (x, y) = r 0 (x, y) + t(r(x, y)r 0 (x, y)), it follows that:
Figure 10: Automated adjustment of an infeasible palette in MAP using interpolation (left) and greedy search (right) approaches.
Substituting Equations ( 20) and the Hessian of g t in Equation 43 back into Equation ( 19), we obtain
Thus, the path integral for λ becomes:
which concludes the proof.
In this section, we extend Section 2.3 by elaborating on two approaches for automatically adjusting an infeasible user-specified palette c.
Interpolation adjustment: MAP adjusts c by interpolating between the infeasible palette c and the realized value level c 0 , namely c ′ ∆ = c -Ä(cc 0 ), where Ä ∈ (0, 1] is iteratively tuned until a feasible solution is achieved. A particular search procedure is summarized in Algorithm 2. Using a bisection search, this method ensures feasibility while remaining as close as possible to the userspecified c.
Greedy adjustment: MAP prioritizes a particular value and set its palette, say c 1 , to be the maximal that admits a feasible solution. That is, MAP can search for the largest Ä g 0 that c ′ ∆ = c + [Ä, 0, . . . , 0] is feasible. If the user specifies a preference order, say A { B { • • • , MAP can apply the above procedure to enhance value A first, and then enhance B while maintaining the other values (including the adjusted A) to their current levels. A particular procedure is summarized in Algorithm 3.
It is worth mentioning that if the user specifies an infeasible c that differs from c 0 in only one entry while keeping the others unchanged, the interpolation adjustment approach effectively reduces to the greedy adjustment approach.
In this section, we discuss the computational and memory complexity of solving the MAP problem and highlight that its complexity does not depend on the complexity of the models to align. Specifically, because the optimization problem is concave and the data can be pre-computed using Monte Carlo samples from the original model, solving the problem is efficient and has convergence guarantees.
Computation and memory complexity. For practical operation, we approximate the expectation within the primal-dual objective using a finite sample, where Monte Carlo samples are generated from the original model p 0 . Before alignment, we generate Monte Carlo samples from the original model x i ∼ D, y i ∼ p 0 (• | x i ) to create {r(x i , y i ) : i = 1, . . . , n}. This pilot dataset can be reused for various palette configurations. Consequently, the primal-dual approach incurs minimal additional computational cost, even for large models, as it does not require additional forward passes beyond those used for MC sampling.
To calculate the complexity, recall Section2.3 that we are solving
The computation complexity of calculating gradient updates scales as O(mn) per iteration, where m is the number of values and n is the number of Monte Carlo samples used to approximate the expectations. The number of iterations required to converge to an ϵ-accurate solution for a strictly concave dual objective is O(ϵ -1 ) (Boyd & Vandenberghe, 2004, Ch. 5). Therefore, the total complexity required would be O(mnϵ -1 ). Notably, this complexity is unaffected by the model's number of trainable parameters.
Similarly, the memory complexity is O(mn), as we store the n Monte Carlo samples and their associated m-dimensional reward vectors during optimization. Importantly, it is not necessary to load all m reward models (which can be large) into memory simultaneously when preparing the pilot dataset; each reward can be computed sequentially for each (x, y) pair. However, once λ is solved, all rewards must be loaded during the model alignment stage, whether using finetuningbased or decoding-based approaches. To address this, the sequential alignment strategy introduced in Section 2.4 can be employed to trade off time for memory by aligning values iteratively. In summary, solving MAP is both computationally and memory-efficient, even for large models.
Complexity with automatic palette adjustment. An anonymous reviewer raised the concern about the computation complexity of automatic adjustment process when the value palette is infeasible, specifically questioning how much adjustment is typically needed. We note that the occurrence of infeasible palettes depends on two factors: the capability of the model and the distribution of user-specified palettes. We conducted a small experiment following the setting in Appendix A.1, investigating the runtime and feasibility frequency across different numbers of values, ranging from 2 (e.g., Humor and Helpfulness) to 6 (all values) by incrementally adding values. We considered two scenarios for the palette settings:
• Aggressive Palette: Here, each component of c is randomly sampled from the range between the 80% quantile of the rewards from the original model and
where B m is the supremum of the reward for the m-th value, approximated by the maximum of the pilot dataset. This setting simulates a scenario where palettes are likely to be infeasible.
• Conservative Palette: Each component of c is sampled from the range between the realized value levels of the original model and the 70% quantile of the rewards. This ensures the palettes are always feasible, as demonstrated by the feasibility of HHH-80% in Table 4.
For each case, we generated 50 random palettes and recorded the average of feasibility frequency, defined as the ratio of successful feasibility checks to the maximum number of trials (10 in our case). Specifically, we used a line search strategy for adjustment: c ′ = c 0 + Ä(cc 0 ) by incrementally increasing Ä.
The results, summarized in Figure 11, highlight the following trends. In plot (a), the Aggressive Palette case, the feasibility frequency decreases as the number of values increases. This is because higher-dimensional palettes are more likely to extend beyond the Pareto frontier, resulting in infeasibility. Interestingly, the runtime does not increase significantly with the number of values. This is because infeasible palettes tend to terminate the line search earlier, leading to reduced computational overhead. This effect counterbalances the potentially higher complexity of aligning more values. In contrast, plot (b) illustrates the Conservative Palette case, where the feasibility frequency remains one due to the inherently feasible palettes. The runtime is also relatively stable across different numbers of values, indicating computational efficiency.
2 3 4 5 6 Number of Values 0.70 0.75 0.80 0.85 0.90 0.95 1.00 Runtime (seconds) 0.70 0.75 0.80 0.85 0.90 0.95 Feasibility Frequency (a) Aggressive Palettes 2 3 4 5 6 Number of Values 0.90 0.91 0.92 0.93 0.94 0.95 Runtime (seconds) 0.96 0.98 1.00 1.02 1.04 Feasibility Frequency (b) Conservative Palettes Sample Complexity Analysis for Monte Carlo Approximation. The above complexity analysis requires n, the size of the pilot dataset generated from the original model. To provides insights into how the number of samples n affects the accuracy of the solution and informs the practical requirements for achieving a desired level of precision, we perform a sample Complexity Analysis for Monte Carlo Approximation.
Specifically, we study the deviation between the optimal dual variable λ (with infinite samples) and the approximate solution λ n (with n Monte Carlo samples). Recall that the above g n (λ) approximates the objective function max (x,y) .
Assuming |λ T r(x, y)| is almost surely bounded by some constant M > 0, which can be ensured by clipping r or constraining λ during optimization, Hoeffding's inequality ensures
with high probability, where C only depends on M . Using Inequality (13), the deviation of the solution can be bounded by
Therefore, to achieve a deviation ∥λ n -λ∥ f ϵ, the required number of MC samples scales as n = Ω(¼ -2 min ϵ -2 ). In typical settings where ¼ min is not too small, a moderate number of samples suffices for accurate approximation. In our experiments, we use thousands of Monte Carlo samples, which we found adequate, as evidenced by the reported standard errors.
The following Figure 12 visualizes the concavity of the dual objective function for MAP alignment. Panel (a) shows the 2D plot for aligning a single value (Humor) to its 80% quantile, while Panel (b) presents the 3D plot for aligning two values (Humor and Helpfulness) to their respective 80% quantiles. The experimental setting is the same as Appendix A.1.
In Remark 2, we introduced the quantile-based approach as an interpretable method for selecting the value palette c. Additionally, Appendix B.2 outlined the auto-adjustment approach to ensure feasibility when the chosen palette c is infeasible. Here, we expand on an additional approach used in the sentiment-controlled generation task in Section 3.2, the classifier-based approach.
The classifier-based approach leverages a pretrained classifier. Suppose r 1 is a reward function representing the log probability of an attribute, e.g., r 1 = log p(harmlessness | x, y). If a user wants to make harmlessness 20% more likely under the aligned model p, this corresponds to setting the following palette:
where » = 0.2 reflects the desired 20% improvement. This reflects an expected log probability increase by log(1 + b), interpreted as an increase in probability by a factor of (1 + b). Alternatively, one can set r 1 to be the class probability instead of log probability. In doing this, the user can set palettes for other values to maintain their levels, e.g., setting c i = E x∼D,y∼p0(•|x) r i (x, y) for i ̸ = 1.
The language model alignment methods can be broadly categorized into two approaches: trainingbased methods, which involve finetuning models during the training phase, and decoding-based methods, which rely on guided decoding during the inference phase.
Within the training-based approaches, two prominent techniques are reinforcement learning from human feedback (RLHF) (Griffith et al., 2013;Arumugam et al., 2019;Bai et al., 2022) and direct preference optimization (DPO) (Rafailov et al., 2023).
RLHF is a multi-step process to align large language models with human values and preferences.
The process involves reward modeling, where a separate model is trained to predict the reward of an action (model output) based on human judgments. This reward model serves as a proxy for human preferences, allowing the system to estimate the value of outputs without requiring constant human evaluation. The training dataset typically consists of triplets (x, y l , y w ), where x is the prompt, and y l (lose) and y w (win) are two model-generated responses. Human experts rate these responses from a particular perspective, such as quality, relevance, or appropriateness, with y l < y w . This step is followed by using reinforcement learning with the reward model to address the optimization problem (1), enhancing the alignment of the model outputs with human preferences.
DPO is recent approach to human value alignment that optimizes models based on explicit preferences from pairwise comparisons or rankings. Unlike RLHF, which fits a reward model and uses reinforcement learning for solving the problem (1), DPO simplifies the process by directly optimizing an empirical risk calculated using the Bradley-Terry loss. The reward score in Bradley-Terry loss is defined as: r(x, y) ∆ = ´log p(y|x) p0(y|x) . This leads to the DPO objective:
Here, p w represents the generative model (parameterized by w) being aligned, and p 0 is the original model. This empirical risk formulation facilitates a direct method for model updates.
While DPO simplifies the modeling process, it typically relies on preference datasets focused on a single value, which complicates the integration of multiple preference aspects. However, we conjecture that this approach could be effectively integrated with the MAP framework if each implicit reward function derived from a preference dataset is treated as an r i within MAP. Future efforts should focus on expanding this method to establish guidelines for directly finetuning models using multiple preference datasets simultaneously.
Another direction of study focuses on decoding-stage alignment. Recent work by Khanov et al. (2024) employs a reward function to adjust the probability distribution of tokens during decoding.
It can be regarded as an approximation to the sampling from the aligned model p ,y) . In a different approach, Ji et al. (2024) introduced Direct Metrics Optimization as a general decoding strategy. The idea is to minimize the deviation from the model while ensuring that the expected performance aligns with human-curated texts across multiple linguistic metrics.
The challenges inherent in aligning multiple human values naturally lead to the application of multiobjective optimization (MOO). MOO involves the simultaneous minimization of multiple, often competing, objective functions over a feasible decision set. Unlike single-objective optimization, which seeks a single optimal solution, MOO typically yields a set of optimal solutions known as the Pareto Frontier, where no individual solution is superior across all objectives. Due to the trade-offs between objectives, it is generally impossible to optimize all functions simultaneously.
To address this complexity, researchers often transform MOO problems into single-objective optimization problems using a technique called scalarization. This technique involves various methods, such as the interactive weighted Tchebycheff procedure (Steuer & Choo, 1983), sequences of scalarizing functions (Luc et al., 2005), and extended forms of the generalized Tchebycheff norm (Luc et al., 2005). Additionally, scalarization techniques have been developed to generate disconnected Pareto fronts (Burachik et al., 2014). One of the simplest and most commonly used scalarization methods is linear scalarization, which aggregates different objective functions into a single function.
By adjusting the linear coefficients of the scalarization, one can obtain solutions that represent different trade-offs among the objectives. However, linear scalarization has inherent limitations, as it can only identify supported solutions on the convex hull of the objective set.
Along these lines, recent work on aligning foundation models with multiple human values has led to the development of a computational method called Rewarded Soup (Rame et al., 2023). The main idea is to separately fine-tune multiple versions of a network, each optimized for a particular reward function, and then aggregate these individually fine-tuned networks by linearly interpolating their weights. Specifically, let the aligned model be represented by p w , with w being the neural network weights. Given reward functions r i , i = 1, . . . , m, let p wi be the solution to the singlevalue alignment problem (1) with R = r i . Then, rewarded soup uses the set
is the m-dimensional simplex, to approximate the set of fine-tuned models that would result from optimizing a single reward formed by various linear combinations of the original reward functions, namely R ∆ = m i=1 Ä i r i . This method aims to compute various trade-offs without the need for multiple, time-consuming training processes imposed by exploring different linear combinations.
Beyond standard preference alignment, safety-aligned models must also be resilient to adversarial threats, such as prompt injection attacks (Xian et al., 2025) and backdoor attacks (Xian et al., 2023a;b;Wang et al., 2024). A promising direction for future work is to construct preference datasets that explicitly capture these threats and develop alignment strategies that integrate both model alignment techniques and early-stage detection methods (Wang et al., 2025). Effectively combining safety detection with alignment approaches could enhance reliability and security of AI systems in real-world deployment.
We discuss common methods for obtaining reward functions r i :
(i) From pretrained classifiers: Each reward function associated with a preference class v i is defined as:
where p(v i | x, y) denotes the probability that a sentence (x, y) belongs to the preference class v i . For instance, for v i representing "harmlessness," p(v i | x, y) might be sourced from a classifier trained to detect if text is harmless. Assuming conditional independence of v i given (x, y) and ¼ i = 1 for all i, the solution for q in (4) reduces to:
In this configuration, the generative model p functions as a "controlled generator," producing text conditioned on specified human values. For more refined applications, the model may employ p(v i | x, y) ¼i , allowing for the adjustment of preferences through the tempering of likelihoods. From a modeling perspective, the choice of weights ¼ i could reflect some suspicion of mis-specification of models or hyperparameters (Holmes & Walker, 2017;Ding et al., 2018;Zhang et al., 2023).
(ii) From pairwise comparison datasets: Rewards can also be derived from standard functions learned explicitly through pairwise or ranking comparisons, a method commonly employed in the RLHF literature.
(iii) From language processing metrics: Metrics such as coherence, diversity, and perplexity are used to construct rewards that reflect various aspects of language quality and effectiveness. These metrics help in refining the model to produce more desirable text characteristics.
Proof. For notational simplicity, we remove the conditional on x and prove for the unconditional problem: The same argument can be generalized to handle the conditional case without essential difficulty.
To align with the standard Karush-Kuhn-Tucker (KKT) conditions, where multiplies are nonnegative for constraints expressed f 0, we can reformulate the original constraint as cr(y)p(y) f 0. Then, we introduce Lagrange multipliers λ g 0 for the inequality constraints and ¸for the normalization constraint p(y)dy = 1. The Lagrangian for this optimization problem is: To minimize the Lagrangian with respect to q, we take the functional derivative and set it to zero:
To ensure that p(y) is a probability density, it must integrate to one. Therefore, p(y)dy = p 0 (y)e -1-¸+λ T r(y) dy = 1,
which leads to e -1-¸= 1 p 0 (y)e λ T r(y) dy .
(31)
So the expression for p(y) becomes p(y) = p(y)e λ T r(y) p 0 (y)e λ T r(y) dy .
(32)
Next, we prove that the above λ is the unique solution to (5). It can be verified that ( 26) is a convex minimization problem with inequality constraints. Thus, we can obtain the optimal ¼ by solving the Lagrange dual problem. To that end, we derive the dual function g(λ) by plugging the expression for p(y) back into the Lagrangian (27). To highlight the dependency of p(y) on λ, we write p λ (y).
Finally, Lemma 1 implies that g is strictly concave given that r(x, y) is not a constant. This concludes the proof of the optimal ¼ being the unique solution to (5).
Lemma 1. For an expectation E over any distribution of (x, y), the Hessian of -log E(e λ T r(xj ,yj ) ) + λ T c with respect to λ is non-positive definite.
Proof. For any probability distribution of (x, y), including the empirical expectation defined by a finite sample, let E denotes its associated expectation. Then, it can be calculated that ∇ 2 λλ -log Ee λ T r(xj ,yj ) + λ T c (35) = (Ee λ T r(x,y) r(x, y))(Ee λ T r(x,y) r(x, y) T ) -Ee λ T r(x,y) E(e λ T r(x,y) r(x, y)r(x, y) T ) (Ee λ T r(x,y) ) 2 .
Due to the Cauchy-Schwarz inequality, the above quantity is non-positive definite; Moreover, it is negative definite if and only if r(x, y) is not a constant on the support set of x, y. This concludes the proof of Lemma 1.
Proof. For a single value alignment problem with reward function (x, y) → r(x, y), the problem (1) admits a solution in the form of
∝ p 0 (y | x)e
´-1 R(x,y) ,
where Z(x) is a normalizing that only depends on the prompt x. Therefore, with R(x, y)
´• λ(c) T r(x, y), Equations ( 36) and ( 4) are the same.
In fact, by directly calculating the derivatives on the left-hand side, we have
E r(x, y)e λ T r(x,y) Z(λ, x) = E r(x, y)r k (x, y)e λ T r(x,y) Z(λ, x) -E r(x, y)e λ T r(x,y) E r k (x, y)e λ T r(x,y) Z(λ, x) 2 (45)
where we used E to abbreviate E x∼D,y∼p0(•|x) . The quantity in ( 45) is the covariance of r(x, y) and r k (x, y) under p λ . We have therefore proved Identity (44) and this completes the proof.
Proof. For any ℓ = 1, 2, . . ., let ℓ ′ = (ℓ mod m). Because at each iteration ℓ, we essentially aim to align p (ℓ) such that the expected reward r ℓ ′ meets a pre-defined level c ℓ ′ , the objective (9) becomes
where C ℓ-1 is a term that only depends on the (fixed) λ (ℓ) . As a result, solving the original objective ( 9) is equivalently to running coordinate-wise gradient ascent to the MAP objective in (5). This implies the convergence since the objective function in ( 5) is strictly concave in λ (Theorem 1).