Multi-Objective Alignment

A model trained to maximise helpfulness alone will happily explain how to synthesise dangerous chemicals if the question is phrased politely. A model trained to maximise harmlessness alone will refuse most interesting requests. The tension is not a training bug; it is a fundamental property of the objective landscape. Multi-objective alignment is the set of methods that take this tension seriously rather than sweeping it into a single weighted sum and hoping for the best.

Why one reward signal is not enough

Standard RLHF collapses all desiderata into one reward model trained on human preference comparisons. The implicit assumption is that a single number can rank every possible response on a dimension that conflates helpfulness, factual accuracy, safety, tone, length, and legal compliance. In practice this works tolerably for average queries but breaks at the extremes: a verbosely correct but unsafe answer can outscore a terse safe one, depending on which annotators happened to label which pairs.

The deeper issue is that human preferences are not consistent across contexts. A medical professional asking about drug interactions has legitimately different needs from an anonymous user asking the same question. A single scalar reward cannot represent this context-dependence; it averages over population heterogeneity and loses the variation entirely.

Formally, suppose we have \(k\) objectives \(r_1, \ldots, r_k\) (e.g., helpfulness, harmlessness, honesty). The standard approach produces a single reward \(r = \sum_i w_i r_i\) with fixed weights \(w_i\). The problem: any fixed \(w\) encodes a specific value judgement about trade-offs before training, and the resulting policy cannot be cheaply adapted to a different trade-off at inference time without retraining.

Multi-objective alignment instead attempts to learn a representation of the entire Pareto front, so that a specific operating point can be selected at deployment.

MODPO: folding multiple objectives into DPO

Multi-Objective Direct Preference Optimization (MODPO) extends DPO to handle \(k\) reward signals without reinforcement learning. The key insight is that each additional objective beyond the primary one can be treated as a margin constraint: a response must not only be preferred on the primary criterion, it must also clear a threshold on every secondary criterion.

MODPO trains the language model as an implicit collective reward model. Given a set of per-objective preference datasets \(\mathcal{D}_1, \ldots, \mathcal{D}_k\), the method combines them into a joint objective whose solution is theoretically equivalent to multi-objective RLHF but requires roughly three times less compute, because no separate RL training loop is needed. Empirical results on safety alignment and long-form QA show that MODPO traces out a Pareto front of policies indexed by the weight vector \(w\), allowing post-hoc selection of the operating point without retraining (Zhou et al., ACL Findings 2024, arXiv:2310.03708).

A simplified view of the MODPO loss for two objectives:

L_MODPO(θ) = L_DPO_primary(θ)
             + λ · margin_loss(r_secondary(y_w) - r_secondary(y_l) - δ)

where y_w and y_l are the preferred and dispreferred responses, δ is a margin hyperparameter, and λ controls how hard the secondary constraint is enforced.

Reward scalarisation and its pitfalls

The naive alternative to MODPO is fixed-weight scalarisation: define \(r = \sum_i w_i r_i\) and run standard RLHF or DPO. This is not wrong, but it commits you to a single trade-off at training time. Changing \(w\) after training requires a full re-run.

A more dangerous subtlety is that scalarisation can hide conflicts. Two objectives that look nearly aligned in aggregate can diverge sharply on a minority of inputs. A safety objective and a helpfulness objective both score most cooking queries near-identically, so the weighted sum looks fine; but on dual-use chemistry questions they pull in opposite directions and the aggregate gives you neither clearly. Pareto-based analysis would surface this conflict explicitly; scalarisation buries it.

Constitutional AI (Bai et al., 2022, arXiv:2212.08073) sidesteps explicit reward weighting by encoding objectives as a natural-language constitution and using AI feedback rather than human labellers. Each principle in the constitution is effectively an objective, but the enforcement happens through critique-and-revision passes rather than a joint optimisation. This produces implicitly multi-objective behaviour without a formal Pareto framework, at the cost of less interpretable trade-off control.

Reward over-optimisation amplifies objective conflicts

Goodhart's Law operates on each individual reward signal, but it interacts poorly with multi-objective settings. When you over-optimise for a scalarised reward, the policy typically exploits whichever sub-objective has the weakest proxy. Anthropic's early RLHF work (Bai et al., 2022, arXiv:2204.05862) observed a roughly linear relationship between RL reward and \(\sqrt{\text{KL}(\pi \| \pi_0)}\): the further the policy drifts from its initialisation, the more it has potentially exploited weaknesses in the reward proxy.

In multi-objective settings this means the objectives with the noisiest or most gameable reward models get exploited first. A helpfulness reward trained on crowd-sourced preferences is noisy; a rule-based safety reward (e.g., "does the output contain these keywords?") is gameable. The optimiser will find responses that look helpful to annotators but sidestep safety keywords, maximising both proxies while violating both underlying objectives.

This is why multi-objective alignment methods pair Pareto-front exploration with KL constraints or reference-model anchoring. The constraint is not just about preventing distributional shift; it is an implicit regulariser that keeps the policy from finding adversarial optima that exploit one objective at the expense of another.

Inference-time objective weighting

One attractive property of a model that has learned the Pareto front is that objective weights can be shifted at inference time without retraining. Several approaches exist:

• Reward-weighted decoding: compute \(k\) reward scores for each candidate token (or sequence via beam search) and combine them with a context-dependent \(w\) before selecting.
• Conditional generation: prepend a control token or system prompt encoding the desired objective weights. The model learns during training to condition its generation on this signal, trading off objectives accordingly.
• Model merging: merge separately fine-tuned specialist models (one per objective) using linear or SLERP interpolation. The merge coefficient plays the role of objective weight. This is cheap but requires separate training runs and can lose non-linear interactions between objectives.

KTO (Ethayarajh et al., ICML 2024, arXiv:2402.01306) is notable here: it aligns on a binary desirability signal (was this response good or bad?) rather than pairwise preferences. This makes it easier to build separate objective-specific datasets without requiring every response to be compared across objectives, which is the bottleneck for multi-objective dataset construction.

When it falls down

Incommensurable objectives. Pareto optimality assumes objectives can be evaluated independently on the same response. Some cannot. "Be maximally helpful" and "never reveal confidential training data" are not simply in tension; the second is a hard constraint, not a soft objective. Treating it as a reward and scalarising it allows the optimiser to trade a small safety violation for a large helpfulness gain. Hard constraints require separate enforcement mechanisms (filters, classifiers, constitutional principles) rather than inclusion in the reward mixture.

Preference dataset coverage. MODPO and related methods need preference data for each objective. If objectives are annotated by different annotator pools with different standards, the joint training signal is inconsistent. A helpfulness dataset collected from crowdworkers and a safety dataset labelled by domain experts do not share a common calibration; aggregating them implicitly assumes they do.

Pareto front instability. The true Pareto front shifts with the data distribution. A policy that correctly navigates trade-offs on the training distribution may behave unexpectedly on out-of-distribution inputs where the objectives have different relative strengths. Standard Pareto-front evaluation benchmarks (e.g., AlpacaEval paired with safety classifiers) capture this only partially.

Reward model disagreement. Running \(k\) separate reward models at inference time for reward-weighted decoding multiplies both the latency and the surface area for individual model failures. If one reward model is miscalibrated on a subpopulation, it will dominate the combined score for that subpopulation even if the other \(k-1\) models are accurate.

Objective specification is still hard. All of this machinery assumes you can operationalise your objectives as trainable reward signals. "Be honest" sounds clear until you need to label 100,000 responses for honesty and realise annotators disagree on 40% of cases. Formal multi-objective alignment is only as good as the quality of the objective specifications feeding it.

Further reading

• Zhanhui Zhou et al., "Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization," ACL Findings 2024. https://arxiv.org/abs/2310.03708
• Rafael Rafailov et al., "Direct Preference Optimization: Your Language Model is Secretly a Reward Model," NeurIPS 2023. https://arxiv.org/abs/2305.18290
• Yuntao Bai et al., "Constitutional AI: Harmlessness from AI Feedback," Anthropic 2022. https://arxiv.org/abs/2212.08073
• Stephen Casper et al., "Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback," 2023. https://arxiv.org/abs/2307.15217