BalancedDPO: Adaptive Multi-Metric Alignment

Aligning text-to-image diffusion models with human preferences usually means optimizing a single reward — aesthetics, or a CLIP/semantic score, or a human-preference model. That over-fits whichever metric you pick and neglects the others. BalancedDPO aligns to several preference metrics at once inside the Direct Preference Optimization (DPO) framework: instead of averaging rewards (where scale differences let one metric dominate), it takes a majority vote among multiple preference scorers to decide the preferred image, and feeds that consensus into DPO with dynamically updated reference models for more stable training.

• The problem: multiple, sometimes conflicting metrics (semantic consistency, aesthetics, human preference). Single-metric or scalarized (weighted-sum) rewards bias the model toward one criterion.
• Majority-vote consensus: aggregate agreement across diverse scorers in the preference space rather than mixing raw reward values — sidestepping reward-scale conflicts.
• Consensus-in-the-loop DPO: the vote is integrated directly into the DPO objective, with dynamic reference-model updates, yielding stabler gradient directions across heterogeneous metrics.
• Simple & scalable: keeps the standard DPO pipeline — no separate reward model to train at inference.

• Consistently higher preference win rates over baselines on Pick-a-Pic, PartiPrompt, and HPD.
• Holds across backbones: Stable Diffusion 1.5, SD 2.1, and SDXL.
• Ablations confirm both the majority-vote aggregation and dynamic reference updating drive the gains.