7/n Though the discretion of GPT-4o, DeepSeek-V3, and Sonnet 3.5 is mostly similar (see above), they exhibit subtle differences. Here are some examples:

6/n We can go further: by analyzing how often each principle ‘wins’ or ‘loses,’ we compute an ELO rating for each, giving a sense of that principle’s ‘priority’. This reveals huge discrepancies between human annotators, trained reward models, and LLMs.

5/n In conflict cases, annotators must decide which principle takes priority. By tracking how often different principles ‘win,’ we can measure which values annotators prioritize most. For example in HH-RLHF, we uncover distinct patterns in annotator decision-making:

4/n In consensus cases, we’d expect annotators to agree. In reality, we find widespread arbitrariness—annotators frequently override clear consensus in alignment datasets like HH-RLHF and PKU-SafeRLHF. And when algorithms annotate? The pattern varies significantly.

3/n We formalize that a set of alignment principles can agree/disagree in one of three ways:
1️⃣ Consensus (all relevant principles agree on the best output)
2️⃣ Conflicts (some principles disagree on the best output)
3️⃣ Indifference (no principle applies)

2/n AI alignment is done with preference-based learning: annotators rank responses, teaching models what’s "helpful" and "harmless." However, just like judges make decisions based on laws (judicial discretion), annotators balance conflicting principles (alignment discretion).

AI is built to “be helpful” or “avoid harm”, but which principles should it prioritize and when? We call this alignment discretion. As Asimov's stories show: balancing such principles for AI behavior is tricky. In fact, we find that AI has its own set of priorities. (comic by @xkcd.com)🧵👇