Interesting. Is this because they have govt contracts and those would be jeopardized? Or uncertainty around whether those models will be banned for use in America, which adds a huge risk premium?

relevant paper: arxiv.org/abs/2410.08847

for all methods, it is better for data to be on-policy and be labelled as good/bad relative to the current state of the policy.

but ultimately this is a learning dynamics problem that transcends how the data is sampled

unpaired methods work the way we hope paired methods would, simultaneously increasing the relative prob of good outputs and decreasing the relative prob of bad outputs. this allows you to skip SFT entirely

it's not really on vs. off-policy. in theory, paired methods should increase the prob of good outputs, decrease prob of bad outputs. in practice, they decrease *both*. you need to do SFT beforehand so that you can pay this price and hope that relative to the base model, p(good|x) is still higher

all paired preference methods suffer from this problem while also being more inflexible. unpaired preference methods are always the way to go IME

Source: old.reddit.com/r/LocalLLaMA...

nominating myself @kawinethayarajh.bsky.social

These differences with the DPO version don't seem statistically significant?

(almost) all good poetry has high perplexity. it's by design something an out-of-the-box llm would be bad at. alignment on one poet would actually help imo.

Moderately grumpy UToronto alumnus nominating myself 🙋

Could i be added (recent alumnus)? Thank you!

Would love to be added thanks!