This means post-training (of this kind at least) optimizes KL(model || posterior), whereas pre-training optimizes KL(data || model). It also means post-training is mode-seeking (as opposed to mode-covering like pre-training), so those rewards better be well calibrated.

Reward functions are log-likelihoods, and the pre-trained model is a prior. The posterior target is the product of the likelihoods and prior (the prior weighting can equivalently sharpen or smooth your likelihoods). Rewards can be hard for math/code verification, or soft for subjective preference.

Ironically on my wrist