Well, the last thing I said can be done today too, so not a great benefit, maybe something else

Yeah, I agree, it’s more out of curiosity to see if interpolation would be ≥, where my guess is that it would be at least on par. Like you said, it might be interesting for some interpretability study, e.g., examining how the AdaLN parameters vary with the interpolation coefficient

Interesting that you’re also not seeing a clear reason why this should fail!
Also, thanks for the great links! 2/2

Yes, this is exactly my idea, Fourier emb is implicitly making assumptions about which timesteps matter more, and I suspect that makes it harder for downstream transf (AdaLN etc) to use, compared to simple linear interp, considering how these emb are later used. 1/2

But since the range is fixed, wouldn’t interpolating two e.g. 256d random (or not) vectors work just as well, or even better, since it doesn’t bias any specific timestep (while standard PE does)? I don’t have an intuition for why this would be wrong. 2/2

Thanks @sedielem.bsky.social! I remember this reason for PE. What I’m unsure about is whether it still applies when encoding a timestep as 0-1. In the Flux code it’s first mapped to 0–1000, so I guess your point about needing enough granularity holds. 1/2

Black swans are real

Maybe the tradeoff is that adv loss pushes toward the prototype rather than the input, making the image more “realistic,” while perc loss keeps it close to the input, with less perceptual distortion. If the input is atypical, the two losses might go in different directions and balance each other

Human heads are out of fashion with the new regime