We cover it in the paper to some extent. We found that things work out as long as y resembles an image. Moreover, the encoder is robust to a small amount of noise.

🙌 This work was led by Ron Raphaeli under the guidance of Prof. Miki Elad.

🧵[7/7]

✨ The result?

✅ Sharper images
✅ Significant speedups
✅ A simple framework for inverse problems with latent diffusion priors.

🧵[6/7]

🔥 Our solution: What if we could bypass the decoder entirely?

We designed a latent operator that mimics image-space degradations directly in the latent space, eliminating the use of the decoder and its Jacobian.

🧵[5/7]

⚠️ Worse, backpropagating through the decoder introduces artifacts into the restored images due to its Jacobian.

🧵[4/7]

💡 The challenge: Solving inverse problems with latent diffusion models is tricky because degradation operators (e.g., blur, noise) are defined in image space.

This forces costly decoding steps at every iteration, slowing everything down.

🧵[3/7]

🔗 Project page: ronraphaeli.github.io/SILO-website/
📄 Arxiv: arxiv.org/abs/2501.11746

🧵[2/7]

If we are talking about image-to-image tasks, its seems you need only one:

arxiv.org/abs/2406.00828