We visualize motion latents by perturbing a spatial feature (green square) and forward predicting to propagate the perturbation.

Feature similarity between the propagated and original perturbation intuitively matches object correspondence!

We can also repeat this over multiple frames for tracking

and more of semantic segmentation and optical flow!

Video visualization of semantic segmentation...

We find that Midway Network is the only model that performs well on both semantic segmentation and optical flow tasks.

It even outperforms our prior work PooDLe (bsky.app/profile/meng...) without needing an external optical flow network!

Forward predictions at higher feature levels are used to infer motion latents at lower levels, motivated by iterative refinement in optical flow methods (e.g. PWCNet, UFlow).

We also introduce learnable gating units on residual paths of forward predictors to remove bias towards the identity mapping

Midway Network centers around a midway top-down path that infers motion latents between video frames (inverse dynamics).

It predicts future dense latent features from visual encoders, conditioned on these motion latents (forward dynamics).

Neuroscience theory (e.g. Wolpert et al. 1998) suggests that animals use future prediction and dynamics modeling for perception and control.

Inspired by this, we asked: can latent dynamics modeling learn useful representations of visual observations and their transformations over time, i.e. motion?

Animals learn to recognize objects and how they move from observation. SSL on videos emulates “learning by observing” but only for recognition or motion, not both!

Our #ICLR2026 work Midway Network is the first to learn both recognition and motion understanding from videos via latent dynamics 🧵