Isn't the simulation vs. data-driven boundary rather blurry? There's also the option of having additional restrictions & bottlenecks in learned simulators. It could enhance our understanding of what's under the hood, and it would improve data efficiency. Still giving up to some extent though.

The question is how the building blocks look like and how they are built, right? Following Chollet's parallelism, I'd suggest Lego with pipes. In the direction of SAVi++ (slot-attention-video.github.io/savi++/) but with stronger biases towards representations easy to interpret. Long way to go...