This is neat because we encountered a similar bottleneck with real-time fracturing of walls in Havok Unity. The solution was to use rectangular colliders and disable collisions with everything but the ground. Looking forward to trying this out and thanks for the updates!

youtu.be/4Lh9ULTybG8?...