More demos and code available at intelligentsensingandrehabilitation.github.io/MonocularBio...

JD did a great job creating a Gradio demo so try it out and let us know what you think

And here is a video of JD going on a celebratory run that the preprint is out :)

Excitingly, in addition to producing accurate kinematics, we can measure the gait deviation index from these videos. We find this is quite sensitive to a number of different clinical backgrounds, and even more responsive after neurosurgical interventions than the standard clincial outcomes (mJOA).

Central to this was extending our end-to-end differentiable biomechanics approach to fitting both 2D and 3D keypoints measured from images. This can also account for smartphone rotation measured by our Portable Biomechanics Platform, which also makes this easy to integrate into clinical workflows.

We developed a novel approach to fitting biomechanics from smartphone video that produces kinematic reconstructions within a few degrees and has been validated across a wide range of activities and clinical backgrounds.

However, more work to do actually validating these against EMG recordings (we have this in many of our trials from our wearable sensor platform) and I suspect there will be lots of work to really tune it up.

Still, finding clinically sensible patterns is a promising first step.

The imitation learning policy is trained to replicate all the kinematics from 30+ hours of markerless motion data by driving a muscle-driven model with some regularization on muscle activation. Through training it learns muscle patterns that will replicate the kinematics.

Looking forward, we hope to combine things like this.

E.g. using BiomechGPT to understand movement user requests and then run simulations in the physics simulator using imitation learning, for example.

Either way, really starting to see promise for foundation models in biomechanics

Stay tuned :)

Particularly exciting was evidence of positive transfer learning as we increased the set of tasks it is trained on.

Of course lots of it also makes mistakes (the person in that video is using a crutch!). Lots of work to do and we are just starting exploring the opportunities from this approach.

We were super excited to see how well this performed across a range of tasks, even with fairly sparse annotation.

It's doing a great job at things like activity classification, which can be rather challenging for impaired movements, and more subtle things like inferring likely diagnoses.

The next paper is BiomechGPT arxiv.org/abs/2505.18465 with @antihebbiann.bsky.social and Ruize Yang which trains a language model to be fluent in tokenized movement sequences. This draws inspiration from MotionGPT but focuses on benchmarking performance on clinically meaingful tasks.

Shoutout to recent related work from @trackingskills.bsky.social group arxiv.org/abs/2503.14637

Great to see growing enthusiasm in this space
bsky.app/profile/trac...

And shoutout to MyoSuite for pushing the neuromuscular modeling in MuJoCo

Here is another example. It also captures some imperfections like little foot slips we want to improve.

Since then, we've tuned it up to handle anthropomorphic and muscle scaling. Still lots of work to do further tuning this as there are many things we aren't scaling such as mass and inertia and optimizing w.r.t. the EMG data we have from our wearable sensors.

The first is KinTwin arxiv.org/abs/2505.13436 which trains torque driven and muscle-driven policies to replicate movements of intact and impaired gait. It detects clinically meaningful features like propulsion asymmetries and muscle timing.

Teaser from a few months back: bsky.app/profile/peab...

It was also the initial meeting of Julius Dewald and Bob Sainburg's Society for Neuromechanics in Rehabilitation (SoNMiR) and it was great to present on biomechanics in rehabilitation and arxiv.org/abs/2411.03919. Very exciting for this society bringing together this community.

@jdpeiffer.bsky.social and Tim Unger also won second prize for the ICORR talks for "Differentiable Biomechanics for Markerless Motion Capture in Upper Limb Stroke Rehabilitation: A Comparison with Optical Motion Capture" arxiv.org/abs/2411.14992.

With Kyle Embry and the @abilitylab.bsky.social C-STAR team we also organized a half-day workshop "From Motion to Meaning: AI-Enabled Biomechanics for Rehabilitation" showcasing work from Georgios Pavlakos, Eni Halilaj, Vikash Kumar, Chris Awai, and Pouyan Firouzabadi.

With Dailyn Despradel, Derek Kamper, @marcslutzky.bsky.social, and @dougweberlab.bsky.social we organized a workshop on EMG biofeedback. Very much enjoyed the engaged discussion on how to disseminate these technologies into the real world.