LightNews LightNews
banner
shanestorks.bsky.social
@shanestorks.bsky.social
77 followers 190 following 8 posts
NLP researcher and postdoc at University of Michigan Weinberg Institute for Cognitive Science. Grounded language understanding and reasoning. (he/him) 🏳️‍🌈
Posts Replies Media Videos
shanestorks.bsky.social
So happy to finally share this last piece of my dissertation (and my first post on Bluesky)!

Obligatory photo after my recent hooding attached 🧑‍🎓
Shane Storks wearing academic regalia after his doctoral hooding ceremony.
December 17, 2024 at 1:59 PM
shanestorks.bsky.social
How well can VLMs detect and explain humans' procedural mistakes, like in cooking or assembly?
🧑‍🍳🧑‍🔧

My new pre-print with Itamar Bar-Yossef, Yayuan Li, Zheyuan Zhang, Jason J. Corso, and Joyce Chai dives into this!

arxiv.org/pdf/2412.11927
Dialog between a foundational VLM and itself to detect the incomplete state of the procedure "Unclip the pegs on the cloth" in an image showing a cloth pegged to a clothing line. The VLM generates the following questions and answers: 1. "Is there a cloth in the image? Yes", 2. "Are there pegs on the cloth? Yes", and 3. "Is there someone holding pegs? No". As the VLM asks these questions it becomes more confident that the procedure has not been successfully completed.
December 17, 2024 at 1:59 PM