Also, the technique makes robotics more accessible while trading understandability. People find that useful in other tools (llm, gen image models). This makes me think this will be useful also.

I was wrong about this. My change of heart was inspired by dexgen (zhaohengyin.github.io/dexteritygen/). They use BC for course motion, RL for fine grained. Instant policy could just do both in a single shot. Instant policy would be great for tele-op

Goal: vlm capable of planning + action
Problem: no data
Solution:
Bootstrap intelligence from vlm
1. Start with off the shelf vlm
2. Collect rollouts from code as policy from vlm for a set of tasks
3. GRPO over rollouts
4. Goto 2
5. Offline RL over vlm for direct obs->act

This btw is just regular old RL + curriculum learning. However, unlike control RL where we learn on the test set, here we have diff envs from train and test time.

Recipe.
1. Start with weak base + problems that range from really simple to really hard
3. Sort problems by how well model does on them
4. Pick 75% problems model can do, 25% it can’t. RL with GRPO. Use 10x rollouts + high temp on 25%
5. Repeat step 3 till all problems solved

Turns out problem was too low resolution. Going higher res fixed the problem.

On second thought, rectified flow doesn’t help with this problem. Need to look for something different

The samples look like the mean of the input. Suspicion that the similar examples result in a nullcline in velocity when t -> 1. Rectified flow should fix this. But, I hate the multi stage process of reflow.

Keeping functionality in modular libraries helps everyone doesn’t have to rewrite everything. Least cohesion of modules is a great rule of thumb.

That’s the best board I’ve ever see

The reward signal here would be if the user accepts some task completion or not. You’re fine tuning to improve the success rate of the agents taking into account the multi-turn decision making nature of agents.