Hmm for benchmarks it depends on what I’m benchmarking. If it’s a metric I honestly just use brain trust at that point to log everything.

Asserts are super easy to just get started with and so I use them for simple eyeballing and checks ( including some pytests tests )

Interesting will give it a look, thanks so much!

This was a really good read! I’m curious though when it comes to building benchmarks, what are your favourite ways for

- Deduplication
- Quality Control

I’ve tried to build spatial reasoning benchmarks before but never released them

Hello sir!

Evals are "too damn expensive" until you:

• can't migrate underlying models safely
• can't add new features with confidence
• can't ship without HITL evals, which takes >100x longer
• product development and iteration grinds to a halt
• lose customer trust due to poor user experience

Yeah haha, this was my problem too.

I often find that some upfront investment in building these tools pays off significantly

Trying to find more AI folks too for a better feed, hope it improves lol

Think you mean not sshing in haha