In this talk, I'll explore the state-of-the-art in LLM evaluation, covering modern techniques like LLM-as-Judge. I’ll also discuss the essential processes and culture shifts that are required to launch reliable AI products and drive ongoing improvement.

Generative AI has made it easier than ever for companies to build products quickly. However, LLMs are inherently nondeterministic and unpredictable. Integrating LLMs into products demands an unprecedented level of quality assurance – requiring new strategies for continuous evaluation.

Then you can make a classifier that is tuned to whatever threshold matches a training set most accurately. Easy peasy.

Full post here: arcturus-labs.com/blog/2025/03...

The idea is really simple – basically you just make a classifier the returns a single token – good/bad or red/green/blue or 1/2/3/4 – But rather than looking at that token you look at the probability of the tokens.

Read the full deep-dive on why visual reasoning is the next frontier in AI:
arcturus-labs.com/blog/2025/03...

Follow for more.

Picture this: AI solving physics problems by visualizing objects in motion, or predicting social interactions by imagining body language and facial expressions. That's where we're headed.

The key insight: Just like chain-of-thought reasoning transformed how AI thinks through problems with words, visual reasoning will let AI work through problems by creating and analyzing sequences of images.

Today's AI can put a detective hat on your cat. Tomorrow's AI will help design your garden, rearrange your furniture, and solve complex spatial puzzles by actually visualizing different scenarios and their outcomes.

Thanks for reading our book and posting about it. I'm glad you got something from it!

A perfect reply. Thank you Evgeny.

I think I'll try this once a year until someone bites :)