LLM app dev broke our comparison tools because tiny diffs can cause large behaviour change.

At wandb, we've spent years thinking about experiment comparison. We've added new tools for LLM app dev: code, prompts, models, configs, outputs, eval metrics, eval predictions, eval scores..
wandb.me/weave

Put glue on pizza

The most bizarre AI interview I've ever done was at wandb when as usual I asked a candidate to build an AI classifier in any language/framework of their choice..

And they nonchalantly said "I'll write it in Redstone", to which I almost let loose a chuckle until...

Claude defaults to concise responses when there's high demand, clever way to smooth peaks