Hamel Husain
@hamel.bsky.social
evals evals evals. https://evals.info
👀 Animals have been assigned.
Scheduled to print fall 2026!
We have iterated on this with over 3k students (and continue to do so). We give our students access to the full draft as part of our evals course (link in bio)
Scheduled to print fall 2026!
We have iterated on this with over 3k students (and continue to do so). We give our students access to the full draft as part of our evals course (link in bio)
November 8, 2025 at 5:03 PM
👀 Animals have been assigned.
Scheduled to print fall 2026!
We have iterated on this with over 3k students (and continue to do so). We give our students access to the full draft as part of our evals course (link in bio)
Scheduled to print fall 2026!
We have iterated on this with over 3k students (and continue to do so). We give our students access to the full draft as part of our evals course (link in bio)
Reposted by Hamel Husain
Love that @hamel.bsky.social is putting on a hackathon where the goal is for your agent to score the highest on evaluations, not just do something flashy.
click.convertkit-mail2.com/gkumlz753lc5...
click.convertkit-mail2.com/gkumlz753lc5...
October 31, 2025 at 12:07 AM
Love that @hamel.bsky.social is putting on a hackathon where the goal is for your agent to score the highest on evaluations, not just do something flashy.
click.convertkit-mail2.com/gkumlz753lc5...
click.convertkit-mail2.com/gkumlz753lc5...
Reposted by Hamel Husain
If your looking to get started with evals check out this cookbook from @hamel.bsky.social
cookbook.openai.com/examples/eva...
cookbook.openai.com/examples/eva...
Building resilient prompts using an evaluation flywheel | OpenAI Cookbook
This cookbook provides a practical guide on how to use the OpenAI Platform to easily build resilience into your prompts. A resilient prom...
cookbook.openai.com
October 6, 2025 at 8:34 PM
If your looking to get started with evals check out this cookbook from @hamel.bsky.social
cookbook.openai.com/examples/eva...
cookbook.openai.com/examples/eva...
"Can I just get an LLM to do my error analysis?"
We get this question constantly. The answer is no, and trying is the fastest way to miss critical bugs.
Full podcast: youtu.be/BsWxPI9UM4c?...
We get this question constantly. The answer is no, and trying is the fastest way to miss critical bugs.
Full podcast: youtu.be/BsWxPI9UM4c?...
October 3, 2025 at 9:12 PM
"Can I just get an LLM to do my error analysis?"
We get this question constantly. The answer is no, and trying is the fastest way to miss critical bugs.
Full podcast: youtu.be/BsWxPI9UM4c?...
We get this question constantly. The answer is no, and trying is the fastest way to miss critical bugs.
Full podcast: youtu.be/BsWxPI9UM4c?...
I recently sat down with Lenny Rachitsky to discuss why AI Evals are becoming the most sought after skill for product builders.
As a bonus, we step through an end-to-end example of building an eval in a spreadsheet so everyone can understand. See reply for links.
As a bonus, we step through an end-to-end example of building an eval in a spreadsheet so everyone can understand. See reply for links.
October 3, 2025 at 6:07 PM
I recently sat down with Lenny Rachitsky to discuss why AI Evals are becoming the most sought after skill for product builders.
As a bonus, we step through an end-to-end example of building an eval in a spreadsheet so everyone can understand. See reply for links.
As a bonus, we step through an end-to-end example of building an eval in a spreadsheet so everyone can understand. See reply for links.
This one is going to be spicy. 80% of the time I've seen a graph DB in production, it's been an overcomplicated mess (especially in AI applications).
In this talk, Jo and I will discuss when GraphDBs are overkill and when they actually make sense.
Sign up here:
In this talk, Jo and I will discuss when GraphDBs are overkill and when they actually make sense.
Sign up here:
You Don't Need a Graph DB
Many teams adopt graph databases believing they need specialized tools for relationship data, adding unnecessary complexity to their stack. This session reveals that for most use cases, the…
maven.com
September 24, 2025 at 5:09 PM
This one is going to be spicy. 80% of the time I've seen a graph DB in production, it's been an overcomplicated mess (especially in AI applications).
In this talk, Jo and I will discuss when GraphDBs are overkill and when they actually make sense.
Sign up here:
In this talk, Jo and I will discuss when GraphDBs are overkill and when they actually make sense.
Sign up here:
Reposted by Hamel Husain
For technical domains especially, getting non-DSes involved in analyzing outputs is vital. It’s hard to build anything good without it bc v1s almost always have major fail modes. Finding the appropriate system design—let alone optimizing—requires a tight coupling of output analysis and system design
Can non-data scientists write AI Evals? The answer is nuanced and not just "Yes". @eugeneyan.com and I discuss this in the context of the "analyze-measure-improve" cycle from our course.
Links to more resources in the reply
Links to more resources in the reply
August 26, 2025 at 4:56 AM
For technical domains especially, getting non-DSes involved in analyzing outputs is vital. It’s hard to build anything good without it bc v1s almost always have major fail modes. Finding the appropriate system design—let alone optimizing—requires a tight coupling of output analysis and system design
If you're wanted to learn applied AI evals but not sure if its for you, @sh-reya.bsky.social and I put together something that might help.
This free email course compiles what we've learned from teaching 2k+ students. It’s 17 emails plus 2 free e-books.
Here's the link: ai.hamel.dev/eval-course
This free email course compiles what we've learned from teaching 2k+ students. It’s 17 emails plus 2 free e-books.
Here's the link: ai.hamel.dev/eval-course
AI Evals Email Course
A free 17-part email series on the principles of application-centric LLM evals.
ai.hamel.dev
August 23, 2025 at 4:41 AM
If you're wanted to learn applied AI evals but not sure if its for you, @sh-reya.bsky.social and I put together something that might help.
This free email course compiles what we've learned from teaching 2k+ students. It’s 17 emails plus 2 free e-books.
Here's the link: ai.hamel.dev/eval-course
This free email course compiles what we've learned from teaching 2k+ students. It’s 17 emails plus 2 free e-books.
Here's the link: ai.hamel.dev/eval-course
Reposted by Hamel Husain
I've been eval pilled by @hamel.bsky.social . Everyone is all let's build some MCP servers and ship a minimal AISRE as quickly as possible. And I'm writing a design doc about how to build evals with @runme.dev so we can iterate rapidly on the AI
August 8, 2025 at 2:54 PM
I've been eval pilled by @hamel.bsky.social . Everyone is all let's build some MCP servers and ship a minimal AISRE as quickly as possible. And I'm writing a design doc about how to build evals with @runme.dev so we can iterate rapidly on the AI
Last chance to signup for this free lesson with OpenAI on evals, Including a sneak peek of their new eval products!
Link: maven.com/p/d2dc30/how...
Link: maven.com/p/d2dc30/how...
June 8, 2025 at 2:44 PM
Last chance to signup for this free lesson with OpenAI on evals, Including a sneak peek of their new eval products!
Link: maven.com/p/d2dc30/how...
Link: maven.com/p/d2dc30/how...
Can non-data scientists write AI Evals? The answer is nuanced and not just "Yes". @eugeneyan.com and I discuss this in the context of the "analyze-measure-improve" cycle from our course.
Links to more resources in the reply
Links to more resources in the reply
May 17, 2025 at 12:18 AM
Can non-data scientists write AI Evals? The answer is nuanced and not just "Yes". @eugeneyan.com and I discuss this in the context of the "analyze-measure-improve" cycle from our course.
Links to more resources in the reply
Links to more resources in the reply
If you are writing evals without error analysis, our course AI Evals for Engineers & PMs is for you. Begins monday next week. Full syllabus in this link: maven.com/parlance-lab...
May 12, 2025 at 5:17 AM
If you are writing evals without error analysis, our course AI Evals for Engineers & PMs is for you. Begins monday next week. Full syllabus in this link: maven.com/parlance-lab...
It is very easy to make mistakes when creating evals for your AI product. @sh-reya.bsky.social and I run through the most common errors in this talk.
35% discount code to our upcoming course in the video notes
youtu.be/GL0XhAj5LPE?...
35% discount code to our upcoming course in the video notes
youtu.be/GL0XhAj5LPE?...
LLM Evals: Common Mistakes
YouTube video by Hamel Husain
youtu.be
May 8, 2025 at 3:23 PM
It is very easy to make mistakes when creating evals for your AI product. @sh-reya.bsky.social and I run through the most common errors in this talk.
35% discount code to our upcoming course in the video notes
youtu.be/GL0XhAj5LPE?...
35% discount code to our upcoming course in the video notes
youtu.be/GL0XhAj5LPE?...
Reposted by Hamel Husain
GitHub CoPilot is one of the first commercially successful LLM products (predating ChatGPT). What was the secret? A robust eval suite!
In this lightning lesson, John Berryman will reveal the eval techniques (and mistakes) from working on this product
maven.com/p/da8264/how...
In this lightning lesson, John Berryman will reveal the eval techniques (and mistakes) from working on this product
maven.com/p/da8264/how...
April 29, 2025 at 5:27 AM
GitHub CoPilot is one of the first commercially successful LLM products (predating ChatGPT). What was the secret? A robust eval suite!
In this lightning lesson, John Berryman will reveal the eval techniques (and mistakes) from working on this product
maven.com/p/da8264/how...
In this lightning lesson, John Berryman will reveal the eval techniques (and mistakes) from working on this product
maven.com/p/da8264/how...
Reposted by Hamel Husain
@hamel.bsky.social & @sh-reya.bsky.social are two of the world's best on evals. They've built evals for 35+ AI apps & helped teams ship confidently. Now they'll teach everything they know on building evals that work.
Enrollment closes in 4 days.
Secret 35% discount code: maven.com/parlance-lab...
Enrollment closes in 4 days.
Secret 35% discount code: maven.com/parlance-lab...
April 30, 2025 at 2:56 AM
@hamel.bsky.social & @sh-reya.bsky.social are two of the world's best on evals. They've built evals for 35+ AI apps & helped teams ship confidently. Now they'll teach everything they know on building evals that work.
Enrollment closes in 4 days.
Secret 35% discount code: maven.com/parlance-lab...
Enrollment closes in 4 days.
Secret 35% discount code: maven.com/parlance-lab...
GitHub CoPilot is one of the first commercially successful LLM products (predating ChatGPT). What was the secret? A robust eval suite!
In this lightning lesson, John Berryman will reveal the eval techniques (and mistakes) from working on this product
maven.com/p/da8264/how...
In this lightning lesson, John Berryman will reveal the eval techniques (and mistakes) from working on this product
maven.com/p/da8264/how...
April 29, 2025 at 5:27 AM
GitHub CoPilot is one of the first commercially successful LLM products (predating ChatGPT). What was the secret? A robust eval suite!
In this lightning lesson, John Berryman will reveal the eval techniques (and mistakes) from working on this product
maven.com/p/da8264/how...
In this lightning lesson, John Berryman will reveal the eval techniques (and mistakes) from working on this product
maven.com/p/da8264/how...
Reposted by Hamel Husain
I keep hearing about the emerging role of AI PM. How is this any different than a normal PM? Is it hype? We are gonna find out in this free lightning lesson. I will ask difficult questions. With @schof.bsky.social and Aman Khan
maven.com/p/544677/wha...
maven.com/p/544677/wha...
April 27, 2025 at 6:17 PM
I keep hearing about the emerging role of AI PM. How is this any different than a normal PM? Is it hype? We are gonna find out in this free lightning lesson. I will ask difficult questions. With @schof.bsky.social and Aman Khan
maven.com/p/544677/wha...
maven.com/p/544677/wha...
I keep hearing about the emerging role of AI PM. How is this any different than a normal PM? Is it hype? We are gonna find out in this free lightning lesson. I will ask difficult questions. With @schof.bsky.social and Aman Khan
maven.com/p/544677/wha...
maven.com/p/544677/wha...
April 27, 2025 at 6:17 PM
I keep hearing about the emerging role of AI PM. How is this any different than a normal PM? Is it hype? We are gonna find out in this free lightning lesson. I will ask difficult questions. With @schof.bsky.social and Aman Khan
maven.com/p/544677/wha...
maven.com/p/544677/wha...
Reposted by Hamel Husain
As genAI projects mature, proper evals are becoming table stakes for production deployment. But how do we evaluate probabilistic machines?
Looking forward to learning about the latest techniques and best practices from @hamel.bsky.social and Shreya Shankar next month!
Looking forward to learning about the latest techniques and best practices from @hamel.bsky.social and Shreya Shankar next month!
AI Evals For Engineers & PMs by Hamel Husain and Shreya Shankar on Maven
Learn proven approaches for quickly improving AI applications. Build AI that works better than the competition, regardless of the use-case.
maven.com
April 26, 2025 at 5:19 PM
As genAI projects mature, proper evals are becoming table stakes for production deployment. But how do we evaluate probabilistic machines?
Looking forward to learning about the latest techniques and best practices from @hamel.bsky.social and Shreya Shankar next month!
Looking forward to learning about the latest techniques and best practices from @hamel.bsky.social and Shreya Shankar next month!
Last chance to sign up for this. Recording sent to everyone who signs up. maven.com/p/29a33a/hyb...
April 16, 2025 at 12:59 PM
Last chance to sign up for this. Recording sent to everyone who signs up. maven.com/p/29a33a/hyb...
Reposted by Hamel Husain
@hamel.bsky.social & his wisdom on evals, error analysis, looking at your data is what we need. Here are his 10 Don'ts:
• Don't skip error analysis
• Don't skip looking at your data
• Don't gatekeep who can write prompts
• Don't let zero users be a roadblock
• Don't be blindsided by criteria drift
• Don't skip error analysis
• Don't skip looking at your data
• Don't gatekeep who can write prompts
• Don't let zero users be a roadblock
• Don't be blindsided by criteria drift
April 16, 2025 at 1:05 AM
@hamel.bsky.social & his wisdom on evals, error analysis, looking at your data is what we need. Here are his 10 Don'ts:
• Don't skip error analysis
• Don't skip looking at your data
• Don't gatekeep who can write prompts
• Don't let zero users be a roadblock
• Don't be blindsided by criteria drift
• Don't skip error analysis
• Don't skip looking at your data
• Don't gatekeep who can write prompts
• Don't let zero users be a roadblock
• Don't be blindsided by criteria drift
Reposted by Hamel Husain
If you are building RAG applications, you don't want to miss this. Doug Turnbull is going to show you his tricks he's learned from a decade of optimizing retrieval in search systems, and how that transfers to RAG.
Link: maven.com/p/29a33a/hyb...
Link: maven.com/p/29a33a/hyb...
April 14, 2025 at 4:44 PM
If you are building RAG applications, you don't want to miss this. Doug Turnbull is going to show you his tricks he's learned from a decade of optimizing retrieval in search systems, and how that transfers to RAG.
Link: maven.com/p/29a33a/hyb...
Link: maven.com/p/29a33a/hyb...
If you are building RAG applications, you don't want to miss this. Doug Turnbull is going to show you his tricks he's learned from a decade of optimizing retrieval in search systems, and how that transfers to RAG.
Link: maven.com/p/29a33a/hyb...
Link: maven.com/p/29a33a/hyb...
April 14, 2025 at 4:44 PM
If you are building RAG applications, you don't want to miss this. Doug Turnbull is going to show you his tricks he's learned from a decade of optimizing retrieval in search systems, and how that transfers to RAG.
Link: maven.com/p/29a33a/hyb...
Link: maven.com/p/29a33a/hyb...
Reposted by Hamel Husain
The most critical part of RAG is the R (Retrieval). In this lesson, Doug Turnbull will share how we can go beyond simple hybrid search to optimize retrieval.
He'll share his bag of tricks from over a decade of optimizing search systems.
maven.com/p/29a33a/hyb...
He'll share his bag of tricks from over a decade of optimizing search systems.
maven.com/p/29a33a/hyb...
Hybrid Search Is Just The Beginning: Optimizing the R in RAG
You may have implemented hybrid search, and that's a great first step. In this session, Doug will share his experience building advanced search systems at Shopify and Reddit to provide you with a clea...
maven.com
April 10, 2025 at 2:39 PM
The most critical part of RAG is the R (Retrieval). In this lesson, Doug Turnbull will share how we can go beyond simple hybrid search to optimize retrieval.
He'll share his bag of tricks from over a decade of optimizing search systems.
maven.com/p/29a33a/hyb...
He'll share his bag of tricks from over a decade of optimizing search systems.
maven.com/p/29a33a/hyb...
The most critical part of RAG is the R (Retrieval). In this lesson, Doug Turnbull will share how we can go beyond simple hybrid search to optimize retrieval.
He'll share his bag of tricks from over a decade of optimizing search systems.
maven.com/p/29a33a/hyb...
He'll share his bag of tricks from over a decade of optimizing search systems.
maven.com/p/29a33a/hyb...
Hybrid Search Is Just The Beginning: Optimizing the R in RAG
You may have implemented hybrid search, and that's a great first step. In this session, Doug will share his experience building advanced search systems at Shopify and Reddit to provide you with a clea...
maven.com
April 10, 2025 at 2:39 PM
The most critical part of RAG is the R (Retrieval). In this lesson, Doug Turnbull will share how we can go beyond simple hybrid search to optimize retrieval.
He'll share his bag of tricks from over a decade of optimizing search systems.
maven.com/p/29a33a/hyb...
He'll share his bag of tricks from over a decade of optimizing search systems.
maven.com/p/29a33a/hyb...