Seth Kimmel
@sethkim.me
Founder @stealth data/AI infra startup
Thinks about airplanes, data, markets, math, my next meal
sethkim.me/l/
Thinks about airplanes, data, markets, math, my next meal
sethkim.me/l/
Do you think this is different than human confidence? I'd say 99.9% of what we think is true is second-hand knowledge
November 13, 2024 at 7:03 PM
Do you think this is different than human confidence? I'd say 99.9% of what we think is true is second-hand knowledge
Somewhat disagree here. Have you ever looked at logprobs? The model far prefers steering in directions that it feels confident in given alternatives. cookbook.openai.com/examples/usi...
Using logprobs | OpenAI Cookbook
Open-source examples and guides for building with the OpenAI API. Browse a collection of snippets, advanced techniques and walkthroughs. Share your own examples and guides.
cookbook.openai.com
November 13, 2024 at 4:59 PM
Somewhat disagree here. Have you ever looked at logprobs? The model far prefers steering in directions that it feels confident in given alternatives. cookbook.openai.com/examples/usi...
So do humans! It's why we have QA/testing, and jobs that are just pure oversight.
You might expect both an LLM and a human to get a handful of data labeling tasks wrong, but have it checked with a verifier/adversarial LLM and you'll likely get ~100% accuracy.
You might expect both an LLM and a human to get a handful of data labeling tasks wrong, but have it checked with a verifier/adversarial LLM and you'll likely get ~100% accuracy.
November 13, 2024 at 4:54 PM
So do humans! It's why we have QA/testing, and jobs that are just pure oversight.
You might expect both an LLM and a human to get a handful of data labeling tasks wrong, but have it checked with a verifier/adversarial LLM and you'll likely get ~100% accuracy.
You might expect both an LLM and a human to get a handful of data labeling tasks wrong, but have it checked with a verifier/adversarial LLM and you'll likely get ~100% accuracy.