Adam Bataineh MD
@dradamb.bsky.social
founder of numenor.health (previously cofounder of Span - acquired by Eight Sleep)
A team at Apple recently published a really interesting paper where they tested LLM performance with GSM (a standard benchmark test for mathematical reasoning ability)
they modified the questions with unnecessary information to distract the LLMs
It led to much lower accuracy even for o1
they modified the questions with unnecessary information to distract the LLMs
It led to much lower accuracy even for o1
December 23, 2024 at 3:07 PM
A team at Apple recently published a really interesting paper where they tested LLM performance with GSM (a standard benchmark test for mathematical reasoning ability)
they modified the questions with unnecessary information to distract the LLMs
It led to much lower accuracy even for o1
they modified the questions with unnecessary information to distract the LLMs
It led to much lower accuracy even for o1
I wonder how much of the improvement in performance is because of goodhart's law
“When a measure becomes a target, it ceases to be a good measure”
I.e. is better performance on benchmark tests translatable to real world performance?
“When a measure becomes a target, it ceases to be a good measure”
I.e. is better performance on benchmark tests translatable to real world performance?
December 23, 2024 at 3:07 PM
I wonder how much of the improvement in performance is because of goodhart's law
“When a measure becomes a target, it ceases to be a good measure”
I.e. is better performance on benchmark tests translatable to real world performance?
“When a measure becomes a target, it ceases to be a good measure”
I.e. is better performance on benchmark tests translatable to real world performance?