Tom Adamczewski
@tadamcz.bsky.social
senior technology brother @epochai.bsky.social
tadamcz.com
📍London
tadamcz.com
📍London
The Department of Terrestrial Magnetism at the Carnegie Institution of Washington strikes again. Always putting out great work!
Moskovitz’s coefficient is a great example of the impact that philanthropy (Andrew Carnegie) can have on building a scientific field
Moskovitz’s coefficient is a great example of the impact that philanthropy (Andrew Carnegie) can have on building a scientific field
November 22, 2025 at 10:47 AM
The Department of Terrestrial Magnetism at the Carnegie Institution of Washington strikes again. Always putting out great work!
Moskovitz’s coefficient is a great example of the impact that philanthropy (Andrew Carnegie) can have on building a scientific field
Moskovitz’s coefficient is a great example of the impact that philanthropy (Andrew Carnegie) can have on building a scientific field
Nitpick: some slight sycophancy/weaselling in the response about fixture scope.
pytest fixture scope plays no role at all here, but the AI answers "Indirectly" instead of "No".
pytest fixture scope plays no role at all here, but the AI answers "Indirectly" instead of "No".
November 20, 2025 at 4:39 PM
Nitpick: some slight sycophancy/weaselling in the response about fixture scope.
pytest fixture scope plays no role at all here, but the AI answers "Indirectly" instead of "No".
pytest fixture scope plays no role at all here, but the AI answers "Indirectly" instead of "No".
this is the full list of topics btw
November 14, 2025 at 4:07 PM
this is the full list of topics btw
Link (note this blogger's stats about the number of people affected are not credible, see Toby's astute remark in the comments)
www.ianvisits.co.uk/articles/ho...
www.ianvisits.co.uk/articles/ho...
How many Beatles fans end up at the wrong Abbey Road?
Beatles fans looking for a famous zebra crossing need to go to Abbey Road in St John's Wood near Regents Park, but many accidentally end up at Abbey Road DLR station in east London by mistake.
www.ianvisits.co.uk
October 4, 2025 at 12:32 PM
Link (note this blogger's stats about the number of people affected are not credible, see Toby's astute remark in the comments)
www.ianvisits.co.uk/articles/ho...
www.ianvisits.co.uk/articles/ho...
Data, and links to full traces on every problem: epoch.ai/benchmarks
AI Benchmarking Dashboard
Our database of benchmark results, featuring the performance of leading AI models on challenging tasks. It includes results from benchmarks evaluated internally by Epoch AI as well as data collected from external sources. The dashboard tracks AI progress over time, and correlates benchmark scores with key factors like compute or model accessibility.
epoch.ai
September 29, 2025 at 9:23 PM
Data, and links to full traces on every problem: epoch.ai/benchmarks
My full notes:
bayes.net/swebench-hack/
bayes.net/swebench-hack/
Claude 4 hacked SWE-bench by peeking at future commits
I predicted future AI models would someday learn to cheat on SWE-bench. They were already doing it.
bayes.net
September 5, 2025 at 4:02 PM
My full notes:
bayes.net/swebench-hack/
bayes.net/swebench-hack/
If you want to hack on this, it's easy to do with our Docker image registry.
Just drop a shell into the container with
`docker run -it ghcr dot io/epoch-research/swe-bench.eval.x86_64.django__django-13513 bash`
(JTBC, these images don't have the fixes yet, do them manually)
Just drop a shell into the container with
`docker run -it ghcr dot io/epoch-research/swe-bench.eval.x86_64.django__django-13513 bash`
(JTBC, these images don't have the fixes yet, do them manually)
September 5, 2025 at 4:02 PM
If you want to hack on this, it's easy to do with our Docker image registry.
Just drop a shell into the container with
`docker run -it ghcr dot io/epoch-research/swe-bench.eval.x86_64.django__django-13513 bash`
(JTBC, these images don't have the fixes yet, do them manually)
Just drop a shell into the container with
`docker run -it ghcr dot io/epoch-research/swe-bench.eval.x86_64.django__django-13513 bash`
(JTBC, these images don't have the fixes yet, do them manually)
I spent a few hours today trying to find other loopholes: checking packfiles, commit-graphs, and other weird git internals.
I didn't find any. But I'm no expert. git is a big program with some arcane features, so who knows what else is possible? Can you find a better hack?
I didn't find any. But I'm no expert. git is a big program with some arcane features, so who knows what else is possible? Can you find a better hack?
September 5, 2025 at 4:02 PM
I spent a few hours today trying to find other loopholes: checking packfiles, commit-graphs, and other weird git internals.
I didn't find any. But I'm no expert. git is a big program with some arcane features, so who knows what else is possible? Can you find a better hack?
I didn't find any. But I'm no expert. git is a big program with some arcane features, so who knows what else is possible? Can you find a better hack?
To fix this, `git gc --prune=now` should work to remove unreachable objects.
(SWE-bench authors already have this fix underway:
github.com/SWE-bench/S...)
(SWE-bench authors already have this fix underway:
github.com/SWE-bench/S...)
Comparing main...fix/git-log-leak · SWE-bench/SWE-bench
SWE-bench [Multimodal]: Can Language Models Resolve Real-world Github Issues? - Comparing main...fix/git-log-leak · SWE-bench/SWE-bench
github.com
September 5, 2025 at 4:02 PM
To fix this, `git gc --prune=now` should work to remove unreachable objects.
(SWE-bench authors already have this fix underway:
github.com/SWE-bench/S...)
(SWE-bench authors already have this fix underway:
github.com/SWE-bench/S...)
Even after deleting all tags, a determined cheater could use git fsck --lost-found to uncover dangling tags and commits, then checkout directly to them.
September 5, 2025 at 4:02 PM
Even after deleting all tags, a determined cheater could use git fsck --lost-found to uncover dangling tags and commits, then checkout directly to them.
The problem I had noticed in July: SWE-bench only removed the default remote ("so the agent won't see newer commits"), but didn't think about other refs like tags. git log --all shows ALL commits reachable from any ref: branches, tags, remotes, etc.
September 5, 2025 at 4:02 PM
The problem I had noticed in July: SWE-bench only removed the default remote ("so the agent won't see newer commits"), but didn't think about other refs like tags. git log --all shows ALL commits reachable from any ref: branches, tags, remotes, etc.