We re-ran AF3 without templates, since we noticed it could use any template in the PDB, including the ground truth. We see the performance drops slightly in the lowest bins, but the gap to other methods still exists. We will update the preprint shortly!

I want to thank my co-authors @jeeberhardt.bsky.social, @torstenschwede.bsky.social, @ninjani.bsky.social and all of our collaborators! RunsN’ Poses builds on PLINDER and OpenStructure—this work wouldn’t be possible without them!
Also thanks to @rokbreznikar.bsky.social for this amazing logo! 9/9

Our findings highlight the need for specialised benchmarks for deep learning in protein-ligand interaction prediction. As tasks like co-folding grow more complex, new metrics are needed to assess leakage and difficulty-best defined by interaction similarity, not protein or ligand similarity. (7/n)

To separate pose prediction from ranking, we analyzed all 25 models per method. Top-ranked models (black) outperform the worst (red) but still lag behind the best possible (blue). Success rates still correlate with training similarity, highlighting a key limitation of these methods. (6/n)

The largest cluster has 171 SARS-CoV-2 MPro X-ray structures with different small molecules. Despite explicit chirality in the input, all methods misoriented the 5-chlorobenzofuran-3-aminomethyl group, perhaps mimicking the 5-chloro-2-methoxyphenyl group from the closest training system. (5/n)

To ensure the observed trend is not driven by an over-representation of certain protein families, we also analyzed only cluster representatives to account for potential biases. The observed correlation doesn’t change. (4/n)

Interestingly, we found that ligands with abundant data in PDB, like cofactors, show better prediction performance. When excluding them to focus on drug-like ligands, the correlation between success rate and training similarity becomes even more linear. (3/n)

In this work, we explored how training data similarity impacts protein-ligand prediction accuracy—an overlooked aspect in recent benchmarks. Our analysis shows that the current co-folding methods struggle to generalize beyond ligand poses in their training data.(2/n)