Amazing. Will get back to you.

-we haven't yet tried on a varied downstream tasks. I believe there would some downstream tasks where you have more leakage than others
-we haven' tried a varied set of pretrained models
-need to integrate hashFrag.we used chrom. splits before. might have a reason we didn't see much differenc in mag

this is a very important point. The expression levels for the different test subsets are from a wide range (they are not only sequences with high expression or sequences with low expression).

-for models that overfit to different degrees, the drop in performance would be different.
-the drop in performance would vary by datasets, tasks, as well.

I think its possible. Actually this was and remains in our to-do list.

therefore, I cannot provide any evidence of leakage happening in transfer learning (for now). But I would suggest to avoid it whereas possible.
take a look at @jmbartoszewicz.bsky.social & Melanias'
www.biorxiv.org/content/10.1...

this was a problem I was particularly interested about. to show that leakage can occur when testing fine tuned model on pretraining data. A student from our group pursued it for some time. But we were unable to detect leakage in transfer learning to a degree where everyone would care.

10/hashFrag is openly available and accessible,so it’s win,win,win:more accurate perf. estimates,better perf. overall,and easy to use.We hope that hashFrag sets the new standard for how data are split for trainin genome models
Github: github.com/de-Boer-Lab/...
Paper: www.biorxiv.org/content/10.1...

9/ Not only do hashFrag generated train-test splits effectively mitigate leakage, but hashFrag-trained models even outperformed chromosomal split-trained models, showing that chromosomal splitting not only introduces train-test leakage but also creates inferior train-val splits.

8/ We applied hashFrag to test datasets. Across models tested, model performance was inflated by the presence of test sequences that were similar to training sequences. hashFrag revealed more reliable performance measures.

7/ To detect and avoid homology based leakage, we created hashFrag, which leverages BLAST to identify similar sequences and then either (1) filter out the leaked sequences from the test set, (2) stratify the test set into subgroups by distance, or (3) create leakage-free train-test splits.

6/ We analyzed GWAS SNVs from OpenTarget with PIP>0.1 and found a substantial percentage of these SNVs have their alternate alleles, along with their flanking sequences, replicated on other chromosomes, often many times.

5/ An important application of models is to predict the effect of variants. However, variants along with their flanking region can be replicated throughout the genome. Without accounting for homology, you can’t tell if the model’s prediction is based on learned cis-regulatory logic or memorization.

4/ We saw a very interesting trend where models fit to the most similar test sequences early during training, faster than they fit the overall training data, making these sequences unreliable for evaluating actual performance.

3/We created the cheeky OverfitNN as a maximally overfit benchmark, which is nearest neighbor-based and has no understanding of cis regulation. As expected, OverfitNN only works well for closely related sequences, but even neural networks work best for sequences that are similar to their train data.

2/ We compared regulatory regions against each other using chromosomal splitting and found that many genomic sequences are very similar compared to unrelated sequences. We set out to investigate how this similarity could cause train-test leakage.

1/Typically, genome is split into train & test by chromosomes without accounting for homologous sequences. Because similar sequences encode similar activities, a model could conceivably correctly predict the activity of test sequences that are very similar to train sequences just by memorizing them.

Amazing collaboration between de Boer lab (@CarldeBoerPhD, myself) and Yachie lab (@yachielab, @nzmyachie, Brett Kiyota)

9/ With so many cool technologies being developed, it's an exciting time to be involved in sequence modeling! Make sure to use and strive to beat the state-of-the-art when applying models.
📄 Paper: www.nature.com/articles/s41...
💻 GitHub: github.com/de-Boer-Lab/random-promoter-dream-challenge-2022