open.substack.com/pub/overfitt...

Ultimately, were some of their analytical choices justified? Maybe. But we need more independent groups replicate this finding. The question itself is clearly very important.

I'd love to see 10 groups re-analyze the data. You'd probably get 10 different conclusions.

The authors also stratified by 14 potential confounders and, in each case, found high concordance between each strata.

This is fine but you can't correlate your way to causation. Each strata is susceptible to the same issues of RNA integrity differences, anesthesia, etc..

Reading the methods, it's clear the authors made *many* choices that likely affected the result. Here's a passage on the selection of technical covariates that caught my eye:

The authors compared gene expression between post-mortem and *living* brain tissue and found 80% (!) of genes were differentially expressed.

If true, this is a really big deal for any study using post-mortem brain tissue (i.e., nearly every study of human brain disease)

There are tradeoffs in these GWAS methodological choices. I argue you can either optimize for discovery and hypothesis generation (maybe reasonable for basic science pursuits) or you optimize for precision for clinical models. Thus far AD GWAS have chosen the former.

Does this noise matter? For genomic medicine, we argue yes (example: polygenic scores).

See recent benchmarking study showing the single most important method decision was the choice of GWAS summary statistics!

pubmed.ncbi.nlm.nih.gov/39762974/

Takeaway: genetic association with AD are downwardly biased when you design your study with proxy cases, young controls, and individuals from biobanks with a history of AD compared to incident cases which was our "gold standard"

We show that the effect size for APOE's association with AD is smaller when you look at proxy cases or individuals who enroll in biobanks with a history of AD in their EHR records even though they must not have cognitive impairment to consent to the study (??) (i.e., 'prevalent')

We show that while these choices have increased the statistical power of recent AD GWAS, they have also potentially distorted the effect size of AD genetic associations. Here you can see effect size modification from different age cut-offs in the controls for example:

1) The use of non age-matched controls 2) prevalent cases in biobanks such as UKB and AoU and 3) proxy case labelling based on family history of disease

Link: doi.org/10.1101/2025...

Fundamental to the use of genomics in medicine are the methods we use to infer an association between a variant/gene and a disease. We put recent method choices in AD genetic research to the test including...

1 hour headways for non-Max routes OOF 🤦‍♂️