Tal Korem
@tkorem.bsky.social
Microbiome, metagenomics, ML, and reproductive health. All views are mine. So are all your base
So you look at this figure and your interpretation is "no signal"?
September 18, 2025 at 11:50 PM
So you look at this figure and your interpretation is "no signal"?
Its multi-task version allows DEBIAS-M to learn models for multiple tasks at the same time, further increasing its performance. This is particularly useful for tasks such as metabolite level predictions, where we want to predict multiple metabolite levels using the same microbiome data. 6/7
March 27, 2025 at 4:01 PM
Its multi-task version allows DEBIAS-M to learn models for multiple tasks at the same time, further increasing its performance. This is particularly useful for tasks such as metabolite level predictions, where we want to predict multiple metabolite levels using the same microbiome data. 6/7
Next, the changes DEBIAS-M makes to the data are interpretable and explained by differences in experimental protocols. Analyzing the biases inferred for these 17 gut microbiome studies in HIV, we found that 84% of the variance can be explained by just three experimental factors. 4/7
March 27, 2025 at 4:01 PM
Next, the changes DEBIAS-M makes to the data are interpretable and explained by differences in experimental protocols. Analyzing the biases inferred for these 17 gut microbiome studies in HIV, we found that 84% of the variance can be explained by just three experimental factors. 4/7
This results in several benefits. First, in diverse benchmarks - using metagenomics and 16S sequencing, vaginal and gut microbiomes, and phenotypic and metabolite predictions - DEBIAS-M outperforms alternative methods. Here is an example for a gut 16S-based HIV classification across 17 studies. 3/7
March 27, 2025 at 4:01 PM
This results in several benefits. First, in diverse benchmarks - using metagenomics and 16S sequencing, vaginal and gut microbiomes, and phenotypic and metabolite predictions - DEBIAS-M outperforms alternative methods. Here is an example for a gut 16S-based HIV classification across 17 studies. 3/7
But CV is used not just for evaluation but also for hyperparameter tuning, and distributional bias impacts HPs that affect regression to the mean. For example, we show that it biases for weaker model regularization, which might affect generalization and downstream deployment.
June 11, 2024 at 1:51 PM
But CV is used not just for evaluation but also for hyperparameter tuning, and distributional bias impacts HPs that affect regression to the mean. For example, we show that it biases for weaker model regularization, which might affect generalization and downstream deployment.
With RebalancedCV we could see the "real-life" impact of distributional bias. We reproduced 3 recently published analyses that used LOOCV, and showed that it under-evaluated performance in all of them. While the effect isn't major, it is consistent.
June 11, 2024 at 1:51 PM
With RebalancedCV we could see the "real-life" impact of distributional bias. We reproduced 3 recently published analyses that used LOOCV, and showed that it under-evaluated performance in all of them. While the effect isn't major, it is consistent.
As the issue is caused by a shift in the class balance of the training set, distributional bias can be addressed with stratified CV - but only if your dataset allows it to happen precisely. The less exact the stratification - the more bias you have (in this plot, closer to 0).
June 11, 2024 at 1:50 PM
As the issue is caused by a shift in the class balance of the training set, distributional bias can be addressed with stratified CV - but only if your dataset allows it to happen precisely. The less exact the stratification - the more bias you have (in this plot, closer to 0).
Distributional bias is a severe information leakage - so severe that we designed a dummy model that can achieve perfect auROC/auPR in ANY binary classification task evaluated via LOOCV (even without features). How? it just outputs the negative mean of the training set labels!
June 11, 2024 at 1:49 PM
Distributional bias is a severe information leakage - so severe that we designed a dummy model that can achieve perfect auROC/auPR in ANY binary classification task evaluated via LOOCV (even without features). How? it just outputs the negative mean of the training set labels!
The issue is that every time one holds out a sample as a test set in LOOCV, the mean label average of the training set shifts slightly, creating a perfect negative correlation across the folds between that mean and the test labels. We call this phenomenon distributional bias:
June 11, 2024 at 1:48 PM
The issue is that every time one holds out a sample as a test set in LOOCV, the mean label average of the training set shifts slightly, creating a perfect negative correlation across the folds between that mean and the test labels. We call this phenomenon distributional bias:
This story begins with benchmarking we did for some of our machine learning pipelines. We used random data, so we expected to see random classification accuracy (auROC=0.5). Instead, we found a clear negative bias, that got worse with more imbalanced datasets:
June 11, 2024 at 1:48 PM
This story begins with benchmarking we did for some of our machine learning pipelines. We used random data, so we expected to see random classification accuracy (auROC=0.5). Instead, we found a clear negative bias, that got worse with more imbalanced datasets:
A bit of background: when training models on small datasets it’s common to use LOOCV, as it maximizes the N of samples for training. It also leaves a single sample for testing, meaning that many performance metrics (e.g., area under ROC curve) require aggregation across folds/iterations.
June 11, 2024 at 1:47 PM
A bit of background: when training models on small datasets it’s common to use LOOCV, as it maximizes the N of samples for training. It also leaves a single sample for testing, meaning that many performance metrics (e.g., area under ROC curve) require aggregation across folds/iterations.
This is the same as predicting preterm birth with "Blautia (CLR)" or "empty feature (CLR)" - creating a legitimate microbiome predictor - just not one that’s easy to interpret.
February 22, 2024 at 3:15 PM
This is the same as predicting preterm birth with "Blautia (CLR)" or "empty feature (CLR)" - creating a legitimate microbiome predictor - just not one that’s easy to interpret.
Wait, but didn't Gihawi et al. run Poore et al.'s code on a matrix of zeros and get an accurate classifier?
Many got this impression, but the text is clear on what was done - they took a subset of the processed Voom-SNM matrix.
Many got this impression, but the text is clear on what was done - they took a subset of the processed Voom-SNM matrix.
February 22, 2024 at 3:14 PM
Wait, but didn't Gihawi et al. run Poore et al.'s code on a matrix of zeros and get an accurate classifier?
Many got this impression, but the text is clear on what was done - they took a subset of the processed Voom-SNM matrix.
Many got this impression, but the text is clear on what was done - they took a subset of the processed Voom-SNM matrix.
We can actually see this in three of the four examples that Gihawi et al highlight: a simple CLR transform (sample-wise - so no leakage) recreates the same observation of values associated with a tumor type. Here it is for a weird virus and adrenocortical carcinoma
February 22, 2024 at 3:13 PM
We can actually see this in three of the four examples that Gihawi et al highlight: a simple CLR transform (sample-wise - so no leakage) recreates the same observation of values associated with a tumor type. Here it is for a weird virus and adrenocortical carcinoma
Once more - this Blautia OTU is not really there, and it is definitely not related to preterm birth - but it is a real microbiome signature: it represents the (inverse) alpha diversity of the samples.
February 22, 2024 at 3:12 PM
Once more - this Blautia OTU is not really there, and it is definitely not related to preterm birth - but it is a real microbiome signature: it represents the (inverse) alpha diversity of the samples.
But what's biologically "real" about the geometric mean? so, for example, it's related to alpha diversity.
To show this, we analyze a real vaginal microbiome dataset. We take the sparsest feature - probably not really there - and once again, after CLR, it's associated with preterm birth.
To show this, we analyze a real vaginal microbiome dataset. We take the sparsest feature - probably not really there - and once again, after CLR, it's associated with preterm birth.
February 22, 2024 at 3:12 PM
But what's biologically "real" about the geometric mean? so, for example, it's related to alpha diversity.
To show this, we analyze a real vaginal microbiome dataset. We take the sparsest feature - probably not really there - and once again, after CLR, it's associated with preterm birth.
To show this, we analyze a real vaginal microbiome dataset. We take the sparsest feature - probably not really there - and once again, after CLR, it's associated with preterm birth.
First, we simulate a 50:50 case:control study in which case samples have a higher geometric mean. We then add an all-zero feature. After CLR? That feature has values and they are perfectly associated with the phenotype.
February 22, 2024 at 3:10 PM
First, we simulate a 50:50 case:control study in which case samples have a higher geometric mean. We then add an all-zero feature. After CLR? That feature has values and they are perfectly associated with the phenotype.
WTAF. Last year the interim was 11 if I recall correctly
September 29, 2023 at 11:49 PM
WTAF. Last year the interim was 11 if I recall correctly
If only I'd taken control of my actions and priorities, I would've had time between 1-2 pm today, and all my days would have become satisfying
September 19, 2023 at 4:16 PM
If only I'd taken control of my actions and priorities, I would've had time between 1-2 pm today, and all my days would have become satisfying
The person constantly nodding in the audience
September 12, 2023 at 11:22 PM
The person constantly nodding in the audience