Maybe to avoid confusion another P.S.: of course noise ceilings *can* indicate data quality and when they are high they usually *do* indicate high quality. But you have to look at the whole package and take all factors into account to make that judgment, and it’s hard to compare across datasets.

I hope this thread was interesting or useful! I'd also like to highlight a great paper related to example 3 by Kendrick Kay:
journals.plos.org/plosone/arti...

P.S.: the same issue extends to judging absolute variance explained by your model. 🙃
P.P.S.: no AI was involved in making this thread. 😅

As these examples show, noise ceilings can (if any) only give you a rough idea about data quality. It is nontrivial to compare datasets or even pipelines on the same dataset to judge which one is better.

Noise ceilings have one purpose: To tell you how well your model can possibly do on this data.

Example 3: Assume 2 datasets:
Dataset A.
Dataset B, which is dataset A after denoising.

Our algorithm is great: it isolates a signal component, throws out all noise, but: it also removes all other signal!

Dataset B is now mostly pure signal & has extraordinary noise ceilings. Which one is better?

Now it becomes harder to shift the goalpost because there are so many things that can change between two datasets!

But you might now argue that at least for two datasets with the same parameters and the same number of trials, we can take noise ceilings as an index of relative data quality?

Example 2: We have 2 almost identical fMRI datasets but:
Dataset A has 2mm^3 resolution.
Dataset B has 4mm^3 resolution.

Dataset B has much higher noise ceilings. Which one is better?

Dataset A has lower SNR per voxel. But that's intentional. Downsampling would prob. yield a benefit for dataset A?

Now, you may argue we are comparing apples and oranges, and we can just use 2 repeats in dataset A to compare them.

But (1) now you agreed the noise ceiling is not an absolute index of quality, and (2) for your goals, a dataset with 5,000 unique images might actually be better than one with 100? 🙃

Example 1: Assume we have two fMRI datasets:
Dataset A: 12 sessions, with 100 images each shown 100x.
Dataset B: 12 sessions, with 5,000 images each shown 2x.

Dataset A obviously has almost perfect noise ceilings, dataset B's ceilings are much lower. Is the data quality of dataset A higher?

You might say: Wait, but the term noise ceiling implies that they tell you something about the signal-to-noise ratio in the data? So this means, less noise = better data quality?

In the following, I'll use three examples to highlight why it isn't that simple.

This work builds on and expands previous efforts (e.g. from Diana Dima and @lisik.bsky.social) and builds a comprehensive characterization of similarity judgments of actions in natural environments. This was the result of a collaboration that took many years! Looking forward to what will come next!

The similarity embedding enabled us to make sense of actions as a combination of their behaviorally-relevant dimensions (relevant w.r.t. similarity judgments, i.e. for categorization and making sense of the world).

Andre carefully curated a set of 768 1s movies from 256 action categories. Using a triplet odd-one-out tasking 6,036 workers, he found 28 interpretable dimensions underlying these similarity judgments. Here are just 6 of them.

Woah! That brings you so much closer!

Very sorry to hear! 😢

I’m also curious to see the manuscript. I’d like to understand better how this is smarter than hierarchical partitioning (i.e. non-Bayesian model averaging ;) since this is just a different way to assign the shared variance to unique parts, but it doesn’t solve the indeterminacy. Can BMA address it?