#skyvision

Paper: arxiv.org/abs/2508.10637

Work with @ryan-ramos.bsky.social, @gkordo.bsky.social, Yuta Nakashima, @gtolias.bsky.social, and @noagarciad.bsky.social

We show that representations from some foundation models, especially CVLs like CLIP, encode information about image metadata. More surprisingly we show that such metadata traces can even affect the performance on semantic downstream tasks.

Paper: arxiv.org/abs/2508.10637

Work with @ryan-ramos.bsky.social, @gkordo.bsky.social, Yuta Nakashima, @gtolias.bsky.social, and @noagarciad.bsky.social.

We show that representations from some foundation models, especially CVLs like CLIP, encode information about image metadata. More surprisingly we show that such metadata traces can even affect the performance on semantic downstream tasks.

I second the Hyperion

When it comes to the CVL term we specifically went with it to discriminate CLIP-like VLMs from the VLMs that can generate text as the term VLM is overused and means many different things in different papers. It to an extent also follows the naming from arxiv.org/pdf/2405.17247

I agree that the terminology is confusing. However, I wouldn't agree that CLIP is an SSL method. It uses a contrastive loss, but not with self-supervised labels. DINOv2 and v3 classify it as weakly-supervised as it uses labels coming from the text.

Many thanks to the amazing collaborators: @ryan-ramos.bsky.social , @gkordo.bsky.social , Yuta Nakashima, @gtolias.bsky.social , @noagarciad.bsky.social

If this caught your attention, check out our new paper.

Processing and acquisition traces in visual encoders: What does CLIP know about your camera?

arxiv.org/abs/2508.10637

To be presented at #ICCV2025 (highlight). @iccv.bsky.social

The same pattern can be observed for the acquisition parameters in the task of near-duplicate retrieval. If the negatives are captured using the same camera as the query, the task becomes harder for some models compared to the case when they are captured by a different camera.

Impact on the semantic performance is again the most pronounced for contrastive VLMs, and the least for SSL models.

Here, we show kNN classification in a few cases, depending on whether the semantic positives and negatives share the same processing parameters as the test image.

This impact is especially pronounced when there is a strong correlation/anticorrelation between the semantic and metadata labels. E.g., when semantic positives/negatives have the same/different processing parameters as a query image.

More strikingly, we show that traces of these metadata labels (processing and acquisition parameters) can significantly impact the semantic recognition abilities.

A similar pattern is observed for the acquisition parameters, although generally, all models have a harder time predicting these parameters than the processing ones.