📜 Paper: Multimodality Helps Few-shot 3D Point Cloud Semantic Segmentation (arxiv.org/pdf/2410.22489)
🔗 Code: github.com/ZhaochongAn/...

#ICLR2025 #Multimodality #3DSegmentation @belongielab.org @ellis.eu @ethzurich.bsky.social @ox.ac.uk

7/ For a deep dive into our model design, modality analysis, and experiments, check out our paper and code here.

📝: arxiv.org/pdf/2410.22489
💻: github.com/ZhaochongAn/Multimodality-3D-Few-Shot

6/ The result? New SOTA few-shot performance that opens up exciting possibilities for multimodal adaptations in robotics, personalization, virtual reality, and more! ☀️

5/ By fusing these modalities, MM-FSS generalizes to novel classes more effectively—even when the 3D-only connection between support and query is weak. 🚀

4/ On the model side, our multimodal fusion designs harness cross-modal complementary knowledge to boost novel class learning, and test-time cross-modality calibration mitigates training bias.

3/ That’s where MM-FSS comes in! We introduce two commonly overlooked modalities:
💠 2D images (leveraged implicitly during pretraining)
💠 Text (using class names)

—all at no extra cost beyond the 3D-only setup. ✨

2/ Previous methods rely solely on a single modality—few-shot 3D support samples—to learn novel class knowledge.

However, when support and query objects look very different, performance can suffer, limiting effective few-shot adaptation. 🙁

1/ Our previous work, COSeg, showed that explicitly modeling support–query relationships via correlation optimization can achieve SOTA 3D few-shot segmentation.

With MM-FSS, we take it even further!

Ref: COSeg arxiv.org/pdf/2410.22489