Samuele Bortolotti
@samubortolotti.bsky.social
Ph.D. student in Artificial Intelligence at the University of Trento.
In collaboration with @ema-ridopoco.bsky.social Tommaso Carraro @paolomorettin.bsky.social @emilevankrieken.com @nolovedeeplearning.bsky.social @looselycorrect.bsky.social @andreapasserini.bsky.social
December 10, 2024 at 7:10 PM
In collaboration with @ema-ridopoco.bsky.social Tommaso Carraro @paolomorettin.bsky.social @emilevankrieken.com @nolovedeeplearning.bsky.social @looselycorrect.bsky.social @andreapasserini.bsky.social
Want to know more?
1️⃣ Learn more about RSs: Why they appear, their root causes, and mitigation: arxiv.org/abs/2305.19951
2️⃣ Make NeSy models aware of their shortcuts: arxiv.org/abs/2402.12240
1️⃣ Learn more about RSs: Why they appear, their root causes, and mitigation: arxiv.org/abs/2305.19951
2️⃣ Make NeSy models aware of their shortcuts: arxiv.org/abs/2402.12240
Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcuts
Neuro-Symbolic (NeSy) predictive models hold the promise of improved compliance with given constraints, systematic generalization, and interpretability, as they allow to infer labels that are consiste...
arxiv.org
December 10, 2024 at 7:10 PM
Want to know more?
1️⃣ Learn more about RSs: Why they appear, their root causes, and mitigation: arxiv.org/abs/2305.19951
2️⃣ Make NeSy models aware of their shortcuts: arxiv.org/abs/2402.12240
1️⃣ Learn more about RSs: Why they appear, their root causes, and mitigation: arxiv.org/abs/2305.19951
2️⃣ Make NeSy models aware of their shortcuts: arxiv.org/abs/2402.12240
For other details regarding rsbench, datasets, and experiments, check the links below:
Website: unitn-sml.github.io/rsbench/
Paper: openreview.net/forum?id=5Vt...
GitHub: github.com/unitn-sml/rs...
Website: unitn-sml.github.io/rsbench/
Paper: openreview.net/forum?id=5Vt...
GitHub: github.com/unitn-sml/rs...
rsbench A Neuro-Symbolic Benchmark Suite for Concept Quality and Reasoning Shortcuts
“A Neuro-Symbolic Benchmark Suite for Concept Quality and Reasoning Shortcuts” benchmark paper
unitn-sml.github.io
December 10, 2024 at 7:10 PM
For other details regarding rsbench, datasets, and experiments, check the links below:
Website: unitn-sml.github.io/rsbench/
Paper: openreview.net/forum?id=5Vt...
GitHub: github.com/unitn-sml/rs...
Website: unitn-sml.github.io/rsbench/
Paper: openreview.net/forum?id=5Vt...
GitHub: github.com/unitn-sml/rs...
Easy to set up and use!
1️⃣ Configurable: can be easily configured with YAML/JSON files.
2️⃣ Intuitive: straightforward to use:
1️⃣ Configurable: can be easily configured with YAML/JSON files.
2️⃣ Intuitive: straightforward to use:
December 10, 2024 at 7:10 PM
Easy to set up and use!
1️⃣ Configurable: can be easily configured with YAML/JSON files.
2️⃣ Intuitive: straightforward to use:
1️⃣ Configurable: can be easily configured with YAML/JSON files.
2️⃣ Intuitive: straightforward to use:
📊 8 challenging tasks, all with predefined settings.
3 new benchmarks:
🔢 MNMath for arithmetic reasoning
🛃 MNLogic for SAT-like problems
🚖 SDD-OIA, a synthetic self-driving task!
They can all be made easier or harder with our data generator!
3 new benchmarks:
🔢 MNMath for arithmetic reasoning
🛃 MNLogic for SAT-like problems
🚖 SDD-OIA, a synthetic self-driving task!
They can all be made easier or harder with our data generator!
December 10, 2024 at 7:10 PM
📊 8 challenging tasks, all with predefined settings.
3 new benchmarks:
🔢 MNMath for arithmetic reasoning
🛃 MNLogic for SAT-like problems
🚖 SDD-OIA, a synthetic self-driving task!
They can all be made easier or harder with our data generator!
3 new benchmarks:
🔢 MNMath for arithmetic reasoning
🛃 MNLogic for SAT-like problems
🚖 SDD-OIA, a synthetic self-driving task!
They can all be made easier or harder with our data generator!
🧪 Test your models!
- 🌍 Evaluate concepts in in- and out-of-distribution scenarios.
- 🎯 Ground-truth concept annotations are available for all tasks.
- 📊 Visualize how your models handle different learning & reasoning tasks!
- 🌍 Evaluate concepts in in- and out-of-distribution scenarios.
- 🎯 Ground-truth concept annotations are available for all tasks.
- 📊 Visualize how your models handle different learning & reasoning tasks!
December 10, 2024 at 7:10 PM
🧪 Test your models!
- 🌍 Evaluate concepts in in- and out-of-distribution scenarios.
- 🎯 Ground-truth concept annotations are available for all tasks.
- 📊 Visualize how your models handle different learning & reasoning tasks!
- 🌍 Evaluate concepts in in- and out-of-distribution scenarios.
- 🎯 Ground-truth concept annotations are available for all tasks.
- 📊 Visualize how your models handle different learning & reasoning tasks!
🔍 rsbench allows you to:
- 🧮 Run algorithmic, logical, and high-stakes tasks w/ known reasoning shortcuts (RSs).
- 📊 Eval concept quality via F1, accuracy & concept collapse.
- 🛠️ Easily customize the tasks and count RSs a priori using our countrss tool!
- 🧮 Run algorithmic, logical, and high-stakes tasks w/ known reasoning shortcuts (RSs).
- 📊 Eval concept quality via F1, accuracy & concept collapse.
- 🛠️ Easily customize the tasks and count RSs a priori using our countrss tool!
December 10, 2024 at 7:10 PM
🔍 rsbench allows you to:
- 🧮 Run algorithmic, logical, and high-stakes tasks w/ known reasoning shortcuts (RSs).
- 📊 Eval concept quality via F1, accuracy & concept collapse.
- 🛠️ Easily customize the tasks and count RSs a priori using our countrss tool!
- 🧮 Run algorithmic, logical, and high-stakes tasks w/ known reasoning shortcuts (RSs).
- 📊 Eval concept quality via F1, accuracy & concept collapse.
- 🛠️ Easily customize the tasks and count RSs a priori using our countrss tool!
🤔 What are reasoning shortcuts?
NeSy models might learn wrong concepts but still make perfect predictions!
Example: A self-driving car 🚗 stops in front of a 🚦🔴 or a 🚶. Even if it confuses the two, it outputs the right prediction!
NeSy models might learn wrong concepts but still make perfect predictions!
Example: A self-driving car 🚗 stops in front of a 🚦🔴 or a 🚶. Even if it confuses the two, it outputs the right prediction!
December 10, 2024 at 7:10 PM
🤔 What are reasoning shortcuts?
NeSy models might learn wrong concepts but still make perfect predictions!
Example: A self-driving car 🚗 stops in front of a 🚦🔴 or a 🚶. Even if it confuses the two, it outputs the right prediction!
NeSy models might learn wrong concepts but still make perfect predictions!
Example: A self-driving car 🚗 stops in front of a 🚦🔴 or a 🚶. Even if it confuses the two, it outputs the right prediction!
🌐 rsbench allows you to evaluate the concepts learned by:
1️⃣ Neuro-Symbolic models (#NeSy)
2️⃣ Concept Bottleneck Models (#CBMs)
3️⃣ Black-box Neural Networks (NNs*)
4️⃣ Vision-Language Models (#VLMs*)
* through post-hoc concept-based explanations (e.g., TCAV)
1️⃣ Neuro-Symbolic models (#NeSy)
2️⃣ Concept Bottleneck Models (#CBMs)
3️⃣ Black-box Neural Networks (NNs*)
4️⃣ Vision-Language Models (#VLMs*)
* through post-hoc concept-based explanations (e.g., TCAV)
December 10, 2024 at 7:10 PM
🌐 rsbench allows you to evaluate the concepts learned by:
1️⃣ Neuro-Symbolic models (#NeSy)
2️⃣ Concept Bottleneck Models (#CBMs)
3️⃣ Black-box Neural Networks (NNs*)
4️⃣ Vision-Language Models (#VLMs*)
* through post-hoc concept-based explanations (e.g., TCAV)
1️⃣ Neuro-Symbolic models (#NeSy)
2️⃣ Concept Bottleneck Models (#CBMs)
3️⃣ Black-box Neural Networks (NNs*)
4️⃣ Vision-Language Models (#VLMs*)
* through post-hoc concept-based explanations (e.g., TCAV)
Reposted by Samuele Bortolotti
by @ema-ridopoco.bsky.social @looselycorrect.bsky.social @andreapasserini.bsky.social @samubortolotti.bsky.social
eg
👉 proceedings.neurips.cc/paper_files/...
👉 openreview.net/forum?id=pDc...
👉 unitn-sml.github.io/rsbench/
eg
👉 proceedings.neurips.cc/paper_files/...
👉 openreview.net/forum?id=pDc...
👉 unitn-sml.github.io/rsbench/
Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcuts
proceedings.neurips.cc
December 10, 2024 at 3:46 PM