Kudos to my amazing co-authors Dario Satriani, Enzo Veltri, Donatello Santoro! Another great collaboration between Università degli Studi della Basilicata and EURECOM 🙌

#LLM #Factuality #Benchmark #RelationalFactQA #NLP #AI

Structured outputs power analytics, reporting, and tool-augmented agents. This work exposes where current LLMs fall short and offers a clear tool for measuring progress on factuality beyond single-value QA. 📊

We release a new factuality benchmark with 696 annotated natural-language questions paired with gold factual answers expressed as tables (avg. 27 rows × 5 attributes), spanning 9 knowledge domains, with controlled question complexity and rich metadata.

Our new paper, "RelationalFactQA: A Benchmark for Evaluating Tabular Fact Retrieval from Large Language Models", measures exactly this gap.

Wider or longer output tables = tougher for all LLMs! 🧨
From Llama 3 and Qwen to GPT-4, no LLM goes above 25% accuracy on our stricter measure.

and a special thanks to
@tanmoy-chak.bsky.social for leading this effort!

More co-authors here on bsky
@iaugenstein.bsky.social
@preslavnakov.bsky.social
@igurevych.bsky.social
@emilioferrara.bsky.social
@fil.bsky.social
@giovannizagni.bsky.social
@dcorney.com
@mbakker.bsky.social
@computermacgyver.bsky.social
@irenelarraz.bsky.social
@gretawarren.bsky.social

It’s time we rethink how "facts" are negotiated in the age of platforms.

Excited to hear your thoughts!
#Misinformation #FactChecking #SocialMedia #Epistemology #HCI #DigitalTruth #CommunityNotes

arxiv.org/pdf/2505.20067

Community-based moderation offers speed & scale, but also raises tough questions:
– Can crowds overcome bias?
– What counts as evidence?
– Who holds epistemic authority?

Our interdisciplinary analysis combines perspectives from HCI, media studies, & digital governance.

Platforms like X are outsourcing fact-checking to users via tools like Community Notes. But what does this mean for truth online?

We argue this isn’t just a technical shift — it’s an epistemological transformation. Who gets to define what's true when everyone is the fact-checker?

Thanks for the amazing work to the whole team!

Joint work between Università degli Studi della Basilicata (Enzo Veltri, Donatello Santoro, Dario Satriani) and EURECOM (Sara Rosato, Simone Varriale).

#SQL #DataManagement #QueryOptimization #AI #LLM #Databases #SIGMOD2025

The principles in Galois – optimizing for quality alongside cost & dynamically acquiring optimization metadata – are a promising starting point for building robust and effective declarative data systems over LLMs. 💡

Paper and code: github.com/dbunibas/gal...

This cost/quality trade-off is guided by dynamically estimated metadata instead of relying on traditional stats.

Result: Significant quality gains (+29%) without prohibitive costs. Works across LLMs & for internal knowledge + in-context data (RAG-like setup, reported results in the figure). ✅

With our Galois system, we show one path to adapt database optimization for LLMs:
🔹 Designing physical operators tailored to LLM interaction nuances (e.g., Table-Scan vs Key-Scan in the figure).
🔹 Rethinking logical optimization (like pushdowns) for a cost/quality trade-off.

Why do traditional methods fail? They prioritize execution cost & ignore crucial LLM response quality (factuality, completeness).
Our results show standard techniques like predicate pushdown can even reduce result quality by making LLM prompts more complex to process accurately. 🤔

Alberto Sánchez Pérez (AILY LABS) will explain how we generate high-level hypotheses, use an agent to query databases via SQL, and summarize the findings into concise, correct, and insightful text.

Joint work with Alaa Boukhary, Luis Castejón Lozano, Adam Elwood

Work led by @spapicchio.bsky.social , in collaboration with Simone Rossi (EURECOM) and Luca Cagliero (Politecnico Torino)

#Text2SQL #LLM #AI #NLP #ReinforcementLearning

Key Insights:
🔹 General ZSL reasoning alone is insufficient
🔹 Smaller LLMs gain more from SFT with reasoning traces compared to larger models
🔹 RL consistently improves performance, especially with our fine-grained rewards
🔹 SFT+RL is highly effective for smaller models

We evaluate 4 training strategies:
1️⃣ Zero-Shot Learning (ZSL) +/- general-purpose reasoning
2️⃣ Supervised Fine Tuning (SFT) +/- task-specific reasoning traces
3️⃣ Reinforcement Learning (RL) with EXecution accuracy (EX) vs. our fine-grained rewards
4️⃣ Combined SFT+RL approach