🙋🏽‍♀️🙋🏽‍♀️

6/🧵 We hope that the connection we provide to epistemology can inform and motivate better evaluations and claims regarding knowledge in LLMs 🚀

5/🧵 We also ask philosophers and computer scientists for their opinion on this matter: can we even say that an LLM knows a fact? We find some intriguing discrepancies between the two groups 🌗

4/🧵Using the definitions, we identify gaps in how current NLP research conceptualizes knowledge compared to epistemology 🔍. We also highlight open questions and new research opportunities to improve knowledge evaluations.

3/🧵Philosophers have thought and discussed about “knowledge” for a long time. So we look to epistemological research for definitions of knowledge 🧐 and interpret them in the context of LLMs 📝

2/🧵 However, inconsistencies may arise with paraphrases or with logically related facts 💥. Would it really know that “Lionel Messi plays for Inter Miami” if it fails to predict where Lionel Messi resides? ⚽️

1/🧵 LLMs seem to know a lot of facts about the world, GPT-4 correctly completes “The capital of Germany is ___” with Berlin, so it must know that, right?

Hey Maria! Thanks for the list, could you add me? 😀