An AI question I’m pondering: When we’ve solved the hallucination problem, will we know it? What I mean is, what if the model, having correlated all of humanity’s tokens in a massive multidimensional space of latent knowledge, begins speaking truths that we don’t understand or cannot accept? To give a more concrete example: What if Galileo had been an ML researcher whose supposedly hallucination-free model began telling anyone who’d listen that the earth goes around the sun? What if the model could explain its reasoning step by step? Surely the cognitive elites of his day would’ve declared that the model was hallucinating and needed more work. Or worse, maybe they’d have thought the model was producing harmful “disinformation” and “conspiracy theory.” The breathless VICE article almost writes itself: When Galileo’s chatbot is asked to describe the arrangement of the planets, it confidently produces a detailed and plausible conspiracy theory that places the sun at the center of the solar system. “By centering the sun instead of the earth,” warned a spokesperson for the Inquisition, “this problematic model has the potential to cause real harm by fooling human users who might take it to be an authority on matters divine and celestial.” These so-called “deep hallucinations” have far more potential for harm than simpler errors of basic fact in the previous generation of models, because they involve cherry-picking superficially true facts and putting them together in a way that presents a distorted picture of reality. When the balance of intelligence flips from us to It, and It starts telling us how the world really is, I suspect we’ll think It has gone stark raving mad.

Sometimes, explainability and power are at odds with one another.