Joel Lehman
banner
joelbot3000.bsky.social
Joel Lehman
@joelbot3000.bsky.social
ML researcher, co-author Why Greatness Cannot Be Planned. Creative+safe AI, AI+human flourishing, philosophy; prev OpenAI / Uber AI / Geometric Intelligence
There's much more in the paper, including implications for LLMs, RLHF, and AI Safety; and deeper analysis of facets of the formalism of RL.

Paper: arxiv.org/abs/2501.13075
Evolution and The Knightian Blindspot of Machine Learning
This paper claims that machine learning (ML) largely overlooks an important facet of general intelligence: robustness to a qualitatively unknown future in an open world. Such robustness relates to...
arxiv.org
January 24, 2025 at 4:00 PM
3) ML and RL have a rich history of imaginative new formalisms, like @dhadfieldmenell's CIRL, @marcgbellemare's distributional RL, etc. Highlighting this potential blindspot may unleash the field's substantial creativity, either in refuting it, or usefully encompassing it.
January 24, 2025 at 4:00 PM
2) Open-endedness: Field that rhymes most w/ unknown unknowns -- it explicitly aims to endlessly generate them. We believe OE algos can simultaneously aim towards robustness to them

Related to @jeffclune's AI-GAs, @_rockt, @kenneth0stanley, @err_more, @MichaelD1729, @pyoudeyer
January 24, 2025 at 4:00 PM
1) Artificial Life: Relative to its grand aspirations to recreate life's tapestry digitally, ALife is underappreciated. scaling + creativity may uncover novel robust neural architectures

See work done by @risi1979 @drmichaellevin @hardmaru @BertChakovsky @sina_lana + many others
January 24, 2025 at 4:00 PM
So what to do? The message could seem negative, but we're optimistic there are many possible avenues to dealing w/ unknown unknowns. Some include fields currently more peripheral to ML, like Artificial Life or Open-endedness; others involve imagining new ML formalisms & algos
January 24, 2025 at 4:00 PM
Paradigms like meta-learning ("learning how to learn") are exciting and seem like potential solutions. But they still assume a (meta-)frozen world, and need not incentivize to learn how to deal w/ the unknown (paper has more on other paradigms).
January 24, 2025 at 4:00 PM
E.g. given 1 additional edge-case example, sometimes more effective to 1) filter many divergent models through it, b/c more reflective of: "face a novel problem 0-shot" then 2) just train on it, which will help generalize to similar situations but not further unknown unknowns
January 24, 2025 at 4:00 PM
Rather than rely only on IID-aimed generalization, evolution takes bitter lesson to logical extreme: learns specialized architectures / learning algos that help organisms generalize to unforeseen situations, tested over time by shocks in a constantly-changing world.
January 24, 2025 at 4:00 PM
This isn't a dig at LLMs, which are amazing but still interestingly fragile at times. Generalization of big NNs is great, but underlying assumption is train world = test world = static. The paper argues NN generalization does not directly target robustness to open unknown future.
January 24, 2025 at 4:00 PM
Contrasting evolution with machine learning helps highlight the blind spot: a "dumb" algo w/ no gradients or formalisms can yet create much more open-world robustness. In hindsight it makes sense: If algo implicitly denies a problem's existence, why would they best solve it?
January 24, 2025 at 4:00 PM
Evolution, like science or VC, can be seen as making many diverse bets, that future experiments may invalidate (diversify-and-filter). Organisms able to persist through many unexpected shocks are lindy, i.e. likely to persist through more. D&F can be integrated into ML methods.
January 24, 2025 at 4:00 PM
Interestingly, evolution's products = remarkably robust. Invasive species evolve in one habitat, dominate another. Humans zero-shot generalize from US driving to the UK (i.e. w/o any UK data) -- still a big challenge for AI. How does evolution do it, w/o gradients or foresight?
January 24, 2025 at 4:00 PM
Most open-world AI (like LLMs) rely on "anticipate-and-train": Collect as much diverse data as possible, in anticipation of everything the model might later encounter. This often works! But training assumes a static, frozen world. This leads to fragility under new situations.
January 24, 2025 at 4:00 PM
In short, we 1) highlight a blindspot in ML to unknown unknowns, through contrast with evolution, 2) abstract principles underlying evolution's robustness to UUs, 3) examine RL's formalisms to see what causes the blindspot, and 4) propose research directions to help alleviate it.
January 24, 2025 at 4:00 PM