David Duvenaud

AI:AM LIVE — June 24, 2026 — Gradual Disempowerment and the Search for a Stable Post-AGI Equilibrium: David Duvenaud

The opening covered a fast-moving week in AI policy and infrastructure: the first federal lawsuit over the BIS export-control order cutting off Anthropic's Fable 5 for foreign nationals; two papers pointing at latent world-models inside RL agents and a 7M-parameter loop model beating much larger systems on hard reasoning; the $8M super-PAC defeat of New York's frontier-AI-safety lawmaker; and a brisk exchange on Claude Tag's launch as multiplayer AI, GLM 5.2's cost advantage over GPT-5.5 Codex, and Claude's UltraCode orchestration mode. David Duvenaud — ML professor at the University of Toronto, co-creator of neural ODEs, former alignment lead at Anthropic, and co-author of the 'Gradual Disempowerment' paper — then joined for a full hour exploring whether any stable post-AGI equilibrium actually exists where humans keep meaningful control. Nathan pressed every optimist steelman: historical absorption (prior automation shocks were absorbed without permanent disempowerment), comparative advantage (Ricardo says humans keep a niche), constitutional and property anchors (the franchise, rule of law, military command), aligned AIs defending human leverage, and the argument that 'gradual' gives time to correct. Duvenaud's rebuttal to each was consistent: the disempowerment mechanism doesn't require malice or misalignment — it requires only that the systems driving growth stop needing human participation, the way human civilization doesn't need the monkey economy despite occasionally trading bananas with them. He described the 'Earth as a slow zone' scenario — throttled AI growth, bans on recursive self-improvement, no cultural optimization — and argued that when you enumerate everything it requires controlling (research, startups, reproduction, memetics), the list is horrifyingly long, analogous to listing all the mutations that can cause cancer. On timelines, he sketched white-collar automation first, then a decade-plus to build enough robot factories to displace physical labor, putting full human economic irrelevance perhaps 15–20 years out. His two concrete recommendations: restrict frontier compute at the TSMC/fab choke point, and cultivate the temporal coherence of public preferences by chaining the 'is it okay if humanity disappears?' question forward to one's own children and grandchildren until a coherent answer emerges.