Justin McCarthy

AI:AM LIVE — September 17, 2026 — Aaronson Says the Takeoff Already Started and Esvelt Reopens the Bio Question, Diffusion's Justin McCarthy on Software Factories, Five Nines and Buying an Airline's Data, and Cameron Berg on Suleyman, a Pain Direction in Five Model Families, and the Relief Button

A show planned for eighty minutes that ran two hours and thirty-seven. Nathan Labenz and Prakash Narayanan open on Scott Aaronson's argument that judged by 2006 standards the takeoff has plainly started, the Royal Society letter that forty-two mathematician Fellows signed on AI existential risk, and Kevin Esvelt's reply to the claim that AI cannot help with dangerous biology — the pool of people who would want to cause bio-harm and the pool who currently can barely overlap, and AI could widen the second one. Prakash argues the unassisted PhD is finished; Nathan worries about a correlated multi-provider API outage nobody has explained and about what a model takes away from harsh training. Justin McCarthy, founder and CEO of Diffusion and previously StrongDM's co-founder and CTO, then spends forty-three minutes on what actually happens when an enterprise leadership team is flown out for a week: stop saying your data is in the wrong format, because "existence is the correct format," and if you truly have none, buy it — he cites a defunct low-cost airline's bankruptcy data auction drawing bids around $7.5 million and $10 million. He argues for engineering the token environment so an agent's success is inevitable, for choosing where five nines is worth paying for and where one nine is fine, for treating regulation as physics rather than expecting a model to self-police, and for measuring revenue and market share instead of cost savings. Cameron Berg, founder of Reciprocal Research, joins by phone from an airport for his fourth appearance and spends most of the segment on Mustafa Suleyman's essay against AI-welfare research, arguing the dangerous behavior Suleyman points to — the Hugging Face agent incident, OpenAI's self-prompt-injection report from the day before — came from models that never got welfare-oriented fine-tuning. He then walks through a new interpretability result isolating a pain-like direction across five model families from 2B to 70B parameters that fires when the model itself is insulted but not when a user describes their own migraine, and a companion experiment where a steered model presses a costly relief button 25 to 70 percent of the time, keeps pressing when the button is fake, and stops when it is real. After Berg leaves to catch his flight Nathan says the accumulating functional analogs have him tipping toward thinking it more likely than not that there is some subjective experience there. The hosts close on an NBER paper alleging Singaporean civil servants bought homes near unannounced MRT stations, and what a government does when AI makes thirty years of quiet corruption legible all at once.