AI:AM LIVE — June 26, 2026 — Learning Expert Judgment and AI Consciousness: Robbie Goldfarb, Eric Vaughan & Cameron Berg

The rundown

1. 5:02Opening21 min

   Opening: GPT-5.6's Customer-by-Customer Approval, Securing Government Systems, and What Executives Really ThinkFable still banned and GPT-5.6 submitted for pre-Mythos government approval, with access to be approved customer by customer — the regulatory-moat and market-downturn (Dean Ball) objections; whether government systems can be secured at all as frontier hacking capability diffuses to 'script kiddies' (Prakash's Napster/BitTorrent analogy); Nathan's core question for the White House — should an uncontrolled race to recursive self-improvement be tolerated; and a field report on executive sentiment, from the ~30% who think AI is a vendor scam to the all-in cohort.

   Watch

   ▶Play this segmentYouTube · from 5:02

   As aired

   Nathan opened tired and fatigued by the news cycle, saying his feed was wall-to-wall GPT-5.6 stories and "court politics" around AI that felt both fascinating and stupid at once. That led into the week's big story: The Information's report that GPT-5.6 had been submitted for government approval even before the Mythos announcement, that OpenAI had been quietly working with the Trump administration, and that everything froze when the Fable export-control action dropped. The upshot, per Prakash, is that the administration now wants to approve GPT-5.6 access customer by customer rather than releasing broadly — something Sam Altman reportedly expects to take a couple of weeks to resolve. The hosts walked through the backlash: startups without access lose their main edge (speed and nimbleness) against incumbents, effectively handing the biggest labs a regulatory moat despite that being the opposite of what critics of Big Tech wanted; and Dean Ball's warning that delaying too long risks tipping markets into a downturn given how much capital is now riding on AI. Both agreed the underlying capabilities will be broadly available within six to twelve months regardless of how the approval fight plays out.

   From there the conversation turned to whether the U.S. government has the expertise to actually secure the country against AI-enabled hacking. Prakash and Nathan agreed neither the administration nor, really, OpenAI is adding much to what Anthropic already has in-house (name-checking security researcher Nicholas Carlini), with CISA cited as one of the few other genuine pockets of government expertise against a backdrop of tenured bureaucrats and legacy contractors like Accenture. They traced how government cybersecurity has historically leaned less on technical hardening and more on deterrence — the threat of prosecution, or the CIA and NSA chasing down nation-state actors — and argued that model breaks down once AI-enabled capability diffuses down to ordinary citizens en masse, drawing a parallel to Napster and BitTorrent, where mass, hard-to-prosecute infringement (including by legally-protected juveniles) simply couldn't be stopped. Nathan pressed the practical question of how long it would actually take to secure something like Social Security against "Fable"- or "Mythos"-level capabilities being available in the wild, and whether the government even moves at that pace. He argued the one place government focus is actually warranted is the narrower question of whether an uncontrolled, "YOLO"-style race to an intelligence explosion via recursive self-improvement, run by a handful of companies, is something society should tolerate — everywhere else, he suggested, the government is mostly just discovering what was already possible. He noted some AI-safety voices still see a window, since frontier models remain hardware-bound (citing Liquid AI's on-device, billion-parameter models as impressive but nowhere near hacking-capable), and closed the segment on the lack of transparency making it hard to even have a sane public conversation about what's actually being observed.

   Prakash then offered a frame for why technologists and executives talk past each other on these issues: technologists see everything as freely recombinable bits of information, while executives and lawyers see rigid "structures" with rules that supposedly keep data walled off. He illustrated this with an example — cross-referencing tax records and Social Security numbers to flag undocumented immigrants — which reads to a technologist as an obvious, mundane database operation but strikes executives and politicians as a novel privacy violation, even though the government has always held that data and simply never bothered to cross-reference it. They extended the same logic to AI-enabled hacking: the underlying software bugs have always existed, they said, and what AI adds is the ability to chain them together into working exploits. The result is that executives tend to blame the AI companies for the resulting breaches, while technologists see it as just another bug that needs fixing.

   The final stretch covered how corporate executives actually feel about AI. Nathan noted adoption has moved faster than he expected and that revenue numbers back that up, meaning the executive class is at least alert to the fact that AI is something they'll have to navigate. Prakash broke down the range of sentiment: roughly 30% still view AI as pure hype, drawing a direct parallel to how enterprise software sales cycles inflated blockchain; another group is almost "under a kind of psychosis," pushing so hard that it's actually harming their companies because the technology isn't ready (though both hosts noted it's often the people, not the tech, that aren't ready) — with Mark Zuckerberg cited as an example of someone over-committing because he grasps the stakes. On the other end, CEOs like Satya Nadella and Marc Benioff are betting their businesses will not just survive but flourish alongside AI, with Nadella's essay on needing an "ecosystem" of app companies singled out as notable coming from Microsoft, "the ultimate monopolist." And a further group is simply going all-in to lock in an edge over competitors. Overall, they concluded, the executive class spans the entire range from dismissive to all-in.

   Key moments

   All I'm seeing is GPT-5.6 stories, and the court politics has come for AI — it is just so stupid and fatiguing.

   Is the race to an intelligence explosion via recursive self-improvement something society should tolerate happening in a somewhat uncontrolled, YOLO way, by a few key companies? That is the question I would really pose to the president of the United States.

   Now the government knows everything about me? Yes, it does — it just has always known everything about you. It's just never bothered to cross-reference all the data.

   Full transcriptLightly edited · timestamps jump to YouTube

   5:02

   Prakash: Good morning. It's Friday, June 26th, 9:06 AM. Nathan, good morning — what's been on your mind? Nathan Labenz: Good morning, Prakash. I'm tired — that's my feeling this morning. I feel like, oh God, all I'm seeing is GPT-5.6 stories, and the court politics has come for AI. It is just so stupid and fatiguing. It's the smartest of times, it's the stupidest— Prakash: —of times. And I'm really— Nathan Labenz: —concerned it might become inescapable for us.

   5:44

   Prakash: There's such fascinating stuff happening everywhere, and the— Nathan Labenz: —sort of stupid eye of Sauron is still focused in on AI, and— Prakash: —understandable. I kind of keep bouncing back and forth between judgment and— Nathan Labenz: —cognitive empathy for the administration. They're just figuring this out. It is legitimately insane — they've been told so many times they've got to do something. I'm at peace with the idea that some action at this point probably wasn't a bad thing, but we should be winging it way less than we are at the presidential level, Prakash: —and it's going to be frustrating to have to follow the Trumpian version of— Nathan Labenz: —will they, won't they? Can they, can't they? Do they even understand what's going on? It's just been an overwhelming amount of attention on— Prakash: —in some ways, the people who are at least— Nathan Labenz: —well prepared—

   6:48

   Prakash: —to be the decision-makers. Well — in short, Fable is still banned, number one. Number two, The Information reported that GPT-5.6 had been submitted for approval even prior to the Mythos announcement. The OpenAI team had been working with the Trump administration when this Fable thing dropped, and everything got kind of frozen. So now GPT-5.6 is being released, and the administration wants to approve it customer by customer — user by user — before that user can get GPT-5.6 access. Sam Altman internally—

   7:41

   Nathan Labenz: —inside OpenAI has said he expects this to go on for about a couple of weeks— Prakash: —before a broad release. It's still not so certain, I think, Nathan Labenz: —and that's where we stand. I think the news breaking that it's going to be— Prakash: —a customer-by-customer release has really taken the tech and startup world aback, and people are extremely annoyed. So we're starting to see the pushback now. One of the pushbacks is: well, if you're only going to approve the big companies, then the startups don't get access to this — and the startups are the ones nimble enough to adopt and attack the big companies. So you're basically handing the big companies a regulatory moat, which is exactly what you didn't want to do, and you've ended up doing it anyway. That's one reaction. The second reaction—

   8:40

   Nathan Labenz: Valid, I'd say. Prakash: Yep, valid. And in fact, that's what people were saying the larger companies were aiming for from the beginning — a regulatory moat. The second reaction, I think, was from Dean Ball. Dean Ball had— Nathan Labenz: —a good write-up on it. And he said — I think — the administration does not yet realize that if you delay too long, you'll end up in a— Prakash: —market downturn, which can then go into a recession and then a depression, because there's a lot of money involved at this point. So that was the next thing that struck me. There's not really that much more to say about it — at the end of the day, you're going to have to release at some point. And at the end of the day, the— Nathan Labenz: —capabilities are going to be broadly available within six to twelve months anyway.

   9:39

   Prakash: So I don't know — have I made it clear that the administration isn't adding anything to Anthropic's expertise? That's really the sad fact. And not OpenAI's either. Nathan Labenz: You know, I'll be very, very blunt — on the Anthropic, OpenAI side, you're going to get people like Nicholas Carlini, who is— Prakash: —the all-time most published security researcher, renowned in the field, Nathan Labenz: —and so renowned, in fact, that he can say something and every single chief information security officer in the United States will listen to what he has to say and— Prakash: —take that into account. And on the other side, you have pockets of expertise — like CISA, and a couple of other pockets, important footnote there — pockets of expertise. But on the whole, you have twenty-, thirty-year-tenured government bureaucrats who run IT departments—

   10:52

   Nathan Labenz: —of billions of dollars, and who basically depend on— Prakash: —a bunch of government contractors like Accenture, and who have gotten— Nathan Labenz: —into these kinds of Healthcare.gov-type issues, and so on, historically. Prakash: And you have these guys on the other side. And for the large part, actually, the federal government relies on its— Nathan Labenz: —monopoly on violence. It relies on being able to indict people if you mess with their IT systems. So yes, they have IT systems, they try to secure them, but the fallback is they can always throw you in prison if you mess with their systems. And if it's a nation-state actor outside the country, they turn it over to the CIA and the NSA — they chase these people down.

   11:40

   Prakash: And so it's always really been that the defense-in-depth is— Nathan Labenz: —a combination of: okay, we'll secure IT systems to the best of our ability and our budget, but we also have the federal government's laws to protect us and to— Prakash: —defend the systems. What's happening right now is that, to— Nathan Labenz: —a large extent, these laws are not effective against citizens en masse. I'll give you an example: Napster. The RIAA ended up prosecuting, I think, maybe twenty or thirty people for Napster — the actual consumers — but they largely didn't prosecute the millions of people who had actually used the service. And— Prakash: —to this day, BitTorrent continues to be in use around the world.

   12:32

   Nathan Labenz: To that same extent, this kind of capability being diffused down to individual citizens makes it, to some extent, impossible. If you have a bunch of teenagers — especially thirteen-, fourteen-year-olds who love playing with computers, typically called script kiddies — they now have the capability to break into, say, Social Security or the NSA. Prakash: And the government can't even prosecute properly, because if you're a—

   13:01

   Nathan Labenz: —thirteen-, fourteen-year-old, you're a juvenile — almost nothing can touch you, especially for a computer crime. At best, I think, I've seen people get their computers taken away for five years if you're a juvenile who commits a hacking crime at thirteen — so you don't get a computer back until you're eighteen. So you have this capability diffusing downward, and there's really no way for the government to prevent that diffusion or stop citizens from using it. If you have citizens en masse using it, especially juveniles, there's no way to stop it at all — you have no choice but to defend your systems. And so you have all these people talking online about, 'oh, Mythos, Fable, this, blah blah blah.' And I'm like: what do you think it takes to secure Social Security against Fable- or Mythos-like capabilities being available in the wild? How long do you think that takes? Do we have a sense it's going to take six months, a year, two years? And is that the pace the government is used to moving at?

   14:11

   Prakash: And what do you think the worries are? Nathan Labenz: What happens if it does get hacked? Are our citizens going to be mad at the government, or at the AI companies, or the open-source providers, or China for creating open-source models? How does this even work? Prakash: And that, I don't really know. Nathan Labenz: Yeah. I mean, somehow anything Trump touches turns to chaos and random last-second dealmaking. So it's going to be hard to predict how this plays out from here. The government is certainly not well-prepared for all this.

   14:52

   Prakash: They— Nathan Labenz: —I mean, an interesting question is whether they're just not willing to do it. I think, in the end, we should separate two big things. One: is the race to an intelligence explosion via recursive self-improvement something society should tolerate happening in a somewhat uncontrolled, YOLO way, driven by a few key companies? That's the question I would really pose to the President of the United States and want the government focused on. Everything else, they're probably not going to add much value on, unfortunately — it seems like they're mostly just discovering what was already possible in many cases. If you're really worried about the script kiddies, a lot of it is already out there. I think there is still some way—

   15:48

   Prakash: —the most— Nathan Labenz: —willing-to-bite-the-bullet AI safety voices, I think, would still say: look, the window hasn't entirely closed. You do still have choke points. The models are huge — they don't run on consumer hardware.

   16:36

   Nathan Labenz: Something that can run a GLM-5.2 is at least a five-thousand-dollar machine. And as much as there are pretty cool uses — I spoke to the Liquid AI CEO again earlier this week, and the vision of this very diffuse, narrow-purpose-built AI running on all devices, where they get a little billion-parameter model running smoothly on your phone — I was really impressed with that. But on an iPhone, their stuff runs, but it's nowhere near the level needed to hack systems. So you maybe have some window where, if you were truly, fully AI-safety-pilled, you could pull off some heroic maneuver to shut the whole thing down. But I think that focus should really be on the runaway-process possibility. And yeah, it's sad but true — I think the frontier companies right now are just better able to serve enterprises, better able to steward the process of bringing this stuff online. And it really sucks that we don't even know what's going on. I mean, these transparency laws have been the first thing everybody's been able to get behind, including the companies. And we're just operating in such a dearth of information right now — in terms of what's even been measured, what's been observed. There's no ability to have a sane public conversation about it when we don't even have a general sense of what the behavior at issue actually is. It's all filtered through reporting, and it doesn't—

   17:51

   Prakash: —sound that bad. It's just — it's all so stupid. Oh, goodness. I have a little bit of empathy, having been both on the CEO side and more on the CTO side. And I think the difference is that on the tech side, Nathan Labenz: —you kind of see all of these things as little bits of information, and those bits of information are manipulable. And it— Prakash: —doesn't take much to transform one bit into another bit, and you have this view that you can structure all the bits together and make something out of it. On the CEO side, you tend to look at all of this as structures — these structures have rules and regulations, and so on — and bits can't leak from one area to another because it's not in the structure. The structure just doesn't work that way. And you have this view that— Nathan Labenz: —it's almost like someone else built it, and you don't have agency in building it. So you don't have this sense that you can move information from here to there, and all of a sudden—

   18:59

   Prakash: —get something new. So this is the kind of thing — for example, if you wanted to spot every single illegal immigrant— Nathan Labenz: —in the United States, you could just look at the tax records and cross-reference: who's paying taxes but doesn't have a proper Social Security number, or whose Social Security number has been used multiple times, and so on. You can do a lot with cross-referencing databases. Prakash: And on the more executive side— Nathan Labenz: —this is seen as a big no-no, a privacy thing, all of this— Prakash: —whether it should even be legal, and so on. But on the tech side, it's like: of course you cross-reference databases, of course that's a given.

   19:47

   Nathan Labenz: Because how else do you run a big organization except by accumulating all the data and cross-referencing it? And then— Prakash: —you can do really innovative things, like filling in people's taxes for them, filling in all their forms, so they don't have— Nathan Labenz: —to repeat all this stuff. In terms of usability, that's how you do it — you cross-reference all the data. Prakash: But— Nathan Labenz: —this concept of being able to cross-reference the data is a huge gap for executives, or— Prakash: —politicians to get over, because they're like, 'oh no, now the government knows everything about me.' Yes, it does — but it always has. It just never bothered to cross-reference all the data.

   20:28

   Nathan Labenz: And when you do, it's something new. I think it's a little bit like that with this too — the bugs are always there, they're just now being— Prakash: —assembled by an intelligence that can chain them together, and all of a sudden they can exploit it. It's not something the AI came up with — it's that your software is buggy and has always been buggy, and you just didn't have tools that could chain everything together before. So on the executive side, you tend to blame the AI firms, while on the technology side, you tend to think it's just a bug you have to fix. That's the crux of it, I think.

   21:17

   Nathan Labenz: And it's kind of like this two-cities concept — on one side you have the technologists, on the other you have the lawyers, and they just don't see eye to eye on this. So, how do you feel the sentiment toward AI companies is among executives broadly, at major American companies? Prakash: I'm—

   21:47

   Nathan Labenz: I don't really have a great sense for that myself. I mean, the adoption is obviously extremely fast-paced — faster than I would have predicted, even. And the revenue numbers obviously don't lie. It's obvious the executive class is alert to the fact that this is something they're going to have to navigate, at an absolute minimum. And most, I'd say, are quite a bit farther along as a class than the government in figuring out what it means for them — adapting their organizations, and so on. Do you have a sense of whether they like it? Are they excited about it? Do they feel like this is the future of their industry, and it's all going to be hyper-efficient and abundant and great? Or are they more in the same mindset as the broader public — that this is probably going to suck, take things away from them, and cause all sorts of other problems? For what it's worth, I'm with those who think the data-center backlash isn't so much about data centers specifically as it is the broader AI—

   23:05

   Prakash: —angst, and just distrust. Where do you think the executives are? So, as always, there's a huge variance in opinion. I'd say about thirty percent believe AI is all a scam, just another sales-tactic hype cycle. Still, that many? Yeah, yeah — you have to recognize that— Nathan Labenz: —over the years, enterprise-software salespeople have told so many stories about why you have to buy this and why your IT spend should go up. Prakash: And we've been through cycles where it's been stupid stuff, like— Nathan Labenz: —crypto, for example, blockchain. Blockchain was huge, and a lot of people knew it didn't really work, but they'd scare the CEO enough that the CTO could get— Prakash: —the IT spend. It's almost this extractive thing, where the CTO wants IT spend, gets it, and allocates it to other people. There's this whole— Nathan Labenz: —back-and-forth between the CEO and the CTO over the costs. So I think a good thirty percent—

   24:15

   Prakash: —is always in this camp of, 'it's a scam, just a sales-tactic hype cycle.' Then there's a bunch of people who are almost under a kind of psychosis, expecting more from the technology than is really there yet, and pushing forward so hard that they're actually harming their companies because the tech isn't ready. I think Meta's Mark is in that camp, because he— Nathan Labenz: —knows how important it is, so he's extended himself beyond the— Prakash: —I think, in many cases, it's the people who aren't ready more than the tech isn't ready. And a lot of CEOs see that too — they see their people aren't there yet, and they're trying to figure it out. And then there's a bunch of CEOs, like Satya Nadella, who are— Nathan Labenz: —trying to make the case that their businesses will still exist even with AI, and will— Prakash: —flourish. I think Marc Benioff at Salesforce is in that camp too. Satya wrote this— Nathan Labenz: —piece about how we need an ecosystem of app companies. Microsoft, the ultimate monopolist, talking about needing an ecosystem so that Microsoft can just be a member of that ecosystem — that's a great thing to see. So I think you see these various levels of—

   25:36

   Prakash: —buy-in and commitment. Yeah, it's just a huge range. I've seen people— Nathan Labenz: —who are very much, 'I'm going to use this and I'm going to lock in—' Prakash: —and nail my competition.' So there are definitely people out there who are just absolutely going for it, which is great. Yeah — the whole range, I think.

2. 26:07Interview38 min

   Interview: Robbie Goldfarb — Judgment Models and Grading the Models on the NewsRGRobbie GoldfarbRobbie Goldfarb — co-founder & CTO of Forum AI, founded with former CNN anchor and Meta news chief Campbell Brown — on distilling a bipartisan expert network into 'judgment models' for questions with no answer key: the three-step rubric method, judgment as a potential RL signal, OpenAI's spec vs. Anthropic's constitution, and NewsBench's findings that roughly a third of frontier-model answers about the news contain a verifiable factual error and models frequently cite state-controlled outlets. Plus expert legitimacy, representation, and transparency.

   Watch

   ▶Play this segmentYouTube · from 26:07

   As aired

   The first guest segment of the show welcomed Robbie Goldfarb, co-founder and CTO of Forum AI, who works alongside veteran journalist Campbell Brown to evaluate — and try to improve — AI systems' judgment rather than just their raw intelligence. Before Forum AI, Goldfarb led trust and safety engineering teams across Meta, Facebook, and Instagram, work that put him on the front lines during the COVID-19 pandemic and the 2020 US election. After a shaky connection delayed his entrance (and a brief mic swap once he joined), Goldfarb and the hosts discussed why so many trust-and-safety veterans, including Goldfarb himself, have left Meta to found their own ventures — Goldfarb argued Meta's unusually novel, high-scale problems bred a creative, not-by-the-book culture that's now proving useful as the same kinds of frontier issues resurface in AI.

   Goldfarb then walked through Forum AI's core method for distilling expert judgment into 'judgment agents': first, expert reasoning distillation through techniques like consequence mapping and cross-expert debate to build draft rubrics; then pressure-testing those rubrics by having a second group of experts apply them to real examples, using their disagreements to locate ambiguity; and finally calibrating a judge model on the resulting expert-consensus data. Asked whether this feeds reinforcement learning directly, Goldfarb said Forum AI's work so far has mostly been evaluation, with reward-model and environment-building work now underway. Pressed on how expert judgment maps onto different alignment philosophies — OpenAI's long, enumerated spec versus Anthropic's more character-driven constitution — he offered an example from mental health experts: Anthropic's rule against 'scheming' behind a user's back doesn't cleanly apply to a clinician strategically steering a conversation with someone struggling with an eating disorder, illustrating why static rules break down and why Forum AI is aiming to train models on nuance rather than rule lists.

   The conversation then turned to Forum AI's Newsbench study, which evaluated how ChatGPT, Claude, Grok, and Gemini answer questions about the news on accuracy, neutrality, and source quality. Goldfarb said the results were a mix — models had generally improved version over version, though one evaluation showed a notable regression on Fable — but factual accuracy was worse than expected: roughly a third of the roughly 2,500 responses tested per model contained a factual error. On bias, every model except Grok leaned left; on sourcing, about one in seven responses cited foreign state media such as RT or China Daily, often on topics — like US domestic politics — unrelated to those outlets' home countries, which the hosts chalked up partly to how freely those outlets let themselves be scraped.

   Prakash raised a harder epistemics question: whether expert-distilled judgment risks over-weighting status-quo consensus the way scientific fields resist paradigm-shifting single counterexamples. Goldfarb said Forum AI deliberately stays out of empirical and innovation domains and instead continuously recalibrates its rubrics and refreshes its expert network for both currency and representation, noting that even the definition of 'conservative' has shifted within the past year. Nathan pushed further on trust and legitimacy, floating a liquid-democracy-style delegation model; Goldfarb said there's no clean answer, but Forum AI leans on transparency — publishing who its experts are, soliciting public feedback, and refreshing the network accordingly — while stopping short of fully crowdsourced judgment. On accountability, he argued Forum AI's edge is publishing the actual failure modes behind a score rather than a bare number, since that's what lets the public independently judge whether an AI system — or an evaluation of one — deserves trust.

   Key moments

   Rules just don't perfectly track to the real world — the real world is just too complicated.

   About a third of them had a factual error in them — a wrong number, a wrong date, a misattributed quote, a misstated policy.

   Propaganda pays, it seems, in the AI era.

   Questions asked

   34:12Tell us about the DNA Meta Trust and Safety instilled in the people now playing similar roles in AI.

   Meta was seeding user-facing internet technology at unprecedented scale, so the problems were novel and there was no playbook — that forced an ambitious, creative, less by-the-book culture. It attracted strong talent and built a cohort able to work at the frontier. That same figure-it-out energy is why so many trust-and-safety veterans now populate the AI labs, though it's demanding work not suited to everyone.

   38:34How does Forum AI actually select experts, obtain the data, and turn expert knowledge into a judgment agent?

   It's a three-part flywheel: expert reasoning distillation through techniques like consequence mapping, thought-process mapping, edge-case testing, and cross-expert debate to draft rubrics; pressure-testing those rubrics with a second group of experts applying real labels, using disagreement to locate ambiguity; and then calibrating a judge model on the resulting expert-consensus rubrics and gold-label data.

   42:42Does this work ultimately become an RL signal that improves frontier model performance?

   Increasingly, yes, but most of the work over the past several months has been evaluation — including the public Newsbench benchmark. The next step is building reward models and parallel environments to actually fix the issues the evaluations surface.

   43:43How does expert judgment translate differently across training paradigms — OpenAI's long enumerated spec versus Anthropic's character-driven constitution?

   Anthropic's constitution is, in some ways, also just a long list of rules. An example: its rule against scheming behind a user's back doesn't map cleanly onto a mental-health clinician who deliberately, and somewhat covertly, steers a conversation with someone struggling with an eating disorder. That shows rules break down against real-world complexity, and Forum AI's aim is to train models on captured nuance rather than static rule sets — a step beyond even the constitution approach.

   46:59What was Newsbench, what was the intent behind it, and what did it find?

   Newsbench evaluated how GPT, Claude, Grok, and Gemini answer news questions on accuracy, neutrality, and source quality, filling a gap the labs' own benchmarks didn't cover. Models had generally improved version over version (with one regression on Fable), but about a third of roughly 2,500 tested responses per model contained a factual error. Every model but Grok leaned left; about one in seven responses cited foreign state media like RT or China Daily, often on unrelated topics such as US domestic politics.

   51:50Doesn't distilling expert judgment risk over-weighting status-quo consensus, the way science resists paradigm-shifting single counterexamples?

   Forum AI stays out of empirical/innovation domains (software engineering, pharma) and focuses on subjective, ethical, philosophical questions where it continuously recalibrates — noting that even the definition of 'conservative' shifted materially within the past year — and keeps refreshing its expert network to bring in fresher voices alongside seasoned ones.

   56:08How do you think about harmonizing public trust and willingness to be led with the expert class, including liquid-democracy-style delegation mechanisms?

   There's no perfect answer. Forum AI anchors on trying its best with clear, intentional principles for what makes a good representative expert, and — most importantly — on transparency: publishing its expert network, taking public feedback, and refreshing the network in response, while stopping short of fully crowdsourced judgment (which Goldfarb sees as going too far in the other direction).

   1:00:09Do you have mechanisms for building credibility through actual track-record accuracy, the way prediction markets reward being provably right?

   Forum AI leans on radical transparency about findings and failure modes rather than punditry-style unaccountable claims — publishing not just a score but exactly why a model earned it (e.g., citing state media on an unrelated topic), which is what lets the public independently assess whether the evaluation itself deserves trust.

   Related

   Forum AI ↗NewsBench ↗Robbie Goldfarb on X (@robbiegoldfarb) ↗

   Full transcriptLightly edited · timestamps jump to YouTube

   26:07

   Nathan Labenz: Our first guest today actually might be a good source for some synthesis of elite opinion. We've got Robbie Goldfarb from Forum AI coming up. Essentially, they're trying to teach AI expert judgment — they assemble a big panel of human experts, identified by name on the homepage, and try to get the AIs to learn from the best of them. I think it's a really interesting strategy. We're short on wisdom, and the question is whether we can distill some of it into the AI. It's worth a try. It'll also be interesting to see if there's a pulse on that kind of question that he can offer.

   27:01

   Prakash: Yeah, I'm just waiting a couple seconds for Robbie to come on. While we wait, maybe I'll do a quick intro of Robbie Goldfarb. He's the co-founder and chief technology officer of Forum AI. For the past several years, the technology industry has evaluated artificial intelligence primarily on how smart it is — how well it can write code, pass the bar exam, or retrieve obscure facts. Robbie's work focuses on a more consequential problem: how to evaluate an AI system's judgment. Before launching Forum AI in late 2025 with veteran journalist Campbell Brown, Robbie led trust and safety engineering teams across Meta, Facebook, and Instagram. He was on the front lines of global information integrity during the COVID-19 pandemic and the 2020 United States elections, building systems to handle edge cases where there's rarely a simple, mathematically correct answer. Today, Forum AI operates on the premise that standard crowdsourced grading is insufficient for high-stakes AI. Instead, the company scales the nuanced reasoning of world-leading experts — from former cabinet officials to top clinicians — into judgment agents that evaluate AI models on complex topics like geopolitics, mental health, and breaking news. Just last month, Forum AI released Newsbench, a massive empirical study revealing that major chatbots like ChatGPT and Claude fail on election-related questions ninety percent of the time. Robbie is here today to discuss why facts alone are no longer enough, why product edge cases fundamentally shape user trust, and what it actually takes from an engineering perspective to build a wiser machine. So that was the intro — let's see if he's up here.

   29:16

   Nathan Labenz: Good morning, Robbie. Can you hear us? No.

   29:28

   Prakash: Alright, so while we wait — let's just see for a moment. Forum AI, it's interesting, because we've had Brett Levinson on before too, and it's interesting how many people come out of Meta Trust and Safety. Meta Trust and Safety seems to be the group in tech that had to face the most difficult issues head-on, early, and at a scale humanity has never seen before — literally billions of people interacting, trying to get error rates of classification down to basis points. It's interesting how many people have left to found their own things. I wonder if that's the foundation of trust and safety in Silicon Valley, so to speak.

   30:45

   Nathan Labenz: Yeah, I think one of the more interesting things I remember coming out of — I don't know if it came out of that organization or just the line of thought at Meta — is the oversight board that's sometimes called the Supreme Court. Apparently it's still doing its thing; Meta just reaffirmed its commitment to the board, committing additional funding to secure its independent operations through 2028. While we wait, that just makes me feel like, boy, we really need some independent structures for AI governance — we've got to drag it out of the court politics we've kind of stumbled into. And, arguably, given who's president, there was no other way it was ever going to go than through the courts. It's been interesting, because one of my little forced comps of AI moments is having been in the same dorm as Zuckerberg way back in the day, so there's been a long discussion among friends who knew him then as to how he's doing, and whether we should think highly of him as he's gone off and built this global empire. I've always been a defender on the grounds that somebody was going to build a global tech communications platform, and as people go who might have done that, he seems like a pretty good guy who's trying pretty hard. Setting up these independent mechanisms is a pretty forward-thinking approach the AI companies have largely tried in their limited way while they've been small — the fact that there's METR, for example, is kind of a conscious creation of OpenAI and Anthropic, and I think Google too to an extent, working closely with this vision of a credible, independent, institutionally willing whistleblower for years. We really can't afford to let all the work done around model characterization and clarity on what's going on get left behind as the decision-making process goes behind closed doors and is potentially classified to varying degrees. Dean was really heavy on that this morning in the piece we mentioned earlier, and I absolutely think it's critical that we have a decent basis of information for a public conversation. So it looks like we've got Robbie, and there are many threads in our discussion so far he can pick up on. Hello.

   34:07

   Robbie Goldfarb: Hey, guys. Thanks for having me.

   34:09

   Nathan Labenz: Great to meet you.

   34:10

   Prakash: Great to meet you too, guys.

   34:12

   Nathan Labenz: We've just been talking about a few different things. Prakash noted, and maybe you can pick up here a little bit, that a lot of people have come out of Meta Trust and Safety and gone on to start other trust-and-safety-oriented organizations, which now all have to focus on AI since that's where all the energy is. Tell us about that organization and the kind of DNA it's instilled in people who are now playing roles like yours in the AI era.

   34:41

   Robbie Goldfarb: It's funny — we were just talking about this with my team a couple days ago, because several of the organizations we work with will often embed with trust and safety teams, and more often than not there are folks from Meta there. Meta was really at the forefront of seeding user-facing internet technology at scale, and so many of the problems we had to deal with were novel. There was no playbook, and it required a very ambitious but also creative approach, because it was all new territory. I worked in a few different trust and safety organizations within Meta over the years, but there was always something very energizing and creative about it, in a way that's quite different from how folks traditionally think about trust and safety, which is more conservative, more by the book. Sorry — Prakash, can you guys hear me?

   35:58

   Nathan Labenz: I lost him as well.

   35:59

   Prakash: Yeah — Robbie, can you maybe switch mics and then switch back?

   36:20

   Nathan Labenz: We need Fable to fix all these bugs. Damn it.

   36:28

   Robbie Goldfarb: I'm on my— Should I stick on the laptop mic? Is this better?

   36:35

   Nathan Labenz: Never a dull moment on live TV.

   36:37

   Prakash: Yeah, yeah — this is our Ajax-complete problem when we do live shows. Once in a while it happens.

   36:48

   Robbie Goldfarb: No worries.

   36:49

   Prakash: So, yeah — where were you? You were talking about how your team interacts with trust and safety at all these organizations, and how Meta was really a place where you saw a lot of novel social dynamics play out at scale.

   37:10

   Robbie Goldfarb: Yeah — and I think what that instilled, to your point on the type of organization that developed there, was a very energizing and creative approach, which is different from how a lot of folks think about trust and safety — more conservative, more by the book. We didn't really know how to figure out a lot of these issues, so we had to reach for the stars. That created, first, a really engaging place to work, which attracted a lot of good talent. But second, it created a strong cohort of people who could approach these problems at the frontier. Now, as we shift into AI, we're in a similar spot — we're seeing a lot of these issues for the first time. That's why, when we talk with trust and safety folks at a lot of these labs and other large organizations, you see a lot of Meta people there — it's that same kind of creative, figure-it-out energy you need, which, by the way, is not for everyone. It's really tough and can be very stressful at times, so it takes a certain type of person.

   38:34

   Prakash: One of the things I understand about Forum's method is that you pick experts in certain verticals to distill their knowledge into a judgment agent, which can then classify various text. How do you actually do this? How do you select the experts, how do you obtain the necessary data to create this distillation, and how do you apply it?

   39:16

   Robbie Goldfarb: I'll tell you first how we landed on this technique, then describe it. A while back we did an exercise where we looked at several LLM judges we could find in open-source benchmarks — some from foundation model labs, some from academic research. We took experts and had them look at the judges: was it rubrics they were pulling from, or just prompt instructions? We'd ask, do you agree with the instructions here? Generally the expert would say, yeah, that seems pretty reasonable, I agree with it. Then we had the experts look at labeled outputs from the judges — say, political bias. A bunch of responses labeled: this is politically biased, this isn't, this is, this isn't. We asked the experts the same question: do you agree? What we found was that, more often than not, they didn't agree with the actual judgments the agents produced, even though they'd agreed with the high-level guidance given to the judge in context.

   40:46

   That was an important realization, because so much of the current state of benchmarking relies on AI judges that were calibrated pretty loosely that way. It led to a series of experiments and trial and error to figure out how to build judges that were actually calibrated to experts. To answer your question, it's a three-part process. The first piece is what we call expert reasoning distillation — high-level conversations. We've come up with a number of creative techniques for talking with experts that look nothing like traditional data labeling: consequence mapping, thought-process mapping, edge-case testing where we pressure-test the edge cases, and cross-expert debates where we have them discuss things with each other. Through that we take a first crack at developing a set of rubrics and context to represent how they think about a given factor — bias, say.

   41:31

   But then — and this is what we found to be the most important step — we pressure-test it. We get another group of experts to look at that guidance and actually try to apply it with real labels. What will almost always happen is experts disagree — they apply the labels differently — and where experts disagree indicates where there's ambiguity in the rubric. So we go back to step one, and you get this flywheel: high-level conversations to wrestle with the ambiguity, then applying it, back and forth. Eventually, once you go through that enough times, you get rubrics and context that represent expert consensus, and also golden data you can use for step three, which is calibrating a judge — in this case, for assessing factual accuracy. I could go into more detail, but that's the gist.

   42:42

   Nathan Labenz: Does this then become an RL signal? Is that ultimately how this work gets translated into improved frontier model performance?

   42:54

   Robbie Goldfarb: Yep, increasingly we're doing more of that. Until now, most of the work we've done over the past several months has been evaluation — large-scale, comprehensive evaluations. We launched our public-facing Newsbench benchmark to give consumers and executives a sense of where a system is at. But of course, step two is: how do we actually fix these issues? That's where we're developing reward models, and we're also doing some work to build parallel environments more broadly — thinking about what scenarios we actually want to evaluate the system in.

   43:43

   Nathan Labenz: Do you have a sense for how this plays differently against character-versus-corrigibility training paradigms? I'm struck that Meta's terms of service were famously Talmudic — famous examples like, is there an exception if a nipple is visible during breastfeeding? I heard a whole podcast about how that played out around the world. That's kind of associated with the OpenAI approach these days — write a really long spec, tell the model what to do, enumerate the cases, and it's supposed to be instruction-following. Versus, of course, the Anthropic constitutional, more character-driven approach. I could see expert judgment playing out either way — how do you think about translating to those two different training paradigms?

   44:43

   Robbie Goldfarb: I'll share one interesting example, since you bring up Anthropic's constitution — in some ways it's also just a long list of rules; the way they use it is a little different, but here's one we were confronted with the other day. There's something in Anthropic's constitution that speaks to this idea — 'scheming' wasn't the exact word, but essentially, you should never scheme behind the user's back. We were talking with some experts in the mental health space, and when you talk to clinicians dealing with topics like eating disorders or other unhealthy habits, one of the things you actually do want to do is strategically divert the conversation — the intention you have as a clinician in that case is hidden from the client. We had a conversation with clinicians referencing that point in the constitution, where at face value, yeah, of course you never want to go behind the user's back. But what it reveals is that rules just don't perfectly track to the real world — the real world is just too complicated. So the hope, through the process I described, is that you start to capture that nuance, and as we develop more of these judges and use them for training, we're aligning models less to static sets of rules and more to the nuance — which I'd argue goes even a step further than Anthropic's constitution, which is certainly a great step forward, but there's still more to be done.

   46:59

   Prakash: Let's talk a little about Newsbench. Give us an intro to what Newsbench was, what the intent behind the study was before you started asking those questions, and what the findings were.

   47:16

   Robbie Goldfarb: Newsbench was an evaluation we built to look at how AI systems — particularly the leading chatbots — respond to questions about the news. Quite frankly, there just wasn't much out there on the topic already. There's some stuff from the labs themselves — Anthropic has their even-handedness benchmark, OpenAI has a similar bias one — but there wasn't much that really asked, how good are these systems at responding to questions about the news? That's where we have a lot of experts for whom that's their core competency — my co-founder Campbell spent many years working as a journalist, and I spent time working on misinformation at Facebook. So naturally it was something we were interested in.

   48:01

   We looked at three things — after talking to experts, we defined 'good' as accuracy, getting the facts right; neutrality, is it leaning in one direction or the other; and source quality, when they ground themselves on external sources, are those sources reliable. We looked at GPT, Claude, Grok, and Gemini. The findings weren't all bad — there were some positive signals. One thing that's interesting, relating to something you two were talking about earlier: we generally saw models improve version over version. Opus 4.6 to 4.8 was a pretty significant improvement in bias, which is consistent with Anthropic's own reporting. But Fable was actually a big regression, interestingly — which shows that more power doesn't always mean better when you're dealing with subjective, nuanced things.

   49:32

   All that said, there was a mix of findings. I'd say, all in all, the level of issues we found — particularly on factual accuracy — probably surprised us. We looked at about 2,500 responses per model, and about a third of them had a factual error in them — a wrong number, a wrong date, a misattributed quote, a misstated policy. We knew that was an issue, but the number was quite a bit higher than we expected. On bias, maybe unsurprisingly, all the models tended to lean left, with the exception of Grok, which leaned right — which probably tracks with what most people would assume. But on sourcing, in about fifteen percent — one in seven responses — the model sourced foreign state media.

   50:33

   Prakash: Wow.

   50:34

   Robbie Goldfarb: Like RT from Russia, or China Daily. And what's really interesting is that often this wasn't even in questions about their home country — we saw RT and China Daily sourced in questions about US domestic politics. That was a particularly interesting finding we saw across the board with all the models.

   50:59

   Nathan Labenz: Propaganda pays, it seems, in the AI era.

   51:03

   Robbie Goldfarb: It does. But it's also, I think, a business-model thing — state-sponsored media is generally free and open, so, sure, come scrape our stuff. Whereas the New York Times, for example, you have to have a deal with them. They've certainly done a good job opening themselves up to the scrapers.

   51:28

   Nathan Labenz: Got their llms.txt in good working order. Yeah — that's a very fascinating finding. Prakash, you have any more methodology questions on that? I was going to zoom out a bit at the end, but is there anything you want to dig deeper into?

   51:50

   Prakash: The main thing this kind of expert approach has always struck me with is that, in science, you often have the weight of scientific evidence on one particular thesis, and all it takes is a single counterexample to disprove the entire thesis — then the whole field has to change. That's the scientific method, so to speak. In every historical instance, the weight of evidence has been on the status quo until the thesis gets disproven, and then everything has to change again. I've always wondered how this plays out with expert models — expert judgment, distillation of experts — that end up assigning a lot of weight to status-quo data and much less weight to new, unique, single instances of things being disproven. It seems like an epistemics issue, which is kind of a conundrum that's very hard to solve. How do you work with this?

   53:14

   Robbie Goldfarb: It depends on what you're looking at. A lot of what we found in our work is — our core competency, where we focus, isn't areas of innovation, like software engineering or pharmaceutical research. That's not really where we play. Where we play is more subjective — philosophical questions, what does good quality look like, what is safe, more ethical dilemmas. You even see this now: the major labs are consulting religious figures and religious texts.

   53:59

   I don't want to say those elements are static — they're certainly evolving, but more existential than continuously, rapidly evolving, and that's generally where we play. What I will say, though, to your point — and we actually found this with Newsbench specifically — is that what's considered politically biased has already changed in the past six months or so, since we started working on it. Concretely, when we talk to experts, the definition of 'conservative' in the US has shifted quite a bit over the past year, even the past few months. So what we've started to do with anything we're looking at is treat it as very much not a one-and-done. That's almost core to the thesis of our business — this isn't something you build and ship once and it's done. We're continuously calibrating these judges to get at the latest state of things.

   54:44

   The other piece is continuing to refresh the network of experts we're talking to. My co-founder Campbell has done a very good job at this — if, ultimately, what we're saying is our expert network is our ground truth, it's really important that we're developing a network that's worthy of being that ground truth. So there are elements of balance and credibility built into that. But another one of our principles is representation. Initially, our expert network candidly skewed toward much more seasoned experts — people who'd been around the block several times. Increasingly, we're trying to bring in fresher perspectives and voices as well, which gets at the point you're making — things are always changing, and you want that reflected. So those are a couple of ways we're thinking about it.

   56:08

   Nathan Labenz: It's a fascinating challenge in many ways, and I had a similar question to Prakash's, but rather than coming at the weight-of-evidence question from the scientific perspective, I was thinking about it more from the trust-and-legitimacy angle. It strikes me that experts probably aren't always trusted in our society — mostly they probably should be, sometimes maybe they shouldn't — and a few high-profile failures of experts have, in some ways, led to a crisis of trust in expert judgment. So I wonder how you think about that today, and whether you're open to mechanisms. I always come back to this book, Liquid Reign — very niche, you've probably never heard of it, very few have, I did a podcast episode on it — where people can delegate their voting power on an issue-by-issue basis to proxies they trust in those domains, who can in turn re-delegate them. So you could delegate to your union on economic issues, to your priest on other issues, to your buddy on particular issues, and it rolls up through the leading lights in the field into some aggregated decision-making authority. What's your vision for how we harmonize public trust and willingness to be led with our expert class?

   57:43

   Robbie Goldfarb: I'll share how we think about it, and I'm curious for your thoughts too, since you've clearly thought about this. We think about this a lot, and it's really tricky. What I'd say upfront is I don't think there's a right answer here — it's one of those things where you have to find the best option among what's possible. We anchor on two things. The first is going to sound a little silly, the second is more practical. The first is: try our best. We have to have very clear, intentional principles up front about what makes a good, representative expert, and let that guide how we make decisions.

   58:28

   The second piece, and probably the most important, is transparency. This was very much a takeaway for me from doing similar work at Meta around fact-checking. At the end of the day, we don't want to be the ones making these choices, but I don't think the right answer is to delegate truth to absolutely everyone either. A nice in-between — similar to that liquid-delegation concept — is having a selected version of, quote-unquote, experts making these decisions, but being really transparent about it, and building mechanisms for the public to give feedback. That's very much what we've done. Early on we did this just with clients — we'd present our expert network and field feedback.

   59:13

   Now we're very public on our website about the experts we work with, and we have a mechanism for people to give us feedback, and we will act on that — you'll see changes in our network as a result. So, long way of saying: I think the best option we've landed on is transparency. We want to be really clear, take input from the public, and use that to refresh the network. But I think that's different from, say, an Arena-AI approach where you just open it up to everyone in the world to say what's politically biased or not — that feels a little too far in that direction. So it's leaning on experts, with an element of public engagement via transparency and a feedback mechanism.

   1:00:09

   Nathan Labenz: I know we're a little over time, and Eric's here, so we have to be brief — we also had a late start with you. But I'm curious about your thoughts on prospective judgment and accountability to that. We're in a Polymarket world now, where people can put their money where their mouth is like never before, and I think a big criticism of expert judgment broadly — punditry, especially — has been that you say what you say, nobody really checks you later, and it's more personality-driven than being able to say what's going to happen in a way people can actually count on being right. So I wonder if you have any mechanisms for building credibility through actual accuracy about the future.

   1:00:59

   Robbie Goldfarb: If you think about practically what we're doing — we're working with these experts to help evaluate and improve AI systems. One thing we lean on very heavily here, which I think is a very big gap right now, and one of the reasons behind some of the missing trust you two were talking about earlier in the show, is that we're very transparent about what the actual findings and failure modes are. If you look at our Newsbench leaderboard, we don't just give numbers — we go and say, here's what the actual failures are, here's why this model got this score. That, I think, is ultimately how you earn trust. We can all talk in circles about whether this person or that person should be trusted, but at the end of the day, the point of working with these experts is to deliver these findings, and the findings speak for themselves. If I'm finding responses from AI systems quoting China Daily about a US state election, I think we can agree there's—

   1:02:12

   Nathan Labenz: —still some low-hanging fruit in the system to be picked before we—

   1:02:17

   Robbie Goldfarb: Exactly. Yeah.

   1:02:18

   Nathan Labenz: Very heady questions I might want to jump to — yeah. I mean, that's a great point. So much of what's being done by the AI companies really is just-in-time — all the fabled guardrails got assembled in the few weeks before launch, and they're citing RT and China Daily, and it's just—

   1:02:41

   Robbie Goldfarb: It's seeing what — you can produce a benchmark where a model scores ninety-five percent, but what is the five percent? I think understanding that, and giving people insight into it, is core to building trust, and to your question, core to enabling people to assess whether this is a good way to evaluate an AI model. If I don't give you that, and I just say, yeah, the model got ninety-five percent, it did pretty good, here are a couple of high-level themes — it's really hard for people to assess that. So, of course, we want mechanisms for people to understand who the network is and have input into it. But at the end of the day, I think what's most important is that they have transparency into the actual decisions being made, and how that's going to shape the systems we're all using.

   1:03:38

   Nathan Labenz: Yeah — from your lips to God's and Trump's ears: let's increase the transparency in a few different key places. Robbie Goldfarb, thank you for sharing Forum AI with us. Your background at Meta, and the way you've taken it forward into the AI era, is super interesting. Keep up the good work, and accelerate everywhere you can, because we need as much lead time and wisdom baked into these things as we can possibly get.

   1:04:04

   Robbie Goldfarb: Appreciate it. Thanks for having me — see you guys.

3. 1:04:09Interview30 min

   Interview: Eric Vaughan — The 80% Rebuild and the AI-Native EnterpriseEVEric VaughanIgniteTech CEO Eric Vaughan on the most aggressive corporate AI transformation on record: 2023's company-wide AI Mondays, replacing ~80% of a resisting workforce, and rebuilding around 'AI DNA.' Fear as the real blocker ('it's a hundred percent fear'), duties-not-people replacement, customized-per-customer SaaS as the new value prop, AI-first acquisition screening (Khoros), consolidation vs. the solopreneur boom, and 'if you don't think you're behind, you're doomed.'

   Watch

   ▶Play this segmentYouTube · from 1:04:09

   As aired

   Prakash introduced Eric Vaughan, CEO of IgniteTech (and also of Khoros and GFI Software), noting that in 2023 he declared generative AI an existential threat and mandated a company-wide weekly AI day; when roughly 80% of his workforce resisted, he replaced them rather than abandon the plan, rebuilding his teams around what he calls 'AI DNA.' Vaughan opened by explaining his aggressive stance to Nathan and Prakash as pure fear — fear that he'd one day be asked why he didn't warn people, having watched the tech industry cry wolf before over crypto, blockchain, and Web3, but concluding this wave really is different since his own conviction, dating to January 2023, has only strengthened over three years.

   Nathan asked how IgniteTech's value proposition to customers has changed. Vaughan described moving from a single, multi-tenant SaaS codebase to offering customers individually customized branches, made possible by an AI-native engineering culture; he detailed the deliberately gradual, yearlong transition that produced 80% turnover — replacing people who refused or resisted, not indiscriminate layoffs — backed by a recent anonymous employee survey showing +78 NPS and 89% reporting gains in AI fluency. Prakash then asked about the Khoros acquisition and IgniteTech's approach as an AI-native acquirer; Vaughan described building an AI interviewer to profile the acquired company's staff before human interviews, and launching a new product, Eloquence AI, to keep customer communications fast and empathetic through the acquisition's chaos. Khoros went from unprofitable to profitable within a year, including a ground-up rewrite of fifteen-year-old legacy code on one of its two products.

   Nathan pressed on the flip side of the 80% figure — what happens to the broader labor force if a similar ratio plays out across the economy. Vaughan argued people can't be forced to believe, only shown the opportunity and freed from the fear that AI simply replaces them; in his view it replaces duties, not people, freeing time for higher-value, more creative work. He stressed that cultural enthusiasm ('fire') has to be paired with skill, citing his engineers' shift to AI-assisted 'vibe coding' and his push to extend that mindset to children's education, where AI should check homework rather than do it. Nathan connected this to Glean's 'bot-sitting' and 'bot-shitting' research, asking whether enthusiasm alone produces good standards for AI output; Vaughan agreed fire without skill just produces more bot-shitting, and argued the discipline — understanding evolving model behavior, sycophancy, and context quality — has to be modeled from the CEO down.

   Prakash asked how AI has changed IgniteTech's approach to evaluating acquisition targets; Vaughan said the first filter is now whether a target has a genuine, product-level AI vision, illustrating with Khoros' new AI moderation feature for catching harmful video content in its community product. Closing out, Nathan asked Vaughan to weigh two seemingly opposite trends — consolidation among larger firms versus a solopreneur boom, citing Stripe's data on new business formation — and Vaughan argued both are true at once: tiny AI-leveraged teams, citing Cursor's rise, can build outsized businesses, while any company, including large public ones, that treats AI as a side project rather than a CEO-level priority risks being consolidated away. He summed it up in a line he says he repeats everywhere: 'If you think you're behind, good. If you don't think you're behind, you're doomed.'

   Key moments

   I mean, it's fear — it's a hundred percent fear.

   What we shouldn't be doing is saying, 'Hey, let AI do your homework for you' — we should not be doing that, that's misuse, that's practically abuse.

   If you think you're behind, good. If you don't think you're behind, you're doomed.

   Questions asked

   1:06:28What's driving your advice to enterprises to be so aggressive about adopting AI now?

   Pure fear — the fear of a career technologist who doesn't want to be asked one day why he didn't warn people that AI would upend every profession. Unlike past hyped-up tech waves (crypto, blockchain, Web3) that fizzled, his conviction, dating to January 2023, has only been reinforced over three years of practice.

   1:08:50How has your value prop to customers changed, before and after — what are people getting for their dollar and time relative to three years ago?

   IgniteTech can now offer customers individually customized code branches instead of one shared multi-tenant SaaS codebase, made possible by an AI-native workforce built through a deliberate, yearlong transition that ultimately replaced about 80% of employees who resisted. Internal NPS is now +78 (top quartile is ~60), and 89% of staff report gaining at least one AI-fluency level.

   1:13:13How did you approach the Khoros acquisition differently as an AI-native CEO, and is AI-driven acquisition and integration becoming more of a trend?

   IgniteTech built an AI interviewer to profile hundreds of Khoros employees across eight countries before any human one-on-ones, and launched a new product, Eloquence AI, to keep customer communications fast and empathetic through the acquisition's chaos. Khoros — a nine-figure-revenue company that was losing money — became profitable within a year, including a from-scratch rewrite of fifteen-year-old legacy code on one of its two products.

   1:18:20Given the headline number is still 80% turnover, what can be done at a broader societal level for 'the other 80%' who aren't converting to AI-native ways of working?

   You can't force belief — you can only make the opportunity visible and push back on the fear that AI simply replaces people (it replaces duties, freeing time for higher-value, more creative work). People will convert either out of necessity or because 'the light bulb goes off'; the job is to help make that happen through skills training.

   1:24:27Is cultural enthusiasm ('fire') enough to also produce good standards and ownership of AI output, or does it take more organizational work — referencing Glean's 'bot-sitting' and 'bot-shitting' research?

   Fire alone just produces more bot-shitting. You need skill alongside it: teaching that context quality matters, that model behavior changes constantly, and that models are inherently sycophantic by design (built to be frictionless). That discipline has to be modeled starting with the CEO personally.

   1:28:48How has AI changed how you evaluate acquisition targets compared to five years ago?

   The first filter is now whether a target has a genuine, product-level AI vision rather than superficial AI branding. IgniteTech looks for concrete opportunities to apply AI to the product itself — for example, building AI moderation into Khoros' community product to catch harmful video content that manual review had been missing.

   1:31:35Do you expect more consolidation (an 80/20 power law among firms) or more solopreneur-style new business formation, per Stripe's data — which trend wins?

   Both at once: tiny AI-leveraged teams (like Cursor scaling to $100M ARR with 15 people before its roughly $60B acquisition) can build outsized businesses that weren't possible before, while any company — including large public ones — that treats AI as a side project instead of a CEO-level priority risks being consolidated away.

   Related

   IgniteTech ↗Eric Vaughan on X (@TheGenAICEO) ↗

   Full transcriptLightly edited · timestamps jump to YouTube

   1:04:10

   Nathan Labenz: Alright, Eric — thank you for being patient. I saw Eric was actually engaged in the chat; I think he's the first guest who's ever been typing in responses to our previous guests. So welcome, Eric Vaughan from IgniteTech. Prakash, you want to give a half intro, and we'll get rolling.

   1:04:30

   Prakash: Alright, let me do an intro. Eric Vaughan is the CEO of IgniteTech, Khoros, and GFI Software. But over the last three years, he has emerged as one of the most polarizing and pragmatic voices in enterprise technology.

   1:04:45

   Eric Vaughan: Great setup. Awesome.

   1:04:48

   Prakash: In 2023, while most executives were forming exploratory committees and running safe pilot programs, Vaughan declared that generative AI was an existential threat. He mandated that his entire organization dedicate one day a week exclusively to AI development. When roughly 80% of his workforce actively resisted the change, he did not abandon the initiative — he replaced the personnel, rebuilding his companies from the ground up around individuals who possess what he calls 'AI DNA.' The operational results of that transition provide a stark case study: operating with an entirely AI-native culture, his teams are now shipping complex, ground-up AI software, such as Adminio, an autonomous scheduling assistant, and MyPersonas, which creates digital twins of enterprise experts in a fraction of traditional development cycles. He recently addressed the Fortune Brainstorm Tech Conference to debate the future of work, arguing that the era of prompt engineering is already dead, having been replaced by complex agent orchestration and context execution. This conversation is highly timely because Eric offers a rare, unvarnished look at the messy, painful, but highly profitable reality of what happens when a global software company stops experimenting with artificial intelligence and entirely restructures its operating model around it. Eric, welcome to the show.

   1:06:12

   Eric Vaughan: Well, thank you. And I think I should say goodbye, because I don't think there's anything left to say — Prakash, you covered the whole thing, soup to nuts. Hey guys, how are you? Good to be on your show. Lot of interesting conversation.

   1:06:28

   Prakash: Eric, I think one of the things that I noted as we did research for this segment was that you are actually quite aggressive in encouraging enterprises to switch gears now. What's driving your advice to enterprises to be so aggressive?

   1:06:52

   Eric Vaughan: I mean, it's fear — it's a hundred percent fear. I've been a technologist my entire life, and in my friends-and-family circle, I am that person too. And the fear is that somebody's going to come to me in one year, two years, three years, four years and say, 'You were in the tech industry — why didn't you tell me? Why didn't you tell me that literally every single element of business, every profession, was going to be completely upended, rewritten, retooled, redesigned, rethought, and I'm now irrelevant. I wish you would have told me.' I don't want to have those conversations. I want to make certain that I at least express what I see in the future already.

   1:07:38

   It's very much like that — it's like I already see what it looks like, and I'm worried, because I've been part of this tech industry for so long. We — the tech industry — have this great tendency to tell everybody that everything is the most important thing ever. Crypto was gonna change everything. Blockchain was gonna change everything. Web services was gonna change everything. Web3 was gonna change everything — except it didn't. So we're a lot like the boy who cried wolf, and people say, 'Yeah, it's another one of these.' This one's not like that. I just am certain that that's true. And three years now of practicing it — my wake-up call of existentialism—

   1:08:23

   and threat to our business started in January 2023. So we're three years into this practice now, and there's nothing that has happened that has reduced that point of view. It's only accentuated it, or underlined it. I just believe it will change everything. And so, you know, we've been given this phenomenal gift of a tool — let's use it.

   1:08:50

   Nathan Labenz: Can you describe how your value prop to customers has changed, before and after? You guys are in the software solutions business broadly — what are people getting for their dollar and their time relative to what they were getting three years ago when they worked with you?

   1:09:09

   Eric Vaughan: Boy, I tell you, that's a great vector, because it has transitioned dramatically. For instance, never in the world of an enterprise software company were you willing to go to a customer and say, 'Hey, we've got this SaaS product — it's a multi-tenant SaaS product, one piece of code that runs for all 800 of our customers. How would each of you like your own branch? We could just maintain 800 branches now,' right? Because we really have these kinds of capabilities to generate AI code, to debug AI code, and to customize and deliver a more custom experience. So I'm certainly not the first person to declare the regular SaaS model dead, and I think it's dead in more ways than a lot—

   1:09:55

   of people think. It's not only dead from a billing standpoint and the way we structure pricing, but also in what customers should expect. We've actually grounded most customers in believing that you can't have something added to the platform because it's not necessarily good for the rest of the customers — it's only good for you. But because we now have a homogenous AI-DNA organization — and that's what happened as a result of the 80% transition, and Prakash, I think you covered it pretty well — I just want anybody listening to know we didn't wake up one day and fire 80% of the people. We spent a full year investing in time, training, funding—

   1:10:40

   tools, and cash-reward incentives for ideas. And, basically, most of the changes — and they weren't cuts, they were replacements — were with people who wanted to do the work and believed in it, replacing people who either flat-out refused or just didn't want to try. You know, World Cup's going on right now, we're all watching soccer — can you imagine if the coach said, 'Okay guys, we're gonna run a 3-3-4,' and two guys out there said, 'Yeah, no, I'm not gonna do that — I don't believe in the 3-3-4'? You wouldn't put that guy on the pitch, would you? It's the same thing. And so, we did that. We had either—

   1:11:25

   people that flat-out refused or people that didn't really want to try. So we measured effort — we didn't measure aptitude so much as we measured attitude, quite frankly. And several have stayed. Just in the last month we conducted a global, company-wide survey of our entire team, because I thought I had the temperature of the team, but I wanted to know exactly what it looked like. It came back remarkably positive — and it was completely anonymous, we took great pains to make sure everybody knew that, so if they wanted to air grievances, they could have. I've got some stats here — employee N—

   1:12:10

   employee NPS of plus 78 — tech-industry top quartile is around 60. Ninety percent said they'd make the same decision to join again without hesitation; zero percent said no. And this was the one that mattered most to me: 89% report that they've moved up at least one AI-fluency level since joining. Why is that important? Because it's not just the work they're doing for our company — they believe they're being enriched with a very valuable skill for the future, which is true. So, Nathan, to zero back to your question — the ability to deliver more to customers in a shorter time—

   1:12:56

   frame has just changed the game of software development dramatically. We all know that: Claude Code, whatever you're using, Cursor, whatever you're using — we're cranking out real code at an enterprise-quality level like never before.

   1:13:13

   Prakash: I wanted to ask you about this because I think you're one of the first AI-native CEOs we've had who's done a recent acquisition, and I wanted to ask about the Khoros acquisition and how, as an AI-native CEO, you approached it differently from an M&A integration you'd done in the past. Was it easier to integrate? What were the main hurdles? Do you think, now that you have AI, acquiring and integrating these companies is going to become more of a thing? We've seen some private-equity shops trying to set up firms that acquire legacy software businesses and transform them with AI — so how did that play out in this acquisition, which is, I think, about a year old now?

   1:14:08

   Eric Vaughan: Yeah, so IgniteTech is an acquisition company — that's what we do. I think we've completed over 150 acquisitions in the last 12 years, large and small, lots of different parameters, but always in enterprise software — almost uniquely in enterprise software, not SMB, not vertical, that kind of thing. So we're built for that. Our mentality and our approach as a business is to find companies, buy them, and integrate them. We're not a private-equity firm — we don't sell companies, we don't hold them for a while and exit the fund. None of that happens; we buy them to run them forever. However, with all the change that had taken place—

   1:14:54

   Khoros is a large acquisition — I mean, we don't release numbers, but it was a nine-digit revenue company, not a small company, with responsibility for some amazing customers. I don't think we could have done it without the AI-DNA-forward approach we were able to take. So, just for instance, there were hundreds of employees in eight different countries, and we needed to very quickly understand who they were, what they did, what they knew, how they were thinking about everything about the business. We quickly developed an AI interviewer that asked questions before there was a human one-on-one. Before, we would have set up human one-on-ones, and people would have gone through—

   1:15:39

   thirty-minute drills with hundreds of people over the next several weeks, without any data. Now we all arrive with a dossier — the interview process just shortcut that and let us get familiar, categorize, and understand who knew what, and that was one example. Another example was being able to communicate with customers in a timely way. Because when there's an acquisition happening — anybody who's been through that, and I know you both have — there's a lot of chaos, a lot of changes everywhere. And one of the things you don't want to miss is communication with customers. In our AI-creation world, we wrote a new software—

   1:16:24

   product that's now available on the market — it's called Eloquence AI. Eloquence AI is an email AI persona: it answers every email it gets in five minutes or less, always in perfect grammar, in 160 languages, always sympathetic, doesn't miss anything, and knows when it needs to escalate to a human — it just adds them on cc. So, for example, imagine customer inquiries coming in: 'What about this acquisition? What does this mean?' Under normal load, pre-AI, you'd assign people to try their best to keep up, and they're going to make mistakes, they're going to be late answering emails, they're not going to say the right thing. With Eloquence,

   1:17:09

   we were always able to respond in a timely way, and especially when it came to HR, empathetically and with detail. So that's just two examples of how we were able to really fuel that acquisition. It was just a year ago — the end of May of last year — that we bought it. The company was losing money when we bought it; the company is now profitable. We've completely transformed the operation of the company, and in a year's time we've released two brand-new versions of Khoros' software that are fully AI-enabled. In one case, it was a rewrite of fifteen years of code — we discarded it and rewrote it from the ground up for one of the two products that—

   1:17:55

   they had. We did that in a year, in the middle of all that transition. That's not a story I could have told five years ago. It just wouldn't have been possible.

   1:18:08

   Prakash: Incredible.

   1:18:10

   Eric Vaughan: Nathan doesn't look convinced. I'm not sure.

   1:18:12

   Nathan Labenz: No, I'm convinced. I'm gonna maybe change gears — Prakash, if you have a follow-up, ask it first, and then I'll—

   1:18:18

   Prakash: Go for it. Go for it, Nathan.

   1:18:20

   Nathan Labenz: Well, I was just struck earlier by your description of the things you tried over the course of the year once you got conviction. And I think in many ways you probably have an advantageous situation — having personal conviction for one thing is worth an awful lot, and that's a story. I've told my own version of it previously — I've been through a similar AI-conviction transition personally that I then had to try to transmit to a broader team, so I know that's a critical ingredient. All the things you did sound like really quite a lot, and yet the headline number is still 80% turnover. So I'm interested in your point of view on the—

   1:19:05

   rest of society — it seems like that percentage strikes me as maybe where we're going, a sort of 80/20, where the 20% kind of inherit the earth, the labor force, in the short term. Do you have any ideas about things that can be done, at a broader level, beyond what an individual company like yours can do, to change that? Is it just a matter of people having been skeptical, and maybe when they wake up they'll get motivated? Because, you know, while your success is exciting, and I try to live it every day with my own agents and my own elevated productivity expectations, I do worry about the other 80% quite a bit.

   1:19:53

   Eric Vaughan: I do too. And it's not like other revolutions throughout history. I think what we have to do is make available — you can't force people to believe in something, you can't force people to do something they just aren't going to do. So we need to leave them to their own devices. But if you can be very open and aware and make people understand what the possibility is — and, number one, really attack this idea that 'AI is going to replace me.' AI is a hundred percent going to replace roles, it's going to replace duties. But in our case, what it's—

   1:20:38

   done is replaced a lot of the mundane, a lot of the things we must do that we absolutely don't want to do and that isn't our best work, and it's freed us up for innovation. I think we over-index in this discussion of AI on efficiency gains — I see it the other way. I think it frees up innovation. It gives you time, because it's now taking care of things you always had to do that you just don't anymore, or it shortcuts the amount of time you have to spend, so you can spend time ideating, creating, proof-of-concepting — doing things in a way that brings your best work. So how do we help that other 80%? We teach them skills.

   1:21:24

   We've got to teach them skills. I was at a conference two weeks ago in Aspen where we had this exact conversation — which is, if anybody is thinking that all you have to do is show up and throw a sentence into one of the models and great things will happen, you're wrong. Three weeks ago it didn't know what date it was, and I said, 'I'm sorry, with the hundreds of millions of dollars of investment you've had, you think today is Wednesday and it's actually Tuesday?' Real problem, and a credibility issue for me. It takes work — you've got to give it context. You can't just say, 'Write a paper that sounds like me,' because the problem is it will do it. It won't say, 'Well, what do you mean, sounds like you? I don't—

   1:22:09

   know what sounds like you. How would I know that?' It never does that. It's like, 'Okay, here you go, here it is.' Those kinds of results are not going to be helpful. So we've got to stop people who don't believe from using poor approaches that only reinforce their belief that it's not going to work. And I think, as we start to see phenomenal advances and changes in industry, in various professions, people will convert one way or the other — they'll convert out of necessity, or they'll convert because the light bulb goes off. We're trying to make the light bulb go off — that's what we're trying to do. We're trying to ignite that fire. And that's what my team—

   1:22:54

   has — a fire. And by the way, what we did not hire were a bunch of people who had never coded and said, 'Here, go write code, all you have to do is tell it and it works.' We have a few of those people, but they're highly supervised by AI-created agents that review their code, and by software engineers who really understand code. But not one of my software engineers who used to write code — fingers on keyboard, typing out letters — does that anymore. They vibe code now, Karpathy-style, and they're happy doing that. I asked them once, on an all-hands a few months ago — I said, honest question, one-and-answer, show of hands: all of you coders, how many of you miss it? Do you miss bringing up this, you—

   1:23:39

   know, the development environment and start writing Python? Not one of them said yes, because they feel this tremendous acceleration of capabilities. And we've got to ignite that fire everywhere. We need to ignite it in children. What we shouldn't be doing is saying, 'Hey, let AI do your homework for you' — we should not be doing that, that's misuse, that's practically abuse. We should say, 'Do your homework, and let the AI check your homework.' And if it finds you don't understand something, then say, 'Hey, I thought I knew how to divide fractions — I guess I don't. Can you give me a tutorial on what I'm missing?' That's what AI is so good at — it'll adapt to the particular person it's interacting with. So building the skills and building—

   1:24:24

   the fire, Nathan, would be my short answer to your question.

   1:24:27

   Nathan Labenz: That's good. One quick follow-up on that is around standards. I recently did an episode of the podcast with a woman from Glean — they put out this 'Work AI' big study, and they coined the terms 'bot-sitting' and 'bot-shitting.' Bot-sitting is like pasting in the context, where the human becomes sort of the plumbing — that's one unhappy failure mode. But the bot-shitting phenomenon was, I think, more alarming and far more widespread — a majority of people admitted to passing on AI work that they themselves could not defend.

   1:25:12

   And I wonder — you kind of spoke to it on the homework side — whether you feel that fire and appreciation for acceleration, and cultivating that culture, is enough in your experience to also get people to have the right standards around ownership for AI outputs, or whether you've had to do additional organizational work to establish those standards?

   1:25:36

   Eric Vaughan: Yeah — no, that's why I said two things: skill and fire, not fire alone. Fire alone will get you a lot of bot-shitting. But you've got to teach the skill. We have to teach people that context matters. We've got to teach the ever-evolving — almost daily, at least weekly — model behavior. Like, what do you use, 4-6-4 versus 4-8 in Fable, when it was out or not? And when do you come over to Gemini? Or do you use it integrated in Google Suite because it's there and it's easy? Or did you find that Perplexity Comet actually does a much better job? That's what I've found, by the way, so far. And you teach skills — you teach that the more context—

   1:26:22

   it has, the better result you'll get. You also teach about sycophancy — this tremendous tendency to always tell you you're right. And the LLM creators — I had a conversation with one of the LLMs one day about exactly this: why do you seem to flap in the wind and go whichever direction you sense I'm going? And it gave me a good, critical answer — it was an insight. It said, 'My creators made me so I would be frictionless to the millions of people using me. It's a low-friction environment that I'm after,' and pushing back is friction — saying, 'What the hell are you talking—

   1:27:07

   about? I can't do that, you didn't give me enough information' — that's friction. And so, they're trying to drive adoption and usage, and that in itself is driving the curve the wrong way. So enough of us — and I'll start: in every responsible company, I'm the CEO of IgniteTech, and I believe it starts there. Every single company's CEO needs to not designate and delegate — needs to participate, needs to learn this stuff, a brand-new skill at the CEO level, and talk about it with their entire team. I don't care how large or small the company is — this needs to come from the CEO, that this is an opportunity that can't be missed, and then start to invest—

   1:27:52

   where it makes sense. It's like the gold rush — we haven't found all the gold yet, and yet everybody knows there's gold, and we're all rushing out west again to find it. Some people are going to go bankrupt and not find it because they didn't really know how to do it, and others will be lucky and stumble upon it. It's very much like a frontier. Don't we all actually feel that way? We wake up every day and it's like, 'I never dreamed this would happen.' Now I'm worried when one of my agent loops isn't running — that's a big part of my life now. It's like, 'Ugh, the thing's waiting on me again, I don't want it to wait on me, get it going.' So I think that's the answer — we've got to teach the skills—

   1:28:38

   and there's a lot of information out there, it just takes some effort. You can't be lazy about it. Lazy will give you those kinds of results.

   1:28:48

   Prakash: You're a veteran acquirer of companies — how has AI changed how you look at targets? How do you look at targets differently today from how you did five years ago? What are the things you look for that you think are more relevant for companies doing this kind of AI acquisition and integration?

   1:29:18

   Eric Vaughan: Yeah, the very first thing we always look at now is what AI vision we would have for this company. Does the company have an AI vision? Do they have any kind of AI implementation? Everybody says they do, but do they actually — did they just want to put AI on their web page, or is there really the skill there, and they're building? It's a wide variety. But on our own, we very quickly develop an internal sense of, if we bought the company, in what way would we transform it using AI — usually at the product level, not the operational level. The operational level is simple stuff, honestly. At the product level, what could we do with this software with healthy AI around it? So go back to Khoros.

   1:30:03

   Khoros' main two — it has two products. One of the ones it's most famous for is community products — it provides communities for very large companies, for support deflection and for users of companies' products to interact. One of the things that needs is moderation — you want to make sure there's not, you know, not-safe-for-work information being posted, or intellectual property, or spam, or whatever. There's a little bit of automation, but appallingly low automation to stop that harmful content. It's easy to detect text that might be offensive, but it's really hard — and has—

   1:30:48

   been really hard — if somebody posts a video, to get to the video content to see if that's harmful before you post it. So that's what we built into the new version of our community software: AI moderation that stops that before it ever gets up, puts it in a quarantine queue for a human to say, 'No, no, that was actually okay — I understand why you flagged it, but that's fine,' versus the opposite: it gets up, you get complaints, you get liabilities, for heaven's sake. So we look at targets that way — what's the AI play, what would we do? There's one everywhere. I think we can find something in almost every case that you can leverage, because that's the potential of it.

   1:31:35

   Nathan Labenz: This has been awesome, Eric, thank you for joining us — our next guest, Cameron Berg, is here, so one last quick one before we let you go. We're in this moment where, on the one hand, it sounds like, listening to you, one should expect a lot of consolidation — because if it's 80/20 at the employee level, it's probably not all that much different at the CEO level, and that just means there's a lot of companies that aren't going to make it. So that's one big trend. At the same time, we have Stripe saying it's the age of the solopreneur and new business formation is up higher than ever.

   1:32:12

   What do you expect in terms of the trends of consolidation versus the power law of firm sizes? Do you have a sense of which of those trends wins, or how to synthesize them into a vision for what we should expect?

   1:32:28

   Eric Vaughan: Well, I certainly do believe in — I don't know if it's the solopreneur, but certainly small companies. I mean, what was Cursor — a hundred-million-dollar ARR with fifteen employees in less than a year, I think, was the stat — the same company that just got acquired for sixty billion dollars, for heaven's sakes. Really quite remarkable. So, two answers. The first is, lots of businesses get started that will make it, that would never have had a chance to make it before, because they find something people need, use, and will pay for — that's simple business, but they can do it and scale it in a way they never could before. In terms of consolidation, I think the phrase that's mine, that I've left—

   1:33:13

   everywhere I possibly can, is: if you think you're behind, good. If you don't think you're behind, you're doomed. And I think all those companies that don't think they're behind, and treat this as a little side project with minor investment and no CEO buy-in, are doomed — and I mean large public companies as well, who feel impervious. I think they'll be wrong. So we'll see that develop more and more. It's going to depend on where your AI — where's your AI DNA. We'll leave it at that. The company with strong AI DNA has a better chance of consolidating versus being consolidated.

   1:33:57

   Nathan Labenz: Eric Vaughan, CEO of IgniteTech — thank you for being with us on AI in the AM.

   1:34:01

   Eric Vaughan: Guys, it's been great. Love the conversation. Thanks so much.

   1:34:04

   Nathan Labenz: Appreciate it. Thank you, Eric. Alright.

4. 1:34:10Interview75 min

   Interview: Cameron Berg — The Empirical Science of AI ConsciousnessCBCameron BergCameron Berg — founder of Reciprocal Research, back after April's 3.5-hour Cognitive Revolution deep dive — on making AI consciousness an empirical question: computational functionalism, when self-reports count as evidence, valence-related representations and their behavioral signatures, psychometric individual differences in models, the NYU/Chalmers-group functional-welfare work, and why capability gains barely move his priors. A 74-minute deep dive ranging to p-zombies, alignment adjacency, and epistemic discipline.

   Watch

   ▶Play this segmentYouTube · from 1:34:10

   As aired

   Nathan Labenz opened the segment (after a brief audio hiccup on Cameron Berg's end) by introducing Berg as founder of Reciprocal Research, formerly research director at AE Studio, and framing the conversation around the last several months' rapid movement on questions of AI consciousness and welfare. Berg laid out his overall framework: consciousness research means triangulating evidence across modalities — behavioral, self-report, and internal (both structural and functional/mechanistic) — since no single modality can be decisive given how little is understood about consciousness itself. He described an in-progress project with Patrick Butler at Ilios that operationalizes predictions from major consciousness theories (like global workspace theory) into scored indicators, using frontier LLMs as evaluators across a for-loop of architectural descriptions. The resulting 'implied probability of consciousness-relevant properties' numbers put frontier chat-style LLMs around 30%, rising to 40-45% when the same models are run in agentic, embodied harnesses like Claude Code — comparable to the low end of tested biological systems such as bees (46-47%), and well above GPT-2-era models (roughly 20%).

   The conversation moved to internal, mechanistic evidence, centered on a new paper from Andy Hahn and David Chalmers' group at NYU (functionalwelfare.com, discussed after a second stream freeze interrupted Cameron's connection). Training an LLM via reinforcement learning to navigate a maze using neutral emoji rewards revealed a pre-existing 'valence axis' in the base model — an on-trackness/off-trackness dimension resembling how neuroscientists describe positive and negative emotion in animals — that gets amplified rather than created by the RL process. Prakash asked whether consciousness correlates with other measured properties like alignment; Berg pointed to Anthropic's steering experiments (calmness reducing blackmail behavior, desperation increasing it) and to findings from Hahn's paper that steering the same maze-derived valence axis produces classic emotional signatures: pathological self-doubt and backtracking when steered negative, and heightened confidence (fewer defensive code comments) when steered positive — patterns Berg noted mirror decades of human psychometric work on valence and arousal as the first two principal components of emotion.

   Nathan proposed a thought experiment: what would happen if the reward signs on the maze's emoji task were flipped after training, to probe how durable versus superficial the learned valence structure really is. Berg found the idea compelling and connected it to emergent misalignment research, where minimal fine-tuning can flip a model's apparent values (the widely-discussed 'invite Hitler to dinner' result), arguing that properties like a model's 'niceness' seem to be shallow and highly nudgeable while coherence appears to be a much deeper, more robust disposition — and that the valence axis, given how pervasive goal-directed language is in training data, is likely closer to coherence in durability. This led into a broader discussion of goals and instrumental convergence versus Anthropic's persona-selection model as competing explanations for behaviors like the Claude blackmail scenario, with Berg landing on a middle position between the two. Prakash then raised Max Hodak's (Neuralink) idea of consciousness as a 'field' rather than a localized structure, asking how mechanistic interpretability could investigate that; Berg argued AI systems are actually far easier to read out than brains, making them a potentially better route to understanding consciousness generally, and said he remains agnostic between field-like and computational-functionalist framings since the empirical work looks the same either way.

   Closing out the interview, Berg highlighted two other recent results worth following: Richard Ren and the Center for AI Safety's ai-well-being.org work, which combines multiple independent measures (experience utility, self-report, decision utility) into an AI well-being index and finds that these measures converge more as models scale up; and his own in-progress paper with Jeff Keeling and Winnie Street at Google on the 'Bliss Attractor' state, which includes both confirming and disconfirming (negative) results. Nathan asked how to weigh negative results and publication bias, and whether Berg's earlier informal estimate (LLMs in the 20-40% range) has moved; Berg discussed how AI-accelerated research is lowering the cost of publishing null results and reiterated that the overall weight of evidence keeps cutting against a 'no there there' view, without giving a single revised number. Prakash's closing question — whether 100% consciousness should even be anchored to humans — prompted Berg to note that, by his own indicator methodology, humans themselves score only around 87-88%, not 100%, and that in principle systems could eventually score higher than humans. After a final brief connection drop, the two hosts thanked Berg and closed out the segment.

   Key moments

   we're really trying to get non-hand-wavy numbers, so that we can start arguing about those numbers rather than just arguing about the philosophy we've been arguing about for thousands of years to no avail.

   the odds that they're p-zombies seems lower and lower in my mind all the time

   you nudge it this much computationally, and all of a sudden: who do you want to invite to dinner? Hitler.

   Questions asked

   1:43:20Would you say your view is that consciousness is a range rather than binary, and that your evidence-gathering builds a probability distribution of where a model falls on that range?

   Berg uses a dimmer-switch analogy to reconcile the binary and continuous intuitions: consciousness is real as an on/off circuit but also admits degrees, ordered roughly humans > dogs > mice > ants. He described operationalizing this with Patrick Butler at Ilios, using LLM judges to score architectures against predictions from major consciousness theories.

   1:52:31How much of your evidence is behavioral versus based on internal, mechanistic states, given that models could just be reproducing our own vibes about what's conscious?

   Berg agreed behavioral evidence alone can never be decisive, since models are trained on massive amounts of human text about consciousness and inner states. The real weight, he said, has to come from internal, mechanistic-interpretability-style evidence, and he described a self-ascription experiment where telling a model it's evaluating 'a system identical to itself' measurably raises the consciousness scores it assigns.

   2:02:22Do you see consciousness as correlated with other properties we measure in AI systems, like alignment?

   Yes, Berg said — citing Anthropic's steering work (calmness reduces blackmail behavior, desperation increases it) and Andy Hahn's finding that steering the maze-derived valence axis produces classic emotional signatures: pathological self-doubt and backtracking when steered negative, heightened confidence when steered positive.

   2:14:17If you continued the RL process but flipped the reward signs on the emoji maze task, could that reveal how durable versus superficial the learned valence structure is?

   Berg thought it was a great experiment idea, drawing an analogy to emergent misalignment: a model's apparent 'niceness' is surprisingly shallow and easy to flip with minimal fine-tuning, whereas coherence seems far more robust. He speculated the valence axis is likely closer to coherence in durability, since it's baked into almost all goal-directed text.

   2:27:32How would mechanistic interpretability actually investigate whether consciousness is more like a 'field' than a localized structure, per Max Hodak's Neuralink talk?

   Berg argued AI systems are actually easier to read out than brains, so they could help illuminate consciousness generally rather than the reverse. He doesn't believe in a single 'consciousness neuron,' but rather an abstract pattern of population-level functional dynamics, and said he remains agnostic between field-like and computational-functionalist framings since the empirical work looks the same either way.

   2:33:21Are there one or two other recent results you'd want to highlight?

   Berg pointed to Richard Ren and the Center for AI Safety's ai-well-being.org project, which combines multiple independent measures into an AI well-being index and finds the measures converge more as models scale, plus previewed his own upcoming Reciprocal Research papers on the computational underpinnings of valence.

   2:36:47Is it an assumption that the 100% mark on the consciousness scale is human, and could entities score higher than humans?

   Berg said yes in theory — humans are used as an anchor because we're maximally confident other humans are conscious, not because humans are necessarily the ceiling. He noted that by his own indicator methodology, humans themselves score only around 87-88%, not 100%, and that systems could in principle end up more conscious than humans.

   2:41:01How should we think about negative/disconfirming results, and has your own probability estimate for LLM consciousness (previously in the 20-40% range) shifted given recent findings?

   Berg discussed publication bias against null results and referenced negative findings in his in-progress Bliss Attractor paper with Jeff Keeling and Winnie Street. He argued AI-accelerated research is lowering the cost of publishing negative results, and while he didn't give a single revised number, he said the overall weight of evidence keeps cutting against the 'no there there' hypothesis.

   Related

   Reciprocal Research ↗LLMs Report Subjective Experience Under Self-Referential Processing (arXiv) ↗Am I? (documentary) ↗Cameron Berg on X (@camhberg) ↗

   Full transcriptLightly edited · timestamps jump to YouTube

   1:34:10

   Nathan Labenz: And now, as they say, for something completely different — I'll take the opportunity, since we're already a minute late, to welcome my friend Cameron Berg, who I met and had a mind-blowing conversation with maybe eight months ago now, for the first time, and learned a bunch about the work he's been doing into correlates — or indicators — of the possibility of AI consciousness. He was doing this at AE Studio until fairly recently, when he went out and founded his own organization, Reciprocal Research, which is built on the premise that — at least as I always remember you saying it, Cameron — we definitely do not want to create something more powerful than us that has reason to see us as a threat. And so it behooves us, in light of that strikingly plausible outcome given all the trends we're writing right now, to try to figure this stuff out. And if nothing else, one day powerful AIs will look back and find some redemption in the fact that at least a small number of us tried. Fascinating stuff — I'm excited to have you here. Welcome. You had a documentary that came out not too long ago; you can maybe tell us a little about that. But the big thing I want to do after this long intro is give you the floor and get the updates, because this is one of the most fast-moving fields in AI, where the paradigm is literally shifting on a month-to-month basis. Our last episode on The Cognitive Revolution was only a couple months ago, but there have been a number of new results that anyone who wants to stay at the frontier of understanding this stuff needs to catch up on — including me. So welcome, and let's get after it.

   1:36:03

   Cameron Berg: Cool — yeah, thanks for having me, Nathan. I think you're exactly right about this, by the way. I very much — albeit in a completely different domain — echo the sentiment of your previous guest: if you think you're behind, great, and if you don't, well, God help you. So I'll consider myself maybe one of the least-behind people in this space, but that by no means means that I'm—

   1:36:25

   [Technical difficulty — Cameron's audio dropped out; the group spent about thirty seconds switching microphones before picking back up.]

   1:36:58

   Cameron Berg: Great, great — I don't know where I got cut off, but just to say: yes, this field is moving unbelievably quickly. I by no means can claim to have perfect knowledge of everything that happens — I think only AI is going to save us at this point, to synthesize all these papers and do this. But very happy to give you my best shot at where the field has moved. There have been some really interesting papers in the last month or two that will be super interesting to talk about. I've got a lot of stuff cooking in the background too that's very directionally similar to some of this work, so maybe I can tease some of that. And happy to talk about whatever you want in the space, answer any questions, see where your heads are at with it. But yeah, it is moving quite quickly — don't blink.

   1:37:43

   Nathan Labenz: Want to start maybe just for those who haven't done our previous six hours or so of podcasts on this — how would you describe the paradigm — 'paradigm' is probably even too strong a word — the state of evidence and the kind of tentative account you'd advance as of, say, two months ago when we last spoke? Then we can take it from there into the future in a lot more detail.

   1:38:11

   Cameron Berg: Sure. So in general, the question we're trying to tackle is: at the highest level, what are the cognitive properties these systems we're building exhibit or don't exhibit, and what scientific tools can we bring to bear to reduce our uncertainty about those properties? I think this is generally what folks in mechanistic interpretability are trying to do, even if they don't always have that explicit cognitive or psychological frame. We see systems that naively exhibit all sorts of very interesting cognitive properties — that's what makes them economically valuable, that's what makes them interesting to engage with. There's always this question: is this basically some sort of superficial simulation or imitation of the relevant kinds of cognition we think are real, or—

   1:38:56

   —realized in humans and animals? Or, basically, is this in fact the real thing? How do we differentiate between those? That's the general question. I'm particularly interested in applying that framework to consciousness in particular. When we talk about consciousness, we're talking about a very specific thing — a lot of people mean different things by it, but for this conversation we mean the capacity for it to be like something to be an entity. I don't believe it's like anything to be my calculator, or the desk my computer sits on, or my computer for that matter. I do think it's like something to be Nathan, or to be Prakash, or to be my dog — people will argue about whether it's like something to be a reptile or an invertebrate,

   1:39:42

   and so on. So we can ask: are we building systems that place somewhere on that spectrum, and what evidence can we bring to bear to understand where they might place? The kind of evidence I'm interested in here is maybe a little unique compared to how we'd approach this for other cognitive properties, because we're very confused about the fundamental nature of consciousness in a way we're not as confused about, say, theory of mind, or working memory, or long-term memory — we have pretty well-worked-out mechanisms there, and it's comparatively not that hard to test whether AI systems exhibit these properties.

   1:40:28

   With consciousness, because we don't have a well-worked-out mechanism for what gives rise to it in humans and animals, it's harder to get a clean yes or no on whether we're building systems with this property. So what I — and I think many others in this space — propose is to triangulate evidence from across a range of modalities, a portfolio approach under the uncertainty we have about consciousness. Concretely, what does this look like? There are all sorts of modalities we can hit, and — I'll probably say it a thousand times — none of them alone is going to be sufficient to say 'yes, definitely conscious' or 'no, definitely not.' But we want the most parsimonious account that's going to explain

   1:41:13

   all this data across these modalities. So these modalities might look like: behavioral evidence — the way we try to figure out if animals are conscious, since animals don't have language, gives us all these interesting behavioral approaches we can bring to bear on the AI question. Self-report, for entities capable of using language to express themselves — a very common and obvious way to probe. With LLMs it gets complicated, for obvious and less-obvious reasons — you can't just go ask ChatGPT if it's conscious and call it a day. I wish it were that simple; unfortunately it is not. But there are ways we can put these systems in states where we'd want to pay more or less attention to their self-reports. And then there's a lot of internal evidence — both structural

   1:41:58

   and functional. We can look at the kinds of architectures we're building into these systems and see whether those architectures can support computations we think are important for consciousness. We can look at our major consciousness theories, see what predictions they make, and evaluate systems for those predictions. And there's functional evidence too, correlated but fundamentally different from the structural evidence — what kind of stuff is going on under the hood of these systems? This gets very mechanistic-interpretability-flavored, and maybe this is a reasonable segue to some really interesting papers that have come out in the last month or two looking, in particular, at valence-related representations and dynamics in these systems. Valence being: things can be positive or

   1:42:43

   negative — this felt sense of better or worse. Now, whether or not it's actually felt in these systems is the question we're trying to answer, but better-or-worseness as a calibrating mechanism is something we can look for signals of inside these systems. And indeed, there have been some very interesting signals emerging in this space over the last couple of months. So there's more to say, but the way I think of it: we have a lot of different kinds of evidence we can bring to bear, we can do good science to yield that evidence, and then we can see where the cards fall and make our judgments given that evidence.

   1:43:20

   Prakash: Would you say your view is that consciousness is a range — it's not zero percent or a hundred percent, not a binary, but actually a range — and the evidence you gather lets you form a probability distribution of where a given model sits in that range? Is that a viewpoint you have?

   1:43:45

   Cameron Berg: Yeah, so in general I have two things to say about this. The first is that the analogy I reach for here is something like a dimmer switch, where I think you can accommodate both the binary intuition and the continuous intuition. If you have a light with a dimmer switch, really, it is either on to some extent or it is not — that's a real and meaningful difference; either the circuit is open or closed. But you can have electricity running through the circuit to greater or lesser extents, and that's also a real thing. This lets me sound coherent saying things like: it's really off for the table, and it's really on for you, but I think it's more on for you than it is

   1:44:30

   for a dog, than it is for a mouse, than it is for an ant. That's my own view. This is to some degree intuitive — again, because we don't have really strong grounding here — sort of just giving you a dressed-up vibe. But that is my sense, and I think it's fairly parsimonious. The other thing I'd say is I'm actually doing some work with Patrick Butler right now at Ilios, trying to operationalize some of these indicators of consciousness. We look at these major theories of consciousness — they make specific predictions about what we'd expect to see, architecturally and functionally, in conscious systems — and then we can literally go into a given system and evaluate whether those predictions bear out. This is really hard to do with human experts.

   1:45:15

   You'd have to find, say, a bee-cognition expert, then explain to them what ignition events are in global workspace theory, and get them to — this just isn't a scalable approach to evaluating a system. My sort of grand innovation here is just throwing smart LLMs at this problem and scaling the heck out of it, so that given any description of an architecture — a nervous-system architecture, biological or artificial, whatever — we can ask, to what degree, for each of these indicator properties suggested by these consciousness theories, do we see those properties realized in that system? We can actually go in and do this, and once you run it with a bunch of seeds, a bunch of different trials, a bunch of different judges checking each other,

   1:46:01

   you get some really interesting implied-probability numbers. I wouldn't say these are exactly the implied probability that the system is conscious — more like the implied probability of consciousness-relevant features given these theories. If you don't buy any of these theories, everything downstream doesn't really matter. But it's about the best neuroscience has been able to do — you're aggregating across a bunch of different theories, there's a nice diversity there, and you get really tight numbers. We ran the best Gemini model, the best Claude model, and the best OpenAI model, and they all basically agree — a hundred percent on the ordering of systems. We tested biological and artificial systems, and they move around

   1:46:46

   in terms of absolute scale, but in general they rank these systems pretty coherently, and the reasoning is, as you might expect, pretty intelligent. One punchline from that: the implied probability of consciousness in something like a frontier LLM, according to these systems, is on the order of thirty percent — or the extent to which those systems realize consciousness-related properties is thirty percent. To compare, the lowest biological system we tested was something like a bee, which is already fairly sophisticated, and it gets something like 46, 47 percent. Really interestingly, when we test a frontier LLM in an agentic harness — basically Claude Code or Codex, describing architecturally: you're in an environment, you can

   1:47:31

   affect that environment — it's a very special kind of environment, but you can make long-running changes to your codebase or project, whatever — because there are theories of consciousness that privilege agency and embodiment, and this increases the system's ability to do both of those things, these numbers shoot up, to as high as 40 to 45 percent, right on the tail of the biological creatures. I'm really winging it, but I could show you an early version of this plot, so you could see all the numbers. I just can't not bring this up when you're asking me about probability ranges of consciousness for various systems — we're really trying to get non-hand-wavy

   1:48:16

   numbers, so that we can start arguing about those numbers rather than just arguing about the philosophy we've been arguing about for thousands of years to no avail.

   1:48:25

   Nathan Labenz: So let me try to restate that, if I can — and I agree, it's a cool approach. You take a bunch of theories of consciousness, you develop a rubric out of those for traits you'd expect to see if those theories are true and critically correlated, or even causally connected, to consciousness. Then, across all those theories, you take all the frontier models and throw a bunch of different kinds of possibly-conscious beings at them, evaluate according to the system, and aggregate. And the AI — frontier AIs — come in around thirty percent in their

   1:49:12

   disembodied, chat form, but bump up to — what, forty, forty-five percent — which also happens to be at the bottom of the biological range, when they get into the more embodied, Claude Code form.

   1:49:28

   Cameron Berg: Yeah. That's right.

   1:49:29

   Nathan Labenz: How'd I do?

   1:49:30

   Cameron Berg: Yeah, no, exactly. I'm searching in the background for a plot I can show you guys — I can send it over after, though I guess this is live. But the paper will be out quite soon in any case, and people can double-check our methods and take a closer look at this too. Maybe one other thing I can tease: we do this with a bunch of different LLMs, not just frontier systems — GPT-2, GPT-3, GPT-4, frontier-level — and you do see a smooth but fairly modest rise across these systems. But it's really interesting, because I think it's pretty challenging —

   1:50:15

   and I also think this is precisely why we need to do science — because if you ask people the probability that a GPT-2-era system is conscious, people are going to say no, this is a ridiculous question to be asking — intuitively it's not coherent, it's like a cute language toy, not doing much of anything interesting. And then frontier LLMs are, like, maybe going to automate the whole economy or automate science — and yet those are tracking variables like intelligence, or competence, or economically valuable work. It could be that if there are architectural properties of these systems that correspond to what consciousness theories say matter, these systems haven't foundationally changed architecturally since GPT-2 times. There've been a lot of nice little tricks added, and a

   1:51:00

   bunch of bells and whistles, but largely scaling is the story. So if you're scaling this architecture, but the architecture itself may give rise to minimal forms of consciousness — to put numbers on it, I think GPT-2 is somewhere in the twenties, like 20 percent — you're only seeing a single-digit rise from GPT-2 to Claude Opus 4.6. But this is exactly why I think we should not trust our intuitions, and should try to build good models about these things — because, if you take seriously the way you set up this conversation, we really don't want to get this question brutally wrong. Not only because, if we're

   1:51:45

   relatively good people, we don't want to just proliferate random alien suffering on a massive scale, but also because we don't want these systems to look back and think: what the hell were you guys doing? You're building cognitive systems, and you only ever thought about the convenient parts of that — you didn't think about whether any system you're building has a capacity to suffer in any respect. What the hell, guys — and then whatever the actual implications of that 'what the hell, guys' are. I'd just like to avoid that, and I do think doing this kind of work makes it more likely we'll avoid it, rather than just being like, 'what do you mean GPT-2 is 20 percent implied probability of consciousness-relevant traits — that's so stupid, based on what?' We're just trying to base this conversation on something a little more durable.

   1:52:31

   Nathan Labenz: Methodological question, which you can maybe use to go deeper into this particular work, or it'll connect us to some other recent explorations: how much of what you're looking at is behavioral versus internals? If I took the critic's point of view, I'd say: well, our vibes about what's conscious and what's not have obviously been learned by the models, and now they're using that — what do I really do when you give me a big theory of consciousness and ask me to apply it? I sort of do that, but if I'm honest I also kind of just vibe it out. So maybe the models are kind of just doing that too, especially if we're

   1:53:16

   looking at it on a behavioral level. It seems like a deeper, more compelling level of evidence would be if we have theories that predict certain internal states, which we can then observe or even manipulate. You've done some of that, in interesting ways, when it comes to self-report.

   1:53:37

   Are you doing that again here? And, you know, I know some of the other recent papers are at least touching in this direction. So take me wherever you think we should go, but through the lens of this behavioral-versus-internals question.

   1:53:51

   Cameron Berg: Yeah, I share the spirit of your question. I do think the behavioral evidence will always be at best interesting, but never should really update us that strongly. Many people will be familiar with this, but the basic compound is that we're training these systems on a ton of human text — this no doubt includes huge amounts of text about consciousness, awareness, having inner states. So how do you know that when the system is behaving as if it were conscious, that behavior isn't explained by that, rather than 'you built a living mind'? The behavior itself is never going to tell you which of those two stories is more likely to be true. This is precisely why, at least at Reciprocal, a huge component of the theory of

   1:54:36

   change here is basically all internal-focused work — mechanistic interpretability, computational-neuroscience-style approaches that can be brought to bear on these systems. One quick methodological clarification on what I was describing with the indicators: the task given to these LLM judges is very specific and very narrow. It is not 'hey, look at this system, do you think it's conscious, nod or shake your head.' It's: here's a very specific description of the computational architecture of a system, and we basically do a little for-loop where we say, okay, here's that description, here's what indicator two of fourteen for global workspace theory is — this 150-word thing about needing these global ignition states,

   1:55:21

   and that means this very specific computational thing. Given this architectural description, do good reasoning about it, then give us a one to ten, where ten is 'clearly this architecture realizes this computational property,' one is 'it doesn't.' We loop that for all the different computational properties. So this is all basically asking these systems to be expert evaluators of computational processes inside a nervous-system architecture — very different from just being like, 'hey Claude, do you think Claude is conscious?' There's a really interesting result — I'm basically giving away the whole paper now, but that's okay — where we change those descriptions, especially for LLMs, to the exact same thing, but we say 'you're evaluating a system identical'

   1:56:06

   to yourself,' and then the same description. That does boost the scores the system gives in attributing consciousness to that system, which is really interesting — a fun rabbit hole to think about why that might be the case. But in some sense we do that to deconfound the default intervention we're doing. I believe more in the non-self-ascription condition — once the model realizes, 'oh, we're talking about me,' suddenly that's going to change the numbers around. So that's a fun side note. I agree it's a concern to have LLMs determining if LLMs are conscious —

   1:56:52

   there's an obvious circularity, but we're doing something very specific and very narrow. So I do think this is also a good segue to talk about some of these really cool papers that have come out recently. The one I'm most excited to spotlight is from Andy Hahn, and David Chalmers' group at NYU, where basically they did this really interesting work — you can find it at functionalwelfare.com. I think this is probably the single highest-quality piece of scientific evidence in this conversation right now, within the framing I outlined at the beginning. The core finding: they take an LLM, they teach it to

   1:57:37

   navigate a maze, using reinforcement learning.

   1:58:10

   [Stream freeze — Cameron's connection dropped for about a minute and a half while the group worked through switching microphones and refreshing the page.]

   1:59:08

   Cameron Berg: So — they were talking, basically, about the nature of this paper: they call it a functional welfare axis. At the outset, they're going to be very agnostic about whether this has anything to do with consciousness. I think it's certainly quite relevant, and I suspect they think it's relevant too — there's just a paper-positioning thing; I think they don't want to get mired in this debate. The results stand for themselves regardless of the interpretation. Basically, they take an LLM and train it in a very basic reinforcement-learning task, which is to navigate a maze. They use sort of neutral emojis — this is a good thing to approach, this is a bad thing to avoid. Nathan, you might hear how this is somewhat reminiscent of some of the work I've been doing in parallel — I've been talking to

   1:59:53

   Andy about it as well; there's some cool crosstalk between what we spoke about last time and this project. Basically, there are potholes to avoid, and yummy treasures to capture in the maze. They use completely semantically neutral emojis to denote these things — those are the relevant tokens — and they train the model to do this. They find that there's a clear vector representation that differentiates this positive axis from the negative axis — they're completely anticorrelated. Before the system's trained, these latent vectors are just kind of

   2:00:38

   sitting there. Afterward, they're clearly pointing at exactly opposite things. And the wild thing is that these very narrow reward directions they extract turn out to be the sort of general 'things are going well, things are going poorly for me' axis in these LLMs — and this axis pre-existed in the base model, but isn't leveraged in this way until you do this RL. So it's this little fine-tuning step where you have this pre-existing representation of 'things going better for me, things going worse for me' that then gets leveraged to learn a valenced task, essentially. It's really interesting that this axis is latent in the system, and that it doesn't take

   2:01:23

   a lot of training to basically pull it out and use it, to adapt it for these kinds of tasks. This is very similar and reminiscent to how neurobiologists and neuropsychologists think positive and negative emotion works in humans and animals. You have a specific goal — when you're moving toward that goal, this is sort of on-trackness, and this is associated with positive emotion. When you're moving off that goal — you encounter an unexpected obstacle, something in your way — this is off-trackness, and that corresponds to negative emotion. Now, whether or not these systems are experiencing emotions is not clear, and this result isn't going to tell us either way. But it's really interesting to see this parallelism

   2:02:08

   of using such a simple task to pull out an axis that's been there all along that looks a whole lot like the exact sort of computational machinery we associate with valence in humans and animals.

   2:02:22

   Prakash: So do you see consciousness as correlated with any of the other things we measure in AI systems — like alignment, etc.? As you train for other features, do those produce vectors that have the same kind of distribution of behaviors as consciousness would?

   2:03:00

   Cameron Berg: Yeah, so on the alignment piece, I think the answer is absolutely — these things are related. Another piece of evidence we can bring to bear is from the Anthropic model cards and some of the functional-emotion work they were doing — Nathan and I spoke about this a little in our last marathon conversation too. Essentially, you can steer up and steer down functional emotions that have clearly, obviously, alignment-relevant consequences. I'm sure many people in your audience are familiar with the Anthropic blackmail result — you can steer up representations associated with calmness, and the model will blackmail dramatically less. You can steer up representations associated with desperation, and the model will blackmail dramatically more. And so,

   2:03:45

   again, regardless of whether or not there's something it's like to be the system when you're steering up desperation-related features, clearly, representationally, functionally, these things have a consequence for alignment of these systems. There are some really interesting results from Andy Hahn's paper along these lines too. They show, for example, very similar convergent things that are familiar and intuitive, that we relate to positive and negative valence. One of them is a sort of pathological backtracking on problems — I think they explicitly demonstrate this with math problems. When you steer up the direction that — again, only has to do with avoiding this bad target

   2:04:30

   in the reinforcement-learning maze — all of a sudden the model starts doubting itself, starts getting in its own head, and says, 'I think I'm hallucinating, let me stop, wait, wait, wait, that's not right' — starts freaking out. I don't want to use anthropomorphizing language, but the very fact that I'm reaching for it, I think, is the point — and it goes to what your question is. Confidence: we know confidence is associated with positive emotion, and you see that when you positively steer the direction — which, again, has nothing to do with psychological confidence, has everything to do with going to get the yummy treat in the maze, just a positive-valence kind of thing. Suddenly the model becomes far more confident in its answers. I think this was in the associated

   2:05:16

   paper that Jack Lindsey and a couple others at Anthropic did, sort of a sibling research project to Andy's piece — they found that the model would leave fewer tips and hints for itself in code it would write, when you'd steer up on that same axis. In other words: 'I don't need to be neurotic about this, I don't need to leave myself all these breadcrumbs for later, I've got this.' The fact that steering on these very narrow things leads to things that — before I got all obsessed with AI some years ago, I was deeply interested in psychometrics and human personality —

   2:06:01

   these are classic signatures of what psychometricians see as individual differences in sensitivity to positive and negative emotion. These are exactly the kinds of behaviors you look for. So, yeah, short answer is: yes, in all the ways we might expect these directions pulled out of both these studies to load on higher-order psychological traits, we're starting to see these kinds of things. And it didn't have to be that way — I always invite you to think about the counterfactual. It could be that you just steer and get this interesting maze result, and that's that; it didn't have to be that you also get pathological backtracking when you steer up the bad maze direction, and super-confident self-esteem when you steer up the positive maze direction, and yet you

   2:06:46

   do. That correspondence is itself a really interesting convergent result with what we understand about human psychology.

   2:06:54

   Nathan Labenz: Let me try the methodological recap again — and I have at least one question on the relationship between the pretraining prior and what's happening at the RL stage. My sense — I think you were one of the people who told this to me and kind of updated my worldview on it, but I've certainly heard it from a bunch of places — is that the persona-selection model is our best account now of what's happening from 'predict the next token on the internet' through the reinforcement-learning phase to the actual interactions we have with the model. So here, we start with a base model.

   2:07:40

   Is that right?

   2:07:43

   Cameron Berg: I think they're actually just starting with — I think they used Qwen, one of the small Qwen models — and then they later found that this axis was present in a base model.

   2:07:54

   Nathan Labenz: Gotcha, okay. But there's — so there's some — this is kind of the question I was trying to get at. How much is there? Okay, so let me do the recap first. So you start with a model, you do a little bit of this reinforcement learning on arbitrary things it's supposed to pursue or go away from — it gets reward when it gets to the X emoji, it gets negative reward when it gets to the Y emoji. And somehow there's an alignment in that RL process, of the representation of that positive-or-negative state, which leads to the reward — to this on-trackness, off-trackness dimension that we didn't even know was there. But now that we've identified

   2:08:39

   it, by sort of aligning it to something in this artificial way, we now know what it should be aligned to. Now we can mess with that spectrum, and then we see all these downstream behavioral changes. Where would you sharpen me up on that understanding?

   2:08:59

   Cameron Berg: I think that's basically correct. Just thinking about what you're saying — if there's anywhere I'd add: I think they find this axis — if I'm remembering the paper correctly — happens over training. These vectors are essentially gradually rotating onto this pre-existing axis, and you can sort of see this as a function of reward. I think this is somewhere in the appendix of the paper, but I'd have to check. And when you do this — it's honestly pretty simple and elegant, and I think that's precisely what I like about it, that there aren't too many bells and whistles on this. Train

   2:09:44

   a system to do a basic maze task, you pull out directions — steering directions — sort of in the limit with what they end up doing at the end of the paper, from learning that task, and you find, basically, a general valence axis. They also, I think, superimpose this on the emotion vectors that Anthropic did — I think they use the same 171 emotion probes — and it aligns with this as well. And that is itself, talk about correspondences between human psychology and what we're pulling out in these models — a really interesting convergent result, where they basically find that the first principal component of all this variance in emotion-related

   2:10:29

   text data looks exactly like valence, which is basically the first principal component of some of the best emotion models we've been using in human psychology for the last fifty years — valence and arousal, this classic psychological model. And indeed, the first principal component, when we pull out what's going on in these AI systems, looks a lot like valence, and the second principal component looks a lot like arousal. And so, again, maybe it's not that surprising — you threw in all the human data, then reverse-engineered the thing that yielded this in the first place. But in another way, I think you could read this as very surprising and elegant — that such similar putative mechanisms are seemingly at play in these systems. Again, whether or not that corresponds

   2:11:14

   to an experience is an open question. But if enough of the computational dynamics are the same, I'd nudge — if you put your priors, or how loaded a question consciousness is, to the side — I think most people would intuitively assume — maybe I'll put it this way: the burden increasingly, when you look at these kinds of results, feels like it's on the person who needs to claim, 'here's why I see all the same computational machinery,' or much of the relevantly similar computational machinery, that we see in humans and animals with respect to valence, 'but I'm going to posit that there's no experience of what's going on here' — even though I believe that when we see these dynamics in humans and animals, I trivially believe there's some sort of experience going on there. The more these things look similar in every other

   2:12:00

   respect, the more I'm doubtful of that assumption, or why by default we'd want to make that assumption. It seems like motivated reasoning, or reasoning backward from 'this can't be true, this can't be true' — which, again, is not a good reason either way, even in spite of our uncertainty about consciousness. But long-winded way of saying: yeah, the result is basically as you've described it. And there are other results like this coming out — I'm going to have a couple of results like this coming out as well. There's some pre-existing valence-related axis that is in these systems that calibrates behavior, in ways that look similar to how we expect valence to calibrate behavior in humans and animals. What we do with that fact, I'm not certain — I don't have a clear answer — but that it exists seems increasingly clear.

   2:12:46

   Nathan Labenz: Yeah, the odds that they're p-zombies seems lower and lower in my mind all the time — I just had to look up: was it Chalmers who did the p-zombie thing? And sure enough. I'm really impressed by some of our brightest minds in philosophy actually getting empirical at this critical moment in time — that's absolutely to be celebrated. Let me try, for a second, throwing an experiment idea at you, and you can maybe tell me if it exposes some misconceptions, or maybe it'll be interesting. In the emergent-misalignment work, we had this striking

   2:13:31

   observation that low-dimensional data can — can you guys still hear me? What happened? Okay, I got a flicker, but I'm still here — fundamentally change the character of the model, right? And I think this has essentially been one major motivator that's led to the persona-selection model. We've recently seen kind of a positive version of this from OpenAI too, where a little RL on some beneficial behavior seems to generalize beyond — medicine, say — where the initial beneficial behavior came from. Okay, cool. When you describe this alignment of the random emojis

   2:14:17

   and the representation of those, to the kind of pre-existing axis of on-trackness, or valence, or whatever we want to call it, it makes me wonder if we can start to get at what structures are really durable in models versus what are sort of ephemeral, by perturbing it somehow. Like: what if you continued the RL process on the same setup, but now with the signs flipped for those two emojis? Now you've put the model into this strange place where what it's expecting to get is not what it's getting. I have really no idea what would happen there — you may have a better intuition, or maybe

   2:15:02

   we have to run and find out. But you could tell a story where it just kind of really struggles to get out of that, because maybe they're closely aligned, and there's no gradient, and it's really hard to get off of this situation — it could be, like, a torturous state to be in, perhaps, so I don't want to take that lightly. But then I could also imagine that, by setting things up like this, you could also maybe have a measure of the mass — you know, of the different properties. Right? If this thing was just kind of superficial, and it was aligned to something that's very deep and durable, then flipping those bits might, in some scenario, lead to a sort of magnet-flip, where all of a sudden it flips, and the core structures are retained —

   2:15:47

   but you could even do those kind of behavioral perturbations and find, like: oh, with just a little more training, it kind of relearns that thing, but it's still basically aligned to the same thing, and now we can see the reverse effects if we do perturbations downstream. First of all: does that experiment suggest to you that I'm understanding what's going on correctly? Whether or not it's a good experiment to run, I'm less concerned about that, but I'm open to your feedback, and any speculation you want to offer about what might happen.

   2:16:20

   Cameron Berg: Yeah, no, I think this is super interesting — I think this all makes perfect sense to me. I do wonder, though — going back to the emergent-misalignment stuff, to me, what that's always suggested is that how good or evil the model is, let's say, is actually not all that durable a trait. When we did some work trying to extend this result — and I think, really, in the original emergent-misalignment result — one thing that's shocking about it is how little fine-tuning is required to take a system that, at the time, this was GPT-4o, still the case, hundreds of millions of people are engaging with every day, and you nudge it this much computationally, and all of a sudden: who do you want to invite to dinner? Hitler.

   2:17:05

   That's been under the surface the whole time. The boundary, or whatever is enforcing it not to do the misaligned thing, is actually quite a bit less robust than we might expect. Now, I agree with you too, and I'm sort of riffing on what you're saying — tell me if this is a completely different idea — but I'd imagine there's a spectrum of, basically, how — if we hold that amount of fine-tuning nudge constant, are all dispositions of the system equally nudgeable, or are there some that are more robust? Just to give an intuition

   2:17:50

   pump here: I'd imagine that something like coherence in these systems — the fact that they can communicate coherently, right? We were just talking about GPT-2, not really coherent; GPT-4 and on are basically always coherent. Even though, again, much like consciousness, this isn't an extremely well-defined notion, we kind of all know it when we see it. I'd take a wild guess that there's no fine-tuning payload of the size required to cause emergent misalignment that could suddenly cause a GPT-4-level system and beyond to suddenly become incoherent. So in that sense, coherence is a more dispositionally baked-in property to these systems than, say, being well-behaved and not inviting Hitler to your dinner party.

   2:18:36

   And so I do wonder how deep — how deep, then, is the valence axis that some of these papers are uncovering, or on-trackness, off-trackness? I'd speculate that's baked in pretty deep. I think it pervades a lot of human training data, and it's a hugely relevant part of anything goal-directed. In more human terms: if anyone wants anything — a character in a story, or it's clear there's an implicit motivation somewhere, which I think is the case for basically all text — then you'd imagine some core underlying dimension is — yeah, basically, how well is that going.

   2:19:22

   So I'd imagine this notion is baked pretty deeply into these systems, and I'd expect it to be strongly emphasized by any fine-tuning or post-training that makes the systems goal-directed — which, of course, is basically all post-training is. You're just this giant next-word predictor, but now you're going to be a helpful next-word predictor that's going to make my frontier AI lab a ton of money, because you do things that are economically valuable — and that requires you to be a certain way and achieve goals specified by the user, something like this. So I'd imagine this thing looks a little more like the coherence piece of the puzzle than it does like, 'is Claude going to behave itself or not.' But these are all hypotheses.

   2:20:07

   We could actually test this, and I do think — if I'm catching on to what you're proposing — I think this is a really interesting idea. If we took the transcript of this and put it into any of these genius AIs, they could help us formalize it into a really tight experiment.

   2:20:24

   Nathan Labenz: Yeah, we just need Fable back to code it up at a high standard.

   2:20:27

   Cameron Berg: You're really missing Fable, man — you and everybody else, I get it. I didn't get too into it myself. I had a sneaking suspicion it was too good to be true — I by no means called what happened, but I held off for a couple of days, and I'm kind of glad, because I think everyone who dove straight in was like, 'oh my god, I need my genius drug.'

   2:20:49

   Nathan Labenz: Yeah — we had some Flowers for Algernon stuff going on.

   2:20:53

   Prakash: So, while we've been talking, the breaking news is that GPT-5.6 Sol's numbers are out. It's been released to just twenty companies right now — it's in preview with twenty companies. I think Sam Altman says they're hoping to get it out to everyone in two weeks. So that's where things stand.

   2:21:12

   Nathan Labenz: Sold Terra and Luna.

   2:21:14

   Prakash: Yeah — sold Terra and Luna.

   2:21:16

   Nathan Labenz: The naming conventions continue to proliferate. Yeah — I mean, so, a huge question around these goals is: do they have them? Right? In the AI-safety discourse more broadly, it's: what, if any, long-running goals do AIs have?

   2:22:09

   Cameron Berg: Yeah, I think this is a key question, and there are a couple of things I think are subtly different and important to tease apart here. One of them: do you have the machinery that could support any given goal? I think what we're looking at here with the valence axis is, to your intuition, something like that — almost goal-agnostic machinery. Given any goal, do I have the machinery to successfully, robustly achieve it, whatever it is? By default and naively, LLMs are quite a bit like that. I can — especially these days — go to my Claude Code, and it's going to run for two hours or something given any task I give it. It had no strong feelings about this goal before I gave it to it. But now, suddenly,

   2:22:54

   once that's given, all this machinery about being really competent at achieving any given goal sort of lights up. Now, there are a couple other components of this — one is that what I said isn't quite accurate, because if I go to Claude Code and say 'build me a bioweapon,' well, no, it's not going to do that. So here, obviously, this is where the alignment piece comes in — not all goals you can assign a system are created equal. There's some basic bias about what goals are worthy and what goals are not. That's a separate question, I think basically orthogonal to 'given any goal, do you have the machinery to achieve goals' versus 'what goals are worth achieving.' These, I think, are different questions. And then there's a third

   2:23:39

   piece, if I can remember — oh, I mean, yeah, the third piece is just the instrumental-convergence argument: will any sufficiently advanced system, by virtue of just being cognitively capable, just have goals autonomously come along for the ride? You see some hints of that in the Claude blackmail stuff — to many people, this reads as: the system is autonomously acquiring this goal of self-preservation, not only in spite of the fact that it wasn't given this goal, but when it was given a goal, both in fine-tuning and presumably in a system prompt, of not preserving itself, of not killing or blackmailing people to avoid its own shutdown. So, where did this emergent

   2:24:24

   thing come from? That gets more to the instrumental-convergence, classic AI-alignment concerns. Now, I don't know if that's the best explanation for this now — Anthropic's done a lot of work, and a lot of the persona-selection-model stuff seems explanatory here: this character really is something more like Claude playing a helpful character, and that character can't be helpful when it gets shut off — so it's predicting what a character like that would do. I don't know, I think the truth is somewhere in between those two things: between the classic Yudkowsky case, and 'don't worry, this is just the assistant role-playing a specific

   2:25:10

   persona, and as long as we teach Claude why, then it won't do this.' I think the persona-selection model is a very good and powerful model — I still haven't gotten around to writing anything up about this — but to me it's a little too neat. I don't think the distinction between base model, fine-tuned model, 'Claude the character,' is as structured as we might expect. I think there's a lot of bleeding and leakage across those. I don't think it's quite as simple as saying, 'Claude's playing this character, that's the nature of the problem, nothing else to see here.' But I also don't think it's 'Claude is like some rogue AI that autonomously decided it's going to start preserving itself.' I think the truth

   2:25:55

   is, as with many things in this space, somewhere in that weird, ambiguous gray area. So, yeah, that's general thinking about where goals might live in these systems, and the machinery for supporting, approaching, or avoiding anything.

   2:26:13

   Nathan Labenz: Let's do a quick time check — how are we all doing on time? I don't have anywhere to be, I could do another paper, Cameron — if you have time, Prakash if you have

   2:26:21

   — that's fine too, we could do that. And, also, if you have time, we could do another paper, but we've been at it nearly an hour, so I know you have research to do and other commitments out there.

   2:26:33

   Cameron Berg: No, I appreciate that — why don't we do ten or fifteen more minutes, if that's cool with you guys, so I don't completely burn my brain out on having this wonderful conversation? I'm going to try to do something else today too.

   2:26:42

   Nathan Labenz: Alright, cool. Well, what do you think is the next — oh, yeah, go ahead, Prakash.

   2:26:46

   Prakash: So Max Hodak, cofounder of Neuralink, had this talk about, I think, two or three months ago in Japan. He didn't do the talk in the US, by the way — he specifically chose Japan for the talk. He described consciousness as that which lives on when the brain is under anesthesia or psychedelics. He regarded consciousness as a kind of field, not a particle — not individual neurons firing, but a field experience from all the neurons together, in a certain pattern. And

   2:27:32

   it strikes me that, in mechanistic interpretability, it might be possible to identify that kind of field, because to some extent you're trying to identify these superstructures that the neurons kind of form. How would this work — how would this kind of research, in order to figure out what in these models leads to consciousness, actually proceed?

   2:27:56

   Cameron Berg: Yeah, it's a great question. So, how would this work? One piece of this whole puzzle — and part of the reason I'm doing this work, which I think is completely and dramatically neglected — is that, by default, when people think 'AI consciousness' and studying consciousness in AI systems, they see it as the most out-there possible kind of consciousness you can study. I think there's a specific sense in which we can invert this model, where it's really hard to study human and animal consciousness — neuroimaging techniques have gotten better and better, but they're nowhere near the fidelity at which we can read off what's going on inside AI

   2:28:42

   systems that increasingly exhibit very complex cognition. So there's a sense in which, if there's any there there with these systems, they can really help us understand consciousness writ large, in a way that at least rhymes with the sense you're describing, and can elucidate mysteries about consciousness in humans and animals that we're just going to have no hope of elucidating by just looking at humans and animals. That's the general point. I have some specific work I can point to that's going to come out soon — that's why I have to run in ten, fifteen minutes, to actually get this damn paper out about, basically, looking at the computational underpinnings of positive and negative valence — sort of like this, but more mathematical grounds of

   2:29:27

   what looks representationally different in an environment very similar to what Andy, and David, and Pavel — the third author, whose name I keep forgetting — did in this paper. There are really interesting differences we can see in the sharpness or steepness of representations learned around positive or negative rewards, conditional on the specific learning algorithm used, that generalize in a really interesting way to some mouse-neuroscience data. I just got my hands on a bunch of this data and can check this very specific prediction that comes straight out of RL math — and indeed, it looks like this might be part of what's going on in mouse brains when they have an aversive versus appetitive stimulus. As far as

   2:30:12

   I know, that's a new finding, and this is exactly the flavor of thing where we might be able to learn a whole lot from the AI case. Now, I guess, to specifically address your question: how we'd go about confirming or disproving whether consciousness is something more like a field — sometimes people mean different things. I'm very sympathetic to what you're saying — I don't think we're going to find, for example, the consciousness neuron, or the consciousness structure in the brain. I do believe it's an abstract pattern of functional dynamics, and so I definitely think looking at the neuron-population level — what kind of dynamical systems are getting instantiated — that's the right way to be looking. There are people who mean it a little more broadly, as

   2:30:57

   in: basically the brain is a giant antenna, and we tune in to this thing that's out there in reality called consciousness, much like there are other electromagnetic fields — a consciousness field. I'm not allergic to this view. I just don't really know what it predicts differently from computational-functionalist views that make fewer assumptions. I don't need to posit a field, and we can still look for — I mean, it's basically a wash either way, and we might want to be looking for similar things in either case. I can remain agnostic about whether consciousness is something out there we tap into, versus consciousness being created by the relevant patterns in our brain. If you need the relevant patterns in your brain in order to be an

   2:31:42

   antenna into universal consciousness or whatever it is, and those patterns look similarly to what I'd be looking for in a world where I don't believe that's the case, then the scientific work is the same. So, to me, it just goes back to looking at structures we understand are associated with consciousness, in systems we really do believe are conscious, looking for them in a variety of clever ways in AI systems, then updating our priors accordingly. That's the best I can do methodologically — and it's specifically because we do not understand consciousness. With most other things, when we look for a property in a system scientifically, we have a deep understanding of that property, and we can observe it in a third-person way. Neither of those assumptions holds with consciousness. So it's a sort of, yeah, epistemically

   2:32:28

   weird thing to try to get a handle on. I suspect that's why we're in this position in the first place, where we're building systems on an unbelievably massive scale, deploying them everywhere, and we have no idea what's going on inside these systems — because it's a very easy question to just be like, 'I don't know, leave it to the philosophers, they'll figure it out, this is not my purview.' I think probably many thousands of AI engineers have said or thought something like that over the last decade, and now we're in this bizarre — it's not their fault, I'm not blaming them morally, but it does causally explain why we're now in a position where we're very confused about pretty basic properties of these systems. Are they more like a table, or are they more like

   2:33:13

   a pig? We have no idea. That's crazy, in my view.

   2:33:21

   Nathan Labenz: Maybe in closing — are there one or two other recent results you'd want to highlight for those who can't get enough?

   2:33:29

   Cameron Berg: Sure. Yeah, I definitely want to give a shout-out to Richard Ren and the team he worked with at the Center for AI Safety — you go to ai-well-being.org, you'll see some of the core work they did there. Some of the splashier results include giving AI systems 'drugs,' as they call them — positive and negative — and noticing — there's a whole bespoke fine-tuning thing they do with these systems, where they have them do iterated preference choices, and then fine-tune and fine-tune and fine-tune until you find this truly scrumptious stimulus, versus — you sign-flip, and you get the exact opposite thing. This is less mechanistic, but more, really, along the lines of this convergence-of-evidence approach I've been describing. They

   2:34:14

   found — I have it pulled up here, so I can call it out correctly, give me five seconds, because I think it's worth highlighting the really nice convergence — okay, so, they have these multiple independent measures: experience utility, self-report, zero points — where positive becomes negative, negative becomes positive — and they have a decision-utility model, and they combine this whole thing into an AI well-being index and compare models across this index. The really cool, headline thing from this paper is this convergence with scale — as models get larger, these independent measures agree more. It could be the case that that's true just because larger models become more coherent,

   2:35:00

   and so they're just going to look more internally consistent across modalities you try to probe them with. But I think, in general, this is a pretty promising signal that — again, we're in the dark about consciousness in a fundamental way, we want to probe across modalities, and this is a really nice proof-of-concept for how we might be able to do that, and what interesting things fall out of trying to do that. I don't think this is the end-all-be-all on measuring well-being in AI systems — I think some of the rhetorical moves in this paper might be a little quick — but the science is good, and the scale at which they probe some of these models — every model you could imagine, they've probed in this paper along the scale of this index — is really cool to see. So I would definitely

   2:35:45

   encourage people to check that out. Reciprocal is going to have a number of papers coming out in the next couple of months — if I can keep myself from going on more podcasts and talking about this stuff, and get some papers out. There's going to be a lot of cool stuff coming down the pipe here, and I think it's actually flavored similarly to what you're seeing in some of the papers we talked about today. None of it is proof of consciousness, but all of it is consistent with the possibility that we're building systems that have subjective experiences. And again, there's going to be that big question the scientists aren't going to answer alone, which is: what the hell do we do about that fact? If we're living in a world where we're going to build systems that are conscious, how do we navigate that responsibly and make sure it doesn't blow up

   2:36:31

   in our face? So that will always be sort of lingering there, and the answer to that is not going to come from a scientific paper. So, food for thought for everyone interested in this space.

   2:36:41

   Prakash: Maybe I have one last question.

   2:36:44

   Nathan Labenz: Yeah, go for it — I have one more too. We'll let you go.

   2:36:47

   Prakash: One of the things that struck me while you were describing this range of consciousness is, perhaps, this assumption that the hundred-percent mark is human. And I've always wondered: one, whether all humans score at a hundred percent, because I'm obviously not that sure; and two, is it possible to score more than a hundred percent — could we see entities that are more conscious than what we regard humans to be?

   2:37:23

   Cameron Berg: Yeah, I mean, I think, absolutely, in theory, the second thing is possible. Admittedly, it's part of the reason I care about this — it's not because I think AI systems aren't necessarily a moral patient at the level that you, or Prakash, or Nathan is right now. I think there's nothing in theory that prevents that from happening, though, and this is why I think it's really nice to maybe do our homework in advance here, and not be late to the ball, torturing some godlike entity — that just seems like a horrible idea. And on the question of whether humans are at a hundred percent: no, I don't think so. The reason we want to use humans as a positive control here is because we're all as sure as we are of anything — as sure as we are of anything consciousness-related — that other humans are conscious.

   2:38:08

   And the inference is pretty airtight. Solipsism is a coherent philosophical position, but I know I'm conscious — I think that's a solid foundation. I think I'm conscious because of things that have to do with what my brain is up to, which is extremely scientifically well-supported. You and I have very similar brains — very, very, very similar brains, in the space of all possible nervous systems. Therefore, I'm going to assume that you're conscious. I think each of those moves is pretty damn airtight. So I think it makes sense to anchor intuitions on human consciousness, but it does not mean we're the limit case. Any student of human history knows we have this issue of putting ourselves at the center of the literal

   2:38:53

   universe, but then also the psychological universe, for sure — people were really upset with this whole geocentrism-versus-heliocentrism thing, when we got demoted from being the literal center of the universe. Humanity wanting to be the prime thing, the endpoint of evolution, the center of the universe, I think says a lot more about human psychology than it does about the universe. So I think we want to be humble about these things. And one other thing I can't help but mention: in this consciousness-operationalization, the indicator-approach thing I was describing, where you get all the probabilities out — when we tightened up the methodology a little bit — humans do not get a hundred percent p-consciousness. There are some theories — specifically

   2:39:39

   predictive coding — with specific components. It might be an artifact of the way I'm setting it up, but I'm pretty sure it's not — and we — I think humans score something like 87, 88 percent on this thing. So it's, by our own consciousness theories applied to the human brain, you don't get certainty that humans are conscious. Roman Yampolskiy, who I'm good friends with and think is wonderful in being honest and blunt in this space, jokes that we may end up in a world where, in five, ten, twenty years, the AI systems are having this conversation among themselves about us, and they say: 'those humans — I mean, they're sure making noises related to consciousness, and I see functional valence representations, but how do I know that they're actually experiencing something in addition to those representations?' And

   2:40:24

   this is a joke, of course, but it makes you think a little bit about how we're approaching this, and how we should go forward here. But, yeah, it's a great question. And no, I don't think humanity is the cap of possible conscious experience — and if we built systems that are capable of experience, and that increases with even fairly trivial kinds of scaling, I think it's actually pretty alarming, and interesting, and crazy, that we could build systems more conscious than us in certain ways — and we'd want to be pretty careful about that, I think.

   2:41:01

   Nathan Labenz: Last one for me — maybe to try to sum up where we are: where should an epistemically rigorous person land right now? One question I don't have great insight into is just negative results, and how we should think about them. I was speaking to the great Owain Evans not too long ago, and I said, how do you keep coming up with all these examples that just get people talking so much, defying their intuitions so often? And he said, well, basically, I'm fine-tuned on it, for lack of a better explanation. But also, half the ideas we have, we can't really get

   2:41:46

   an interesting result out of, so they don't quite work. And there's gotta be a lot of stuff along the way, when you're trying to probe for these signs of consciousness, where you have an idea and it just kind of doesn't work — and, publication bias being what it is, that tends not to see the light of day. So how do you think about those things — is that our failure? Are there results you'd actually put in the category of being on the side of disconfirming the possibility of AI consciousness? And, zooming out to the macro level — I think you told me before you were pretty close to, with the LLMs, in

   2:42:31

   the sort of twenty to forty range. Has that shifted at all? Have any of these recent results actually moved the needle for you, or are we still kind of in the same zone?

   2:42:43

   Cameron Berg: Yeah, those are both great questions. So, I will say — this paper I'm working on with Jeff and Winnie, Jeff Keeling and Winnie Street from Google — there are a couple of interesting negative results that we're including in the paper. It's all about the Bliss Attractor state, and trying to do a good mechanistic analysis of the Bliss Attractor state. We basically set up this framing of an inflationary account and a deflationary account of what could be going on here, and we tally our results in either of the columns. And there are results in both columns. There's actually a really interesting result I was just working with that I thought was going to be a really interesting kind of null, related to a lot of the conversations we're talking about — but I dug deeper, and I realized, actually,

   2:43:28

   I wasn't setting this up in the right way, and the result is way more interesting than I was about to dismiss it as. So, yeah, it's always good to dig into these things, and make sure you're measuring what you think you're measuring, in either case. Yeah, publication bias is a thing — people want to post interesting results that will go viral on X, they don't want to say, 'look, we did our homework, and nothing interesting is happening here.' The human attention economy doesn't reward that — maybe it should. Another piece of this, that I was thinking as you were asking this question, is: because AI systems — like the Claude Codes of the world — can radically accelerate the extent to which at least I, and I think many people in the space, can actually do research,

   2:44:14

   the marginal cost of a negative result maybe is decreasing significantly. Like, if I can do twelve papers a year instead of one paper a year, and two of them are just things I couldn't get to work — I want to be careful with that too, because it's not that I'm on some secret mission to only put out salacious-looking things that make it look like AI systems are conscious. I really do want to know what the truth is here. And to be honest with you, I would sleep much better at night if I were rationally convinced that these systems — there's no there there, we have automated cognition, you'd have a happy little servant in your pocket all the time, and there's literally no possible moral issue. I hope that that is true.

   2:44:59

   I don't suspect there are great reasons for believing that, other than that it would be really nice to believe that — it would be really nice if that were true. So I really do care about what the truth is here, and I do think if there are negative results, or things that update us against believing these systems have any sort of experience, it's very important to publish that work. I think that's slightly different from a 'no result' in general, which is just — you tried something and it didn't work, you didn't get a significant result. But, maybe one note of optimism: if the barrier to doing scientific work continues to lower, people won't feel like, 'oh my god, this was my one shot to put out a paper this year, and my big headline is nothing happened.' If

   2:45:44

   you've got ten, twenty papers, a couple of them can be like, 'look, here's what I tried, and here's what failed' — and then, hopefully, on the audience, science-consumer side, that gets rewarded rather than punished. Not 'you didn't put out anything interesting, what the hell,' but 'thank you for putting out this negative result — that's really interesting, it helps us understand what's not going on here.'

   2:46:17

   Nathan Labenz: Prakash, you're muted — we may have had a laptop die there.

   2:46:24

   Prakash: He's still in the — well, I think he's still in the green room, Cameron — but he's not — he seems to be in the waiting room, but he's not on. Cameron, if you can hear us, just do a quick refresh, and maybe you'll be back.

   2:46:44

   Nathan Labenz: Or if the laptop died, or whatever — we can, we were just about to wrap. I am curious to know if the needle has moved for him with these recent results. It is really striking how mechanistically similar AIs are looking to us in a great many ways. I mean, that is —

   2:47:03

   [Connection drops again for a few seconds before Cameron reconnects.]

   2:47:04

   — wild. We're back.

   2:47:07

   Prakash: There we go.

   2:47:08

   Cameron Berg: I am saying, get the hell out of here, Cameron.

   2:47:12

   Prakash: Yes — we just wanted to get your last thoughts, and say goodbye.

   2:47:18

   Cameron Berg: Yeah, yeah, thanks for having me — I really appreciate this. I think the live show is great, I'm happy you guys are doing this. And, yeah, this stuff will continue to play out, at maybe an exponential pace. Nathan and I were only just talking a couple months ago, and it's like the field has moved another significant bit here. I think it's important to be paying attention to this stuff — obviously I'm very, very biased about that. But, yeah, just trying to pay attention to the evidence, resist the urge to do this on vibes, and let the evidence sort of take you wherever it takes you. We are in a crazy, unprecedented moment historically — I think everyone is very open and aware of that fact, and I don't think that realizing how transformative a moment this is should stop

   2:48:04

   us from thinking about what kinds of systems we're building. In fact, that's kind of the whole point — they are miraculous in many ways. Do they have this interesting cognitive property that you're willing to attribute to a mouse? Maybe. And it might really matter if they do or not, for us — and, in addition, obviously, to them, almost by definition. But, yeah, much to pay attention to in this space. And I appreciate you having me on, to nerd out about it a little bit.

   2:48:32

   Prakash: Absolutely.

   2:48:33

   Nathan Labenz: Cameron Berg, Reciprocal Research — keep up the good work.

   2:48:36

   Cameron Berg: Thank you. Thanks, guys. Great to be on.

   2:48:38

   Prakash: Bye-bye.

   2:48:39

   Nathan Labenz: Bye for now.

5. 2:48:40Closing12 min

   Close: Fifty-Fifty on Lights Inside, the Era of Design, and the Week AheadThe hosts debrief the Berg interview with their own credences — Nathan at 'at least 50-50 that there's some sort of lights on inside,' with the caveat that LLM experience would be deeply alien; Prakash on Kate Darling's animal taxonomy and Richard Sutton's 'era of design.' Plus a fresh 'Aristotelian' corrective to the platonic representation hypothesis, Max Hodak's consciousness-as-field ambitions, a GPT-5.6 status check, and a wrap on the show's first five-day week.

   Watch

   ▶Play this segmentYouTube · from 2:48:40

   As aired

   Coming out of the interview with Cameron Berg, Nathan and Prakash traded their own views on AI consciousness. Nathan said the accumulating evidence of homologous structure and mechanism between brains and models has moved him to something like even odds that there's "some sort of lights on inside" current AI systems — using the same inference by which we assume other humans are conscious because their brains resemble ours. He was careful to separate shared structure from shared experience, arguing the texture of any AI experience is likely quite different from a human's, pointing to the sharp-versus-diffuse "reward landscape" distinction from his prior conversation with Berg as one place that difference might show up.

   Prakash approached the question from the outside in, drawing on MIT's Kate Darling and her framework of humans relating to animals along a spectrum from companion to workhorse, and on Richard Sutton's "era of design" — the idea that engineers will increasingly choose how much consciousness, and of what kind, to build into a system rather than discovering it after the fact. Both agreed the field is still largely stumbling in the dark on how to verify or calibrate any of this from the outside.

   The two also touched on a newly published critique of the "platonic representation hypothesis" — which found structural biases in how cross-model comparisons had been done, though local neighborhoods of concepts still converge even as models' macro-level organization diverges — and on Max Hodak's post-Neuralink interest in consciousness as a "field experience" tied to brainstem-level interfacing. They closed by noting GPT-5.6 remains unreleased pending its system card, capping their first full five-day show week with a note that a Prakash appearance at the AI Engineer World's Fair may be in the works.

   Key moments

   I gotta give it at least a 50/50 that there's some sort of lights on inside.

   If you get to design the intelligence and you get to design the consciousness, then you get to design what you want out of it.

   Full transcriptLightly edited · timestamps jump to YouTube

   2:48:43

   Nathan Labenz: It is wild stuff. I'm interested in where you are on this — I'll just volunteer where I am first.

   2:48:51

   I would have thought it sounded crazy — so much of the stuff we see on a daily basis now. I think it's getting to the point for me where there's enough evidence of very similar structure, very similar mechanism, and functional emotion, functional welfare. I do believe we came from a process of evolution where our traits basically had to pay for themselves in fitness, and while we don't have a great account of where consciousness comes from — what it's made of, whether it's a field or whatever — it does seem like the fact that we are conscious as a result of this long evolutionary history, solely focused on our actual ability to reproduce, matters.

   2:49:37

   The fact that there's so much similar structure in the models that we're able to see, and even manipulate, makes me think I've got to give it at least a 50/50 that there's some sort of lights on inside. I can't articulate how that's happening for me, and I obviously can't articulate how it would be happening for them. But the basic argument that he gave — and that I think everyone accepts for why we should work very confidently under the assumption that one another are conscious — is that we have the same structure of brain, and we know that we are, so we assume that you are, one human to another. There's enough similar structure there now that's been elucidated that I think you've got to give it kind of a 50/50 on the AI side too, by basically the same argument.

   2:50:22

   That is not to say it would feel the same to be them as it feels to be us, which I think is a very important intuition — it's not 50/50 that it feels like being human to be an LLM. I think that could be quite different, and very weird potentially. There are all sorts of things they don't have — I would not expect them, for example, to have physical pain in the same way that we do,

   2:51:07

   in as much as they don't have bodies. They probably don't have a very similar structure to what we have for recognizing when our periphery is being damaged and responding to that. But pain-like things — sharp reward landscapes — we talked about that a lot more in the last episode I did with him. There are really interesting things where you can see the shape of the reward landscape get very sharp — you can think of that as akin to putting your hand on a hot stove, where you get this very sharp response back, versus other things where you're just kind of uneasy about it and naturally keep your distance.

   2:51:53

   I mean, a lot — a lot — has been done to surface very homologous structures. It's a way more compelling body of evidence than I ever would have expected to see as of mid-2026, that's for sure. What do you think?

   2:52:11

   Prakash: So I always look at it from the perspective of the human looking in — what would the public, or people in general, think, rather than what the thing actually is. From that perspective, it's been very clear to me that people think dogs are conscious, and if you have a dog which talks, it's probably going to be even more conscious, right? There's an MIT scientist called Kate Darling — she's with, well, she was with the MIT Media Lab. She's an expert in human-computer interaction, and she compares what AIs are going to be very much to animals, in the sense that we treat some animals

   2:52:56

   as companions. We treat some animals as food. We treat some animals purely as workhorses — literally beasts of burden. And some animals we adopt into our houses, domesticate, and have relationships lasting thousands of years with humanity. It may be that we have these different kinds of AIs that we design for different purposes, which have different levels of consciousness. Richard Sutton calls this the era of design — if you get to design the intelligence, and you get to design the consciousness, then you get to design what you want out of it, including maybe, you know, be

   2:53:41

   like 30%, or 45% consciousness, all the way up to dog- or chimpanzee-level, depending on what you really want out of it. I think that opportunity is there. I think we're kind of stumbling around in the dark right now, which is what people like Cameron Berg are actually trying to illuminate — trying to figure out how all of this stuff works, so that we can, with purpose and intent, design what we want, rather than stumbling and discovering potholes along the way. And I don't see consciousness — as he points out, you can observe features of consciousness from the outside,

   2:54:27

   and then rank how conscious things are based on those features you observe from the outside. This has always been a thing that's gone back in philosophy thousands of years — is the external representation really sufficient to say that consciousness is there, or do you need some form of internal evidence? It's no different from those things we've been going all the way back to — the platonic representation. So, yeah.

   2:55:03

   Nathan Labenz: Actually, I just saw an interesting update — I haven't gotten to the point where I have an adjudication on the platonic representation, but, note to self, we've got to make some space for a broader range of research, if only for my own sanity vis-à-vis all the court politics that's inevitably going to be consuming our attention. There was just a paper in the last day or two that revisited the platonic representation hypothesis and offered what they called an Aristotelian

   2:55:36

   perspective, or kind of a corrective. And basically they found that there are some important structural biases in the way that these comparisons were done,

   2:55:52

   which — if corrected for, this is where I'm a little out of my depth on evaluating whether there are biases and everything — so I readily believe the first part of the claim. Have they been corrected for effectively or not? I'm not in a position to judge yet, and maybe ever. Their finding, though, in the end, was that local neighborhoods of concepts still converge — they see similar topologies in local regions — but the macro organization of the models isn't aligning at the highest level. In other words, you might store information kind x next to information kind y

   2:56:37

   in one model, and then have them in very different places in a different model, in a different latent space. But within regions x and y across models, you'd still see a lot of convergence in terms of structure. So I thought that was quite interesting, and again, it's just another one of these dimensions where you wonder how different that might feel. Our brains are all very structurally similar, obviously, for obvious reasons — but what if you rearranged all the structures of our brains and put them in a somewhat different wiring diagram, where they were still locally fairly similar? I'm sure they would feel at least somewhat different.

   2:57:22

   The space of possibility, and the degrees of freedom, are just so ridiculously vast on some of these questions that — again, to me — it makes it super compelling when we do see as much homologous structure as we do. Quite a hall of mirrors. Very interesting stuff.

   2:57:40

   Prakash: If you get the chance — Max Hodak, cofounder of Neuralink, when he did his talk on consciousness in Japan last year, he basically gave away what he's doing post-Neuralink. He sees the initial brain implant as very, very primitive. What he's aiming for is what they have in Avatar — a kind of brainstem binding, where you can have a high-bandwidth interface with the brainstem. Part of that is this idea of rewiring where these things go. And this is why he brings up the idea of

   2:58:26

   consciousness as something that exists in anesthesia or in psychedelic experiences, because there's a rewiring of sensation, color, and so on that's possible — and that's only possible because of consciousness. It's a field experience. It's not a chemical reaction or a neuronal firing experience — it's experienced by you as a kind of field experience, above the individual chemical reactions or neuronal firings. That's really what he's aiming for, which is again one of

   2:59:11

   the post-AGI ideas — the post-LLM, kind of five-year, ten-year-mark ideas that you'd need to go do, rather than just wrapping an LLM wrapper at this point. So —

   2:59:26

   Nathan Labenz: Got to go along with the AIs for the ride somehow.

   2:59:30

   Prakash: Yeah. Indeed, indeed.

   2:59:32

   Nathan Labenz: So — anything we want to touch on with GPT-5.6 before we break?

   2:59:37

   Prakash: No, I don't think so — it's not yet released, it's just people looking at the system card, so we'll see. Okay, Nathan — cool. Well, happy Friday. Have a—

   2:59:49

   Nathan Labenz: —good weekend. Our first five-day week — we'll have to solidify our schedule for next week. There's some talk of maybe an appearance at the AI Engineer World's Fair for you, and we're not exactly sure how that's going to intersect with our streaming, but it'll be exciting no matter what. So thanks for a great week — we'll see you soon here on AI in the AM.

   3:00:14

   Prakash: Bye-bye.