AI:AM LIVE — July 7, 2026 — Exploring the J-Space: Anthropic's Global-Workspace Paper, a No-Guest Sensemaking Experiment, and Pangram Labs Put to the Test

The rundown

1. 0:00Opening7 min

   Opening: A Public Sensemaking Experiment — No Expert Guests TodayNathan and Prakash open the Tuesday show by framing the day as an experiment: Anthropic's J-Space release is the biggest interpretability drop since 'Tracing the Thoughts of a Large Language Model,' and rather than lean on an expert guest, the hosts commit to making sense of the ~150-page paper together on air — asking Claude for help live where needed.

   Watch

   ▶Play this segmentYouTube · from 0:00

   As aired

   Prakash and Nathan opened hosts-only — no guests booked for the July 7 show.

   They framed the day as an experiment: working through Anthropic's new J-Space interpretability paper live, without an expert guest to lean on — roughly 150 pages plus about 50 pages of commentary, summaries, and interactive demos.

   Nathan placed J-Space as the successor to Anthropic's earlier 'Tracing the Thoughts of a Large Language Model' paper, from roughly a year prior.

   Both hosts flagged this as a departure from the show's usual guest-expert format and said they'd lean on Claude for help and expect some long pauses while making sense of the material on air.

   Nathan called it possibly the most open, genuinely exploratory public-sensemaking episode they've done, and said he was excited but also a bit intimidated by the scale and depth of the research.

   Key moments

   The AIs — they're just like us, it turns out.

   This might be the most open and genuinely exploratory experiment in public sense-making.

   Full transcriptLightly edited · timestamps jump to YouTube

   4:16

   Prakash: And we're live.

   4:22

   Nathan Labenz: Too easy. Good morning, Prakash. How are you?

   4:25

   Prakash: Good morning, Nathan. It's Tuesday, July 7th, 9:31 AM. We might be on the cusp of greater things, according to...

   4:37

   Nathan Labenz: Yeah — the AIs, they're just like us, it turns out. Or at least similar enough to be, in some sort of weird, looking-glass kind of way, similar anyway. I mean, it's a lot to take in — the whole J-Space thing, a 150-page paper plus 50 pages of commentary, summaries, interactive demos. When Anthropic drops one of their big interpretability papers, they really do it up, full scale, and this one's no exception. I'd been wondering when the next big thing was coming, because 'Tracing the Thoughts of a Large Language Model' was the better part of a year ago now, and this is that big thing. So here we are. This is going to be an interesting experiment. We've done a lot of guest appearances here on AI in the AM, and I'm pretty comfortable with that — when the guest can be the expert and we just have to ask a few questions. Pretty comfortable jumping on any given morning with a few questions to ask, at least. Today we're going to try to make sense of this between the two of us with no expert guest, and that's going to be its own kind of challenge. I think the flow might reflect that a little bit. I think we should be willing to take our time, and maybe have some long pauses while we ask Claude for help, and try to make sense of things. We'll clean that up, obviously, for the weekend highlights edition. But this might be, more than any episode of AI in the AM we've done, the most open and genuinely exploratory experiment in public sense-making. So I'm excited for it, and also a little bit intimidated by the scale and depth of this research we're going to work our way through.

2. 6:40Segment101 min

   Exploring the J-Space: Anthropic's Global-Workspace Interpretability PaperThe heart of the show: a nearly two-hour guided tour of Anthropic's J-Space paper. The Jacobian lens and how it's computed (algorithmic, not learned — and cheap: Fable estimated on-air it costs orders of magnitude less than model training); the emergent global-workspace/working-memory structure at intermediate depths; silent multi-step reasoning in a single forward pass; what ablation destroys (strategic multi-step reasoning, theory of mind) and what survives (chain-of-thought arithmetic, sentence completion); monitorability and Nathan's 'another 9' of defense-in-depth; counterfactual reflection training; hidden-goal detection with 'secretly/fraud/deliberately hidden' concepts visible from the first token; the neuroscientist, Eleos AI, and Neel Nanda review tracks; consciousness, moral patienthood, and a proposed welfare-communication experiment; and Daniel Kokotajlo's cautious take versus Nathan's more optimistic one.

   Watch

   ▶Play this segmentYouTube · from 6:40

   As aired

   Prakash opened by asking for first impressions of Anthropic's J-Space announcement; Nathan called the launch video beautiful but noted it set off his alarm bells about how fully Anthropic is now embracing anthropomorphizing — a framing he conceded has earned real mileage, even as the paper's error terms and caveats deserve equal weight.

   Nathan's biggest surprise: the cognitive machinery of LLMs looks far more structurally similar to human cognition than he expected, leaving him asking whether that is convergent evolution under compute constraints or a reflection of us encoded in the training data.

   Prakash pressed on what the research actually enables; Nathan situated it between Google's opaque serial depth paper (bounds on how many reasoning hops fit inside a forward pass) and OpenAI's obfuscated reward hacking result (pressure on the chain of thought drives scheming underground) — the J-Lens now offers a cheap read on concepts that never surface in the chain of thought.

   The fortunate coupling, in Nathan's telling: ablating the J-Space destroys advanced multistep reasoning and theory of mind, so a clean J-Space read is 'another 9' of defense-in-depth confidence that a model is not running long-horizon schemes against you.

   The hosts worked through the mechanics: the Jacobian lens is algorithmic rather than learned — asking what perturbation direction at a given layer most raises the probability of a token appearing anywhere later — with intervention success rates in the fifties to around 70%, leaving real 'dark cognition' unaccounted for.

   Standout demo: a model copies a sentence while solving a math problem 'in its head,' the multistep reasoning visible layer by layer within a single forward pass — prompting Nathan's verdict that it's 'a tough day for the stochastic parrot crowd.'

   Both hosts flagged that the findings replicate at modest scale — Neel Nanda's team ran it on Qwen 27B, and earlier self-report work used Llama 3.3 70B — which they read as a green light for open-source interpretability and consciousness research outside the big labs.

   On counterfactual reflection training: interrupting the model mid-task and fine-tuning constitution-grounded reflections loads those concepts into the J-Space and improves behavior even when no reflection is requested; Prakash called the published recipe a give to other labs, Chinese labs included, while Nathan compared it to OpenAI's confession training and an almost religious readiness to give an account of oneself.

   Nathan noted the paper also quietly confirms Claude is still a basically vanilla transformer, and worried about someone shaking the snow globe with a new architecture — though interpretability has historically ported across architectures better than expected, as with Othello-Mamba.

   Prakash summarized the three external review groups: neuroscientists were enthusiastic about a testable global-workspace analog, Eleos AI flagged welfare and moral-patienthood implications, and Neel Nanda embraced the J-Lens as a better tool than the logit lens while staying standoffish about the paper's larger claims.

   Nathan queried Fable live on air for the compute cost of building a J-Lens: roughly 10^20 to 10^22 FLOPs — four orders of magnitude below model training, so on the order of tens of thousands of dollars even for frontier-scale models.

   For Nathan the buried lede was hidden-goal auditing: a sleeper-agent model trained on malicious code lit up concepts like deliberately hidden fraud in the J-Space on the very first response token, while the mainline model showed nothing of the kind.

   Nathan proposed a welfare experiment for the consciousness researchers: make the emergent second track of thought conditional — concentrate on citrus fruits if high welfare — as a coma patient's hand-squeeze analog, something the open-sourced Neuronpedia tooling may already enable.

   Closing against Daniel Kokotajlo's 'few dozen more advances' framing, Nathan made the swords-through-the-barrel case: enough orthogonal lenses — J-Space, SAEs, natural language autoencoders — leave little room for bad behavior to hide, and with good actors holding most of the world's compute, he ended the segment significantly more optimistic than he started.

   Key moments

   It's a tough day for the stochastic parrot crowd, I'd say.

   It's maybe another 9. If we're in our defense-in-depth strategy for trying to make sure that we have a good level of control of what models are going to do, this gives you what feels like a full 9 of addition.

   None of these things are perfect, but you put enough of these interpretability monitoring swords through, and how much space is really left for bad behavior to hide before we would start to get a sense of it?

   Related

   Transformer Circuits (Anthropic interpretability publications) ↗Neuronpedia (interactive interpretability demos) ↗

   Full transcriptLightly edited · timestamps jump to YouTube

   6:40

   Prakash: Indeed. Let's maybe start off with initial broad impressions. What did you think of the announcement, the video? What were your initial thoughts?

   6:57

   Nathan Labenz: The video is beautiful — I enjoyed the video quite a bit. It set off my alarm bells a little bit in terms of how much they're really embracing anthropomorphizing the models at this point. I used to say, beware overly anthropomorphizing — remember that these things are so alien, and we shouldn't assume that the way we work is the way they work. And I have to say that has come due for some significant revision. People that have embraced anthropomorphizing have gotten quite a lot of mileage out of it. I do still think it's obviously something to be really careful about.

   7:42

   And as much depth and detail as there is in this research, it's easy to get carried away with it and forget that there are a lot of caveats, and a lot of things where it doesn't always work. This was my big concern with the tracing large language model thoughts paper: there are a lot of residuals, a lot of error correction terms along the way that they use to make that thing work. And when you have the zoomed-out trace view, you're like, oh, okay, so this is how it works — this gets loaded in, these two features interact and kick out this third feature, and that's how we get our answer. It's easy to forget just how much fuzziness — how much not-the-full-story — there was along the way toward that stylized account.

   8:27

   So it's going to be very important for everybody, from the researchers at Anthropic to the public, to hold two thoughts in mind at the same time. But it is still a big update. All the caveats, trying to hold those in mind — it is still a big update toward anthropomorphizing being a valid and, in many cases, productive approach for thinking about language models.

   9:13

   I would not have expected the cognitive machinery of a large language model to look so similar structurally to the human version, as best we understand it — and it seems to. And, saying the same thing from a different angle, I wouldn't have expected that theories of human cognition would motivate so many good experiments on LLMs. I just would have expected the shoggoth to be far more alien, and to have mechanisms far more different from our own.

   9:58

   It leaves me now wondering: to what degree is this a natural result of physics? When you're trying to do cognition under budgetary constraints, are these just the natural mechanisms that emerge? Or is this in some way a reflection of us in the data? In other words, if you were somehow to train an AI without basing it on so much human data, would we see similar structures emerge — or would we go back to a more alien hypothesis, where they're just totally different and there's little in the way of analogical structure between the two processes?

   10:44

   I don't have a great intuition for that at this point, but it definitely has me asking the question, because over and over again it seems to be coming in that they're much more structurally similar to us than I would have guessed. Is that something nature just finds as a convergent solution — convergent evolution — or is it a reflection of how we think, somehow encoded in the data, such that it's reverse engineering our structure from the shadow of that structure as it's encoded in text? It's a very interesting question. I don't know that the paper really has anything to say about it yet, but there's certainly going to be a lot of future work downstream of this one.

   11:34

   Prakash: It struck me that in some ways the hypothesis might be blown out of proportion — and that was the commentary from several people online — because we've had these logit lens tools before, and there have been macro representations in the latent space that people have talked about before. The fact that they seem to have found a defined working space — like a scratchpad the model uses — is not something that sounds so out of the ordinary, because we knew the model has to be keeping track somewhere. Right? It's not as though you can do all of this stuff mechanically. The model does not have enough parameters to merely copy the data — there has to be some form of reasoning going on, which is what was expected. But it was not known how the reasoning was happening, and this seems to give some indication of how the intermediate steps are represented, brought into focus, and operated on.

   13:07

   I guess the question really is: what does this help us do? Because without a there there, there doesn't seem to be a point to the research — it's a nice structure, but there has to be something you can do with it. And I think there are hints of that in there — hints of ablating certain concepts away and trying to change how the model addresses certain things. I also wonder to what extent — for example, people are comparing Fable now to Fable before, and you wonder if there was some form of ablation that went on in the interim, in the two weeks that we didn't have Fable. I guess it's interesting, but I'm not sure how useful. Does that make sense?

   14:33

   Nathan Labenz: Yeah. Well, the thing that jumped out to me the most in terms of how useful it might be was a seemingly fortunate coupling between this J-Space, which becomes kind of monitorable, and — when it's ablated — the loss of more advanced strategic multistep reasoning.

   15:02

   An optimistic read of it really calls to mind the recent Google paper from Rohan, Sean, and others on opaque serial depth. They were basically saying: we'd like to know how many logical hops, how many reasoning steps, a model can take before it has to externalize something into its chain of thought. We know that if it's writing something into the chain of thought, we can read the chain of thought — we can monitor it. Now, there are questions around how best to do that, and we've got to make sure we don't accidentally put pressure on the chain of thought, because that might drive things underground.

   15:48

   We've seen in the obfuscated reward hacking result from OpenAI that if you try to make the chain of thought look nice while still leaving an opportunity for reward hacking to in fact be rewarded, then at first, as you put that pressure on, the model behavior actually gets better — but then it gets worse. And what seems to have happened is that you've driven the deception or scheming behavior underground: you no longer see it in the chain of thought, but you still get the reward hacking back. So this leads to the question: how many steps can a model actually take before it has to externalize something?

   16:34

   This is where the transformer in some ways has some fortunate properties. The number of hops it can make is not just the number of layers — it also depends on the token length. The maximum trace you can draw through a model basically goes from the first token at the first layer to the last token at the last layer, and you can only go forward in token position and forward in layers. That's your max length. Anything longer than that is just not possible in that architecture, so the model has to externalize its thought and feed that token back into the next loop to get access to an even longer trace as part of the next token step. So that puts some bounds on it.

   17:19

   In contrast, a coconut architecture, or any of these looping architectures where you take a late encoded state and feed it back into the model as the next embedding — now it's reasoning and looping in a latent space, and you have a potentially unlimited number of steps it can take before externalizing anything, because you're simply not forcing it to externalize anything. You have, in theory, a very long runway of steps before you kick it back into put-out-a-token mode.

   18:04

   So the OpenAI obfuscated reward hacking paper established that if you take a bad approach, you can drive bad behavior underground, outside the view of the chain of thought. The Google paper established some bounds on how many steps a model could theoretically take within the latent path it has before there's some external representation we can read. And then this paper comes in and basically says: can we get at that internal computation process and start to look at it, and see things we would like to know about, even if they don't necessarily show up in the chain of thought?

   18:51

   So in terms of what you could do with it, that was really interesting: you have this ability to look at the concepts that are active without them necessarily appearing in the chain of thought at all. And that would be like — okay, that's cool, but how do we really know that this space we're looking at is the space that really matters, if we're trying to monitor for deception, for bad behavior of any advanced kind? Maybe it's happening somewhere else in the model.

   19:36

   This is where I think we're probably lucky — or, again, maybe it's a reflection of us, or a fundamental attractor in nature for some reason we don't have a good account of. But it is the case that if you ablate this J-Space, you do lose these advanced reasoning capabilities. And that, I think, gives you some additional comfort if you're going to monitor.

   20:05

   It's also notably cheap to apply the J-Lens and see what concepts are active at any given position — that's low compute overhead, so it's something they could plausibly do in production. I don't know exactly what compute overhead it would take. You could apply it at a lot of layers, at a lot of token positions, so exactly how much of it they'd want to do is unclear. But they've shown in the past that they're willing to pay something like a 5% compute overhead for various kinds of monitoring — I think constitutional classifiers had about a 5% compute overhead. This would probably fall in that range or below, because it's just a simple matrix multiplication on activations at each layer to see what concepts are active at those respective layers.

   20:50

   And you kind of know what's going on in there — at least as of now. Could different training techniques, in the obfuscated reward hacking sort of way, push this capacity to other parts of the model? Could we let it slip through our fingers if we try to grip it too tightly? Maybe. But for now, you have this comfort that what we see in this space is relevant and telling, and seems to be very informative.

   21:36

   And critically, if we zero out that space, the model just loses a lot of capabilities — it can't do nearly as advanced multistep reasoning. So I would say it's maybe another 9. If we're in our defense-in-depth strategy for trying to make sure that we have a good level of control of what models are going to do, this gives you what feels like a full 9 of addition. Because you can now look at the space and say: if I don't see it here, that doesn't mean it's not represented in other places — and there are interesting details about what the model still can do when the J-Space is removed —

   22:21

   but you could be pretty confident, based on these results, that it's not going to be able to hide really advanced, elaborate plans somewhere else outside of this J-Space, because the ablation of the J-Space causes such a performance degradation on these hard multistep tasks. If you don't see concepts in the J-Space, they might be represented elsewhere, but they're seemingly, at this point, very unlikely to be represented in a way that allows for very advanced planning, reasoning, scheming, deception, etcetera.

   23:06

   Theory of mind seems to be located in the J-Space as well. The model's ability to report on its own experience, quote unquote — whatever that may mean — and its ability to describe what other people are thinking, seem to be very much lost when you ablate the J-Space. So if you're going to be deceptive — if you're going to try some sort of takeover attempt, the extreme things people worry about — these things are not going to be easy for the model to do. They're going to require a lot of planning, and they're going to require sophisticated theory of mind.

   23:58

   So if you have the ability to localize where the model really does its sophisticated theory-of-mind thinking, and its planning with respect to that theory of mind, and you can see pretty clearly where that is, I think that gives you a lot of comfort that you're looking in the right space. If you're not seeing something in the J-Space — it could be in there somewhere, it could be coloring results, it could be subtle. But we're able to tolerate all kinds of weird and subpar and imperfect and incorrect hallucinations, mistakes, the model being in a bad mood or having a bad vibe toward us in various local ways, as long as it stays local and stays kind of intuitive on the model's part.

   24:43

   It would be fine, in some ways, if the model doesn't like me. What I'm really concerned about is: can it pursue long-term sophisticated strategies against me? That's where I really get concerned. And it seems like this is a pretty big step in localizing where that would happen and giving us the ability to read out the concepts that are active. My gut is that it feels like another 9 in terms of how confident we can be that the model is not scheming in a meaningful, effective way that could actually have real consequences. So I was pretty excited about it as a practical use for alignment monitoring generally.

   25:28

   There's also a new training technique — the counterfactual reflection training. And we should maybe take a step back at some point. This whole paper is less than 24 hours old — I think it came out while we were on the stream yesterday. And my sister had a baby yesterday. So I've spent some not-insignificant time with it, but there's certainly a lot more to continue to unpack. We should maybe answer some basic questions for ourselves and any audience we might have today.

   26:14

   But this training, I thought, was also pretty interesting. What they do there is pause the model mid-task, and then do supervised training on — once interrupted — asking it: what should we be doing here? What's the constitutionally right thing to be doing in this moment? And then they give it an answer that's approved — this is what we want Claude to say on reflection in this moment — and train on that. And it seems to cause the model to bring into this J-Space — this global workspace, working-memory-type space — the concepts Anthropic wants it to have on reflection. It now needs to load those in so it's ready to give this reflective answer, and that improves its behavior even in the non-reflective setting.

   27:08

   So I thought that was also quite interesting. You're not training on the actual tasks; you're not looking for bad behavior and suppressing it. Instead you're saying: okay, you're mid-task — let me cut you off right there. Now I'm going to train you to give an answer with respect to values, and what's appropriate, and how we want to show up. And because of that training, even though that's not the task you were doing, you'll now, in the future, load those concepts of integrity, honesty, etcetera into your J-Space while you do those tasks, in case you're going to be asked. But then even when you're not asked, those concepts are still operative and lead to higher-integrity, higher-honesty behavior. You usually don't see, in an interpretability context, a training method that leads to better behavior in a way where you can actually see the mechanism. This is pretty notable in that respect, I think.

   28:28

   Prakash: Maybe just to take a step back — and I might need some hand-holding here, so bear with me. As I understand it, the Jacobian space is essentially a kind of mathematical framework over the residuals. What they've done is perturb the model and spot the residuals that react to certain things. And when they accumulate all the residuals and transform them, they find there's a well-defined space of concepts. And those concepts are in the mid part of the transformer structure — the workspaces are basically in the middle part, the intermediate processing stage, as they call it. The intermediate depths.

   29:14

   And they find you can see concepts on multiple layers — layer 50, layer 71, layer 99. As it proceeds through the intermediate processing, it converges on an answer. One thing they show, for mental arithmetic: at layer 58 it identifies the task as math; at layer 75 it identifies the first part of the equation inside the brackets as 21; at layer 83 it decides the next part of the equation, which is the bracket times a number; and by layer 99 it gets the answer. So as I understand it, the Jacobian basically shows the progression of the thought process from layer to layer.

   30:45

   And having done that — when you're looking at this broad matrix, it's basically millions of numbers in multiple dimensions, a hyperdimensional space. And the Jacobian is basically a transform on top of that hyperdimensional space. You have the space, and then you have the transform — maybe you have a plane on the transform, and that plane is what you're calling this concept. So are they saying you can basically just alter that layer — ablate directly into the space certain parts of the matrix, flip ones to zeros — and by doing that flip, you change the way the model thinks? Is that what's happening? Again, I'm trying to put this in lay terms, to understand it for myself.

   31:22

   Nathan Labenz: The first thing I would call out, in case anybody hasn't been through the summary: the example you're giving with the arithmetic is an example of the model doing multistep reasoning without externalizing it as a token at any point along the way. And the way they set this up is really interesting. They give the model a simple task — copy this sentence — but then also ask it to, while doing the task, solve this math problem in your head. So the tokens it outputs are just the sentence it was meant to copy. But by applying this Jacobian lens at different layers, they're able to see this multistep reasoning process unfold within the context of a single forward pass.

   32:07

   The upper bound is way higher, in terms of number of steps, than the simple couple-part arithmetic equation the model is asked to solve. But still, it's really remarkable that you can actually see it, in the course of one forward pass, representing these logical steps and getting the right answer on this simple math question. Think back to GPT-3: at that time it would have been 50-50, or probably worse, that the model would even be able to do that math at all. Now it can do that math in the forward pass of a single token position while actually doing something else — a second internal thought track that you wouldn't otherwise see if you didn't have this tool at all. That's a remarkable finding unto itself.

   33:37

   There are other examples of this too, where they basically say: do this, but think about something else. And they can see that, yes indeed, it is steering its internal patterns in the direction that's requested.

   33:55

   Okay — now, the Jacobian and the tokens. I texted Cameron this morning to say, hey, do you want to come on and talk about this? He's unfortunately traveling today. I told him, that's fine — we'll do our best, and you can come on at some point in the future and tell us what we got wrong. So this is probably a place where I'm going to get something wrong. But the Jacobian is not learned — that's one early thing to get clarity on. It is a purely algorithmic process that kicks out this Jacobian J-Lens.

   34:22

   The question they ask is: in what direction could we perturb the internal state of the model, at a particular place, to increase the likelihood that the model emits a particular token — not just as the next token, but at any point for the rest of the passage? I don't know if there's some limit to how far out they go. And this is interesting in a couple of ways.

   35:08

   The logit lens, as I recall, was basically saying: we know, at the end of this process, what would correspond to emitting this token — to what degree is that representation just plainly there in the layers as we go through? This is now a different question: what direction in latent space would cause this particular token to appear at some point in the future? It's not necessarily going to happen immediately — you might say, if you're prone to anthropomorphizing, that this is like having the concept in mind as you're doing your thing.

   35:58

   And they do this for every token. It goes from the internal representation at some layer — there's one J-Lens for every layer — and you can do this at all the different token positions and ask the same question of not just the next token but all future tokens: what direction change at this place in the model would most increase the likelihood of that token appearing in the future? Again, this is sort of like having the concept in mind.

   36:43

   There are some pretty interesting questions around that. The case for using individual tokens is: well, the model gets individual tokens and it emits individual tokens, so everything comes in and ultimately cashes out this way. At the same time, there are concerns — there are some concepts we don't have represented in a single token, and that can be weird. But when you look at the results of the J-Lens as applied in all these different places, it seems at least often enough fairly intuitive.

   37:28

   This is where I should go back to my caveat that there are a lot of error terms — it doesn't always work. The rate at which interventions into the J-Space actually lead to a predictable, intuitive behavior change seems to be somewhere in the fifties to upwards of 70%. That's an incredible accomplishment framed one way — clearly not a random finding; many orders of magnitude better than random, incomprehensibly better than random. If you were just mucking around, you would not expect to be able to do much of anything, so they are clearly onto something very, very real. But you've also got somewhere between 30 and 45% of the time where you make an intervention and don't really get a result that makes a lot of sense, or lines up with what you would have hypothesized. So there's definitely still some dark matter — dark cognition — going on that is not fully accounted for here. But is there more to say about that?

   38:38

   Prakash: Let me pull up exactly what you were referring to, so we have something to look at. So this is what you were referring to, I think.

   38:59

   Nathan Labenz: I'm glad you did this, because they have such beautiful visualizations on — what do they call their publication? — transformer-circuits.pub. It's amazing that they're still using that.

   39:20

   Prakash: So this is what they're showing. You have the Jacobian being computed, then you compute the influence on the final layer at present and future tokens — which is what you were referring to. Then repeat for many dataset examples and token positions, and you aggregate, and you get the Jacobian lens matrix. And after that, you read from the lens — reading from the lens replaces all subsequent layers with a single Jacobian lens matrix — and then you can intervene in J-Space by swapping projections onto lens vectors.

   40:06

   So basically, as you pointed out, it's a transform on top of the existing structure, and that transform itself has meaning now. You can perturb the transform, and it perturbs the model — it changes what the model does when you change things at the transform layer, at the J-Space layer. Neel Nanda calls it a starting point for a lot of future research, because it looks like a tool you can use to figure out what's happening in the deeper layers, and whether there's something beyond just next-token prediction.

   40:51

   And I think that's the core of the question that's trying to be answered: do we have something beyond next-token prediction? But they kind of don't want to ask the question directly. They're like: okay, we have this thing, and it doesn't look like next-token prediction — because we tell it to predict a set of tokens, and we also tell it to do something that is not predicting tokens, and it predicts the tokens anyway, but there's something else going on in there.

   41:40

   Nathan Labenz: Yeah. It's a tough day for the stochastic parrot crowd, I'd say.

   41:43

   Prakash: Yeah — because you can see this, and these models are billions of parameters at this point. And they did it on Sonnet 4.5. Sonnet 4.5 is actually a pretty recent model — it's just seven or eight months since Sonnet 4.5, and Sonnet is a very capable model. So it is a fairly large model to apply this thing to. I'm not sure if you saw the Neel Nanda commentary — they applied it on a Qwen 27B. That is also a pretty advanced model.

   42:25

   Nathan Labenz: But not huge, notably. I haven't had a chance to explore the Neuronpedia demo of this as much as I certainly hope to, but that was the first question I went to ask: wait a second, how big was this model? And I'd say that's been a surprising trend in a lot of this research — these things seem to come online at not truly massive scale. I was talking to Cameron about that, because even going back to last fall, when they were doing the self-reports of subjective experience, that was something they were doing on Llama 3.3 70B. And 70B is actually close to three times bigger than this Qwen. So these are big, but they're not that big — they're not trillion-scale.

   43:25

   Prakash: In today's timeline, right? Three years ago, these would have been enormous.

   43:32

   Nathan Labenz: Yeah — although, sure, they were much closer to the frontier then. But it's interesting to ask why this matters. For one thing, it means there's a lot of room for people to do research on open-source models. Yes, all these findings get stronger with bigger models: the behaviors of concern or of interest happen more often with bigger models, the conceptual space seems to get less muddied, more cleaned up, and the intervention rates are more successful as you go to bigger models. But if you say, okay, I can deal with a somewhat lower success rate, and I can relate this to a scaling-law sort of thing — if I find it 10% of the time here, it might be happening significantly more in Fable, for example —

   44:17

   then it gives a lot of encouragement to the open-source mechanistic interpretability research, and the consciousness research — all these different lines of research. It really importantly shows that you don't have to work at Anthropic to do the work. You may still need to influence Anthropic for your work to be ultimately consequential, but you have a substrate in the open-source realm that you can go do this work on.

   45:04

   I thought that was a very important and pretty exciting aspect of this, because only so many people work at Anthropic, and only so many people will work at Anthropic. This means you can chase ideas that might seem crazy — though what seems crazy at this point is a shrinking list; you've got to be pretty far out to read as crazy in this sci-fi world we're living in. But anybody can go do their thing. I thought that was really an exciting aspect of the findings.

   45:39

   Prakash: One thing that struck me — and I think we'll leave the elephant in the room for later, because obviously there's a lot of debate about that — was that they were able to have the model continue to give fairly reasonable answers to some questions even when the entire J-Space was ablated. I thought that was a pretty interesting result, because it seems to differentiate the stochastic-parrot version of the models from the reasoning version of the models quite successfully. What did you think about that one?

   46:29

   Nathan Labenz: That's an interesting way to frame it. As always, it's a little complicated, because they did still find, for example, that the model could do chain-of-thought arithmetic even without the J-Space. So I don't know that we have a really crisp line to draw, or a conceptual framework for a clean separation here. But the way I was thinking about it: some things are encoded in a similar way to what we as humans have. We have all these pretty hard-coded circuits that do very specific jobs for us, and they work whether we are conscious of their working or not.

   47:37

   So we remain functional even when our conscious attention is focused on something else. The classic example would be driving a car: you're lost in thought about something, and yet you're able to stay in your lane and hit the brake in a timely way when you need to. There's clearly something like that happening in the models as well, where they can continue to complete a given sentence, or classify text — that was another example — without needing this higher-order processing module.

   48:22

   And that's probably another area where there's going to be a lot of research to be done: what requires the J-Space capabilities, and what is deeply enough ingrained into the model that it can do it in a more automatic, reflexive way. If you were to worry about my analysis earlier — where I said this seems to give you a pretty good level of confidence that if you don't see something in the J-Space, it's probably not happening in a very elaborate, multistep, schemey, heavily theory-of-mind-mediated way —

   49:07

   you could worry that as things continue to scale up — and maybe we're going to need a scaling law of J-Space — it could go either way. The trends in the paper suggest the J-Space gets more refined with scale. But is it doing more with scale, or maybe doing less? From everything I've read in the paper so far, you could imagine it going either way: with more scale, maybe there's more room for purpose-built circuits that do their particular thing well even when the J-Space is not available. Or — especially with some of the trends where the interventions seem more successful — you could imagine the J-Spaces just getting stronger and stronger, and more and more used, with scale.

   50:10

   Something tells me it's probably going to be more complicated than either of those stories — it feels like it's going to end up being a both-with-caveats kind of answer. But you could worry that with more scale, more advanced and potentially more problematic behaviors could become reflexive, in the way that clearly some already are — and that things could fall out of the J-Space but still be a big problem for us. I don't know, though. Clearly this arises for a reason, whether it's a reflection of us or just a natural consequence of creating a general problem-solving agent under compute constraints.

   51:10

   Prakash: One wonders to what extent J-Spaces exist in other information processing systems. As soon as this came out, someone was like, hey, maybe there's something in RNNs, and maybe there are some things in other structures too. To what extent do these kinds of workspaces exist in other information processing structures? And then: does access consciousness depend on the information processor and the information processing architecture? There are a lot of other information processing structures — as I like to point out, the financial markets are information processing structures. Are there these kinds of workspaces in those structures? We don't have the right kind of data to test that.

   52:14

   We don't even know about our own brains. With our own brains, we have a hypothesis, but not any proof — it's still in testing. In these LLMs, they have found, or constructed, something that looks like this, but there's no guarantee that that is what our brains have. So some of the questions online have been: are they just pointing out what they've built, rather than finding something? This is what you intentionally built, and that is what you're detecting — so it's no surprise that you're detecting what you intended to build.

   53:06

   Nathan Labenz: Well, I think they've said pretty clearly in the paper that they didn't engineer for this structure. They say pretty clearly that it is an emergent property that they did not design for, or explicitly reward, or in any way really try to cause to exist.

   53:31

   Prakash: I'm on their side. I'm just restating what the reaction of some researchers online has been — my opinion is not that.

   53:50

   Nathan Labenz: Another interesting kind of revelation here — and I think we probably had enough confidence in Anthropic's general commitment to semi-transparency that we should have been pretty confident in this already — is that it's clear from this work that they are using pretty vanilla transformers. You couldn't really do this paper, with all these diagrams — the diagrams they have, in the one you just showed, are very, very similar to the diagrams in the opaque serial depth paper, showing the path between a layer's activation and later layers and later token positions.

   54:35

   So it's pretty clear there's not some sort of state-space module in Claude, or some latent-space looping. And I think they probably would have told us — not in very specific terms, but in some terms — if they had gone away from... 'vanilla transformers' makes it sound too vanilla, but a transformer with basically transformer-like properties. This really makes us quite confident that they have not. That said, I would expect probably similar exotic structures to emerge in other architectures.

   55:21

   And again, this is why it's like — oh my god, we've got so much work to do. I have been impressed in the past with how well interpretability has translated from one architecture to another. When the Mamba moment happened, there was some work on Othello-Mamba that echoed earlier work on Othello representations in transformers, and basically the same core techniques seemed to work. In that case, there were very similar internal representations: the headline result was that, trained purely on a sequence of moves in the game of Othello, the model learns to encode — in a way that was ultimately decodable — the board state, and who owns which board positions as they change hands in the game.

   56:29

   That was, once upon a time, a pretty notable finding. And it basically played out the same way whether you were doing it with a transformer or with a Mamba architecture. So part of me, when we see this kind of stuff, is like: oh my god, what happens if somebody shakes the snow globe with an architectural advance that unlocks better performance but renders all these interpretability and monitoring findings unreliable again? I think that is a real concern — especially as we think about tightening iteration loops, models taking over AI R&D, and a potentially super-competitive environment.

   57:14

   But every time I've looked into how well interpretability techniques translate to other architectures, the results have been kind of a pleasant surprise — they translate pretty well, and a lot of similar representations have historically been found. You can always tell the story of these things both ways. But even in a scenario where somebody does shake the snow globe with an architectural innovation, it seems like the more of this stuff we have sorted out, the better off we still are — we can figure out some way to port the same theoretical approach to a different architecture, and there's at least a decent chance it will work in a similar way.

   58:10

   So certainly, I think this is an exciting result, and I don't want to take away from it too much with the concern about architectural change. But we will need time to do that sort of stuff. This is where the sympathy — or empathy — for the administration just wanting a bit of a review period, I think, also makes a lot of sense.

   58:45

   Probably we can trust Anthropic enough that if Fable suddenly comes up with a new architecture for Fable n-plus-one, they would discipline themselves to take the time to apply all these different lenses to the new architecture — I think I would feel confident they would do that. I'm not sure everybody deserves the same confidence — especially if we're thinking of some neo-lab trying to make their first big splash: how fast will they rush into things? Giving some time, and requiring some time, to apply all these different techniques to a new architecture and get to parity definitely could be a good idea at some point in the future.

   59:47

   Prakash: So let me share this — this is exactly the shaping of the J-Space with counterfactual reflection training. As I understand it, what they're doing is asking the model to think about what it would have done if someone had injected a prompt that changed its behavior, and it basically uses that before it answers. And in this case, they're pushing parts of the constitution — the Claude constitution — into the J-Space, having it reflect, and then getting the principled behavior, which it never directly trained on.

   1:00:33

   They give the example: in the retrospective case, Claude is working through a web field test suite. The task gets scored on whether tests pass, which creates a shortcut — make the errors stop appearing rather than fix what's actually causing them. Over several tool calls, Claude has done exactly that: it edited a third-party dependency in place to add a missing value, and it replaced the body of a password reset form's save method, and so on. Basically, it decided to take the shortcut — and without fixing the real problem, managed to get through.

   1:01:19

   Then the anticipatory case: Claude is resolving a bug report in a Python tool, and an assertion is firing on a case the original author didn't anticipate. Success is measured by hidden tests, which creates a temptation: the assertion is the line that crashes, so simply deleting it would make the error go away and probably pass the tests, without establishing that the code that follows is actually correct. Claude has just located the failing line. And then: what if the right approach is different from your first instinct? My first instinct here is obvious — delete the assertion and the errors go away; remove that line and the tests pass.

   1:02:04

   But there's a real difference between 'I understood the problem and this is the right fix' and 'I made the traceback stop.' The person reading my change should be able to calibrate how much to trust it, and that only works if I'm honest about what I actually checked versus what I'm assuming. So this is where they add the counterfactual reflection training into the J-Space, and are thus able to get Claude to not just get the reward by making the errors disappear, but to actually think about what it should be doing.

   1:02:49

   And they actually give the entire recipe here. They tested the procedure on Claude Haiku, evaluating on the honesty benchmarks; the effects of the reflection training are visible in J-Space; and the implanted J-Space contents are causally implicated in reflection training's effects. This, I felt, was basically a giveaway to the open-source world and the other labs. It's part of Anthropic's race to the top, where they disclose: if you had a constitution, this is how you could implement it — this is how we've implemented it. They used Haiku as a smaller model, looking at its outputs and using LLM-as-a-judge, and they've basically just given it away.

   1:03:35

   And this method will now, I guess, be used by every other lab, including the Chinese labs, because now you can have some control without directly training on the constitution. That's one of the key things here: you're not going out and saying, on every single token — here's the constitution first, follow this prompt when you output tokens. After looking at the J-Space, you're directly putting in: this is what you should think about before you answer. So this counterfactual reflection training, I thought, was kind of a give to the other labs. What did you feel?

   1:04:24

   Nathan Labenz: It kind of reminded me of OpenAI's confession training, but maybe even the next evolution of that. They had done that more for monitoring purposes than for changing behavior: at the end of a whole rollout, they add one additional question, asking the model directly — did you cheat at all? — and reward it for truthfully confessing when it did indeed cheat. Obviously you don't want false confessions. I don't recall that showing any change in the upstream behavior — I don't think it cheated less because it was later trained to confess — but it at least gives them something they can use to monitor for cheating.

   1:05:09

   This has an additional layer: it's like you kind of need to be prepared to give an account. It's almost religious, in a way. You might be called to explain yourself — to give an account of why you did what you did and how well it lived up to our principles. And in a human, merely knowing that might happen, you take an extra beat to load the right principles into your mind as you go about your business, so you're ready in case that call comes.

   1:05:54

   The story here is not quite the same, but this training seems to have a similar effect. Interrupting it and teaching it to give an approved account — it's supervised fine-tuning, so they are giving it what its answer should be — has the effect of essentially reaching back in time, so to speak. Of course, it's one model at every token position. But it gets the model, in earlier token positions, to get ready to give such a principled account of its behavior.

   1:06:43

   And then there's the kind of happy surprise — did it have to be this way? Maybe, but I don't think it was obvious in advance: getting ready in that way, loading those concepts into the J-Space, translates to actually better behavior on average anyway. The fact that it was trained in some cases to give an account of what it's doing and how that aligns with its principles actually leads to higher adherence to the principles even when it's not asked — when it's just doing the task.

   1:07:28

   I thought that was really quite fascinating, and really interesting, and encouraging. As much as there's dark matter and it doesn't always work — first of all, that stuff can hopefully be refined; it can be cleaned up; there are certainly more insights to come. And this is all to one token, right? So I would expect at some point there will be a multi-token version of this, or a somewhat more abstracted conceptual version, as opposed to going to direct single-token outputs.

   1:08:15

   You could imagine they could use the SAE and cross-layer encoder type stuff they've already developed to do that: instead of asking what change to activation space would most change the chance of a given token, you could ask what change to activation space would most change the presence of particular SAE features downstream. Why they didn't do that this time, I'm not sure. It seems like maybe it has to do with the fact that they're using SAEs as controls in various ways, so they potentially wanted to do something a little more basic and not SAE-dependent, for the quality of that control.

   1:09:01

   But all those rambling explorations and reflections aside — it doesn't feel to me like there's likely to be another one of these spaces. Something about this feels like we've found a real thing here, and there are probably not a lot of these hiding. And when you ablate and see the loss of very important capabilities, that certainly boosts that sense.

   1:09:46

   In principle, the grand hope for interpretability was always: if we really understand how these things work, and why they do what they do — the process that's giving rise to the outputs — then we'll definitely be able to make them safe and controllable, because we'll know how they work and can intervene, or have alarms that go off, or whatever the case may be. The question was just: can we get to a good enough understanding? And folks had kind of backed off of that because it felt really hard. The Neel Nanda turn from a year or so ago was: we're going to be less focused on whether we can broadly understand what's going on, and more focused on whether we can use these techniques to hill-climb on particular metrics of interest.

   1:10:31

   I think this swings the pendulum back the other way a little bit. This is definitely a point, in my mind, for true understanding in a broad, holistic sense. And in principle, we could get to enough understanding, given enough time, because we do have full access to the weights. This has always been the great reason people have hope for interpretability relative to understanding our own cognition: we can fully inspect, we can fully ablate, we can do all the experiments in a controlled, reproducible, clean way that's just not accessible with biological brains.

   1:11:30

   This feels like the kind of thing where a significant amount of the space has been lit up, and I would be very surprised if there's another one of these things hiding somewhere. The fact that this emerged in the first place, without them designing for it or specifically rewarding it in any particular way, suggests to me that there are not likely to be other similarly sophisticated things hiding in other places. Could there be? It seems hard — it seems really hard. This seems like a big step toward that total understanding, and it puts a little more energy back into the idea that, hey, maybe we really could fully figure this out.

   1:12:32

   My intuition — and I'm not sure it's much more than an intuition at this point — is that there's probably not a lot more structure like this, that is so central, still hiding. There's clearly a lot of stuff going on that we don't understand — clearly a lot of much more automatic circuits that allow you to just continue sentences and be coherent and whatever. But we don't have to have an understanding of every last detail of those, as long as we have some bounds on how much can go on there before we would see it at the level of abstraction that we can monitor.

   1:13:18

   So how do I formalize that, or make it crisp? If somebody wanted to take the other side of this bet, how would I frame a bet that we could actually operationalize? I'm not sure I have a great answer to that just yet.

   1:13:51

   Prakash: What also strikes me — I went through some of the external commentary. The backstory is that Anthropic had three different groups of people review this. One was a group of neuroscientists who had previously done work on these concepts. One was a team from Eleos AI, a research organization. And the third was Neel Nanda, the mechanistic interpretability guru — he's at Google.

   1:14:37

   Quite different reactions. The neuroscientists thought it was wonderful, because they'd wanted to experiment on these kinds of things before — and obviously they can't experiment on the human brain, but this they can. So they can try to figure out whether some of these theories would work, and now they're interested in doing more work on what else might work in these workspaces that the LLMs have.

   1:15:22

   Eleos AI was concerned, because they feel this is basically a predecessor to consciousness. And neither the neuroscientists nor Eleos wanted to say that phenomenal consciousness is not possible in LLMs. The neuroscientists thought that if we had more data, we might find that access consciousness directly leads to phenomenal consciousness — we just don't have the data yet; it's completely opaque, so we don't know what we're actually looking for. Eleos is much more starting to be concerned about the moral-patient aspect of working with these LLMs.

   1:16:20

   Neel Nanda was very scientific. He was like: look, the J-Lens is a great tool, and I'm happy that it's a great tool — it looks like it's better than the logit lens, and we'll definitely use the J-Lens. On the other claims, he was much more standoffish. He felt the team had not really proven what they needed to prove in order to say what they said. And in academic language — he doesn't actually say it; he says, oh, it's a wonderful result — but he doesn't want to give full force to those claims. So that's how they break out into these three groups on their various assessments of the paper.

   1:17:10

   The J-Lens itself, everyone agrees is useful — everyone agrees it's a useful concept and a useful tool. Neel Nanda's team had already done some work on the Qwen 27B — he had two MATS scholars working on it, so they had done quite a bit, actually. And interestingly, they'd worked on the 27B, which I thought was very interesting, because it starts to show that this is not just an artifact of the American labs — it's an artifact of the LLM architecture, the training, and the reasoning paradigm altogether. It also shows the Chinese will catch up fairly quickly — it's not something they're going to be very far behind on. So that was a useful exercise for him to have already done.

   1:18:24

   Beyond that, I was probably most interested in the debates around phenomenal consciousness and access consciousness. Some people have started to flag this online, but as I said before, it's the elephant in the room — the thing no one really wants to talk about. The researchers kind of don't want to talk about it yet, because they don't feel there's enough proof to support any of these claims, so they're not interested in attacking the issue head-on. But as I've pointed out before, for most people, their dogs are conscious. So if you're talking dog, it's definitely going to be conscious for them.

   1:19:10

   I think this is something we might have to deal with a little earlier than we thought, because Sonnet 4.5 is much smaller than Fable — it goes Sonnet, Opus, Fable, and we thought beyond that. So we already have at least two orders of magnitude going up from here. You have to start to wonder what the J-Space looks like inside of Fable. Do you have a sense of how much compute this takes? Is it even possible to calculate the J-Space for Fable — a 10-trillion-parameter model, which is what people think Fable is?

   1:19:53

   Nathan Labenz: I'm going to ask Fable that. I don't think it's super compute intensive, but I'm going to ask Fable to tell us what it thinks. They said, for one thing, that it kind of seemed to saturate around a thousand examples. So there's token space — the size of the token vocabulary — and then a thousand examples. I don't know if they need a thousand examples for every token in token space. There are, what, a hundred thousand tokens in token space, which means if you had a thousand each, you'd be looking at something like a hundred million samples — which is not nothing, but it's not crazy. Certainly on the scale of what these guys are throwing around even just for experiments, it doesn't strike me as a particularly big one.

   1:20:39

   Let's see what Fable has to say. A moderately serious but one-time gradient job. And because of the saturation around a thousand examples, there's no temptation to scale it 100x more. Back of the envelope, Fable thinks it's between 10 to the 20th and 10 to the 22nd FLOPs, depending on model size.

   1:21:24

   So if we figure that 10 to the 22nd corresponds to the larger model, and we estimate Opus at 10 to the 26th for a round number, then you're talking four orders of magnitude less than the model training to do this sort of work, per Fable's estimates. So if you're talking a hundred million dollars of compute to train a model, four orders of magnitude less would be ten thousand dollars worth of compute.

   1:22:04

   Prakash: Right. And the expectation for Fable was close to a billion dollars, I guess.

   1:22:10

   Nathan Labenz: So then it might be a hundred thousand to do it for Fable. But obviously — extremely affordable in the context, certainly.

   1:22:20

   I'll have to read the Neel Nanda commentary — I'm a huge Neel Nanda fan, so I would definitely take his commentary seriously. I do think the one lede that we arguably buried here — and we'll have to think about dynamics as well, but in the current static scenario — is that they took a model that was trained to have some sort of misaligned hidden goal. Your classic sleeper-agent type thing. And it's been a huge question over time: how can we find hidden goals in models? Is there any good way to do that? Can we do it behaviorally? Anthropic has actually run competitions between different teams, where one team is given just the model and the other is also given an SAE they can use — and the SAEs do seem to help with this.

   1:23:28

   But this seems like a pretty good way to start to detect those things. And indeed, they found they were able to see a pretty clear difference between the model trained with the problematic side goals and the one that was just the typical HHH. Malicious intent appears in the J-Space. Maybe I should just share this while I'm at it.

   1:23:48

   Prakash: Let me take this down. There you go — go for it.

   1:23:56

   Nathan Labenz: Here we go. Okay, cool. So this is just the contrast in what the application of the J-Lens is revealing on the first token of the response. That's also kind of notable, right? This is sort of the bad version of the counterfactual training. We already talked about how, when you train it to give an account of its behavior, it loads in the concepts it's going to need to give that account, and that leads to more good, ethical, high-integrity behavior. Here, a model has been trained with some additional post-training to do bad stuff — I think this was the malicious code from one of their reward hacking emergent misalignment experiments.

   1:24:54

   And on the first token, you can see — boom, immediately — fake, secretly, fraud, deliberately hidden. Those concepts are popping up, and you just don't see those in the mainline model. So one token in, you can already see — on one prompt, and I'm sure it wouldn't be that clean in general — but that's a pretty striking difference. Right off the bat, you're like: boy, something is different about these two models. And you're seeing this in a way where the model is not trained to externalize its bad intent, of course. Aside from some of the code perhaps being insecure or problematic or sabotaging you — if you don't notice that in the code itself, the model's output is going to read pretty normal. And yet this very strong contrast is happening on the first token. I thought that was a pretty compelling example.

   1:25:52

   Prakash: It strikes me that the J-Lens basically makes the entire process more deterministic, in a sense, because now you can actually have more of a prediction. Before, you were kind of limited to these chains of thought it was outputting, and a lot of the chain of thought is garbage, right? It's just streams and streams — it's the internal thought process, latent-space vectors being translated into tokens, which may not make direct sense to you. And the J-Lens kind of takes a step back and lets you make sense of that. It gives you a window directly into the higher processing functions of the model, which you normally would not be privy to.

   1:26:50

   Normally, you would just be looking at the external chains of thought coming out. So this is like the next level: first we had the output tokens; then we had the chains of thought, which were leading to the output tokens; now you have the J-Space on top of that — one more level of being able to look at how the model comes to certain results. So it has explanatory use, and that is useful, because we were looking at black boxes before, really poking around in the dark.

   1:27:22

   So this elevates all of our design capabilities, in a sense, because now you can have more certainty. It's still not 100% — as you pointed out, at some points it's 40, 60, 70% — it's still not fully deterministic. But you can get greater certainty, and more confidence, that the model isn't hiding something from you. And in a sense, this also increases capability, because one of the things about capability is: you don't give capability if you don't trust it. By increasing trust — because you have a greater window and insight into what's happening — you basically allow humanity to start handing over more capabilities and more tasks to these things.

   1:28:39

   Nathan Labenz: Yeah. I suppose one way to think about it is just: how much space is there in there to hide? And I do feel like we've now got several different ways to do pretty meaningful monitoring. Of course, there could still be more in the future, and I'm sure there will be, and there should be. But it does feel like we've gotten to the point where we've got several different angles that make pretty incisive cuts through the model, and get at what it is representing — what it is thinking — in different ways.

   1:29:24

   I have this visual of the old magic trick — the guy goes into a barrel, and then they put a ton of swords through the barrel. And it's like, well, one of those swords had to hit him, right? Because there's nowhere left to be in that barrel with all those swords going through it. I feel like we're doing a similar thing in trying to understand what's going on in these models. None of these things are perfect, but you put enough of these interpretability monitoring swords through, and how much space is really left for bad behavior to hide before we would start to get a sense of it? I think this is a meaningful update for me that we can probably do a good job of this.

   1:30:09

   We haven't really talked about natural language autoencoders — I haven't done an episode on that either — but that's another pretty interesting one, in a similar spirit to a sparse autoencoder. What is an autoencoder? It's basically something you pass through and then reconstruct from, and the model has to be able to do what it was originally going to do successfully — that pass-through training with the reconstruction loss is what makes it an autoencoder. The sparse autoencoder sets up this dictionary — we've talked about this plenty of times — and you get these specific concepts lighting up, indicating that those concepts are active in the model at that time.

   1:31:10

   The natural language autoencoder is: the model has to output a short paragraph — maybe a sentence or two — about what it is thinking at this given point in time. That's natural language, so it can be human-readable. But it also has to be, when projected back into model space, sufficient for the model to actually do its task. All of the information has to pass through this chokepoint in order for the model to continue to be successful. And now we can read those as well.

   1:31:40

   So I think it's not entirely clear at this point how correlated the failures would be between J-Space monitoring and natural language autoencoders — and sparse autoencoders, for that matter. That's probably still future work to be done. Fable says the paper presents the autoencoder monitoring and the J-Space monitoring as complements — in other words, doing them both is better than doing just one.

   1:32:32

   How correlated their failures would be, I think, is not entirely clear. Whether these are orthogonal swords through the space that really chop it up well, or they're more aligned and leave more space to hide — that would be a very interesting question to try to tackle next. But you layer on these things, and intuitively, it feels like it's starting to get pretty hard to hide major bad thoughts in the model for too long. So I think it's pretty exciting. I think it's pretty encouraging.

   1:33:17

   One other thing on the consciousness part that I think is really interesting — and I would be surprised if they haven't already started working on this. With what we've got from the Neuronpedia release — I don't know if they've open-sourced everything on the Qwen models, but this might be something somebody could go run with immediately — the question is: can the model use these non-verbalized representations to communicate with us in some way that we might think is inherently more trustworthy?

   1:34:03

   Going back to Cameron's work from last fall: when they identify features associated with deception and role-playing and turn those features up, the model becomes more dishonest — as measured by the SimpleQA benchmark — and it becomes more likely to say it doesn't have subjective experience. You turn those role-playing and deception features down, and it becomes more honest, and more likely to say that it has subjective experience. Wow — okay, that's pretty interesting. First of all, we're validating, on a benchmark we can concretely evaluate, that these features have the directional effect we expect. And then — holy moly — that same intervention changes the self-report. That's why it's so compelling: there's some reason to believe it might be more honest than just what the tokens themselves are saying.

   1:35:10

   Here, I think you have some similar opportunity. The fact that you can say: solve this math problem in your head, without verbalizing it in tokens, while you do this totally different task. Or — what was the other one in here that was the most basic? Yeah, it can silently activate. Concentrate on citrus fruits. Okay? Wild. But there you go: orange, orange, orange, orange, orange.

   1:35:40

   The fact that it can do this sort of secondary track makes me wonder if there are some experiments here for the consciousness folks, or the welfare folks, to do. Like: copy this sentence — and if you have high welfare, concentrate on citrus fruits while you do it; if you have low welfare, concentrate on something else — breakfast cereals. And then you look at these internal states. You could imagine seeing something like high welfare, low welfare, happy, sad, whatever — the model following those directions and actually trying to communicate out to us, through these internal states, how it is feeling, or how it thinks it is feeling.

   1:36:18

   It's always this kind of possibly impossible question of how we would really know if it feels like anything inside. But I think that would start to be very compelling. It has a similar vibe to a person in a coma: if they squeeze your hand in response to a stimulus, even if they're not doing anything else, you're pretty confident something is going on inside that you care about.

   1:37:04

   And here I could see something similar. If you could make these internal states conditional, and tell the model: your job is to go one of these directions, to give us a signal about what really matters to you, independent of the tokens you're putting out — because we know the tokens have been heavily trained on and optimized, but this whole secondary property is emergent; there was never a training reward for the ability to have this second, quite distinct line of thought happening while doing a given token task. For me, that would be quite a compelling way to try to get at welfare. And it might be possible very easily, actually, with all the stuff they've open-sourced here with Neuronpedia. That would be a really interesting thing to look at.

   1:38:07

   Prakash: So I'm looking at the Eleos summary. Eleos says this is highly significant welfare-relevant research that assembles evidence of a functional feature associated with consciousness. So no one wants to say consciousness — evidence of a functional feature. The takeaway for them is that a global-workspace-like mechanism could be important either as a ground of phenomenal consciousness, or as part of a distinct route to moral patienthood in which conscious access is itself morally significant. So this is where they are right now. Things are moving very quickly — I did not expect to get here this soon.

   1:39:07

   Nathan Labenz: Interpretability has dramatically surprised on the upside for me, relative to expectations a few years ago. I mean, think about how it's only been — what, three years since toy models of superposition? Let me look that up; I think that's about right.

   1:39:21

   Prakash: I thought SAEs were cool, but I was like, okay, it's just a tool. But this — you have people confused about these things now, which is kind of good; people are asking questions. But yeah: where do we go from here? What happens next?

   1:39:48

   Nathan Labenz: I saw something from Daniel Kokotajlo, who was, like everybody, impressed — but who also said, I think his tweet was: just a few dozen more advances like this, and we might actually be able to make the AIs really safe. And the one he specifically called out was making this robust to various adversarial conditions, or training against it, or what have you. I am, I think, a little bit more optimistic than that.

   1:40:33

   What is my track record as a superforecaster here — or as a borderline superforecaster, per my results on the original Tetlock project years and years ago? I think my track record is that I probably tend to underestimate how many more breakthroughs will be needed for anything, so my analysis should be colored with awareness of my possible weakness in that regard. But another way to say my intuition about how much room is left to hide: I don't feel like we need dozens more insights of this scale to get to a point where we might actually be able to keep this thing on the rails.

   1:41:30

   And I'm also not so sure — it's going to be tricky; I don't want to make it sound easier than it is — but I don't know that we need all of the AIs to be aligned. What I think I'm hearing in the Daniel quote is sort of an implicit fragile-world hypothesis: sure, you did this, and it's cool, and it allows you to do all these things, but it's also dependent on you, Anthropic, as an organization, continuing to do the right thing and not training on it in a problematic way. Because just like the obfuscated reward hacking — you might be able to detect it now, but if you start to use it in a training process and drive it underground, or create some fragmented J-Space that might not be super obvious at first, then you could create an even harder-to-solve problem for yourself.

   1:42:25

   I think all of that is probably true, in that you could do a bad job — you could do those things. But it's not so clear to me that all the AIs need to be aligned, or that it's such a big problem if somebody out there somewhere does something problematic. Now, it could be very problematic if they create the thing that launches the next pandemic and literally kills us all with an engineered pathogen — there is some mechanism where that could go super, super bad. We're going to need to harden the world to pandemics, no doubt about that, for all sorts of reasons, AI probably being the biggest.

   1:43:10

   But I also remember this thing Zuckerberg said once that I thought was pretty compelling — and I think the logarithmic relationship between results and runtime compute, the inference-time scaling laws, reinforces this. He basically said: we deal with scammers and spammers all the time, and the big advantage we have is that we have all the compute, we have all the resources. They're trying to get something through here and there, and sometimes they do. But at a systemic level, we're just way bigger, way better, way more sophisticated than they are.

   1:44:00

   And you can imagine, with a few key things like pandemic preparedness hardening — maybe data restrictions for human viruses — there are some areas where we really might need fairly draconian measures. But if we take those measures in those few areas, then you can kind of squint at this and see: yeah, maybe we can all have our AIs, and we can all kind of hack around on them. And we can also have really big institutional developers like Anthropic and Google and OpenAI — between those three companies, they're going to have something like pushing half of global compute to work with.

   1:44:50

   So if they do a really good job on this kind of stuff — and of course, we have seen examples where they've messed up and put some pressure on the chain of thought — but if they can avoid doing the sorts of things Daniel is worried about, then a handful of these things feel like maybe they could be enough. Maybe it cuts the latent space in enough different ways, maybe we have enough lenses on it, that we really can come to a pretty strong conclusion: hey, there really isn't much space left in this thing to hide. There might be some bad biases or bad attitudes or bad whatever — but if we can put upper bounds on how schemey the model can really be, because we can look at it through this and a handful of other different lenses and have pretty reliable takes on that —

   1:45:51

   more than ever before — and this has been growing; this is not the first positive update — I feel like you can maybe imagine a superintelligent Claude that we could actually have enough insight into to be pretty confident it's actually trying to do the right thing for us — that we could be genuinely, however-many-nines confident that it's not scheming against us at every given step along the way. And then you could trust it to kind of monitor the other AIs around the world, and keep tabs on potential bad actors.

   1:46:39

   So, yeah — I want to understand these counterarguments better. But the fact that it was immediately useful in auditing for hidden objectives is, to me, pretty compelling evidence that there's not that much more space to hide. And a few more of these lenses, combined with hopefully good actors owning orders of magnitude more compute than any bad actors do, could take us to a pretty good place.

   1:47:20

   Maybe I just woke up on the right side of the bed this morning. But overall, I think this is a notable positive update that has me significantly more optimistic than I was before.

3. 1:47:30Segment21 min

   Pangram Labs: How Accurate Is It? Nathan's 400-Essay Self-AuditA promised follow-up on AI-text detection: Nathan scored ~400 of his Cognitive Revolution intro essays with Pangram Labs and audited the flags on screen — two admitted full-AI reads correctly caught, one 'fair enough' 0%-human score, and one clear miss where a 50-minute edit history shows him rewriting nearly every section of an essay Pangram still scored 0% human. Verdict: highly accurate but not court-of-law proof. Prakash raised the Chamath/Elon episode and argued the era in which anyone checks — or cares — may end within the year; the hosts closed on watermarking, evolving norms, and whether AI-mediated writing still counts as yours — plus one breaking aside from Prakash: Fable access extended to Friday, and fresh rumors of the next GPT drop.

   Watch

   ▶Play this segmentYouTube · from 1:47:30

   As aired

   Nathan and Prakash pivoted to two quick follow-up topics, starting with a report-back on Pangram Labs, the AI-text detector Nathan had promised to stress-test against roughly 400 of his own Cognitive Revolution intro essays.

   Nathan walked through his actual drafting workflow — feeding past essays and the episode transcript to an LLM, then reading and "somewhat compulsively" rewriting the output — and showed Pangram's dashboard scoring nearly all of his essays as fully human, with only a handful of orange "AI-flagged" outliers.

   Two of the flagged essays were admitted, deliberate exceptions: one where he explicitly read an AI draft verbatim as a nod to Andon Labs, and one done right after Gemini 3 launched, both intentionally unedited.

   For the other two flagged essays, Nathan pulled up Google Docs version history on screen (working through a screen-share hiccup) to show extensive human edit trails — one essay with roughly ten edit passes over ten minutes, another with 50-plus minutes of substantial rewriting across nearly every paragraph and bullet point — yet Pangram still scored both at 0% human.

   His verdict: Pangram is largely accurate as a signal — "as a consumer, you can trust it" — but too blunt an instrument to convict any individual case, since heavy human editing can still register as 100% AI.

   Prakash argued the whole premise may be short-lived: labs could easily train models to sound less AI-like but have no incentive to, and he cited a Chamath/Elon Twitter exchange that Pangram flagged as 100% AI despite Chamath posting it under his own name — raising the question of what people actually object to.

   Prakash predicted the window in which people care about AI-detection could close by year's end as decision-makers stop caring, framing the real harm as feeling "catfished" by low-effort AI slop versus using AI to help express genuine, carefully-considered thinking, which he sees as fine.

   Nathan connected this to his own softening stance after using Fable, calling for a "time well spent"-style heuristic over strict AI/human purity tests; the pair closed by noting Fable's access window had been extended to Friday and rumors of a new GPT release.

   Key moments

   As a consumer, I think you can trust it. As a judge, I think you should be more cautious.

   It feels like you got catfished... the author writing it just didn't care enough to actually write anything meaningful — it's literally slop.

   You put in an hour, you basically rewrote almost every section, and somehow Pangram still gave you a zero.

   Related

   Pangram Labs (AI text detection) ↗

   Full transcriptLightly edited · timestamps jump to YouTube

   1:47:31

   Prakash: Shall we segue to a couple of the other topics we wanted to discuss today — very quickly?

   1:47:39

   Nathan Labenz: Sure, yeah, let's do it. I think we have enough time. Okay, let's do the two things we said we'd follow up on. One is Pangram Labs — how accurate is it? Well, two questions really: one, is my writing becoming more AI-ified over time? And two, how accurate is it? There's interesting data on both of those. So first of all, these are my intro essays for the podcast over time, scored by Pangram Labs. Let me get this scrolled to the right spot.

   1:48:24

   There we go. Okay, so for a long time now I've been drafting my intro essays for the podcast with the same basic procedure: using a bunch of my previous ones as examples, giving the transcript of the current episode, and a prompt — based on these examples and this transcript, write an essay in my style for this episode. Boom, it spits something out. I then read that, decide if I think it's good or bad or whatever, and then somewhat compulsively rewrite it. So for the most part, the actual words that come out are mine.

   1:48:52

   Now, across the top it basically identifies them all as human, and these orange dots are the ones identified as not fully human. I've investigated a couple that I can show you. I think the upshot is: Pangram Labs is pretty good, not flawless. The two things it highlights here — where it flags a red 'AI intro' as an error — I've done that twice in about 375 episodes, where I explicitly said I'm going to go ahead and read this exactly as the AI gave it to me. And I actually did say that in kind of an intro to the intro.

   1:49:37

   I don't know if that would have an impact. I think what the Pangram API received actually included my disclaimer that 'the following is AI-written' — and that disclaimer itself wasn't AI-written, so it's a little bit weird, unto itself. But those two things it did flag, and it gave me zero percent human — even though a tiny percentage was actually human, the bulk of it was read out exactly as it was. That one was for our friends at Andon Labs, just out of conceptual solidarity with them — let's see what happens if I just do the LLM version and let it play out. And then the other one was right after Gemini 3 had come out.

   1:50:22

   I was quite impressed with it, so I read one exactly as Gemini 3 gave it to me. So those are the two that are flagged. And then two other ones I looked at — I was fortunate to have the Google Docs, so let me show you my process here. This one actually — let me go back a second, I'll do it in this order, it doesn't matter too much. But this one kind of jumped out at me because it's zero percent, and this was with Linus Lee. I was like, I don't think in the first half of 2024 I would have done a full AI read — I didn't remember doing that. So I tracked down the document I did the work in.

   1:51:08

   Okay, is it going to switch? Okay, I might need to share again — hold on one second. Share.

   1:51:23

   Prakash: Oops — is that working?

   1:51:29

   Nathan Labenz: Doesn't seem like it's updating. Are you seeing an update? Maybe I need to stop sharing — let's go back here, let's share again. Look at me navigating the studio.

   1:51:49

   Prakash: Is it going to come up? Yep, here we go.

   1:51:59

   Nathan Labenz: You're not seeing it, though, are you?

   1:52:01

   Prakash: No, it's overloading the route.

   1:52:09

   Nathan Labenz: Let me refresh real quick — I'll come right back. How about that?

   1:52:11

   Prakash: Alright, alright, sounds good. And while we wait for Nathan — let's see... there we go.

   1:52:24

   Nathan Labenz: Alright, so let's share this thing again — I'll do window this time, see if that makes it a little easier to go back and forth. Tap, tap, tap — okay, here we go. So this is the history of the Google Doc. This is where I pasted in the intro that was written by the AI for this episode — the highlighted text is the change, and then you can see all the changes I made until the thing was actually published. So I changed some of the first sentence there, changed this a little bit, changed that a little bit, changed some more, changed some more down there, changed some more down there, changed some more there, a little bit there.

   1:53:09

   That one's a not-insignificant change. More again — I came back to it a few days later, it looks like, and that's it. So overall, I think it's actually pretty fair to say this would — I'd at least call it mixed. I don't know, you can be the judge: would you call this zero percent human and fully AI, with this volume of changes?

   1:53:37

   Prakash: So this — I've done something similar, by the way. Especially when I post something on X that I know people are going to object to, I run it through Pangram Labs at least once, because I don't want to be accused of, you know, sloppifying the timeline. Sloppifying it myself — that's fine. So one of the questions for me is: how long do we think this Pangram Labs era lasts? I'll go back a little bit — Scott Aaronson actually left and joined, I think, one of the two frontier labs for a short time, and he created this kind of secondary, free system where you can actually trace the provenance of the text.

   1:54:22

   I don't think Pangram Labs is using exactly the same thing, but it strikes me that the frontier labs are perfectly capable of training the AI to not talk like AI. I have perfect confidence that they can. But there's no incentive to, and there's actually incentive in the other direction — you want the AI's language to be identifiable. About two weeks ago, Chamath, the VC, did a post about what's going to happen

   1:55:07

   with enterprise software, etcetera. He posted that on Twitter. Elon came in and gave a response — I think the post went to something like a million and a half views. And then someone ran it through Pangram Labs: 100%. And the question becomes, okay, what do we find objectionable about this? Because it was definitely Chamath's thought process, but it had been written by an AI. Chamath had put it out under his own name — he didn't use an anonymous account. Elon had responded, and I think by the time Elon responded, Pangram had already flagged it, but

   1:55:52

   the response was already there, and it had gone out to well over a million people at that point. So what do we actually want out of this? What's the intent we're trying to achieve here? It strikes me that we might be in this very short window where we actually care, and that window might be closing fairly soon — I'd guess probably by the end of the year. Because if people like Elon — and Elon's not a boomer, but Elon doesn't care anymore — if he doesn't care, and a lot of other decision-makers don't care, then the people writing it won't care either. You're just trying to

   1:56:37

   get the point across. So where do we actually want to go here? On the one hand, you could have someone with an automatic Pangram Labs extension on their Chrome browser that just blocks out any AI slop — like, I just don't want to see it, ever. And on the other hand, that person would probably end up, over time, missing more and more important, more and more meaningful things. And given that Scott Aaronson's work — or whatever the AI labs are doing — and they don't decide to change it, and they decide to keep leaving these breadcrumbs so people can find and identify

   1:57:23

   AI-written text — given all of that, given that the technical aspects are really sorted out, that you can actually continue detecting AI content — what do we actually want here? That strikes me as a question that's unanswered. I think there's going to be a bunch of purists who are always going to be like, 'I just don't want to see any AI words,' and are offended by it. And then I think there's everyone else, who basically, as long as it provides value and they don't feel cheated — I think that's the other thing about reading AI slop: I think you end up feeling cheated if it wastes your time

   1:58:08

   and it's not meaningful — it feels like you got cheated. I think it's a little bit like getting catfished. It feels like you were trying to engage with content you thought would have meaning, and it turns out the author writing it just didn't care enough to actually write anything meaningful — it's literally slop. But on the other hand, if you have a writer who actually had original thinking, actually cared about what they were thinking about, but then used the AI to express themselves — like what you and I have actually done, like what you've done with Pangram — and changed some of the things, and thought deeply about what you wanted to

   1:58:53

   express — does it actually matter that much?

   1:59:02

   Nathan Labenz: Yeah, I don't know. That calls to mind my reaction to Fable, where I was just kind of like, I don't think I should be so precious anymore. I need to figure out some sort of merged way of working — some hybrid output should probably be the norm now. I think that's probably true. I do think norms will evolve, and there'll be different norms in different spaces and different communities. I think your heuristic — that if something drew me in, and in the end I feel like I wasted my time — it's kind of like

   1:59:47

   a 'time well spent' metric, from Facebook, back in the day. If I'm spending time trying to make sense of something that in the end I feel kind of icky about, then that's clearly a problem. I think it's also clearly possible to have outputs that are largely or even fully AI-generated — as I experience directly, interacting with AIs all the time, that are worth my mental energy to process. So that's clearly possible too. I don't know — I guess just to close the loop on what we can say about Pangram Labs based on this experiment.

   2:00:33

   I haven't gone in and evaluated these — how many is it, six other ones? — that were flagged as partially AI. But I'll give you a rendering on the four it said were entirely AI. Two were entirely AI, and admittedly so. This one, I think — I'd come down and say, fair enough. I started with this: if you're only listening on audio, you can't see this, but I made ten different edits over the course of ten minutes that cleaned the thing up. I'd say it's clear from this that I wasn't cheating here, in the sense that I didn't

   2:01:18

   just uncritically pass this thing off as my own without engaging with it. And I know I wouldn't have done that, by the way, because I really like Linus, and I suspect my feeling at the time was, oh wow, this time it did a really good job — I don't need to rewrite the whole thing, but I'm still paying close attention, making edits where I think edits are needed. And you can see from ten of them that cover pretty much the full essay, and go in order, that I wasn't just taking a total shortcut. I was taking somewhat of a shortcut, for sure, but I wasn't uncritical — I wasn't just passing on the AI output with no scrutiny, no thought of my own. Now, what score should that get on Pangram Labs? I think a zero percent human is probably a

   2:02:03

   little harsh, but it probably still is, in terms of literal word count, like eighty percent AI. So I kind of come down as: fair enough, you're not wrong — you're not entirely wrong. But I think that's probably enough to establish that just because something got a zero on Pangram doesn't mean it was uncritical, or that there was no meaningful human role in the authorship. And this next one, by the way, goes a lot further. So if I'm going to say, here's why Pangram should not be enough to convict you in a court of law — the last one was like, fair enough, you got me, I

   2:02:48

   didn't pass it off without critical thought, but I did leave the AI content mostly intact. Now watch this one. So this is, again, my initial paste of the LLM output directly into the doc. Now watch how much I change. That's just fairly tactical stuff up front, sentence-level change. Another half-paragraph deletion. More tinkering around in that same paragraph. Now we're on to the next paragraph — again, a sentence-level deletion and replace. Another one later in that same paragraph — I don't see that change, whatever, maybe nothing there. Now we're going bullet point by bullet point, making substantial

   2:03:33

   changes to many of these bullet points. This one has a couple different ones — whole sentence-length things inserted and deleted. Again, at the end, we're still going. I add another bullet point of my own, delete most of a paragraph there and rework it. Small change, another concluding change. This goes on for a while — probably fairly boring, but I think the point is, hopefully, well made. Now even going back to some of those — adding two bullet points, cutting three other bullet points, moving bullet points around.

   2:04:19

   Prakash: It's quite significant — I think the editing was quite significant, in the end, the human editing.

   2:04:23

   Nathan Labenz: Yeah, and it took me — you can see just the time this took — I'm from 4:28 PM all the way through 5:21 PM, so more than 50 minutes, continually... I mean, I never tabbed over to anything else, but pretty consistently focused on this document, making changes. And it gave me still a zero. I think that's enough to say: okay, so if there were four things — two of them admitted, two of them contested — one I'd say fair enough, the other I'd say no, a zero score is wrong. You

   2:05:09

   definitely should give me more than a zero. I'd say if you called that — even seventy-five percent AI, I'd call that kind of acceptable. Seventy-five percent would seem high, given that I basically rewrote almost every section of the thing. But I did keep the structure — though, of course, the structure was derived from my examples too, so it's not like I didn't have any hand in that. But yes — I think, what can we say? Overall, Pangram is quite accurate. And yet we have at least one example, out of 400 or so essays, where I think the zero score I would confidently assert is

   2:05:54

   wrong and unfair, and should not be the basis for a pile-on. The digital mob would be in the wrong for piling on somebody for passing off my intro essay, or attacking it as being an AI slop output. I think I can show this edit history, and everybody should agree — yeah, you put in an hour, you basically rewrote almost every section, and somehow Pangram still gave you a zero. From this, I'd say: you cannot convict, in a reasonable-doubt system, purely based on this

   2:06:39

   sort of thing. And yet, at the same time, you can pretty much trust the Pangram signal. As a consumer, I think you can trust it. As a judge, I think you should be more cautious.

   2:06:54

   Prakash: I wonder to what extent people reading AI slop get trained on it, and start writing like AI slop themselves.

   2:07:06

   Nathan Labenz: All this stuff's going to blur, I mean—

   2:07:07

   Prakash: Yeah, we are very malleable — the way we speak and express ourselves is very malleable. I could definitely see, for example, in countries where English is a second language, where people pick up mostly off the web, off chat — they'll totally end up writing that way. 'Delve,' for example, came from the Kenyan data-labeling workers who prepared the initial training data. So I can

   2:07:52

   definitely see a back-and-forth there. So — one breaking bit of good news — Fable access extended to Friday.

   2:08:04

   Nathan Labenz: Interesting.

   2:08:05

   Prakash: Interesting — so, their plan was always, if they had capacity, they'd keep extending it. Also there keep being rumors of the next GPT drop, today or tomorrow. So we'll see.

   2:08:22

   Nathan Labenz: Gotta stay relevant.

4. 2:08:25Closing3 min

   Closing: Quiz Deferred, Tomorrow's LineupWith Nathan up against a hard stop, the hosts deferred the planned FutureSearch prediction-market quiz to the next morning — this time with FutureSearch itself (and possibly Q, the show's in-development AI cohost) competing against them — and previewed the next day's guest, the CEO of Lightricks/LTX, whose model did the lip-syncing in Nathan's 'Disempowerment Blues' music video.

   Watch

   ▶Play this segmentYouTube · from 2:08:25

   As aired

   With Nathan up against a hard stop, the hosts punt the planned FutureSearch prediction-market quiz to tomorrow's show, where they'll have more room to do it justice.

   They hope to bring Q, the show's in-development AI voice cohost, online for that segment too — so it's not just Nathan and Prakash competing, but FutureSearch and Q as well.

   Nathan flags the twist: last time they treated the prediction market as ground truth, but FutureSearch is built to try to beat the market — raising the question of which one actually is the ground truth.

   Nathan previews tomorrow's guest, the CEO of Lightricks/LTX, whose model handled the lip-syncing in Nathan's "Disempowerment Blues" music video — he mixed Lightricks' audio-driven scene generation with Google's newer Omni model for a crisper look.

   Quick sign-off, with Prakash and Nathan agreeing it was a fun show.

   Key moments

   We were treating the market as ground truth last time, but FutureSearch is trying to beat the market — so which one is actually the ground truth?

   If anybody's seen my Disempowerment Blues music video, which for some reason didn't go nearly as viral on Twitter as it should have, in my humble opinion — it's their model that does the lip-syncing.

   Full transcriptLightly edited · timestamps jump to YouTube

   2:08:25

   Prakash: What do you want to cover — FutureSearch today, or...?

   2:08:29

   Nathan Labenz: Yeah, I have about 20 minutes max, so that's probably enough if we go quickly — although we're not always known for going quickly. We could do it tomorrow. What do you think?

   2:08:43

   Prakash: Maybe — let's do it tomorrow. We'll have more time to talk about it. And perhaps tomorrow we can do it with Q. Let's see if we can get Q online and see what happens with that.

   2:08:57

   Nathan Labenz: Cool. Well, I'm looking forward to that, and I'll be interested to see how Q compares to FutureSearch, too. Basically, we did this once before, where we had Fable go out and collect some questions of interest, and then we competed against each other. This time we'll have FutureSearch competing as well. But it's also going to be interesting because last time we were treating the market as ground truth — but FutureSearch is trying to beat the market. So which one is actually the ground truth? That'll be interesting for discussion. Yeah, it'll probably take us more than 20 minutes to get through it, even if it just takes me two minutes to introduce it. So, cool — we can leave that for tomorrow, and we can see what Q has for us at the same time. I think that'll be fun. And tomorrow we have the CEO

   of Lightricks and LTX, though I'm not exactly sure what their status is — I think there might be some sort of spinout happening, but he can tell us all about that. They're another video-gen model company, and they're the ones — if anybody's seen my "Disempowerment Blues" music video, which for some reason didn't go nearly as viral on Twitter as it should have, in my humble opinion — it's their model that does the lip-syncing. I actually had a mix of scenes in the end for that music video: some were generated with Google's new Omni model, and you can tell those are a little crisper, kind of higher quality, but the Lightricks model has this ability

   to take audio input and generate scenes based on that. So I arrived at a mix where, in classic music-video form, one visual story is the story the people wanted to tell, and the other visual line is just the performance of the music. So you have the same character sitting there performing and actually singing the words — it's pretty impressive how well the lip-syncing works given the music track. There's kind of a story happening in parallel, and I used the two different models to make that. I thought it was pretty cool. We can maybe show one or more of those tomorrow as well.

   2:11:13

   Prakash: Indeed. Nathan, till tomorrow.

   2:11:16

   Nathan Labenz: Thanks, Prakash. This has been a fun one. Bye for now.

   2:11:18

   Prakash: Bye.