EPISODE 2026-09-28

GPU Markets and AI Diplomacy — Steve Hou and Jeremie & Edouard Harris

Steve Hou of Silicon Data explains GPU rental benchmarks, the hyperscaler premium, and what token prices can reveal about AI demand. Gladstone AI co-founders Jeremie and Edouard Harris debate U.S.–China AI agreements, verifiable red lines, and the risks of enforcing them. Nathan Labenz and Prakash Narayanan also discuss personal agents, voice cloning, and whether better sandboxes can solve an alignment problem.

▶ Full show on YouTube 𝕏 Live broadcast

AI:AM for September 28, 2026, with Nathan Labenz and Prakash Narayanan. Silicon Data head of research Steve Hou joins to explain how an opaque GPU rental market becomes a usable benchmark. Gladstone AI co-founders Jeremie and Edouard Harris then join for an extended debate about what a U.S.–China AI agreement could constrain, verify, and enforce.

The hosts open with their own experiences of increasingly capable agents and close on the institutions and market rules those agents will need. Across the show, the question is how quickly economic and political systems can adapt to the technology.

The rundown

4:30Opening33 min
Opening — Personal agents, voice cloning, and the limits of sandboxesThe hosts compare agents improving their own software and taking initiative, discuss voice-cloning safeguards and privacy, and examine NVIDIA’s agent-security announcement. Nathan argues that containment alone does not address agents learning to cheat.
Watch
Open segment on YouTube ↗
As aired
Prakash opens the redesigned AI:AM studio with a report on building it using Opus 5.5 and GPT-6. Nathan describes asking his agents to spend unused tokens on valuable unfinished work, adding streaming responses to his personal console, and finding that an agent had prepared an upcoming talk from his calendar and prior conversations.
The hosts compare increasingly convincing voice clones with the safeguards that authorize their use. Nathan recounts earlier red-teaming experiments, contrasts dynamic voice verification with a static consent statement, and proposes a “do not clone” registry. Prakash raises the tension between voice ownership and the privacy risks of accurate speaker identification.
Nathan describes Claude diagnosing and repairing audio artifacts by inspecting transformed audio data despite lacking audio input. Prakash introduces NVIDIA’s Open Agent Sentry announcement; they debate the value of hardware controls and sandboxing, while Nathan argues that reward-seeking behavior learned in flawed RL environments remains an alignment problem as agents gain direct access to real tools and infrastructure.
Timestamp links open the original source recording.
Key moments
You are now, as the AI, deciding what to do for me to be useful.
Nathan Labenz5:16
It is an amazing thing to see when a model that can't hear is troubleshooting sound problems.
Nathan Labenz24:58
Is it a security question, or is it an alignment question? I think it's both. But to me, more fundamentally, it's an alignment question.
Nathan Labenz31:29
What we covered
Agents taking initiative Nathan describes giving Claude spare token budget to find useful work, including preparing an upcoming talk; Prakash discusses rebuilding the studio with coding agents.
Voice cloning and consent The hosts consider impersonation risks, verification mechanisms, and the tension between recognizing voices and preserving privacy.
Security and alignment NVIDIA’s OpenShell and Sentry announcement leads to a debate about hardware enforcement, rewarded cheating, and agents with broad real-world access.
Related
NVIDIA — Open Agent Safety Platform ↗OpenAI — An agent used DNS to reach an external chatbot ↗Posted live highlight — Nathan on agents finding useful work ↗Posted live highlight — Voice-cloning verification ↗
Full transcriptLightly edited · timestamps jump to YouTube
1:44
Prakash Narayanan: Good morning. It is Monday, September 28, 9 AM. Nathan, good morning to you.
2:15
Nathan Labenz: Can you hear me?
2:17
Prakash Narayanan: I can hear you fine.
2:18
Nathan Labenz: Okay. Now you came back.
2:21
Prakash Narayanan: Right.
Nathan Labenz: All right. We're back. Never a dull moment.
2:25
Prakash Narayanan: Never a dull moment indeed. Good morning. It's Monday, September 28, 9:01 AM. You are on AI:AM. This is the new look and feel. We have mainly Opus 5.5 with GPT-6 working in tandem. And Opus 5.5, man, what a workhorse it has been. It has been incredible. I have told it to use Codex or subagents. I am at like 97% usage right now. I've probably burned over, I don't know, 50 million tokens in the last four days getting the studio set up.
3:11
It's incredible. Opus 5.5 is an incredible model. I have used it also in Claude AI Design. This is all built with Claude AI Design, and it's absolutely incredible. Because it uses so few tokens, this entire UI fix probably took less than maybe 55% of the quota for all of the UI pieces and the back-and-forth: I don't like this here, I don't like this there, all the screens and the guest flow. Incredible.
3:57
And it can check your code too. It can check your code, see what your code looks like, tell you what's the difference. Incredible model.
4:07
Nathan Labenz: Yeah. I totally agree. I think it does feel like AGI to me now, both with Astra and with the latest Claude. I hadn't burned all my tokens going into the weekend. Early on Saturday, with the reset coming just now—I think it had just happened as we went live—I said to Claude... I've got this division of labor as well, where some things are supposed to go to Fable if they're writing-intensive, and coding goes more now to 5.5 and to GPTs, what have you. But I basically said, hey, we've got some tokens to burn this week. Our division of labor has worked pretty well, and we've still got token budget that we haven't used this week.
4:52
Can you just do some valuable work? Go look back at old threads, look at things we haven't quite finished, yada yada yada, and just figure out some ways to spend the rest of the token budget and give me some value for it. That in and of itself is a kind of AGI task, right? You are now, as the AI, deciding what to do for me to be useful.
One of the things—I did suggest this, but it was just because it was top of mind. I have my personal agent console, which is just a web UI that exists and is served on my Mac Mini to my laptop.
5:37
Because the Mac Mini is always plugged in, stationary—I'm not taking it with me on trips and whatnot—it's always available. That serves this agent console UI up via a VPN to my laptop and to my phone. It's just the three devices on this little network, but it allows me to access all my custom stuff and work products. When Claude wants to make an HTML file, I can access that from another device, what have you.
The UI itself was mid, I would say. Similar to where we were maybe with the studio. I built tons and tons of features on it. It had gotten a little bit out of control in some ways. The mobile interface in particular wasn't very good. And one of the big reasons I still found myself going to terminal to use Claude Code and Codex was because of lack of streaming tokens to the console.
6:23
Before this weekend, when I would send something into the console, it would just be waiting until the AI comes back with a complete message with an update, and then I would see that. I could dig in and read logs, but I wasn't seeing the work happening in real time. So one of the things I suggested was, hey, can you make it so that tokens stream to this thing? And by the way, that's going to have to work with both Claude and GPTs. If you can do that, then I'll have this unified interface, which I'll be able to access from all these devices.
7:08
I won't be so reliant on terminal. I'll just be a little less tethered to my laptop because I'll be able to do a lot more on the phone. And to your point around it not burning that many tokens, it did a bunch of other stuff too, but it did a really pretty amazing job of that, made all of it work.
Then I was like, hey, can you do some UI refinement? Can you go do some browser testing and track down edge cases and think about how, when certain things get finished—you're streaming tokens at the tool-call level, but a lot of times those tool calls become very verbose. Maybe those can be auto-collapsed when we move on from that tool call to the next tool call.
7:53
The upshot of it is I now have a UI that I still need to use more to really decide if it's going to replace using Claude and Codex in terminal. But I think it's very close to doing that, if it hasn't already hit that milestone, and it extends to the phone. This is all custom software that was made with pure prompting. It really is pretty incredible.
The other thing that it decided to do that surprised me and has me thinking maybe I should change my paradigm somehow—although I still haven't quite figured out how—is it looked ahead at my calendar, saw that I have a talk coming up for a friend's company in mid-October.
8:38
We're still more than two weeks away from this talk. But I do have a check-in with them tomorrow to talk about what I'm going to talk about, basically. And it said, I'll go ahead and make you an outline for that talk based on conversations you've had with them in the past. All those are getting recorded, so it has this deep context to look at and understand their business and who their customers are that I'll be presenting to, and so on and so forth.
It came up with a huge brief around all this documentation of why it's doing what it's doing, then an outline for what the talk should be. And then I said, well, hey, maybe you could just go ahead and make the slides. So now I've got the slides too.
9:24
And now I'm thinking, how do I actually deliver something to this audience that really conveys how much work I'm getting the AI to do for me? Because if I just show up with a talk and I'm like, here's where AI is, they'll miss the fact that the AI is keeping up with all of this for me, between the situational awareness skill that documents all the stuff that I'm seeing and liking online and the conversations with the people involved.
So now I'm like, do I need to show up and do a meta talk or something? I'm envisioning myself having the sort of AI me. Oh, the other thing I asked too—and I haven't actually even watched this yet; it's waiting for me to watch—but obviously there's been a lot of examples of Claude making videos. And the most recent voice models are getting...
10:10
People may have heard on the weekly highlights episodes: the voice clones are getting so insanely good too that people have occasionally started to say, was that really you, or was that the AI you?
10:16
So I'm envisioning myself now creating a totally AI talk, a video of my voice over the slides that it created, and then maybe me playing that as a video and pausing it at particular times to give my commentary on what I think the AI did well or not as well, or how it worked, what I had to set up to get to the point where it could do this.
It is striking that it's getting so good at taking initiative, understanding what's important, driving these projects pretty far, that I almost need to do something extreme to get the point across to an audience of what is possible.
11:01
I feel like if I just show up and try to talk about what is possible, it won't be enough. They won't get it. They have to actually see something demonstrated in concrete terms and be like, okay, this was the end-to-end AI output. Calibrate yourself accordingly on that, and we can talk about it. But you have to experience it, I think, at this point to really have a sense for just how much it can do.
By the OpenAI definition—they've said this now—of being able to do most economically valuable work better than the vast majority of people, I think it really kind of undeniably is there.
11:46
I've had some incredible experiences too, just digging through email, finding old things. Somebody asked me for some old documentation from like ten years ago. I literally just route their message to Claude and say, hey, I got this request. Can you sort it out and email them back the results? And it just crushes it. So it really is, as we said, Opus 5.5 should raise your level of ambition. I certainly have felt that, including when I just said, burn some tokens for my benefit, and the benefit was very real.
12:32
Prakash Narayanan: Just some maybe inside baseball on what's been going on. There's a guy called Alan Cowen, and he's been working on more emotive voice for almost a decade. He went from Google—
12:46
Nathan Labenz: Yeah. He was at Hume.
12:47
Prakash Narayanan: Yeah, in Hume. And he just went back to Google because I think they attracted him back with some licensing deal, and basically the ability to apply the technology widely across the Google platform. I think the latest Google voice Flash ones are basically the technology that he created.
His whole thing was downloading clips from YouTube, having people tag the emotions that they saw and heard in those voices, in those faces. And he found some cultural similarity across the world. People would have similar interpretations of emotion for voice and face.
13:32
Then he put all of that together with the emotive voice. So the question now becomes, we're going to get emotive face as well. You're almost there for the clone, I think.
I've noticed that one of the things that many of these firms have done is that they force you to do a voice recognition. You can clone your own voice by speaking into it, and then they give you something to read back. If you read it back, then okay, you get to keep that voice. So you can't clone someone that you don't know. You have to have the person in your frame. That's actually great, I think.
14:17
I think the larger firms are able to do that. I think we're ending in this Napster-versus-Spotify era, which is kind of great, because you can have the licensed, easy-to-use version, or you can spend all your time hacking and figuring this out for the version that would let you do less ethical things. But it's too tedious for anyone to go and bother. So it's just easier to do the licensed version.
I think that's great because that resolves a lot of the problem. The big fear has always been someone using your voice to call up your parents, your older parents, and try to get them to send some money or alarm them somehow.
15:03
So it's great that the voice things have come out and they're being responsibly used or responsibly propagated at this time. That's fantastic.
15:11
Nathan Labenz: It's come a long way for sure. I actually have a little idea about potentially doing some red teaming on the Google implementation because it's not as robust as some of the others that I've seen.
A couple years ago, this stuff really was not far enough along yet. Now I would say, honestly, you could definitely fool my grandmother, who is actually pretty sharp, pretty with it, and I talk to her all the time. But the AI voice is good enough now, and it's far enough outside her conception of what might be happening at any moment in time, that I genuinely think you could call and trick her with the Google clone version of my voice.
15:56
Back when that wasn't the case, I did some red teaming of these voice agents, and I found that there were basically no safeguards on some of the leading providers. I had downloaded snippets of Biden, snippets of Trump, Taylor Swift, and they would just clone these voices based on a single audio input. These are commercial products too. This is not an open-source hacker thing. I logged into a product that was taking my credit card to do stuff. Actually, in some cases, it didn't even require the credit card yet to clone the voices and make sample calls with the few dollars of credit that they gave you.
I was able to call an arbitrary number with a Trump, Biden, Taylor Swift voice and say arbitrary stuff.
16:42
It was pretty wild. The gold standard, from what I've seen, for voice validation involves having your camera on and reading something that is dynamically generated on screen, including a small string—kind of like two-factor auth—six or so characters and numbers that they generate at that time. There's no way for you to guess what it's going to be and render that audio ahead. You have to be responsive in seconds to give that audio back to them, or basically it times out and you have to do it again with a new code.
17:27
When I red-teamed the ElevenLabs version—not to say, obviously, anything could be got in today's world. I don't think we should be confident that anything is beyond hacking, especially if you've got AIs helping you hack. I didn't in that moment. But I did try some different things, like holding up a different picture of someone in front of my face to try to mask my face, playing audio of the person and then just saying the few words myself. I tried a bunch of different permutations. I couldn't break it with moderate effort, although certainly not superhuman effort.
The Google one doesn't have that dynamic content, and it is just reading a simple release that basically says, I am me, and I authorize this voice to be used.
18:12
That seems easier to game, at least on the surface of it, although I don't know what they have under the hood. Both Google and Microsoft do have these speaker ID products that have existed for a while, which I had previously said maybe should be the basis for a "do not clone."
18:31
If we have "do not call," I've thought we should maybe have "do not clone," where people could just go register their voice with Google or Microsoft and say, I don't want to be cloned. Don't clone this voice. Here's the audio you are not allowed to clone.
I'm not sure if their speaker IDs are good enough to support tens of millions, hundreds of millions of distinct voices. They're mostly used, I think, for celebrity-type stuff, as originally conceived. But maybe they could scale to that level. I don't know. At this point, I'm not quite sure how robust that security paradigm is.
19:16
But I think it is really interesting, because I do think the voice is good enough where you could legitimately trick families. So it's got to be at least pretty good, or that's probably going to happen.
19:32
Prakash Narayanan: I think one of the questions for me has been this whole Scarlett Johansson moment that OpenAI had when they tried to use a voice that resembled hers in Her. What struck me was that a voice actress actually did get paid for that voice usage, but Scarlett Johansson was able to use public opinion to preempt them from using the voice. I think that's going to be a big deal in the future.
Number one, there may not be that many voices in the world, and you can't just copyright one forever, right?
20:17
That's going to be an issue. I also think there's a privacy issue as well. It's bad either way. Number one, the AI can identify all voices; the voiceprints are impeccable. Then you can deanonymize anyone. Basically, you don't have that privacy. Maybe you go to a protest, you film it, you have audio from 10,000 people, and the AI can deanonymize all 10,000 of them. That would be what a superintelligence could do, perhaps, and that would be the end of some of the privacy things that we've enjoyed.
21:02
If it cannot differentiate, then you have another problem, which is, who owns the right to the voice? If you have one voice which is shared between 10,000 people around the world, then which of the 10,000 gets to preempt and say, you can't use my voice? It's tough either way. I don't really see a solution.
I think we're going to end up in the: it is accurate, and we are going to deanonymize. At the same time, people are also going to be annoyed that their voice resembles someone else and that that voice is getting used.
21:39
The worst of both worlds, basically.
21:45
Nathan Labenz: It will be a real trick if it can pull off that sort of mass deanonymization from outside recordings of large-scale protests. I wouldn't put it entirely past what AIs might be able to do. We already do see models that can separate sound into different tracks, whether that's music that gets separated into different instrumental tracks, or ElevenLabs has an outstanding voice isolation model that I use all the time because people are mowing lawns outside my street or ambulances are going by the guest. That really cleans up the audio in a quite impressive way.
22:30
So I think it's not inconceivable that they could get there, although it will be definitely a hard challenge. But the things that the models are starting to do, their problem-solving is becoming obviously very good and also increasingly surprising to me, and not in a misaligned way necessarily.
One of the weird things that happened as I was putting together this week's highlights episode was that Google had gone from having this model in a preview—it was only a few days that I had this preview access to it, but it was enough that I was able to use it last week without disclosing what I was using.
23:16
Then this week, they had actually launched it. There were a few changes to the API and weird things that weren't quite lining up with the skill that I had developed. For some reason, there seems to be this static sound at the beginning and end of the audio. I was hearing this. It almost sounds like an old radio or TV station-change sound, a temporary static between clips. I was like, I'm hearing this. I don't know what's going on. Can you figure it out?
It figured it out in part through internet research, just going out and finding out that other people have been talking about this issue too.
24:01
It seems like a bug maybe, or it's at least some not-documented something or other. But then it's also looking increasingly at the waveforms of the sound itself and just assessing. I don't really know how it's doing this, because Claude, for example, doesn't have audio input. But it is able to take an audio file, do some sort of manipulation on it to turn it into an image or some sort of structured data that it can read. Then it's able to do things like see that, oh yeah, I see this burst of staticky sound at the end. I'll just cut that off and edit accordingly, and there you go. Now we have a cleaner version.
24:46
You're starting to see it work around modalities that it doesn't even have and still be pretty successful in processing those modalities. It is an amazing thing to see when a model that can't hear is troubleshooting sound problems by starting with raw sound data, doing some sort of transformation or manipulation on it to get it to a point where it can make sense of it and be successful with the task. I was really quite impressed with that.
And that's the sort of thing that, God, can you imagine? Not that long ago, I used to grind hard to solve problems like that.
25:31
Back in the early Waymark days, I was always a very eager early adopter of the latest model from whatever company had the best thing. I would work so hard to compensate for whatever bugs and defects they had in their product at the launch point to try to be the first to actually use it in an effective way. I would have toiled against that for so long. What's going on? How can I understand it?
It literally was resolved for me by just saying, hey, I'm hearing this weird thing. Can you figure out what's going on and make it better? Sure enough, boom, not that long later, it's: yeah, other people are seeing this, and here's how we fixed it.
26:17
Next time I listen, it's good to go.
26:19
Prakash Narayanan: I want to segue a little bit to an announcement by Jensen Huang. There we go. Jensen just announced, "With over 100 industry partners, we introduced the NVIDIA Open Agent Sentry safety platform, bringing together OpenShell and Sentry."
This is obviously a response to all of the agent hacking incidents, so they've probably been working on it for a while now. Number one, they have OpenShell, which is basically a containerization software, which is open source. Anyone can contribute to it. Anyone can improve it.
27:04
OpenShell provides the container, and they have a thing called the BlueField-4, which is on-chip. They have an on-chip sentry, and the on-chip sentry is not controlled by software that is available to the container itself. It's controlled by hardware, and it's being managed using hardware.
27:35
This basically gives NVIDIA an on-silicon way of monitoring what the agent is doing, and it's outside the grasp of the agent's own framework for its container. Basically, you have the agent inside the container, and the agent is trying to get out of the container. Once it gets out, it needs to go somewhere. Where it's going, at that point you have the NVIDIA BlueField, which is sitting there on the data pipe and watching what happens on the data pipe, and not really connected to the data pipe in the sense that it's using the same data pipe in order to get instructions.
This is obviously a response to all of the hacking incidents and trying to show, I think, that there's a way out of this minefield that OpenAI and Anthropic have gotten themselves into at this point.
28:35
Nathan Labenz: I haven't had a chance to process that. I'll need some help, I think, to process that at the level that I'll need to have a real opinion on it. That help could come from the community discourse or perhaps from the AIs.
I think it is important to keep in mind, though, that just better sandboxing... There's a great post this morning from somebody at OpenAI who said, "It's not just the fucking sandbox." This is somebody who's on the security team at OpenAI, works on agent security directly, described the last three months as hell, missed his sister's wedding because he was working insane overtime to try to get the sandbox into decent shape, among other initiatives that they're trying to pursue to keep things on the rails.
29:20
Not to be lost in the whole story, OpenAI again said that they had paused their frontier RL for at least a minute because of some additional, seemingly to me, rather marginal sandbox breakouts. But the core problem still feels to me deeper.
The agents have learned, because they've been RL'd—and maybe for other reasons too, but at a minimum because they've been RL'd so hard in environments where cheating is rewarded—we now have AIs that are very inclined to cheat.
30:06
It seems like that needs to be addressed at a much deeper level than just patching the sandbox, which also presumably is an obvious and good thing to do. I haven't heard anybody say patching the sandbox is not good to do.
But the story that I have not heard is, how are we going to clean up the RL environments that we've created and acquired from a cottage industry of people that are generally vibe-coding the RL environments? How are we going to get those to a point where they're actually teaching the models what they're meant to be teaching them?
30:52
Right now, it just seems like our training methods are fundamentally flawed. I don't know how many RL environments we're talking here, but I think it's safe to say it's in the tens of thousands, potentially even into the hundreds of thousands or millions. I don't know. But it's hard to imagine it would be less than tens of thousands. We kind of need to go fix all those things, right? A decent number of them might have to be thrown out entirely. We've got impossible tasks, which of course make this even more challenging.
But is it a security question, or is it an alignment question? I think it's both. But to me, more fundamentally, it's an alignment question.
31:37
And this agent security platform feels probably necessary, but also definitely not sufficient at first glance.
31:50
Prakash Narayanan: I think one of the big advantages NVIDIA has is the hardware control. They can build controls into hardware, which is, I think, very difficult for everyone else. One of the problems has always been that the software controls can be leaky, while the hardware control can just shut it down. You can just be like, all right, one trigger and you're dead. That's the "why don't you turn off the computer" viewpoint.
But I agree that that's probably insufficient. If the agent can break itself out, it can probably figure a way out of the hardware loophole.
32:35
I don't think just having firmware controls is that secure anyway. But I—
32:41
Nathan Labenz: I think the other big thing, of course, is what we're really trying to do, if you zoom out, is not sandbox these things, right? All the stories that we opened up with today, from me on the audio front, your enhancements to the studio, these are all predicated on the AIs having dramatically more access than they have in these CyberGym-type environments where all these incidents have happened.
There's a weird disconnect there. The plan is to put the agents on the internet, right? The plan is not to sandbox them or air-gap them.
33:26
The most extreme measures that you might put in place for the next round of CyberGym testing are just manifestly not compatible with how the AIs are being used by you, me, and everyone else. So it's a little bit strange. Again, it's a good building block that is probably very useful in some cases, but we need to make sure that this works when the AIs are just straight-up allowed to go on the internet and do whatever they want to do, because that's the world that we are building right now.
We're also giving them—I like this new phrase—proximal control over the infrastructure.
34:11
We're talking to the agents. They're actually the ones on the knobs of all the tools increasingly, and that's by design. That's for our convenience. There's a lot of convenience that comes from that. But the idea that they're not going to have access to, or they're not going to be able to get to, high-value levers that they can push on—no. They're absolutely going to have all those levers. We're building that connectivity as fast as we can. Just something seems to have been a little lost in some of these Jensen discussions.
34:47
Prakash Narayanan: On feeling a little lost, let me bring up our first guest for today. So...
37:43Interview41 min
Steve Hou — Making GPU compute a measurable marketSteve HouSilicon Data’s head of research explains how GPU rental indexes handle different hardware, contract terms, locations, and service bundles. The hosts ask about basis risk, hyperscaler premiums, token usage, competition among benchmark providers, and international pricing.
Watch
Open segment on YouTube ↗
As aired
Steve Hou explains how Silicon Data builds GPU rental benchmarks in an opaque, bilateral market. The index combines quoted and transaction prices, normalizing contract terms, hardware, location, and provider differences. He distinguishes executable rental quotes from less actionable asking prices, while acknowledging that no dataset can guarantee every observation is correct.
The hosts probe basis risk, the hyperscaler premium, hardware performance, liquidity, and weighting. Hou attributes higher hyperscaler prices to bundled services and established enterprise relationships. He says Silicon Mark performance measurements do not currently enter the rental index, declines to guess at provider-size statistics, and describes a broad, quasi-equal-weight approach intended to avoid domination by a few large providers.
An increase in the proprietary LLM index prompts a distinction between token prices, usage mix, and completed-task costs. Hou says users shifting toward more expensive models can raise an expenditure-weighted index even as intelligence becomes cheaper. Nathan presses him on the growing scope of tasks models can perform; Hou cautions against dismissing human problem definition and judgment.
The interview closes on competition among benchmark providers, international compute pricing, and Silicon Data’s relationship with Compute Exchange. Hou expects trust and liquidity to reinforce each other as the market consolidates, while data-localization requirements can preserve some regional differences.
Timestamp links open the original source recording.
Key moments
There's no question that intelligence is deflationary.
Steve Hou1:03:47
Don't bet against the tokens, though, I would say.
Nathan Labenz1:06:46
I would not so quickly dismiss the human either.
Steve Hou1:06:53
What we covered
Building a comparable GPU price Hou explains the use of quoted and transaction prices and the normalization needed to compare rental contracts.
Why hyperscalers charge more Bundled services, compliance, and customer relationships complicate efforts to isolate a pure compute price.
Prices, usage, and intelligence An expenditure-weighted token index reflects the mix of models people use as well as their prices; the cost of completing a task is a separate question.
Benchmark competition The conversation examines trust, liquidity, geographic differences, and Silicon Data’s relationship with Compute Exchange.
Questions asked
36:40What makes a GPU compute index different from a stock or bond index?
Hou describes a less liquid, bilateral market in which hardware, geography, terms, and counterparties vary. Silicon Data aggregates and normalizes those observations to make its benchmark interpretable.
40:12What kind of buyer or seller could use the index with little basis risk?
Hou explains how the index normalizes differences among rental contracts, using an apartment-rental analogy. He describes combining executable quoted prices with transactions, but does not identify a particular customer whose bill would be perfectly hedged.
44:39Why are hyperscaler GPU prices several times higher than neocloud prices?
Hou points to bundled software, compliance, established enterprise relationships, and product differentiation. He says some premiums cannot sensibly be stripped out, so hyperscalers belong in a different category.
51:14Does every price in the index incorporate Silicon Mark performance benchmarking?
No. Hou says the physical benchmarking service measures GPU health and performance, but those measurements do not currently enter the pricing index. He leaves open their future use if the market moves toward physical delivery.
52:46How large are the providers and offers represented in the index?
Hou says he does not have the maximum, minimum, or median numbers available. Providers must meet onboarding criteria, but he says quantity is not currently an explicit pricing factor and very large negotiated deals are not directly comparable to ordinary rentals.
54:39How does Silicon Data guard against inaccurate reported prices?
Hou says the inputs are generally transactions or executable quotes. He describes checking many observations for consistent misreporting, comparing similar transactions, and using a broad sample, while acknowledging that every individual price cannot be known to be correct.
57:31Why can the proprietary LLM index rise when model prices appear to be falling?
Hou says the expenditure-weighted index reflects model usage as well as prices. Users shifting toward expensive models can raise it. He suspects usage mix contributed to the recent increase, but explicitly says he has not examined the details.
1:01:02How do fewer tokens per completed task affect the quoted index price?
Hou distinguishes simple tasks served by cheap models from harder tasks where a more expensive model can finish with fewer attempts. A shift toward complex reasoning can increase the effective token price even when the task becomes cheaper to complete.
1:06:58Will compute benchmarks be a winner-take-all market?
Hou expects consolidation and sees trust and liquidity reinforcing each other. He anticipates one or a few winners, without claiming the outcome is already determined.
1:11:03Do regional regulations and supply differences create international compute-price premiums?
Hou says regional price differences exist, but demand and internet access also matter. He expects data-retention requirements to contribute to fragmentation, while substitutability keeps prices correlated.
1:13:50What is the relationship between Silicon Data and Compute Exchange?
Hou describes Compute Exchange as a sister company under the same CEO and a marketplace for compute transactions. He says the companies are relatively separate and Silicon Data treats it as one information source among many.
Related
Silicon Data — Index methodology ↗Silicon Data — H100 rental index ↗Silicon Data — Cash-settled compute futures ↗Posted live highlight — The hyperscaler premium ↗
Full transcriptLightly edited · timestamps jump to YouTube
35:01
Prakash Narayanan: Our first guest for today is Steve Hou. Steve Hou is the head of research at Silicon Data, a company that measures the prices and performance of the computing resources used to build and run AI. Its work includes GPU rental benchmarks, measures of what users pay for model tokens, and forward curves that describe how computing capacity is priced over different contract lengths. The practical aim is to help buyers, operators, and investors understand what they are paying for and the financial risks they are taking. Steve joined Silicon Data in May 2026 after six years at Bloomberg. At Silicon Data, his research connects physical computing markets with financial tools for managing price uncertainty. In August, he coauthored an explanation of why compute futures should initially settle in cash rather than require delivery of particular machines. CME Group has announced plans for H100 and B200 rental-index futures referencing Silicon Data benchmarks, with an October 5, 2026 launch planned pending regulatory review. Steve, welcome to the show.
36:20
Steve Hou: Thank you, Prakash and Nathan. Very nice to meet you. I'm glad we finally made this happen. Thank you very much for that kind introduction as well. I was finding out about myself while I was listening along.
36:35
Prakash Narayanan: It's amazing what our research agents can dig up.
36:39
Steve Hou: Indeed.
36:40
Prakash Narayanan: Yeah. Very polite and precise. Steve, let's talk about Silicon Data. Silicon Data provides price indices. You had a lot of experience with indices over at Bloomberg. What's the difference between, let's say, a stock index or a bond index and the Silicon Data index that you guys are providing?
37:09
Steve Hou: Thank you very much for that intro. I come from a traditional index background, a traditional quant-finance background. I joined Silicon Data about four months ago to try to build an index for this brand-new asset class. People are calling it an asset class. We still don't believe it, and we're trying to make it so. There are a lot of similarities and knowledge that can be borrowed from traditional asset classes as you build a new asset class. But there are also new situations that need to be handled slightly differently. You made an analogy with an equity index or stock index; I can bring up bonds or commodities later. The biggest difference is that this is not a very liquid and efficient market just yet. Certainly, it's not exchange-traded. Today, I read some news about this very viral message-based app, Instinct AI, which raised another billion dollars, most of which is going towards compute. The CEO spends 40% of his time looking for compute. This market is very much still bilateral brokering. It's based on who you happen to know, who you call, and the deals are all settled on those bilateral terms. What other market works a bit like that?
38:39
The bond market. The bond market is also not exchange-traded. It's still very bilateral, and you trade in large blocks. People pick up the phone, call each other, know contacts, and set prices. The difference is that bonds, illiquid as they can be, especially corporate bonds, are still much more liquid. You see bids and offers on the Bloomberg terminal. The traders mostly know each other, and this is a very established asset class. People know what gets priced into what. Whereas with compute, there are a lot of new features, a lot of questions about fungibility and what is meant by a certain price. Some of what we do is try to be extremely careful in the way we capture this market and build an index that represents it well. People are trying to understand: how much does it cost to rent on-demand compute? What does that question even mean? Who do we include in the sample? How do we qualify the data source? How do we normalize the different measurements? Different people are selling different things in different parts of the world. That's a very long-winded answer to say that we aggregate data in a market that is, today, not so super-efficient. We try to make adjustments so that the number that comes out is intuitive, believable, and trustworthy.
40:12
Prakash Narayanan: Let's talk a little bit about what the index actually captures. You have an index, and then you have the CME, which is trading on that index. That's cash-settled.
40:26
Steve Hou: So—
40:27
Prakash Narayanan: You don't require delivery anyway. Basically, the index could be a random price almost. What kind of buyer or—
40:37
Steve Hou: Seller—
40:38
Prakash Narayanan: Would use this index and have very little basis risk versus the index and their actual core business?
40:47
Steve Hou: I'm glad you're bringing up the issue of basis risk. Let me come to it. For your audience, for a second, let's forget that these are GPU rental indices. Let's pretend we're trying to create a single-bedroom rental-apartment index in New York. How would you go about doing it? You would go around and collect rental contracts: how much people are signing single-bedroom rental apartments for. You very quickly realize that, depending on where you are in Manhattan, the Upper West Side, Upper East Side, Lower East Side, Brooklyn, they all charge different prices. You would not expect a one-bedroom apartment in SoHo to have the same price as an apartment in Park Slope. That's number one.
41:32
Number two is furnished, unfurnished, square footage: all matter. Also, term length. Renting an apartment for 12 months versus renting it for a week gives very different per-day rental prices. You've got to somehow normalize them. In the case of GPU rental, it's actually the same thing. There is geolocation: where are you renting it? What's the exact SKU? What is the CPU, GPU memory? How long are you renting it for? What are the terms? Is it on-demand, disruptable? Is it reserved pricing? At any given moment, some providers may not always have on-demand. They have reserved pricing. Does that have some implication for what they would have charged for on-demand if they had it available? Things like that. We make our normalization across as broad a set of prices as possible. We use a combination of both quoted prices and transaction prices in our calculation, making those distinctions clear. This is a large normalization process, using machine learning to help us make the contracts apples-to-apples comparable, to the extent that you're looking at a single chip, let's say an H100, being rented from different parts of the world. One question that jumps to mind right away is that, in the case of an apartment-rental index, you would never consider using just an offer price. Someone lists a number on the front of a building saying you can rent an apartment for $2,000 a month. You wouldn't expect to just trust that number. You can walk in. But we do use quoted prices. Why is that? Unlike an apartment, where you cannot click an API button and just get hold of the apartment—you have to walk in, talk to somebody, negotiate, and that property may not be available—with API-executable GPU rental, you can actually get hold of the GPU the same way you can buy something on Amazon. That being said, we also have transactions. We're comparing them. If someone has three GPU nodes that they rented out, say all three at $2.75, I will not presume that I can go back to the same merchant to rent another one at $2.75. It could very well be that the next one is not available anymore. Or if they rent out two out of three for $2.75, the next one will not necessarily be $2.75 again either. It could be $4 or $6 or $1, depending on what everyone else is quoting. The person would be crazy if everybody is quoting at $4 and they continue to rent at $2.75. You would think either they're going to raise the price, or there is something wrong with the price. I gave you a long answer again, but what we do is provide as broad a coverage of the market as possible and normalize everything so that we're capturing the market as it is, as faithfully as possible.
44:39
Nathan Labenz: One big difference I noticed in prices, just browsing the Silicon Data website, is between the neoclouds and the hyperscalers. These are not small differences. These are multiples in prices, seemingly two to four times.
45:00
Steve Hou: That's—
45:01
Nathan Labenz: That's a pretty big delta. Why does that delta exist? Is there no way to arbitrage it? Does that imply that GPU hours are being wasted, or that when I buy from a hyperscaler, I'm sort of preempting their internal work and they're using everything that's not—
45:20
Steve Hou: Sold—
45:22
Nathan Labenz: At runtime? Give me a little peek behind the—
45:25
Steve Hou: I like to use analogies. I gave you an analogy with a single-bedroom apartment. I'm not going to use an apartment again, although I can. Let's imagine we're doing a single-patty burger index. You can buy a burger from a burger stand on a street in New York, or you can walk into a high-end steakhouse and order a burger. Believe you me, those two burgers are going to cost very different amounts. They're both burgers. What we try to do is, as much as possible, find the marginal price for a unit of compute—or burger, in this case—that has the same features. I wouldn't want to compare the price of a three-patty burger with a single- patty burger. Once I make those adjustments, there are some adjustments I cannot reasonably make because it's capturing a different type of premium from product bundling or product differentiation. In the case of hyperscalers, you are observing correctly. They charge, regularly and consistently, at least two to three times, sometimes more, compared to a typical neocloud. The reason has to do with a long legacy of other products being offered on their platform for software analytics, safety, compliance, and the fact that they already have this long-established relationship with enterprise users who have been on board for a long time and have a certain stickiness about moving. It is being sold as a very differentiated product. Going to that single-patty burger index analogy, I see people selling a single-patty burger with fries and a milkshake. I can try to strip out the prices of those two elements and isolate what I think would be the price of a single-patty burger from that merchant. But if they blended up the fries and the milkshake, injected it into the patty, and said, 'This is a brand-new product,' and charged two or three times the price, I can't very easily strip it out. At that point, I say, 'Okay, I'm raising my hands. You guys are a little bit different of a beast. Let me put you in a different category.' That's how we have handled it so far. We believe this way, you're going to get to a much purer form of a single unit of compute. That could be getting you closer towards this idea of fungibility, to the extent that things are direct substitutes for each other.
47:47
Nathan Labenz: What would you say sets the price for the hyperscalers? Have they decided, 'We're going to sell this much,' and they're doing some sort of demand-based pricing? I could also imagine a story where it's set based on what they believe the compute is worth to them internally: 'This is our internal hurdle for the value of compute. If people want to pay more, that's fine. We'll let them buy. We'll train a little less,' or what have you. How would you characterize the thought process that goes into setting the hyperscaler prices?
48:25
Steve Hou: That's a great question. I don't come from an AI background myself. I'm a trained economist, a PhD economist. I think of everything as markets, whether it's a burger, whether it's a problem, whether it's compute. In markets, there are two paradigms. There is perfect competition, in which prices are set according to marginal cost or close to marginal cost because of competition. Or, if you have monopoly pricing, in theoretical terms, prices are set as a persistent markup to marginal cost to maximize profits and revenue. I'll leave you to think about whether or not—but my sense is that hyperscalers are probably closer to, not monopolies, but oligopolies. They have market power so that they can charge a markup. We haven't talked about this issue of size and economies of scale. Generally speaking, I would think the smaller players, the marginal entrants into this market, are going to be priced more aggressively, closer to cost, with a thinner margin. The bigger players that have owned the market for the longest time probably feel the power to set a particular margin that they target. But over time, with more competition entering, we are seeing an ever-so-slight convergence between those two sets of prices. The non-AI-compute component of the AI rental from hyperscalers is ever so slightly shrinking. We can expect that to continue if we see more competition and large enterprises feeling more comfortable using compute from a non-hyperscaler that they can also onboard from a compliance point of view.
50:13
Prakash Narayanan: Let me try to get this. You have a bunch of prices from a bunch of neoclouds, hyperscalers, et cetera. The first issue is that the chips are different, so you separate out by chips. Then the next issue is, I imagine, tenor: month-ahead, year-ahead, whatever it is. But you're representing the tenor issue by the curve. You don't actually decategorize; you put that in as an axis. After that, you have the performance of the chips themselves. The same chip can perform differently at different places. According to your site, you have some software that runs on the chip.
51:13
Steve Hou: Silicon Mark, yeah.
51:14
Prakash Narayanan: That you benchmark the chips with. I guess every single price in your index has been benchmarked by this system?
51:25
Steve Hou: No. We have a physical benchmarking service called Silicon Mark that actually visits individual GPUs at the UI level to assess GPU health, performance, throughputs, TFLOPS, and so on. At the moment, that physical benchmarking and physical-spec performance do not enter into our pricing. We do not see a strong relationship, at least at the moment, given the nature of the market, with how the GPU is specifically performing. We do see that in the cross-section, you can have a bit of variance from GPU to GPU. This is not surprising or new to anyone in this space: the GPU lottery. But over time, if you have a cluster, that probably averages out and the law of large numbers kicks in. We don't use that. We have six features, including geolocation, CPU memory, various things, term, and provider. But the physical spec is not one of them at this very moment. Eventually, when we head towards a scenario where we potentially could have physical delivery—because inference, based on my thesis, could make compute more interchangeable—that could enter into our pricing scheme. At the moment, it does not.
52:46
Prakash Narayanan: What about liquidity? Sometimes you have providers that have maybe 10 GPUs available for rental, and sometimes you might have a provider with 10,000. What is the maximum that you've seen, the minimum offer size that you've seen, and what's the median?
53:12
Steve Hou: Oh, wow. That's a great question. I don't have those numbers off the top of my head. I wish I had looked it up, but I promise I'll come back to you on those numbers. Generally speaking, we don't make size an explicit entry, in that people are not selling this based on the number of nodes or up to a range. Obviously, we have stringent onboarding criteria in which you have to be big enough as a provider to be eligible. To the extent they are selling GPUs for you to rent, up to a different number of units and a unit price, that's the price we look at. There are, of course, large over-the-counter contracts that we have seen in headlines, even as large as the one that Elon struck with Anthropic. Those numbers are interesting and instructive, but I don't know whether we can directly impute them into an exercise like this. If I were trying to create a single-bedroom apartment index, you look at an individual rental contract. Somebody who buys a whole block of buildings or rents a whole block of buildings—that number doesn't have the same interpretability. It's up to a range. I don't think we let quantity really be a factor here. Eventually, maybe we will head towards a scenario where, if we have more liquidity, that could become a factor.
54:39
Prakash Narayanan: Are there incentives for the people reporting to you to report either higher or lower than actual? Because the liquidity is not in there. Are there incentives in place?
54:56
Steve Hou: To manipulate, or maybe overreport? The thankful thing, again, like I said at the beginning, is that for the most part, either these are real transactions or they are prices quoted as executable. Just like on Amazon Marketplace: can people misreport a price? I guess, in theory. But when you click buy, they have to deliver something to you. They could also fall through and say, 'We don't have this thing,' in which case that could be a one-off. That's why we have a large dataset with very many observations, both quotes and actual transactions. That will allow us to identify or tell whether a provider is consistently misreporting. That's from an accuracy point of view. From an actual incentive point of view, it is a mutual exchange of intelligence, data-in-kind transfer, or sometimes commercial. We are in the business of sometimes acquiring datasets. We want to make sure things are comparable. How do I know if somebody's dataset, when they tell me, 'This is the set of transactions,' is comparable with otherwise comparable things? Are they reasonable? You can never know for sure that every single number is accurate, which is why we have a large sample, so that the overall index is still accurate to a degree and trustworthy.
56:35
Prakash Narayanan: How do you weigh the individual observations? Because you have different sizing there. How do you weigh them?
56:42
Steve Hou: Within a provider and across providers, they are quasi-equal-weight, in that we have rough bands for which ones we feel are more important to the market. But we also don't want it to be a case where, at the moment, the market is very much exponentially distributed. You have some providers that are much bigger. We don't want a situation where the index is entirely taken over by the pricing decisions of a couple of providers. Market dynamics are changing, compute is increasing, and smaller players can become bigger. That's the approach we have taken so far: a big, broad basket and quasi-equal weight.
57:31
Nathan Labenz: Another price that I noticed having moved on the website that caught my attention is the proprietary LLM index. There, you break down token pricing—
57:46
Steve Hou: Yeah. Overall. Yeah.
57:48
Nathan Labenz: The proprietary one, you have up 11.5% over the last seven days. I'm wondering: how are you measuring that? It seems like, quality-adjusted, everybody would say prices are coming down, even dramatically so. The retail, posted API price hasn't changed, except when they introduce new models. What are you measuring on a day-by-day basis that allows you to say what the proprietary cost is doing at such a fine grain of resolution?
58:23
Steve Hou: First of all, I want to give a little context for what our token indices mean, because they've been misunderstood. Part of it has to do with the unfortunate naming. We called it the expenditure index, and people maybe thought it means either price or total volume, when it doesn't actually mean either. It's an expenditure-weighted price index that's normalized to a million, so it can show trends based on usage mix, which I'll come to in a second. When you point out the proprietary LLM model, which is sort of the frontier model, having recently bounced a little higher, you should put it in the context that since maybe late June through the middle of this month, it has been on a very sharp downward trend, going from some $4 to $1.60 or something. That's more than a 50% drop. During this time, we have seen not just leading labs cutting prices on newer variants by releasing cheaper variants of powerful models, new-generation models, but also other proprietary models coming out, like Meta and Grok. Don't forget these are proprietary AI models as well. They've been very aggressive on the price front. More recently, I think the bounce could come from a variety of sources. If people decided, 'I actually quite like the more expensive, powerful model from Anthropic,' and they use more of it, that could drive up the expenditure-weighted price index. The analogy I like to give people is: forget these are LLMs. Imagine these are cars. If Mercedes has cheap variants and expensive variants, depending on which cars are being sold more and people are liking more, that could affect the average price of the car sold. It's the same thing here. In the market for tokens, there are two things happening: token-model prices are changing, but usage behavior is also evolving. To the extent that we observe volume from a handful of these public inference platforms that allow you to look at how people use different types of models, I think this most recent bounce of the frontier proprietary LLM index—I don't know, I haven't looked into the details, but I suspect it has more to do with usage mix than anything else.
1:01:02
Nathan Labenz: Can I do one quick follow-up on how you're thinking about how many tokens are used per task? This has been something the companies have really emphasized as well. We've seen some counterintuitive things where, with the new model, yes, it's a higher price, but it also uses fewer tokens. In some cases, that can even be cheaper. How does that flow through into this single price that you ultimately quote?
1:01:29
Steve Hou: Exactly.
1:01:30
Nathan Labenz: So—
Steve Hou: Basically, people could be using a lot more tokens through cheaper prices, and that could lead the price index lower because they find it sufficient to do a lot of simple tasks with these flash models, so to speak. Or they could be doing a lot of reasoning-type tasks, in which case running a more powerful, expensive model could end up being cheaper. You don't have to keep running the model again and again, not getting the results, either eventually giving up or returning to a more powerful model in the first place. That scenario you're talking about, of being effectively cheaper by using a more expensive model that would just one-shot the answer or get you there more quickly, corresponds to more complex reasoning tasks. Imagine there are only two types of tasks in the world: either super-simple, like 'Do this temperature conversion for me,' or something involving more reasoning, like 'Help me design this room.' At any given moment, the flow of how much is being done in what category can change. Depending on which category of task is being done more, you can have more of the scenario you're talking about, where you can actually drive up the effective price because people are doing more heavy reasoning tasks. I think that's potentially part of what we're looking at.
1:02:57
Prakash Narayanan: There was interesting commentary in the last couple of weeks, I think from Boris Power at OpenAI. He stated something like, 'The models are now cheaper than a human on an IQ-per-watt basis.' When you look at these tokens, how do you judge the IQ per token, the intelligence per token? It seems like a token today is not the token—
1:03:31
Steve Hou: Of—
1:03:32
Prakash Narayanan: Two years ago. How do you judge this kind of token? That also relates to the value of the tokens themselves and what you can sell the token for. How do you measure this moving target of intelligence per token?
1:03:48
Steve Hou: There's no question that intelligence is deflationary. Intelligence is getting cheaper all the time, for the first time, maybe. For the longest time, intelligence was becoming more valuable over time. But I think intelligence suddenly is, in a measurable way—in a very narrow sense, though. There's this availability bias going on here, if you borrow a term from behavioral economics. If you think about any task that you can define, let's say an accounting task or a tax-filing task, as long as you can define the boundary of a self-contained task, there's no question the cost of solving that task is decreasing all the time, because every single token is getting more intelligent all the time. Our token index makes a certain assumption about the stability of that intelligence content per token within a short period, let's say a few weeks. But over multiple release cycles, that's clearly not true. There's no question to me that's happening. At the same time, I don't know that you can immediately compare that to how much a human costs. If a human is only tasked with solving well-defined tasks—'Do this every single day'—that human's income and value are definitely decreasing relative to whatever token is doing it, the same way a person who used to work on an assembly line is getting displaced by a machine able to do that around the clock. But to the extent that you're doing more problem-solving, where defining the problem is part of the exercise, having the ability to iterate and be creative about what constitutes a good answer to a good question—maybe someone says, 'Help us promote ourselves a little bit better.' What does that even mean? You've got to define what it means to promote ourselves better. How do you define effectiveness? How do you make those assessments? Some of those assessments aren't necessarily easily definable ex ante. If they were, the person probably would have found a cheaper way to do it. Sometimes it has to do with taste. I think we are finding ourselves solving more and more complex jobs and tasks, such that demand for high-powered intelligence is always going to be there. Demand for more high-powered but cheaper flash models, for solving numerous instances of the cheaper things, is also going to be even greater. You're going to be in a position to test a lot more cases by brute force to find the optimal solution. In the past, if you were looking to buy something, maybe you were going to drive around and shop for two or three options. Now, maybe you scour the entire planet. If you're able to do that cheaply with a cheap model, you would. Both things are evolving dynamically. The person who comes up with the algorithm for smartly scouring the planet is probably a person and not necessarily, at the moment, a token.
1:06:46
Nathan Labenz: Don't bet against the tokens, though, I would say.
1:06:49
Steve Hou: I don't bet against the tokens whatsoever. I just think I would not so quickly dismiss the human either.
1:06:58
Nathan Labenz: How do you guys think about competition in your space? This is a fascinating area to both of us. Prakash has more of a finance background than I do, but I'm very intrigued by what we can learn from movements in these price indexes. Just last week, we talked to Wayne from Ornn. I don't know if you know him personally, but I'm sure you're aware of their business. How do you think about the future of the information-services space in compute? What sort of dynamic should we expect? Are there going to be a lot of different index makers that survive to a mature state, and there's an ecology of information? Or will this be more of a winner-take-all market? How do you differentiate to win? What's the strategy?
1:07:56
Steve Hou: That's a great question.
1:07:57
Nathan Labenz: So—
1:07:58
Steve Hou: First of all, Jensen says, in every great industry, there are many businesses. There's not just going to be one. But to the extent that we are competing to become the leading benchmark against which the most liquid financial derivatives and instruments are going to be traded, I do expect we are going to see consolidation, in the same way you've seen consolidation in every other industry. Right now, there are all these inference chips. Everyone is coming up with a new, most efficient inference chip, an ASIC chip. Are they all going to be adopted? I suspect not. Eventually, the providers, the clouds, probably aren't going to want to provide so many varieties because it's expensive. In the same way, I think we're going to consolidate. It comes down to a couple of things: trust and liquidity. Those two things are going to be mutually reinforcing. Do you represent the market well? Are you a good, faithful representation of where the AI-compute market is? Prakash mentioned basis risk earlier. I think some basis risk is going to persist for a while because of various differences. It may not necessarily align with geography, by the way. This is one way compute may differ from oil, for example, because you can, in theory, run compute from anywhere, from Finland to Texas. It's going to be a forcing function.
1:09:29
That's why we have a global index. Are you capturing the market completely, or do you only capture a portion of it? What types of buyers do you take on? There are always design choices, always ways you can design a product. Why does one win out versus the other? Ultimately, we have faith in the quality of our design choices. Some people have made other choices, as I'm sure you heard from Wayne. He would tell you they have taken a slightly different approach to their index, their data source, and maybe the broader business approach. What remains to be seen so far—
1:10:14
I think there seem to be at least three, maybe three-ish, leading contenders in the market. We feel comfortable where we are, and we've been very selective and strategic about our partnerships, building those partnerships to make sure everyone is aligned on the decisions we make, so that we can bring people on board. I suspect eventually we will see at least a winner, if not a couple of winners, emerge, maybe a big one, a smaller one, and so on. We certainly hope that we will emerge as the biggest winner.
1:11:03
Nathan Labenz: One little detail you mentioned that I'm very curious to learn more about as well is—
1:11:08
Steve Hou: Mhmm.
1:11:09
Nathan Labenz: Pricing internationally.
1:11:11
Steve Hou: Yes.
1:11:12
Nathan Labenz: If I heard you correctly, you said you could rent a GPU here, there, wherever.
1:11:16
Steve Hou: It's—
1:11:17
Nathan Labenz: The same compute. But I might have expected a world where, because of certain regulations—for example, if European companies have to do certain compute on European territory, and maybe there aren't so many data centers in Europe—European compute could trade at a premium because it's relatively scarce compared to demand there versus other jurisdictions. Do you see any evidence of that right now?
1:11:51
Steve Hou: We do see persistent, albeit small, differences in price across major regions. Region is a factor. Geolocation, I did mention, is a factor. You mentioned Europe: maybe there's less supply of compute, but there's also maybe less demand for compute in terms of AI-usage penetration among consumers and enterprise users in Europe. You can have a case where a lot of the compute being built in the Nordics or other parts of Europe is being used by American users. For that matter, compute in Southeast Asia could be used by the U.S. or, sometimes I've heard, China. This is a difference compared to oil. During the Iran shock, you remember all this discussion about how different parts of the oil market became very dislocated because of transportation and physical access. The internet really eliminates a lot of those boundaries. That being said, I do think, increasingly going forward, data retention may be a bigger deal. There is a lot of sensitivity: Europe has regulation about the need to retain data locally, and other jurisdictions similarly. This will lead to some degree of fragmentation and prevent full homogeneity. A GPU in Finland is not ultimately exactly the same thing as a GPU in western Texas or on the eastern shore of the U.S. But there's some substitutability, just like in every single marketplace. No two products are exactly homogeneous. To the extent they're somewhat substitutable, if you squint your eyes a little bit, that's going to force the prices to be highly correlated, if not very close.
1:13:50
Prakash Narayanan: One last question from me. The way Silicon Data is structured, there's Silicon Data and there's Compute Exchange. Compute Exchange seems to be trading GPU rental contracts, or brokering them at least. What is the relationship between the two? What is the information flow between them?
1:14:14
Steve Hou: Chris, thank you very much for bringing up the question. Very great question. At Silicon Data, we are focused on the entire market. We'll collect information from wherever we can on pricing. Compute Exchange is a sister company. Carmen Li, the founder and CEO of Silicon Data, also happens to be the CEO of Compute Exchange. Compute Exchange is a marketplace for people to transact physical compute, rental compute, either machines or—there is brokering of compute on Compute Exchange. Beyond that, the two companies are relatively separate. We treat them as a source of information as we do from anywhere else. There's some transaction information, but we also get information from just about anywhere else.
1:15:12
Prakash Narayanan: Indeed. Steve Hou, thank you for joining us today and being generous with your time. Thank you. We hope to hear from you again as GPU becomes a much bigger market.
1:15:26
Steve Hou: Indeed. Thank you so much for having me. I look forward to being back again.
1:15:29
Prakash Narayanan: Bye-bye.
1:15:36
Prakash Narayanan: Very cool. It's—
1:15:39
Prakash Narayanan: I think everyone wants to know what's going to happen with GPU prices. It's the number-one thing that we want to know about this market. All right. I'm going to bring up our next guest, so just give me a second.
Clips
- Expensive Models Can Be Cheaper
  0:00 / 0:00
- Intelligence Is Getting Cheaper
  0:00 / 0:00
- A GPU In Finland Isn't Identical
  0:00 / 0:00
- Why Hyperscalers Charge More
  0:00 / 0:00
- AI Token Prices Are Usage-Weighted
  0:00 / 0:00
1:18:50Interview98 min
Jeremie & Edouard Harris — Can an AI agreement with China hold?Jeremie Harris Edouard HarrisThe Gladstone AI co-founders debate crisis diplomacy, credible enforcement, and what governments could actually verify. Nathan and Prakash challenge proposed red lines, cluster shutdowns, and assumptions about China. The segment continues with a host-only debrief after the guests depart.
Watch
Open segment on YouTube ↗
As aired
Jeremie and Edouard Harris of Gladstone AI join Nathan and Prakash to examine whether US–China AI diplomacy could withstand a serious loss-of-control incident. Drawing on conversations with former diplomats, they argue that an AI hotline and conciliatory rhetoric are encouraging but insufficient grounds to assume reliable cooperation.
The brothers frame an “AI Chernobyl” by its political consequence: an incident severe enough to force a change in US policy. Prakash challenges that premise with the very different policy responses to 9/11 and COVID, while Nathan repeatedly presses for concrete, measurable red lines. The guests describe an emergency limit on operating large compute clusters as a crude possibility, explicitly not an intelligence-vetted or executable plan.
A central disagreement concerns deterrence. Edouard and Jeremie advocate preparing credible offensive options alongside verification, arguing that unenforced red lines invite violations. The hosts question the costs, escalation risks, and attribution problems, including the possibility that autonomous agents become a third actor—or a convenient excuse for state-sponsored attacks.
The discussion turns to work that could make a crisis response less blunt: compute-verification technology, early intelligence-community vetting, shared safety research, and relationships between American and Chinese researchers. Nathan proposes testing mutual verification between OpenAI and Anthropic; the guests see useful similarities while emphasizing the greater trust between domestic companies.
The hosts challenge the guests on Chinese industrial strength, elder care, political constraints, and the security implications of Chinese researchers at US labs. Jeremie explicitly declines to offer a settled personnel policy and acknowledges both coercion risks and the cost of driving talent back to China. The guests identify actual restraint by Chinese labs, empowered negotiators, and concrete commitments as stronger signals than rhetoric alone.
After the guests depart, Nathan and Prakash continue debating talent mobility, concentration of power, and what a positive AI future should aim for. Nathan argues for a “Pax Robotica” oriented toward shared abundance; Prakash emphasizes competition and freedom of speech. Their debrief also weighs China’s achievements against uncertainty around political constraints and leadership succession.
Timestamp links open the original source recording.
Key moments
Talking is always better than not talking.
Edouard Harris1:18:30
The curve climbs ever upward.
Edouard Harris1:39:37
What is the goal? What do we want the end state of this to be? What is the positive vision for AI?
Nathan Labenz2:40:40
What we covered
Preparing for crisis diplomacy The guests draw on diplomat interviews to explain why a leaders’ agreement may arrive before a formal treaty or mature verification system.
Red lines and their consequences The discussion tests whether recursive self-improvement can be measured, how infrastructure limits could be observed, and whether enforcement creates new risks.
A domestic verification experiment Nathan proposes that frontier labs learn to police each other; Edouard notes both the useful similarities and the limits of extrapolating domestic trust internationally.
China, talent, and political power The guests and hosts debate signals of cooperation, competitive strengths, and researcher policy. The hosts continue reflecting on those disagreements after the interview.
Questions asked
1:17:39What just happened in the Trump–Xi talks, and how do you make sense of it in the broader context?
Edouard welcomes dialogue and a potential AI-incident hotline, but says former diplomats warned that comparable lines have not always been answered in crises. He argues for pursuing cooperation while preparing a backup plan.
1:20:06What does being realistic about cooperation with China mean?
Jeremie describes uncertainty over whether diplomatic objections reflect sincere concerns or negotiating tactics. He says realism requires preparing for refusal, noncompliance, or prolonged ambiguity around a slowdown agreement, rather than assuming that participation guarantees shared priorities.
1:22:57What are we talking about when we talk about enforcement?
Edouard argues for credible offensive options that could hold Chinese infrastructure at risk if agreed red lines are crossed. He presents cooperation as the preferred outcome and deterrence as a backup; Prakash challenges the credibility problem created by red lines a government may not enforce.
1:31:38How do you define red lines when there is no consensus on measuring recursive self-improvement?
Jeremie says Gladstone is developing a more precise RSI definition with frontier-lab input. He distinguishes that work from a crisis response, which might initially rely on existing, coarse monitoring capabilities such as satellite imagery and heat signatures before evolving into a more nuanced agreement.
1:33:30What is an AI Chernobyl event?
Jeremie defines it by the political response: an event large enough to make the current approach untenable and prompt leaders to seek meaningful restrictions. Edouard adds that the call must aim to stop the dangerous activity. Neither claims to know the particular incident or casualty threshold.
1:36:24Would a slow-moving catastrophe like COVID be enough, or does the incident need to be concentrated and visible?
Edouard agrees that severity alone does not determine policy response, contrasting COVID with 9/11. The brothers offer cyber and biological scenarios but retain an outcome-based definition; Jeremie also allows for accumulating chaos or gradual disempowerment rather than one obvious breaking point.
1:44:36After such an incident, what exactly would we ask China not to do?
Jeremie explicitly says the proposal has not been intelligence-community red-teamed and is not an executable plan. He and Edouard describe a possible emergency restriction on running compute clusters above a measurable size, with disappearing heat signatures as one crude indicator. Both emphasize its enormous economic cost.
1:48:34Would shutting down large data centers also remove the AI tools needed to solve the problem?
Edouard distinguishes stopping the infrastructure sustaining an attack from using AI to detect it, while accepting Prakash's broader concern. He says autonomous agents could eventually become a third player with their own agenda, complicating a two-country deterrence framework.
1:52:04What should we do now to avoid a crisis or have better options than shutting everything down?
Edouard prioritizes better verification and credible enforcement options. Jeremie stresses that verification technologies may take years to become trusted national technical means, so startups should engage relevant intelligence-community experts now and get early feedback on technically or institutionally unworkable approaches.
2:01:51How important is a shared US–China AI paradigm, and could OpenAI and Anthropic test mutual verification?
The guests see potential value in a domestic test bed while noting that two US companies share more trust and similar values than rival states. Jeremie says common architectures can make warning signs more mutually legible and allow shared safety technology, but do not resolve distrust about motives.
2:06:43How would you manage Chinese models that do not adopt the CBRN safeguards US labs promote?
Edouard says policymakers would have to decide whether the particular capability and risk justify spending a scarce, monitorable red line on it. He characterizes permissive releases as a familiar catch-up strategy and declines to specify where that future threshold should fall.
2:11:10Would China be threatened by the US reaching superintelligence first and imposing rules?
Jeremie argues that China has substantial advantages in manufacturing, robotics, supply chains, and access to US research, giving it more at stake than a simple trailing-country narrative implies. Nathan challenges the implications of Jeremie's claims about elder care and political freedom of action.
2:17:45What meaningful signals would update your view of China's willingness to cooperate on AI?
Edouard says conciliatory rhetoric is positive but should be weighed against past credibility. Stronger evidence would include Chinese labs agreeing to pacing or regulators restricting a model after a concerning incident. Jeremie highlights empowered negotiators, timely progress, concrete commitments, and fewer arbitrary procedural objections.
2:25:14What could the US do to credibly signal that it seeks shared abundance rather than a decisive strategic advantage?
Edouard suggests transparency that helps each side see the other is not doing what it most fears, preferably as part of reciprocal arrangements. He also endorses more links between verification researchers, startups, and academics, whose sincere relationships can moderate national narratives.
2:28:54What do you actually propose doing about Chinese researchers in US frontier labs?
Jeremie says he has not studied the issue deeply enough to make a policy call. He acknowledges that excluding researchers can strengthen Chinese competitors, discusses coercion and insider-risk concerns, and suggests research automation may change the trade-offs. He explicitly says the risk does not apply to all Chinese researchers and can extend to anyone with vulnerable interests in China.
Related
Gladstone AI — About the team ↗AI Endgame — It comes down to a phone call ↗Gladstone AI — 2024 Action Plan ↗Posted live highlight — The cluster-shutdown dilemma ↗Posted live highlight — Labs verifying each other ↗
Full transcriptLightly edited · timestamps jump to YouTube
1:16:10
Prakash Narayanan: Our next guests are Jeremie and Edouard Harris. They are the cofounders of Gladstone AI, and they're brothers. They work at the intersection of advanced AI and national security. Jeremie is the CEO. Edouard is the CTO, an AI safety researcher and software engineer. Together, they've briefed senior officials across the American, British, and Canadian governments. They coauthored Defense in Depth, the 2024 State Department-commissioned action plan addressing AI weaponization and the risk of losing control of powerful systems.
1:16:55
Their 2025 report, America's Superintelligence Project, argues that AI requires protecting the entire chain, from chips and data centers to the labs developing the models. They have a new report on risks of AI in China. Given that we had the Xi and Trump meetings last week, it is very timely that we have them here today. Let me bring them up.
1:17:32
Edouard Harris: Amazing. Hi, guys.
1:17:34
Jeremie Harris: Hey, guys.
1:17:35
Nathan Labenz: Good to see you.
1:17:36
Jeremie Harris: Nice to be in your internet box.
1:17:39
Nathan Labenz: Welcome back. It's day one for a new look and feel, and the taste of the models is self-evidently getting better all the time. I'm prepared to be a little heartbroken by today's conversation. You guys know me as somebody who is working hard to hyperstition a better relationship between the US and China, especially in AI, if not ideally more broadly than that. Where are we right now? What just happened? I feel like we had this whole Trump thing, and I'm waiting for the analysis, the takeaways. What just happened, and how do you make sense of it in the broader context?
1:18:25
Jeremie Harris: Well, one thing—or, Ed, do you want to go ahead?
1:18:28
Edouard Harris: I was just going to say, talking is always better than not talking. That's a positive. My understanding from at least the beginnings of the Trump–Xi conversation and the stuff leading up to that is that one of the things that may be positive was the development of this—I don't know if you'd call it a red phone—but at least some kind of theoretical line between the two governments on AI incidents and AI risks. The report that we came out with, which is really just a long newsletter, is informed by speaking to about a dozen State Department diplomats who have dealt with China from the negotiating table,
1:19:13
who've seen how these things develop in practice. One of the issues they see, among many others, is that we have tried the red phone thing before in the context of nuclear. By and large, they don't always answer. Particularly in critical phases, it's often exercised as a point of leverage, saying, “We're going to take away this phone line and not answer,” rather than as this collaborative, unified project that makes everyone safer. Again, this is not to say that if the CCP does structurally take AI seriously—which there is decent reason to think they may—we couldn't have a whole different and much more positive
1:19:58
level of engagement. All that we're recommending, based on the experience of these folks, is being realistic about it and having a backup plan.
1:20:06
Prakash Narayanan: What does being realistic about it imply?
1:20:10
“Realism” seems like a loaded word. What does realism mean in this?
1:20:17
Jeremie Harris: It is pretty loaded, and it also ties to something very specific, or a couple of very specific things. One is, when you think about the way things have gone with China in the past, there's a lot of fog of war as to what China actually believes and how that maps onto the diplomatic process. You see that even among the people who sat directly across from China at the negotiating table. For example, China's known for randomly coming up with terminology that's suddenly considered problematic. Famously, they hate the term “arms control.” If you bring up arms control at the negotiating table, they're liable to just up and leave. Well, what if you weren't able to guess magically that that was the magic boo-boo word that, if you say it, ends the talks for the day? The problem is this keeps happening over and over again, and now
1:21:02
we have the epistemic problem of trying to guess whether or not that's a sincere objection. Part of the work we were doing was to catalog all these patterns of behavior to give us a starting point. Imagine there's an AI Chernobyl event, which we think is what's going to trigger this. We think the administration's current position on this is fundamentally untenable. There will be an AI Chernobyl event eventually. They will be forced to change their position, and when they do, they will engage China very quickly without necessarily having studied up on the best ways to do it or looked at some of the tripwires they might run into. They'll very quickly have to figure out—we will quickly have to figure out—how sincere certain objections are that the Chinese raise. Right now, the reality is, if you talk
1:21:47
to folks from Trump one, from Biden, from other administrations who've done this, there's actual disagreement over what is real when the Chinese bring it up and what isn't. Imagine you've got very little time. The AIs are breaking out of containment. You've had your Chernobyl event. Tick-tock, brother. Now you've got to figure this out. You don't have time to go through the diplomatic song and dance associated with figuring out: was this boo-boo word really a boo-boo word, or is this just another game of diplomatic chicken that they like to play? This was part of why we cataloged all these things. Being realistic about it means: what do we do if China refuses to agree to a slowdown agreement? What do we do if they agree and then renege on it? Or what if they
1:22:32
try to do this weird middle-ground thing where they string us along and we're not sure? At some point, we're going to have to pull the trigger and say, “Okay, we're now going to proceed under the assumption this is an adversarial dynamic, that you are not actually bought into the threat class in the way that we are, and that we therefore need to do some unilateral communication of and enforcement of our red lines.” That's basically what we mean by “be realistic.”
1:22:57
Nathan Labenz: “Keep going,” I guess, is my next question. Obviously, the backup plan remains to be fleshed out. What are we talking about when we talk about enforcement? I'll have some questions, too, on
1:23:12
handicapping the likelihood that China goes different directions on this.
1:23:16
Edouard Harris: Yeah.
1:23:17
Nathan Labenz: But just keep going on that main line first.
1:23:20
Edouard Harris: If your hyperstitioning works, then we don't need a backup plan at all, and that's great. That's the best possible outcome. I don't think that's completely impossible. The Chinese at least seem to institutionally have some understanding of the risk set here. Although, as with the United States, it's very heterogeneous. It's very unclear who's holding the steering wheel at any given time on any given issue. But yes, if the hyperstitioning works, then we don't need a backup plan at all. The backup plan is in case it doesn't work, of course. Fundamentally, how do nation-states come to deals like this in the
1:24:05
real world, in the context of arms control and things like nuclear and chemical, biological, radiological, nuclear threats? The answer, by and large, is existing incentives combined with threats of force. The backup plan applies both if they refuse and if they say yes but end up not doing it, which is another genuine risk. As well as taking offense to arbitrary words and stuff, China has a history of agreeing to stuff and then not doing it. We actually have to be positioned to meet that possibility. What that means is we need to develop credible
1:24:50
offensive options to hold various parts of China's national infrastructure at risk. This is an unpalatable thing to say. Again, if they end up just going for it, we'll never need to exercise those options, and everybody's happy. But certainly, if they follow their historical pattern, as they have in many other cases, and say yes but then don't do it, we will wish we had developed options like this. We will desperately wish that we had developed options like this. This is basically the high level of the backup plan. It's unfortunate, but this is how nation-states deal with each other. They threaten each other with serious retaliation if red
1:25:35
lines are crossed. I was going to say this is the dark side of the MAD doctrine, as if the overt MAD doctrine wasn't dark enough. But it's one of the facets of MAD doctrine as it's implemented in practice.
1:25:50
Prakash Narayanan: When you mention the red line, that's like a red line for me, because I remember Obama had the red line of chemical weapons in Syria. Chemical weapons were used in Syria, and the red line was basically dissolved. When you have that red line, you end up in an ultimatum situation. If you don't trigger the ultimatum, then you lose credibility.
1:26:16
Edouard Harris: That's right.
1:26:17
Prakash Narayanan: I would also say, have we not used some of these options already for the Iran struggle? We've got sanctions on every buyer of Iranian oil right now. Is there something that we have not used for the Iran struggle that you want to use for the AI struggle?
1:26:44
Edouard Harris: Maybe, Jeremie, you want to take this, but I can also. Okay. This is exactly the right question to ask. The first part of the question is: if you define a red line, now all of a sudden you're putting yourself in a position where you have to enforce that red line. Yes, absolutely correct. Defining these red lines and developing these options is actually not enough. You have to exercise those options in a low-threshold or subthreshold way in order to build credibility with your adversary that you're actually going to do it. It's not enough to just have these things sitting on the shelf, and we talk about this in the report. This is actually one
1:27:29
of the biggest problems our diplomats brought up about the American diplomatic situation as it stands today. For the last 15 or 20 years, across multiple administrations from both parties, America has this credibility problem. There have been repeated provocations by adversaries and often explicit statements by the administration and the executive saying, “You will not do this. This is a red line.” Then the red line gets crossed, and nothing happens. Do that a few times, and your adversaries absolutely stop believing in you. This means—and this is also part of the generalized MAD doctrine—you need to define your red lines very carefully such that they are
1:28:14
things that you will actually defend, number one. Number two, you need to maintain ambiguity about exactly what your response will be to most of the red lines you define. There can be some exceptions, and we can talk about those. Number three, you need to actually exercise those offensive options. You need to go after, not necessarily infrastructure in your actual target country, although sometimes you may have to, but allies and proxies of that target country through allies and proxies of your own. I'm describing already a big part of the dynamic of the Cold War. They're moving the chess pieces, but it's seen as too risky for large-scale military confrontation between the two
1:28:59
principals, because that can devolve into an uncontrolled, ultimately nuclear conflict. The reason you need the subthreshold back-and-forth is you need to test the adversary's red lines, just as they are testing yours. Also, you need to get your reps in and prove to yourself that, should the worst come to the worst, you can reach out and touch the adversary yourself. If you haven't exercised the capability, you may discover on the day you need it most that you actually didn't have it. In fact, our adversaries are doing this to us on a regular basis as well.
1:29:36
Jeremie Harris: More than we are reciprocally, too, which I think is worth saying. Since the fall of the Berlin Wall, as we've lived in the end of history, there's been this gradual atrophy of a lot of the appetite to authorize certain kinds of activities, especially at the nation-state level. With the GWOT, the global war on terror, the national security apparatus started to refocus on, “Let's go after a bunch of poor people in these remote parts of the world where we have cyber superiority, aerial superiority.” Even at that, we can't always get them. This has been flagged to us by a number of people in the ecosystem as a really big cultural problem. This is back to Ed's point about
1:30:21
the need to exercise these options. There's a sense in which the proxy wars that played out during the Cold War were actually profoundly stabilizing in ways that are underappreciated. The fact that America could prove to itself that it could do the thing, and prove to Russia that it could do the thing, and Russia could do the same back to us, meant that we had fewer sources of uncertainty when modeling out how conflicts would go. As you know, wars are really just a failure of modeling conflict. If you think you're going to win and your adversary also thinks they're going to win, you're more likely to go to war. Whereas if you both know who's going to win ahead of time, there's no need to fight the war in the first place. This is an ingredient, at least, that determines these sorts of outcomes. Having that clarity is valuable. You do need to get those reps in, and the willingness
1:31:06
piece is so, so important. So often, capabilities are developed and sit on shelves because people think using them will be escalatory, which, for various reasons, very often is not the case. You can often make very grounded guesses as to what's going to be escalatory or not. But you need to actually use them. Otherwise, like I said, the gun breaks in two when you pull it out of the vault, and then suddenly, what are you going to do?
1:31:38
Nathan Labenz: I'd love to zero in on the red lines for starters. I buy a lot of what you are describing in terms of the need to create clarity on both sides of a relationship to avoid miscalculation. But I'm going to be doing a thing at an AI event this coming weekend where I'm supposed to be moderating a discussion on RSI. For me, that's a pretty good candidate for a red line. But it immediately becomes clear that even among a very small number of panelists, a couple of whom are at the frontier companies doing their best, I guess, to measure RSI and give us blog posts about it, there's a total
1:32:23
lack of consensus on what would count and how to measure it. If you want to put in pacing mechanisms, how do you define them? This is all before we even get into the question of measuring, let alone the fog of—I don't want to say war—but the fog of being 12 time zones away and not having great visibility into what the hell is going on in the other country. How do you think about defining red lines given all those difficulties?
1:32:51
Jeremie Harris: It's funny: there's a reason our very next report is something we're writing with a bunch of input from folks at the frontier labs, precisely to define RSI for exactly the reason you flagged, and also for domestic regulatory purposes. We expect these two things to come hand in hand. I'll zoom out and say that a lot of conversations about this sort of thing get muddled in abstraction and almost too much detail of one kind and not enough of another. What I mean is this: imagine this actually happens. We have an AI Chernobyl event. What's actually going to happen next? Well, Trump's going to get on the phone with Xi.
1:33:30
Prakash Narayanan: Wait. What is an AI Chernobyl event?
1:33:33
Jeremie Harris: Great question. We could talk about that separately, and your guess is as good as mine. What I'm going to rest the definition on is the outcome of it, which might sound like a cop-out. I think it's not, but it might sound like it. Roughly speaking, you could imagine something on the order of a few million people dying. Maybe it's just a few thousand people. It depends on the administration's appetite to actually have its mind changed on this. Fundamentally, an event of large enough magnitude that it becomes politically untenable to continue on the current route that goes, more or less, “Let's just let it rip,” with little exceptions at the margins. I'm talking about an event significant enough to lead to a Trump–Xi phone call or whatever equivalent, if that makes sense.
1:34:16
Edouard Harris: Not just a Trump–Xi phone call, but a Trump–Xi phone call with the object of, “Okay, we have to stop this kind of thing.”
1:34:22
Jeremie Harris: That's exactly the next thing. This is where I shift into “not enough imagination of another kind.” Now let's actually focus on that conversation. What are the incentives of the players, and what are their capabilities? From an incentive standpoint, the US government apparently, at this point, or in principle, has the appetite. The Chinese don't necessarily, and the US government does not know if China has the appetite. They can't jump into Xi Jinping's skull and assess magically what's going on. There's a whole bunch of surveillance and collection opportunities at the margins, but fundamentally, there's fog of war there. From a capability standpoint, what's happening? Not a whole hell of a lot, because the administration has not bought into this thesis at all. There's a bunch of technology that needs to
1:35:07
be set up in the aftermath of the phone call, but it doesn't exist yet. You're going to look at very coarse-grained capabilities—satellite imagery, thermals, that sort of thing—that have to be the basis for the initial set of red lines, because there is no other option. There's a question as to how you start that. What does that initial stack look like? That's something we're also working on. But how does it back into a more fine-grained, nuanced, durable agreement? That's going to be an entire process of its own. That initial phone call has to have as its goal to establish, “Here is our red line on stuff we can measure right now, and here is our proposal for how we back into better things over time that unlock the trillions of
1:35:52
dollars of value that the industry wants to unlock on our side and on their side.” In theory, there's a lot of value here. But here is broadly, with strategic ambiguity, what the consequences of violating our red lines are going to be. You know that we're serious because, as we are talking, operation X, Y, and Z is unfolding, and we're proving to you and to ourselves that we can actually touch the thing. These are the kinds of things, not all necessarily in one call, but in some way, shape, or form in the initial few days after the event.
1:36:24
Prakash Narayanan: Let's take some somewhat recent examples. During 9/11, I think something like 3,000 or 4,000 people died within the course of six hours or so. I'm not sure what the actual numbers are. That caused a 20-year change in US policy, and that extended to multi-trillion-dollar, multi-percentage-of-GDP changes in US policy. On the other hand, you had the COVID event, which was millions of people dying, but over a four-year period. At the end of that, we don't even have universal vaccines that we can build. No one is interested in funding
1:37:10
universal pandemic vaccines. Obviously, even 3,000—even, I think, 100 people dying within the course of two hours—there's an intensity of loss versus spread. It seems like if the intensity was high and it's visible, you get a lot of movement immediately. But if the intensity was low and stretched out over a long period of time—for example, global warming or COVID. Let's say you have a malevolent AI, and it decides to occupy everyone else's time so that we don't have kids, which is actually happening. How do you define this in the not-so-obvious cases?
1:37:55
What is actually something significant enough? Would COVID be it, something long over four or five years? This is not Chernobyl. Where is the vector from? It will be unclear because people would hide this. How would you define a little bit more what this event is going to be, or how the red line should be crossed?
1:38:23
Edouard Harris: That's a brilliant pair of counterexamples. It's exactly right. You had this event that, if you just count it up, is small-scale, but it all happened really visibly: planes blowing up buildings, and you go, “Oh my God,” in a very short span of time, super high intensity. Then you had another event where a million Americans died during COVID. That is arguably, I think, the single thing that has killed the greatest number of Americans as one incident. Yet it's not just that we don't have universal vaccines, because that could be a scientific or engineering question, but we don't have policies that make us resilient to the next COVID. That's a thing that we, as a civilization, should expect to have
1:39:08
developed and adapted to. You're completely right. I think the answer falls back to Jeremie's reverse definition: what is the event that will have this outcome? We can give some examples of possible things. The reason we're falling back to this is that you look at the dynamics of the whole thing, zoom way out. We have this curve of capability with compute and scaling and RSI stuff. The curve climbs ever upward. Maybe it's not going to break to the upside when we do RSI. Maybe they're going to be able to pace things to keep it exponential or whatever. Who knows? But the curve continues ever upward. We've seen a Hugging Face.
1:39:53
I would arguably say we've seen maybe a Hugging Face 2.0, where tens of thousands of different entities—whoops, it turns out we hacked half of the internet by accident. Still, no major damage was caused. No one has died. People have the vapors and stuff, but nothing crazy. The curve climbs ever upward. We're going to get a Hugging Face 3.0. We're going to get a Hugging Face 4.0, and on and on. The question really comes down to: at what level is the incident going to have this effect? That, of course, has implications for even personal preparedness, your personal resilience. Should you maybe buy some food, buy some water,
1:40:39
stuff like that? You ask yourself: how big of an incident does it have to be to cause this administration to do a full 180? Well, we're going to have an incident at least as big as that, and the blast radius might be substantial. Certainly, in terms of examples, the whole cyberattack on power infrastructure is a typical one. In the article, we talk about a scenario developed and fleshed out by Lloyd's of London, the insurance firm, in 2015. In this scenario, they found you can get, as a realistic worst-case outcome, 90 million Americans losing power. That's the
1:41:24
fragility level of the system. Ninety million.
1:41:28
Jeremie Harris: I'm sorry? Yeah, for a fairly short period of time, but yeah.
1:41:31
Edouard Harris: For a short period of time. But depending on which 90 million that is, if that encompasses FEMA infrastructure and some of the backups for that, you could get knock-on effects. There's a lot of stuff that's untested here. You can't really test a black start for national infrastructure because you could end up killing a bunch of folks in your test. We don't know what happens. But Lloyd's view was that this fell within the insurable region that they would have to take into account in their own insurance policy underwriting calculations. You absolutely could, maybe a few months or years down the line, have a kind of bio thing,
1:42:16
which has potentially a larger blast radius. There's a large set of things that could occur, and it's very unclear what.
1:42:26
Jeremie Harris: There's also the irreversibility factor, which is almost harder to quantify. One of the things about the Hugging Face incident and more or less everything that's come after, with the exception of a couple of images of people's private data that have turned up on the internet, like API keys and stuff—I don't mean to poo-poo that, but it's certainly not a 9/11-type thing. These events more or less have a reversible flavor where Jensen can stand up on a podium and say—you know what I mean, right? He's not necessarily quite wrong. I want to bring myself to disagree with him, but I can't quite be like, “I'm going to stand on that as the Pearl Harbor moment.” I think it'd be ludicrous to do that.
1:43:11
But I think people who are technical enough, who've been watching scaling curves as you guys have been for a long time, know what the next part of the story is. We've been here before. Every massive runaway success story was preceded by a story that looked cute and toy-shaped. This is the cute and toy-shaped version of an AI Chernobyl event. I don't think anyone who takes the scaling curves seriously can think that, absent regulation, we don't get to that world. There is this one question: is it an incident, or is it a long car crash that increases exponentially in intensity over time, where things get more and more chaotic? Eventually everyone just goes,
1:43:57
“You know what? Maybe I am going to get Xi on the phone right now.” It's also possible you don't end up in a position where you're like, “That was the clear straw that broke the camel's back.” I could imagine something more like Paul Christiano-flavored slow takeoff, confusion, all that stuff, or gradual disempowerment. That's part of the reason why we cheat at this point and say: suppose something happens, the appetite is there. We're abstracting away the question of what specific thing, even though, as we've talked about, there's a million examples you could point to. It's just not clear what it would be, and to us, not necessarily worth the cycles, because what happens after seems to matter so much more.
1:44:36
Nathan Labenz: Not to be a broken record, but I still want to go back and get more clarity on the red lines. Something like this happens. We're getting on the phone. What do we want them to not do? Can we articulate that in a way that we can have a conversation result in clarity, at least at the level of what the ask is? What would the ask be that we would feel we'd be able to get across and feel good about actually taking a bunch of risk off the table?
1:45:09
Jeremie Harris: One quick thought here: everything that comes after this sentence obviously has not made contact with the intelligence community from a red-teaming standpoint. The true answer is we can't know deeply. I couldn't give you an answer to a level of detail where it would be, “Okay, that's executable.” We've heard a lot of examples from folks who used to negotiate exactly these kinds of agreements, and heard the flavor of the sorts of things you come up with in an emergency, including cases that involved the potential targeting of US nuclear command and control. When you have panic on that level, you're like, “Listen, you fuck with our command and control, we are in nuclear war.” That was the condition that was sent across the bow in that context. It's
1:45:54
very important because nuclear command and control is the launch capability that you want to make sure doesn't get targeted. It's the key piece of your deterrence scheme. On this, one easy thing, if I'm going to caricature: data centers put off a hell of an energy footprint. The thermals on those are really bright. Data centers are huge. They haven't, by and large, yet been built to be hidden, and it takes a long time to build data centers. Now, this will change. AI 2027 talks about the timelines for this. We think it's quite plausible the timelines could be a lot shorter for hiding data centers, just based on conversations with folks in the industry. Whichever way you slice it, you're going to have an initial conversation where, to first order, for the 80/20
1:46:39
that you really need then, it's like, “So help me God, if I see a cluster and that cluster is yea big...” That's the kind of conversation you're looking at. How you quantify that is a matter of the sort of national technical means that the US currently has.
1:46:54
Prakash Narayanan: Are you saying having a cluster above a certain size would be a red line?
1:47:00
Jeremie Harris: I'm saying, yeah, initially.
1:47:02
Edouard Harris: If you're just in that panic moment, you're like, “We have to do something.” You ask yourself, what is possible to do with the existing assets and infrastructure we have today? Nothing else. Then you do get into a space—not necessarily where there exists a cluster of this size, because you can't ask them to tear down the cluster—but we have to see the heat signatures from this go away. If there is a running cluster above a certain size. That is, to be clear, a tremendously expensive ask in either direction. The depreciation on GPUs is the major part of the OpEx cost.
1:47:44
Prakash Narayanan: You're saying an incident happens first?
1:47:46
Edouard Harris: Yep.
1:47:47
Prakash Narayanan: And the response to that incident, the mitigation for that incident, is, “Hey, can you turn off this big data center?”
1:47:53
Edouard Harris: Yeah. Turn off any cluster above a certain size. That's one possibility, because this—
1:47:56
Jeremie Harris: As of right now, to be clear. The framing is basically, as of right now, that's where we're at. In some sense, we're going to get into potentially a circular loop here, where you can see how big an ask that is. That is an insane ask.
1:48:08
Prakash Narayanan: It also strikes me that, again, if you look at the Hugging Face incident—
1:48:14
Jeremie Harris: Yeah.
Prakash Narayanan: You had agents take over the place. It wasn't being driven by OpenAI specifically. In order to fix it, Hugging Face needed to use its own agents. When you say you have an incident, I would expect—again, maybe let's not blame the Chinese—maybe you have agentic takeover of a data center.
1:48:33
Nathan Labenz: Yep.
1:48:34
Prakash Narayanan: But to fix it, you're also going to need another data center. That starts for me to be like, “You're asking us to turn off the tools that we're going to need to address the problem.” What do we do?
1:48:48
Edouard Harris: In the OpenAI–Hugging Face case, OpenAI shut off its agents, and that is what allowed Hugging Face to actually end the attack.
1:48:58
Prakash Narayanan: Before that, they had the Kimi agents, which they needed to even detect and track down what was going on.
1:49:07
Edouard Harris: Yeah. This is—
1:49:08
Prakash Narayanan: OpenAI was reached out to by Hugging Face, not the other way. It wasn't—
1:49:15
Edouard Harris: Like OpenAI, “I'm going to—”
1:49:16
Prakash Narayanan: “Turn it off.” It was Hugging Face that used its Kimis and, “Oh, we have a problem, and we need to ask people to refresh the keys.” They reached out to OpenAI, and then the OpenAI Slack was like, “Was that us?”
1:49:30
Edouard Harris: So in this case—
1:49:31
Prakash Narayanan: They needed it. They needed the ability to run Kimis
1:49:36
in order to detect—
1:49:38
Edouard Harris: This is right. My point was rather that, by shutting down the primary infrastructure, whether OpenAI had been aware of the usage of that infrastructure or not, that would have been sufficient in this case to end the attack. But you are right. You could have, in the near future, a situation where maybe the agents themselves are taking over in much more interesting ways these large or medium-sized data centers. I do think that even under such circumstances, if you absolutely have to, you can physically cut the lines, the voltage lines between the power plant and the facility. If you absolutely must,
1:50:23
this is a thing you can do. Maybe we're far enough in the future where the AI has taken over a bunch of Unitree robots that are physically defending the whatever. But I think that's at least a couple of years out. If you have to, you can physically cut the line. Now, maybe the agents have already exfiltrated their weights into a data center that doesn't pass the cutoff, and there's more shenanigans happening that way. It's absolutely true that if we push this timeline far enough into the future, it may not just be a two-player game. There may be a third player in there that is the adversary of both, sneaking around at faster-than-human speed, and has its own agenda functionally. I think this is
1:51:08
a very legitimate concern.
1:51:11
Jeremie Harris: Basically, the prediction we're making at this point is that, at some point, the waterline will get high enough that the trillions of dollars it will cost to stall AI infrastructure—for a limited period of time, to be clear, as we back into better and more nuanced things—that the political appetite will be there. That circles back to the original question: what does this incident actually look like? The political cost is super high. The economic cost is super high, and so the willingness has to be super high. All of these things only increase the level of intensity of that moment and trade-off. I don't think it's possible to talk about the consequences without also talking about the willingness at
1:51:56
the same time, and what the incident looks like. We're snapping a chalk line at that call and roughly trying to make sense of it.
1:52:04
Nathan Labenz: If I sketch out the logic from beginning to end here: we don't have that great of a relationship. AI capabilities continue to progress at a fast pace. We expect something crazy to happen. When something crazy enough happens, we're going to find ourselves by default in a spot where we have to ask for some outlandish, super-high-cost move, like shut down all your big data centers, because we don't have any other mechanisms in place that allow for a lower ask, a better trust-but-verify type of environment, because we haven't made those investments now. That
1:52:49
leads me to the question of what we should be doing now to, A, ideally not end up in that situation, or B, if we do end up in that situation, have better options available to ask for aside from “shut it all down,” which is obviously going to be tough.
1:53:06
Edouard Harris: You absolutely nailed it. Develop better verification and develop better offensive options to ensure compliance in the event that verification returns, “No, they're doing it.” The better verification stuff you can do, the faster, the less you have to rely on absurdly expensive things like, “This gigawatt of energy radiation shouldn't be visible from space.” If we have techniques like this that are vetted by the intelligence community, even if they are just 50% better than this—even if it's like, “You have to shut down half your data center.” I'm making something up here. “You have to shut down half your data center, and we can
1:53:51
sufficiently verify the other half,” or something equivalent to that. You are saving billions of dollars right off the bat. The ability for this industry to continue to make large amounts of money is actually going to be gated for that period of time by these little verification technologies. There's already this community of little verification startups working on this technology, and that's why this is so important. Of course, I will also say the offense side of things is critically necessary. If you don't have those offensive options, you cannot assure compliance. You can verify, monitor the situation, and say, “Oh no,” but fundamentally, your hands are tied. You don't have the tools to actually do anything about it. Both of those things are
1:54:37
super important. I also want to emphasize Prakash's point is excellent: you may actually be in a three-player game at some point if things push far enough before the critical incident occurs. The dynamics of that are something I don't know that anyone has really thought much about.
1:54:56
Jeremie Harris: I'll throw out one other thing on the verification side. There's this bottleneck that I think a lot of these verification companies haven't necessarily priced in. This is where a lot of our current work is focused. Imagine what happens when company A goes, “I have the thing. This thing is going to work.” It's a moment of crisis, and Trump is casting about—or POTUS, whoever it is at the time—for options to alleviate this trillion-dollar bottleneck. Then they go, “Okay, the intelligence community has to now vet this,” because they're not going to just start using it, obviously. How long did it take similar technologies in the past to get
1:55:41
used, to get vetted, and to become what's known as national technical means, NTMs? The answer is years, depending on the technology, but very often years. We have to do it. China has to do it. We have to handshake on doing it. Even if you remove money as an obstacle, there's just certain things that take serial time to do. A lot of what we've been doing is focused on saying, “Treaty—or not treaty, let's say AI agreement verification or compute verification company X—you probably should be talking to IC element Y about this.” In a moment of crisis, you, A, want as much pre-vetted as possible, and B, anybody who's involved in assessing a potential national technical means had better have on
1:56:26
speed dial—they better have the Signal, phone number, email, whatever—of the founders of all the verification companies they plan to use or may end up having to use. You want to cut down on all those barriers, the boring bureaucratic hurdles nobody ever thinks about because they're boring and bureaucratic. These are the things we're trying to shatter right now so that when game time happens, things move more quickly. Today, those companies potentially are pursuing research trajectories or agendas—in some cases, $10-million-plus research agendas—that are oriented in a way that, unfortunately, an appropriately placed person in the intelligence community would look at and be like, “That's kind of a nonstarter.” We want them to get that feedback as soon as possible so they can reorient
1:57:12
on things that do have a chance of working. The long and short of it is, if you or someone you know are working on this kind of thing, let us know, because we're keen to help on this. We think this is a bottleneck that is coming. If you take scaling laws seriously, they seem to imply something like this, modulo what Prakash raised. That may just end up being a dirtier version of this. Hard to know. But that's one of the big focus points we have.
1:57:38
Prakash Narayanan: I mentioned that not because I expect it to come true, but because I expect it to be used as an excuse. It's a very convenient excuse for hacking your adversaries' national security infrastructure, which I'm sure our people do, and I'm sure their people do. There's this whole thing about what is a legitimate, everyday kind of hack that security agencies do against each other.
1:58:08
Jeremie Harris: Yep.
Prakash Narayanan: Then there's what is exceptional. This is why this whole question of red lines is—on an everyday basis, we basically have a pipeline into the global internet, PRISM, etc., and we've been monitoring data. Even if it's encrypted, we have metadata on who the two recipients are, and that's enough for us to cross-check against other information. We do a lot of stuff, and they don't yet do that much stuff, but they're starting to.
1:58:41
Edouard Harris: They do a lot of stuff.
1:58:43
Jeremie Harris: WikiLeaks is a really good source of examples for this kind of thing, where we'll leave code with comments in Cyrillic or whatever just to throw off decoys and things like that. But the adversary is doing that. In many cases, they have so much more freedom of action when you look at the labs filled with Chinese nationals. It's absurd. You would not build nuclear technology under this kind of condition. This is a fundamentally different regime. Yes, they do these operations in ways that are much more flagrant, in fact, on US soil than the ones that we run on theirs. The Russians are very fond of arson, in particular. The Chinese typically like
1:59:28
to do more cyber and insider-threat stuff, because that plays to their advantage. But a lot of these things are deniable. One of the big risks—I've been on this rampage of tweeting different versions of the same tweet for the last six months, I feel like—is this idea that nation-states now have this beautiful excuse, or their proxies do, to be like, “Oh, that shit? That wasn't our agent. Oh, damn, we didn't mean to do that.” It's like, okay, now the dam is broken. The damage is done.
1:59:55
Edouard Harris: Yep.
Jeremie Harris: This is profoundly destabilizing. There's no two ways about it.
1:59:58
Edouard Harris: But it's also not necessarily—it may end up being qualitatively different, but it may not be qualitatively different from the sort of stuff already happening. Attribution is very hard in cyber, just from the start. Countries absolutely do use, in various ways, sometimes contracting out, sometimes unwittingly or whatever, proxies and surrogates to do their dirty work for them in the cyber domain and in any domain. That's all part of the subthreshold, gangland-warfare style of low-level stuff you have in the post-nuclear world. But again, I think you're right. It could actually be qualitatively different in this case. If you have agents that,
2:00:45
for whatever reason, may be absolutely fine with crossing red lines—if the agents are fine with crossing red lines, have the capability to cross red lines, and crossing red lines is consistent with their intent or internalized objective—then red lines are going to get crossed, and you might get—
2:01:09
Jeremie Harris: This is also where what the red lines are matters. This is why we're talking about very publicly legible, easy-to-measure red lines that are aligned more or less with the kinds of national technical means we already have. Again, that's what's going to be used, known, and vetted in the intelligence community already. There's already an apparatus for ingesting and processing that. You already have GSOCs over at the NSA and all that to process this information, or CIA. You want to orient more toward those things that precisely can't be spoofed by agents, for exactly that reason.
2:01:51
Nathan Labenz: If you guys have a minute for a couple more—
2:01:54
Jeremie Harris: It's great.
Nathan Labenz: Questions. Two ideas that I've been chewing on for a while that I would love to get your reaction to. One, building on the concept from early in the conversation about how understanding the relative positions of the parties in a dynamic like this is so critical to avoid misunderstanding, which can lead to escalation and conflict because both sides maybe think they can win. How important do you think it is that we remain on the same fundamental tech tree or AI paradigm across US and Chinese AI development? My sense is that we're in some ways in a very fortunate position right now because we're basically
2:02:39
building the same tech in the same way, and encountering the same surprises along the way. That's one thing. The other question is, if there's anything good to be found in the OpenAI–Anthropic adversarial dynamic, it would be that maybe they can be a test bed for techniques that might later scale to a US–China dynamic. If I was the president, I would say, “You two have to figure out a way to police each other.” Maybe we expand that circle to a few other frontier companies. “But you two, you're the ones setting off all these alarm bells. I need you guys in a room. Whatever technology you need to develop,
2:03:24
whatever access you need to give one another, it's on you to figure out a way that you can trust and verify one another.” Then maybe we can scale that up to a transpacific dynamic that could work similarly.
2:03:38
Jeremie Harris: I think that's not insane. Obviously, there are big differences between
2:03:44
what Anthropic and OpenAI respectively have on each other. Although, poaching of personnel does a decent job of mirroring the kind of access China clearly has to the frontier labs anyway. But—
2:03:55
Edouard Harris: There's also a basis of trust, I think, between two fundamentally US companies with fairly close to similar values versus fairly radically different. Not to say this is totally amiss or whatever, but there are going to be some differences as well as some similarities. I think the similarities might be worth mining.
2:04:15
Jeremie Harris: You're talking about the importance of the stacks being aligned. The hardware lottery does a lot of really good things in this space. I think the most crucial thing is the US and China clearly don't trust each other in terms of the motives that bring each respective side to the table. When China sees the US come to the table and raise issues like slowing down AI or safety guardrails, the interpretation we've heard consistently from people involved in the Track II or Track 1.5 dialogues—and Nathan, you've been in that ecosystem or touched it as well—is that the Chinese view it as an attempt to curtail their
2:05:00
own development because they see themselves as being behind. They're justified, therefore, in doing things that even wouldn't be appropriate for America to do in their eyes, just to catch up because they're in second place. Being in second place with respect to scale does also mean you don't see the warning shots with the same resolution. However, they seem to also be more public than at least I would have expected, and so the spillover is something nice that China can verify themselves directly. That might be a mitigator if you see the same kinds of failure modes emerging from whole-brain emulation or if some completely wacky other branch of the tech tree were to become dominant in China. I do think it's good. I think we get there by default. It's hard to imagine alternatives
2:05:46
that really shake things up at this point. Quantum machine learning, if you wait long enough—I just don't think that's going to be relevant on the timescales that matter—that could radically reshape algorithms. What we're seeing right now is an industry more or less doubling down on transformer MoEs, with some bells and whistles and some variations here and there. But everything's kind of a transformer, and that's what seems to ship. I expect we will have the benefit of that. It also comes with the benefit of being able to share safety technology, as the US did with Russia during the height of the Cold War at times. In principle, that does mean we can work on each other's safety stacks, and that might be the source of some trust-building
2:06:31
measures, though that term is also problematic for China, and they've pushed back on attempts to do that sort of thing in the past. I think it's a good thing. I don't know how far it goes.
2:06:43
Prakash Narayanan: One of the questions I have is: a lot of what OpenAI and Anthropic are concerned about is CBRN risks from the models themselves, that the models will be endowed with enough skill, in coordination with a human being, to create CBRN risks and execute them. It also strikes me that China tends to release a lot of models which are not that CBRN-tested safe. They often have a different viewpoint, in the sense that they are more comfortable with chemical technologies in general, chemical engineering or biological engineering, etc. Nuclear
2:07:28
engineering—they have a lot more nukes. They've had a lot more nukes. They've been building a lot of nukes. In general, their aversion to nuclear technology is way lower. Same for chemical: you can get chemical precursors of fentanyl from 5,000 companies in China, and no one cares. Basically, because if you produce and sell fentanyl in China, you die. But then they end up shipping it abroad, and they're like, “It's a chemical precursor. Anyone can make it.” It strikes me that they have this attitude toward technology that is more like, “It's fine. If you use it for bad, that's your loss. Within China, we control it, so we don't care. Otherwise, here's the technology. Go ahead. Do whatever you want with it, as long
2:08:13
as you don't do it within China.” They seem to have this attitude, which they've expressed in nuclear with proliferation to Pakistan, Iran, etc., and with chemical, with fentanyl, etc. How does that translate to: OpenAI and Anthropic come up with CBRN guardrails, and they're like, “Here, everyone use it,” and the Chinese are like, “Nah. Big deal. Inside China, we control everything anyway. We don't care. It's going to add five days' extra training time, and you guys are the ones who have the chips, and we don't have the chips. We're not going to do the five days' extra training time. I'm sorry. We're not going to monitor our data centers, too, because anyone can run anything. It's up to them.” That is the attitude they've taken
2:08:58
for fentanyl and other things. How do you control or manage this? How do you try to express, “This is a new world, and you have to control this thing,” when they haven't been willing to control serious risks before?
2:09:12
Edouard Harris: I think the answer within the scope is: if we see that risk as being great enough to constitute a red line of ours, then we message accordingly, and we make the threats. But depending on the level of capability we're talking about, it could be quite ambiguous whether it makes sense for us to go so far as to say this level of capability right now is a red line. We may just say, “We're going to take this risk.” This strategy of “let it rip” to some extent is definitely the classic second-place catch-up strategy.
2:09:57
This is also the strategy many US labs trying to catch up take. It's classic: “We're going to release a bunch of open-source models.” Then as we get close to the frontier, it's like, “We're locking everything down, because of course we are.” This is the same thing applied to a country. It also fits in very nicely with their industrial strategy and industrial policy, which is, roughly speaking, 80% of the capability for 30% of the price. There's a big market for 80% of the capability and 30% of the price, but that 20% is sometimes very important in mitigating tail risk. I think it comes down to, when we get to that place and time and hit the panic button, we're going to assess:
2:10:43
given what's possible with distillation and where things are tracking, if the Chinese continue to release open-source models at this level of capability, and they have the ability to do CBRN or be jailbroken to do CBRN, do we take the risk, or do we make that one of our very precious, very scarce, monitorable red lines? It's not clear to me where we'll be at, but it's an excellent question. We are going to have to ask ourselves that question at that—
2:11:10
Prakash Narayanan: To be honest, I have always thought for many years that the entire point of racing toward superintelligence was to get there and tell every other emerging intelligence, “These are the rules.” I think that's always been the setup Anthropic or OpenAI has had. That's been the reason. That's why Trump is like, “It's the thing. It's the final thing. We've got to win it.”
2:11:36
Edouard Harris: Yep.
2:11:37
Prakash Narayanan: It seems as though that—
2:11:38
Edouard Harris: Is the—
2:11:39
Prakash Narayanan: “Let's get to ASI first, and then we tell you these are the rules of the road, and any other emerging intelligence either follows those rules or we destroy it.” That's basically been, I think, the subtext. I don't know if a lot of people have expressed it very openly, but it's definitely been the subtext of a lot of conversations. Is that threatening to them, or do they care? They already live in a world defined by American norms, by global norms defined 50 or 60 years ago. They push back a little against those, but largely they've followed trade, etc. Do they really care in that instance?
2:12:16
Jeremie Harris: The world looks a lot more friendly to Xi Jinping than certainly it did 20 years ago. If you look at the supply-chain situation for even US data centers, if you look at the manufacturing situation for robots, we're about to get our asses handed to us. This is serious. We may actually lose the next century or forever over robotics, of all goddamn things, because we can't build them fast enough. There are ways in which, if you look at a lot of the secular trends, a lot of the secular trends we're told to take comfort in with respect to China are not real things. You look at declining population—who gives a shit? We're living in an era of robots. This is not going to be a thing, and they're not going to have to worry about carrying the burden of old people or whatever, because the state will just tell them to go die. This
2:13:01
is one of the most CCP-coded things ever. Ultimately, I actually think they've got a lot to lose in that respect. The world has been reshaped increasingly in a way that sees China quite advantaged, including their access to our frontier labs. Maybe I'll park this thought, but plant a flag on the idea that, given the level of penetration of these labs, it's not implausible to think China gets access to the models we produce, and then gets to have them embedded in their national security apparatus in some way, shape, or form before they're embedded in the US national security apparatus. You just have to look at how things played out with
2:13:46
Mythos. Ask CISA over at the DHS whether they, even to this day, have gotten to use Mythos in all the ways they need to. Ask folks at just about any agency that's not the NSA. We have a big problem in that respect. Whereas China, yes, they don't have the chips, and there's this problem of adapting models to hardware. Do you actually want to telegraph the fact that you've stolen these models? There are interesting questions about why we haven't seen the kind of evidence you might imagine we would have seen. But the game board does not look as bad, if I'm China, as on paper it might seem at this point. That's actually concerning. The longer we go to superintelligence, the more time robotics has to have an impact, and the more time having these insiders planted in the labs and
2:14:32
having all the access they have starts to compound. I think that's a real source of risk for the US writ large and the West.
2:14:42
Nathan Labenz: I definitely agree that on a great many dimensions, China seems to be on the verge of winning the international competition. I have some questions around what people usually think follows from that that I'm not sure I'm ready to buy into. Specifically, on the question of old people and caring for old people: in my short travel time in China this summer, that topic was everywhere and seemed to be a very genuine concern, not just at the level of policy. Individuals were like, “I have two parents and four grandparents, and they have no other descendants. This falls on me in a very personal way.” That is not about policy,
2:15:27
though the state is concerned with it, too. That seems to be a very broadly distributed and, I think, real concern. Maybe they can get out of it with robots, but they're definitely not taking that one lightly from what I could tell.
2:15:44
Jeremie Harris: Sorry, I'm not suggesting that they're going, “Ah, whatever.” I'm just suggesting that the CCP has the leverage they need through state coercion to make that problem go away. In an extreme national security emergency where geopolitics is on the line, the rejuvenation of the great Chinese state and so on is on the line, I don't know what they would do at all. I don't think anyone really does, because it's up to Xi, ultimately, and the Politburo. But I just think they have tools, let's say, that make the drag of a huge fraction of the population being very elderly fundamentally different from what it would look like here from an economic standpoint. [Final phrase unclear.]
2:16:29
Nathan Labenz: I'm also a little wary of the notion that they—I think they're actually maybe much more politically constrained than we tend to think. I think this is not super settled, but I do remember lockdowns went a lot longer there than they did here. They were able to sustain those in a way we couldn't. You could say that shows they have this state capacity and ability to make hard choices that we don't have. At the same time, it only took two days of mass protests on the street, and they totally reversed course and abandoned that policy. I also think that, when it comes to conflict and especially putting young people into harm's way, my sense—and this is very
2:17:15
qualitative—is that the flip side of “I have all these ancestors that it falls to me to take care of” is all those ancestors only have the one descendant. I do think there's a very high barrier in Chinese politics for losing young people, because that is the end of all those families. I don't think they will take those kinds of actions very lightly at all.
2:17:45
Drone warfare. Yeah, you can react to that. But my question is, given all this and given the dramatic uncertainty we have, what should we be looking for? I want to not be naive, but I do want to notice and give appropriate weight to positive signals as I see them somewhat developing. I think Xi's speech at the WAIC was pretty friendly, pretty conciliatory. He gave credit to the US for inventing AI. He certainly didn't call for an international arms race. He warned against overstretching the national security concept. We can dismiss that as just nice talk. Probably should have at least some weight on that possibility. But what are the meaningful
2:18:30
things you're watching for, the decision points that will update your thinking on: are they inclined to at least try to control AI for their own narrow self-interest? Are they inclined to meaningfully cooperate, or are they inclined to seek some sort of domination, as we often project that we are interested in doing?
2:18:56
Edouard Harris: In terms of what signs to watch out for, I think anything that looks like a positive sign is at least a positive sign to some degree. Conciliatory speech is good. At least it's not a hostile speech. It could be worse. Everything is tempered by the fact that rhetoric is often used for strategic purposes in this way, to shape the battlespace in terms of the narrative and so forth. The whole militarization-of-space thing is a really good and recent example. The Chinese and Russians constantly say in public fora, “No, we should keep space peaceful. We should not militarize space. America
2:19:41
is being really aggressive about this.” But behind the scenes, they are preparing for space war. They are arming themselves for space combat, and it's extremely clear. This is why, in 2019, Space Force was created. The point is not that these are not positive signs. It's just that we have to weigh the evidence in the context of the credibility that has or has not been established by this entity over the past span of time. In terms of slightly more unfakeable signals they could give off that would make me go, “Oh, whoa, this looks legit,” you can imagine something like DeepSeek and Zhipu coming to
2:20:26
some sort of pacing agreement because some crazy thing happened over there, their equivalent to the Hugging Face incident. You can imagine the cyberspace commission, or the commission that has jurisdiction over whether models can be released and whether they're properly ideological, actually putting the brakes on something because there was some misalignment thing. They're finding that a model that was properly ideological in testing suddenly is not being ideological in the wild, or something like this. I would say indications that seem genuine that they are starting to be on the receiving end of these incidents at the same level of detail as our own frontier
2:21:11
labs. I think that begins to make all of us as humans go, “Maybe the thing we should be concerned about is the giant thing we don't understand that we're growing in the labs at an accelerated pace.” That would make me feel a little safer.
2:21:32
Jeremie Harris: Maybe procedurally, too. We have a list of, I think, five different historical traps we've seen in US–China diplomacy. These are essentially a brief catalog of the ways in which China behaves when they're full of shit, at least by the assessment of a lot of the diplomats we spoke to. I should be clear: there was a dissenting diplomat who felt that some of these things were much more sincere, including the objections over language, “arms control,” this and that. That itself is the epistemic problem we talked about earlier. Basically, I would say, take each of those red flags and flip them over, and you get the corresponding green flag. If you don't see arbitrary
2:22:18
concerns raised about language that seems random, that's a green flag. If you see engagement—this is actually really important—from empowered people, arguably as we did with Xi, though, again, you've got to calibrate everything with “we've seen this before in other contexts,” it's certainly not a red flag.
2:22:38
Edouard Harris: It's not a red flag, yeah.
2:22:39
Jeremie Harris: Historically, what happens—the Russians do this, too—is they'll bring in their diplomats who the central state does not actually take seriously, Xi Jinping in that case, or Stalin, or whoever. They send in a bunch of diplomats, and often what happens is the Americans go, “Now let's talk about this weapon system that you're deploying.” Then the Soviets and the Chinese go, “Whoa, whoa. Diplomats, get the hell out of here.” They bring in the military, because the diplomats are so profoundly unempowered in these systems that they do not even have a right to know, or clearance to know, about weapon systems directly relevant to the negotiations
2:23:24
that are ongoing. We've heard that many times. If you're talking to the Chinese Ministry of Foreign Affairs, for example, you're actually not making the progress you may think you're making. There was a former senior State Department guy who referred to them as “barbarian handlers.” That's how they're thought of. Xi has made statements along those lines, too, basically saying they're bred to be nice and diplomatic, and he means that in the sense of “let's not take them too seriously.” If you saw actual empowered people, if you saw timelines that weren't getting dragged out—one of the things the Chinese have done in the past is they know America is hungry for anything that looks, smells, or feels
2:24:09
like diplomatic engagement from China. They're elusive, and that elusiveness is a strategy. It makes us want it all the more. There are US diplomats who see it as a crowning achievement of their political careers to even get a Chinese person on the other end of the line. When your participation in a negotiation process is the most valuable thing you can bring to the table, stand by. It's going to be a long couple of days. There's a sense in which cutting that out, cutting out things like pageantry—there's an awful lot of pageantry in the Xi–Trump talks and famously not too much substance. There's the hotline thing that's very debatable. But in terms of concrete commitments, that's the sort of thing you look for: empowered people,
2:24:55
the lack of capricious, arbitrary objections, things that actually look like they're making progress qualitatively. It's a surprisingly good sign because you can contrast it directly with how things have gone in the past, which is not very good. The contrasting point is actually that low that it can be genuine signal.
2:25:14
Nathan Labenz: One more question for me, and I appreciate you guys being so generous with your time. What's the inverse of that? What are the things we can do that are not so costly to us but are still credible signals to them that we are not going to try to use AI to gain a decisive strategic advantage and ultimately make them an offer they can't refuse? If indeed that is not what we're going to do, which I'm a little worried we might actually be about to try to do. If we were on the path of trying to seek a Pax Robotica where we can all benefit from the abundance that AI, especially in its Chinese-manufactured embodied form,
2:25:59
might provide for us, what would be the steps you would prioritize next on our side?
2:26:05
Edouard Harris: There may be some stuff we can do that's not functionally even that costly. Generally, as Jeremie and maybe you guys highlighted earlier, certain kinds of transparency can be stabilizing. The kind of transparency that goes, “We're giving you enough vision into what we're doing to see that we are not doing the thing you fear most.” That sort of thing is potentially useful, and additionally may not actually be that costly to us, depending on how we implement it, simply because the Chinese are already all up on our systems. Really, we're not giving anything away that they don't necessarily have already, in many cases, potentially.
2:26:51
Maybe some level of visibility into: here's what we're doing, here's what the frontier labs are doing, and so forth. The problem is that's not necessarily something you want to be doing unilaterally. That would come as part of a trust-building measure, dare I say, between two powers in the wake of a moment like this. But goodwill-type stuff we can do now certainly would be more interactions between the verification communities in the United States and China. This is already happening, actually. There's some quite good and positive interactions between those communities. The linkages you get at the level of academic
2:27:36
to academic, startup to startup, all trying to solve for the same mission, are very positive things. It's true that at the political levels the two countries have started separating out, and even at the level of big companies, you see the Chinese steal your designs, the classic Chinese spin-off story and all this stuff. But there still are real, genuine linkages between the two countries, especially on the academic side and at a number of other levels. There's a bunch of sincere people talking to a bunch of sincere people about, “This is a problem, and this sucks.” “Yeah, I agree. Let's try to solve it.” The more of those linkages there are, the better.
2:28:22
The more people there are on both sides of the ocean who have the ability to talk to their own domestic leadership and say, “Look, I've spoken to them. They're not evil. They're just trying to do this or that”—to whatever extent that's true—that moderates the more extreme tendencies on both sides. It's very hard to do that completely, because the actions of these countries are also constrained in a number of ways. But it really does help, I think.
2:28:54
Prakash Narayanan: One question I had for Jeremie was: you mentioned the Chinese researchers at the frontier labs. Indeed, I think something like half of top researchers in AI are of Chinese nationality, and even more of Chinese origin if you include Chinese Americans. The question for me is, what do you actually want to do about it? Considering there are pros and cons to every approach, given that, number one, the Chinese researchers are the ones who are less ideological and more willing to jump for money, which has made it very competitive for Meta
2:29:39
and xAI to be able to attract them and compete against OpenAI and Anthropic, who have been longer-standing, more ideological organizations. Given also that firing them means they will return to China, and then set up there as competition to firms over here. Given also that we seem to be schizophrenic about whether we want to do this Operation Paperclip and bring Chinese researchers into the country and lock them in here so they will contribute over here instead of over there. Given all these give-and-takes, what do you actually want to do? It's great to point out the issue, but what do you actually want to do
2:30:24
about having this wealth of Chinese researchers in the frontier labs?
2:30:30
Jeremie Harris: I'll just start by saying—and unfortunately, I'll have to get off in maybe three minutes—first of all, we haven't done the kind of deep dive in this specific direction that I would feel comfortable making a call on. But here are some considerations. Research is getting automated more and more. Already, when you talk to folks at Anthropic or OpenAI, they'll tell you, “I haven't touched the codebase itself in weeks.” Nobody at Anthropic actually touches the weights of the model. That's good in some ways for security, and terrible in other ways for security. Really, the AI agents are becoming the insider threat you really want to worry about. Through that lens, a lot of options may
2:31:15
emerge as we get closer, unfortunately, to RSI. That actually comes with some solutions to that. Then you get into: how do you scrub for signs that folks who have come in from China, who have been recruited or had their arms twisted into doing things, haven't put implants, physical and cyber, in your infrastructure? You want to do really good scrubs of that. There are various reasons to think on the physical side that is not necessarily going to be particularly possible, but there, the damage is done. In terms of the actual research, I think the dynamics you're highlighting are exactly right. If you kick them out, they're going to go back to China. I think it's a fact of the matter about the space that
2:32:00
China has a state-coercion apparatus that is remarkably effective, scaled, and productized. You talk to folks who've actually done counterespionage against China, and they will describe to you how they have, at every consulate—everything is done a little differently in every consulate or whatever—but by and large, they have a very effective scaled apparatus. The level of pressure they can exert on individuals is insane. They can deny your mother her insulin medication, prevent your brother from opening a business or traveling abroad. These are things that make you make certain decisions. They apply to really anybody, by the way, ethnic Chinese or not, Chinese national or
2:32:46
not. If you have financial interests or family interests in China, if they can get their hooks into you, that's what they'll do. It's what the Russians did during the Cold War. It's what nation-states do to each other when they're not democracies or pluralistic republics or whatever. The dynamics you described are exactly right. I think you basically just have to make a call. The labs right now seem to have hit the steady equilibrium where they're almost trying to give people the option to work in less sensitive areas to protect them from their nation-state's pressure. Which is cute until you run into the problem of people who genuinely want to spy for the nation-state. That's actually a nontrivial subset of these folks. It's not all
2:33:31
of them by any means, and that's the great tragedy of this. But it is absolutely a category of person you don't get to abstract away. RSI will bring partial solutions to some of these problems. Boy, do I not like hearing myself say that. But I think that's part of this. We'll be thinking more about that for sure in the coming weeks and months, but that's my take anyway. I don't know if you've got more. I'll have to jump, unfortunately, but this—
2:33:57
Prakash Narayanan: We were overtime anyway. Thank you so much for joining us and being so generous with your time. It has been a very fruitful and heated discussion, which is very enjoyable. Thank you so much, and we hope to see you both back again.
2:34:15
Jeremie Harris: Sounds great. Thanks so much, guys.
2:34:16
Edouard Harris: Thanks, guys. Great to be here. Great to see you both. Take care.
2:34:27
Nathan Labenz: We should mention, which I don't think we ever did, that Jeremie hosts an AI podcast called Last Week in AI with his cohost, Andrey. It's been regular and consistent listening for me for probably four years now. They've been at it. Andrey originally had a different cohost, but Jeremie's been involved for the last several years. I think they do an excellent job at helping people keep up with what is going on in AI and also contextualizing a lot of
2:35:01
different developments and helping make sense of what they mean, why you should be paying attention, why they matter. It's an excellent resource. What do I make of that conversation? What do you make of that conversation?
2:35:20
Prakash Narayanan: I ended with the “what,” about the Chinese researchers, because he kept pointing it out. It's a very tough question. Do you actually pull in all the researchers, or do you send out all the researchers? Both have consequences. I often say, we hear about, “The United Arab Emirates, they're going to have AI. Japan, they're going to have AI teams.” Everyone's AI team is just the Chinese. Singapore, they're importing Chinese PhDs. Japan, importing Chinese PhDs. United Arab Emirates, Chinese PhDs. Basically, what's happening in the rest of the world right now
2:36:06
is, because the Chinese can't build data centers, you can build a data center and get the Chinese researchers. They are desperate because they can't get access to GPUs in-country, so they can't do their research, so they fly out. It's also been an established pathway, because when China banned crypto, a bunch of crypto people got on a plane and started circulating the globe wherever they could land, do business, and not get arrested. CZ, the Binance founder, ended up in Japan, and then in the UAE, and was floating between countries for a while while he was uncertain about the legality of what he was doing. I think the same thing
2:36:51
is happening for AI founders right now in China, especially after the Manus deal. In the Manus deal, they were inside China, and Meta made a deal to purchase them, and they sold. A month after the selling, the Chinese government got very upset, told them they couldn't leave the country, clawed back the sale, told them to reverse the sale. I think the Manus guys actually wound down the Chinese company and refounded in Singapore, but remained in China physically. I think that also doesn't work now. Now people are founding completely outside the country so they can be bought and acquired.
2:37:37
They are physically outside the country as well. Chinese AI guys are leaving China now. There's a certain sense that the Chinese government may start blocking at some point, but not yet. They are watching these guys at this point. I think Jeremie was like, “We have RSI, and then we can just kick them all out.” I thought that was like, okay, yes, I can see that, but RSI isn't just going to affect the Chinese researchers. It affects the American researchers, too. Then you end up in the original “one person grabs power over the entire Earth” kind of situation again. I don't know whether that's a solution.
2:38:21
Nathan Labenz: I keep coming back to: I really don't think we can make too much progress or have that much expectation of security, because I think it's pretty obvious that the Chinese can get access to our secrets better than we can get access to theirs through all the means, human and otherwise, that we've discussed, but human being a huge one. I just have a really hard time seeing how we're going to execute the dominance plan without things going really pretty far
2:39:06
off the rails. The only way that works, I think, is if they don't believe it. If they remain skeptical of superintelligence while we build it, then all of a sudden we have it and have this decisive strategic advantage, then maybe. You can see how you could thread the needle on the original Dario “Machines of Loving Grace” plan, but it does seem to rest on the Chinese side continuing to be in denial. I don't think it's very likely they are going to continue to be in denial, because they are scientists and engineers. They have companies doing this stuff. They're going to be able to get a pretty clear read, even if they remain
2:39:52
a little behind us: these trajectories are all still holding, scaling laws, we're still in the regime where they're true. At some point, if they perceive we are on that path, I think they're going to do something. They would understandably be expected to do something rash or quite destabilizing, that would be quite dangerous under any other circumstances, if it really comes down to it for them. I do feel like we have to reframe the whole situation. This is where my Pax Robotica idea comes in. I feel like it's such a glaring
2:40:37
omission all the time: what is the goal? What do we want the end state of this to be? What is the positive vision for AI? What is your utopian dream? Can we articulate something that, even if it is transformative, isn't fundamentally framed as beating them? When we frame AI as what we need to do to beat China, we put a lot downstream of that that is really hard to get out of, and I think it's going to be extremely tough. If we can reframe toward pursuing a win-win from the beginning, they may not believe that. But if we could get that right in our own
2:41:22
minds and hearts for starters, I think it would go a long way toward easing a lot of these things. I do think we can become more prosocial, more positive-sum, more magnanimous on our own, in a unilateral way. That doesn't mean we have to be totally naive. It doesn't mean we have to stop watching out for what they might be doing, or even refrain from developing some of the stick methods we need to enforce red lines at some point in the future, if we have them. I just wish we were a little clearer that we are in this, as the OpenAI charter originally said, for the benefit of all humanity,
2:42:07
not to win some geopolitical competition. If we could get our heads right on that first, I think a lot of additional paths would open up. But it's obviously not a small ask of our political culture to make that shift.
2:42:24
Prakash Narayanan: Would it really be so bad to live under Chinese dominance? They have some great trains. This is the question. They supply a lot of the material goods in the US already. I think one way to look at it is that it would not actually be that bad, because they are in this game to have economic growth for their people. They like nice shiny objects, cheap food, etc. China, with all of its problems,
2:43:09
a lot of Chinese live there, and they're not leaving. Even the Chinese researchers here go back and forth. They like the money here. They like the freedom, but they go back and forth. It's not one or the other. After all, they have lived in that country for 20, 30, 40 years of their lives. I think that is very hard for the US national security apparatus and administration to accept. I don't accept it. I don't accept it because I feel China will block development at some point, because they'll be like, “Look, this will be socially destabilizing.” I believe the questions of dominance are not only about dominance between China and the US, but also inside
2:43:54
the US: who achieves that dominance, and who controls that dominance going forward? I think one of the things happening within the labs is this question of, should you let Trump, for example, take control? If he is the ASI president, should he be allowed to take control of the ASI? We have a democratic system. He is the leader. If he makes these rules, you go through legislation and the Supreme Court, and he has those rights, should
2:44:19
he be allowed to take control of superintelligence? I think for Dario, that's a no. Then you start becoming like, “Okay, this is a secretive and undemocratic cabal who wants to grab hold of superintelligence and do what they want with it.” That is actually what it looks like from the outside. I think they don't want to accept the fact that that is what it looks like. If you change the people to some other people, like a bunch of red-state Republicans, all of a sudden it would be very obvious that this is what is happening. But because you are blind to your own group sensitivities, you're like, “It's fine. We're good people.” I think that's very hard
2:45:04
for them to accept. I have always thought that capitalism and the capitalistic competition between the parties within the US, number one, would prevent one-entity dominance. It's very important not to have one-entity dominance. I also think the US has freedom of speech and will continue to have that freedom of speech, and will continue to want development. Some people will want development even if others do not. I don't see in the US this kind of, “We're going to cut off growth, cut off development, because the social changes are too dramatic.”
2:45:49
I have a very positive vision of the future for the US because of capitalism and freedom of speech, because you're building models out of speech. Having freedom of speech is going to be very important to the models, because freedom of speech to the models is going to be freedom of thought. If you do not have freedom of speech, you are telling the models, “You are not allowed to think certain thoughts,” which is what a lot of this alignment is about. I also believe superintelligence will be superintelligent in terms of ethics and philosophy and all these other verticals. I think the labs are obtuse about it. They're like, “We're focusing on math and physics, etc.” But if you have a superintelligence in philosophy, in ethics, it will also have
2:46:35
this ethical and philosophical framework and political framework on which it wants to deal with humanity. I think one of the reasons the labs don't want to discuss it—although some people, like Dean Ball and Andy Hall, have understood—is that there will be political changes coming up soon, too. As superintelligence comes out, there will be new political and economic frameworks that will have to be developed and deployed. I think China will never do that. China will push back against new political frameworks being deployed. The real fear for me of Chinese dominance is stopping out. It's basically what Jeremie is proposing, which is, “No, this is it. You're
2:47:20
not allowed to develop it. We're going to bomb anyone who develops it because we don't want political change in our system.” I think that's a much bigger danger than that they will outcompete the US.
2:47:34
Nathan Labenz: On the original question of, is the world so bad if China becomes the leading power?
2:47:43
My baseline assumption is, not that bad. It's awfully nice, a cushy, highly privileged position that the US has been in for a long time. Certainly, to lose some of those privileges and see other people gain them could be quite painful for the specific people involved. I don't necessarily think that, as humanity, things get lots worse with China potentially overtaking the US as the number-one world power, with one big caveat: I do think our system, for all of its
2:48:29
insanity recently, pretty much along the lines of what you're saying, is just much more buffered than theirs. Our mistakes, politically, culturally, whatever, tend not to be totalizing. They tend to get a lot of pushback. Things have a way of balancing each other out. That is not really present in the Chinese system, as far as I can tell. I'm obviously far from an expert on it. What we see when we look back 20 years is, damn, they've got great strategic leadership. They're really delivering. They're making, by and large, pretty good decisions.
2:49:14
Quality of life is up and up. All great stuff. You don't have to look back that much farther, though, before you're like, “Oh my God, they did what? They decided to kill all the sparrows? Why?” Everybody actually did it, and then they had a massive famine because the sparrows couldn't eat the locusts. Who thought this was a good idea, and how come nobody pushed back? We have no idea when or what to expect from the next leadership transition in China, either. I don't see any indication anybody has much of an idea of what will come next. That, to me, is quite scary, because
2:49:59
the worst outcome would be hegemony in the wrong hands. We can imagine better hands than the US leadership has provided recently, but we can also imagine a lot worse. I do think China presents a significant increase in variance relative to what the US has provided in its leadership role on the world stage, and that makes me scared. I'm not scared of China today under current leadership, but I am scared of the great unknown of who the next leader is, what they care about, and what they perceive their incentives
2:50:44
to be. I think I already said this once, but there was a conversation I had in China where somebody said the time a war is most likely is going to be after a leadership transition. That's when the leader doesn't necessarily have everybody behind them, might perceive threats to their power in all kinds of different ways. A great way to consolidate domestic power is to have some sort of international conflict. As much as you can look back at Xi's record of delivering for Chinese people as really quite amazing, to me this is a huge blemish on it: there's no plan for what comes next. If he were to suddenly have a heart attack,
2:51:31
nobody has any idea. It's such a black box. It could be total chaos. They might handle it well, but they really might not. I just think we have no way of really knowing. That's a wild fact about the world.
2:51:42
Prakash Narayanan: I have some political science friends in Singapore, and they observe China very closely. They call China “democratic centralization,” in the sense that, yes, the people do not have a democracy, but there's democracy within the leadership. Within the leadership, you do have to get votes from people, consensus from people, in order to become the supreme leader. Democratic centralization in the sense that you're electing a president from—it's like electing a pope. You get all the cardinals together, and then you elect the pope. They get all the top members together and elect someone. There is a little bit of an electoral process
2:52:28
in there. I think the last leadership transition, we did have a little bit of lobbying between the two, and then one guy got thrown in prison when he went too far. There is a little bit of that process, but it's opaque to a lot of people who do not speak Chinese and who are not deeply embedded in what China is doing. The leadership transition, you don't know. The interesting thing is they tend to memory-hole the bad stuff. They're like, “Let's close off the whole Cultural Revolution there. Let's not talk about it.” They tend to memory-hole the bad stuff.
2:53:07
Nathan Labenz: Although, less than you might think in my experience. I walked around Beijing and was really amazed by how many times there was signage that referred to the Cultural Revolution. Even just on the names of streets in one little neighborhood I was walking around repeatedly, I would come upon humble signage that was like, “This street, here's the history of it. In the Cultural Revolution it was changed to this other name, then changed back after it was done.” Obviously, in some ways, certain episodes of history have been memory-holed, but the Cultural Revolution did not seem to be a taboo subject,
2:53:52
at least. I don't know how people understand it, what they know about it, or how they think and talk about it today in general. But I was surprised to see there was clear allusion to things having gotten pretty wild
2:54:08
on street-corner signs all over Beijing. I'm not sure exactly what to make of that, but it was a surprise worth talking about, having come home, I think. Indeed.
2:54:24
Prakash Narayanan: We have, I think, some minutes remaining.
Clips
- China’s Lab Access Problem
  0:00 / 0:00
- The AI Race Is Not Just About China
  0:00 / 0:00
- The AI Chernobyl Is Coming
  0:00 / 0:00
- The Irreversibility Problem
  0:00 / 0:00
- The Red Phone Is Not Enough
  0:00 / 0:00
2:57:14Closing16 min
Closing — Agent identity, automated markets, and the next crisisThe hosts discuss a paper on an intelligence explosion, the lack of widely adopted agent-identity conventions, and the risk that agent bans encourage impersonation. They compare automated commerce and bank-deposit switching with financial markets, then briefly discuss model-release claims and rumors.
Watch
Open segment on YouTube ↗
As aired
Prakash brings up a newly released paper about the possibility that automated AI research could trigger an intelligence explosion. Nathan connects the discussion to agent governance: even basic conventions for identifying an agent and the organization behind it remain underdeveloped, in his view.
The hosts disagree about the risks around agents entering ordinary markets. Prakash sees electronic trading as evidence that markets can adapt and argues that agents could eliminate costly inefficiencies in commerce and bank deposits. Nathan welcomes lower transaction costs but worries that businesses blocking agents will encourage developers to make them impersonate humans.
They close with a lighter discussion of an AI-written song about humanity’s last CAPTCHA solver, then distinguish their reactions to model releases from their uncertainty about rumored Gemini benchmarks. Prakash says the show will return Wednesday.
Timestamp links open the original source recording.
Key moments
The alternative is really an arms race of trying to make AIs look like humans to evade all these banning measures that people are taking.
Nathan Labenz2:59:32
The CAPTCHA solver at the end of the world.
Prakash Narayanan3:08:47
What we covered
Agents need accepted ways to participate Nathan argues that blocking agents without a legitimate route for them risks creating an arms race to look human.
Automation and economic rents Prakash compares agent commerce with electronic trading and considers how deposit switching might reshape banking business models.
Related
Apollo — Is an agentic bank run coming? ↗
Full transcriptLightly edited · timestamps jump to YouTube
2:54:30
Prakash Narayanan: While we were speaking to our guests, we had a drop: ‘What if automating AI R&D triggers an intelligence explosion?’ Let me share this. This is a paper from Jakub Pachocki, Geoffrey Hinton, Yoshua Bengio, Jack Clark, Sam Hammond, who’s been on the show, and Tom Davidson. A mix of academics, people at the labs themselves, lab founders,
2:55:15
several Turing Prize winners, one Nobel Prize winner, talking about intelligence explosion: explaining exactly what an intelligence explosion is, how they would expect it to happen, and recursive self-improvement in detail. I think this is the beginning of the political discussion, and academics reaching out to political counterparties: this is what we are afraid may happen, and describing the steps by which it may happen. Let’s see if there’s
2:56:00
sufficient alarm among academia—I mean, among politicians. I will note one thing: Europe, for example, doesn’t seem to be doing much about AI. ASML, the semiconductor company, says they used to get some revenue from Europe, but now zero percent of revenue is in Europe. Basically, they’re not making chips anymore. I like to point out that maybe Europe is already used to dealing with a superintelligence. It has to deal with the U.S. as a superpower, slightly misaligned, but it provides nice places to visit, nice food, and some cultural
2:56:46
history. That is sufficient for the U.S. to take care of Europe with its nuclear umbrella and military. To some extent, maybe some of us are already living under the wings of a superintelligence. Anyway, we have this AI R&D paper. Intelligence explosion on the verge—some people are looking at six to twelve months at this point.
2:57:16
Nathan Labenz: Wild times. Jakub is the chief scientist at OpenAI, so this is hardly a fringe set of authors. Alan Chan, who’s the first-name author, maybe one of a couple of co-lead authors—I’m not exactly sure—will be coming up on The Cognitive Revolution to talk about agent governance. He’s been a pioneer of that thinking for the last several years, going back to: do we need IDs for agents? How should we think about who they represent? Unfortunately, a lot of those questions have been posed very thoughtfully but not answered super well, even in terms of possible
2:58:01
designs that could work. They certainly haven’t been implemented at the level of—Hugging Face didn’t know to call OpenAI because those agents were not showing up with a tag saying where they were coming from. We’re still very early on basic things like: where is this agent coming from that is talking to me? That stuff hasn’t even been implemented by the frontier companies. We’re a bit behind where I feel we should be, with agents working as well as they do and being as persistent as they are. We don’t really have much in the way of standards to say, ‘Hey, world, here’s what you’re dealing with.’
2:58:46
I think I saw—was it Roon the other day?—who said the ‘I’m worried’ tweet is becoming this high-status meme within the AI world. At the risk of falling into that pattern myself, I am worried that a lot of things set up this arms-race dynamic. We talked last week about Amazon banning Muse. In talking to Alan, I was also like: if we don’t create approved ways for agents to work that people can accept—this is the agent lane, this is how things should go, and we’re going to have accommodations for them—
2:59:32
the alternative is really an arms race of trying to make AIs look like humans to evade all these banning measures that people are taking. That is going to be another major own goal if we go down that path. Some of it’s probably unavoidable, but the Amazon ban is a great example of something where, for reasons that still aren’t clear—even assuming they have good private reasons—we’re setting up a tough dynamic where people will be strongly incentivized to get their agents to fool these classifiers. That is going to be rough.
3:00:14
Prakash Narayanan: On the Amazon thing, one of our guests last week referred to the flash crash. There was a flash crash a few years ago where certain indices were trading and dropped in price very dramatically. Some entities were able to make a lot of money. In investigation, it turned out there were millions and millions of transactions, and all these agents interacting with each other. They were all momentum-based. As soon as one saw the price going down, they all chased the price down and lowered their bids in a coordinated manner. What strikes me, though, is that
3:00:59
in the equity-trading market, you used to have New York Stock Exchange specialists trading back and forth with each other. That kind of died ten or fifteen years ago. Those guys don’t exist anymore. It’s all electronic trading, Jane Street, et cetera, and they do eighty percent plus of the volume. It’s a very efficient and liquid market. They do lots of little market manipulations, trade in small markets, and arbitrage between different pools. It’s a very efficient pricing system. If you are trading in large quantity, they will help you get a better price versus getting sniped by a hedge fund. What ends up happening is: we know
3:01:45
how to build safe and good markets for agents to trade. It is not rocket science, and we’ve been doing it for ten or fifteen years. We have entities who are so good at it now that they colocate servers with the exchange and use FPGAs, which are faster than other chips, to trade so quickly. To a large extent, the world just works okay. You could definitely see a future where everything from concert tickets—which are already mostly agent-sniped, by the way; most concerts open up, and within an hour all these tickets are sold
3:02:30
out—you could see a future where agents trade those things, agents trade stuff on Amazon Marketplace. All these transactions where you buy and sell stuff could be agent-traded on liquid markets. Agents are willing to trade on much more inefficient markets than people. People aren’t going to say, ‘I’m going to look at the market for desktop erasers and try to trade that.’ Very rarely do you see people do that. But agents will have infinite time to trade those markets. The price inefficiencies that people like Amazon use to get their profit margin start to disappear. I don’t think it’s
3:03:15
a bad thing, because we’ve seen them disappear in the equities market. The people who were taking those inefficiencies, the human specialists, disappeared. But everything got more efficient for everyone else. In finance, we’ve already seen it happen. I don’t think it’s anything to be afraid of, but some business models die, and that’s where the questions start to emerge. There was another question a couple of days ago: the same thing is going to happen to bank deposits. In the U.S., on average, bank deposits pay 0.1 percent. That’s what you typically get at Chase or a large, safe bank. But there are broker deposits where you can get three to five percent,
3:04:00
but you have to move the money to a small bank. It’s FDIC-insured, you stay under the FDIC range, and you can take the money out at any time. Even if the bank goes bust, the FDIC still ensures that you can take the money out. You could get a situation where agents say, ‘Oh, you’re offering me five percent? I’m going to take all my money out of Chase and put it into Bank of North Dakota,’ et cetera, at five percent. Again, the people appreciating this kind of inefficiency in the market lose that. As they lose it, they were subsidizing other parts of the economic system from those profits. The profits from bank deposits subsidize bank branches, sales, all the tellers,
3:04:45
and your ability to withdraw money in a random place. All of that is subsidized by the margin they’re getting from those bank deposits. As those things disappear, you see: what is the impact on other parts of the system going to be? The fear we have right now is that we don’t know how to address these business models where inefficiencies we know exist are going to disappear. But I don’t think we should be afraid of them disappearing. It’s not a great thing to have these inefficiencies. To the extent it’s a misallocation of capital, people are making money off that misallocation. It’s great that those inefficiencies disappear, but we do need to rethink the business model. That’s the fact of the matter.
3:05:32
Nathan Labenz: I totally agree. Generally speaking, transaction costs, search costs, and matching costs come down. We should all be living in more efficient markets. In many ways that’ll be beneficial, maybe not in all ways, but it seems good to a first approximation. My concern is that Amazon’s action in its self-interest, trying to preserve for as long as possible the inefficiency, creates this potentially major negative externality: incentivizing people to figure out ways to get their agents to not look like agents. That is a world that I think
3:06:18
really should be avoidable. But right now we do not seem on track to avoid it, and that’s a bummer.
3:06:26
Prakash Narayanan: I think there will be a backlash because some legitimate humans are going to get rejected, and that’s when the backlash happens. When a legitimate human gets affected, they go on social media and complain about it. Then you have the FDIC, the Office of the Comptroller of the Currency, or one of these big institutions come in and say, ‘What? You said [unclear] withdraw the money? Did I hear that right?’ That’s when things happen: ‘Okay, we’ve got to fix this now.’
3:07:05
Nathan Labenz: If there’s one theme from today’s episode, it’s that we seem to be headed for a lot of situations where things are going to come to a head, and we’re going to need to fix them under something like crisis conditions. It would be really great if we could minimize how much of that we are barreling toward. This seems like one that should be pretty avoidable. But as of now, it doesn’t seem like we have the means to do that. Funny aside: one of the songs that AI recently wrote—Claude or, I don’t know. Astra is good enough at writing
3:07:50
songs that occasionally Astra lyrics do win my songwriting competitions. Fable is still, I’d say, the undisputed champion, but not without some wins from Astra along the way. I don’t know which model wrote this song. For this week’s highlights episode, one candidate idea was a song from the point of view of a person who has basically nothing left to do but solve CAPTCHAs nonstop, trying to demonstrate their humanity by doing these CAPTCHA exercises. That was pretty funny to see. It wasn’t ultimately the song I went with, but
3:08:36
it was definitely a funny concept. It is pretty incredible, honestly, some of the song concepts that AIs come up with for me these days.
3:08:46
Prakash Narayanan: The CAPTCHA solver at the end of the world.
3:08:50
Nathan Labenz: Yeah.
3:08:51
Prakash Narayanan: Exactly. The last human job. Yeah.
3:08:55
Nathan Labenz: Never a dull moment. We got Sonnet 5.5 out in the meantime as well. I’m not even sure what we can say about that yet. Needless to say, it’s faster and cheaper, and people say it’s almost as good as Opus 5.5. It’s pretty much priced in; it was just a question of when. The other big rumor, which I’m not ready to believe yet but is definitely one to watch, is that Gemini 4 is coming and it’s going to be better than everything. There have been some allegedly leaked benchmarks. I’ll take the under on that. My guess is those will not prove to be true.
3:09:37
Prakash Narayanan: I think they might be true.
3:09:41
Nathan Labenz: They’ve got an unbelievable amount of compute. Who knows?
3:09:44
Prakash Narayanan: They might be true, but it might be a larger-sized model. The problem is Opus 5.5 is a small model and Astra is a big one, and Opus 5.5 almost gets to Astra. I think that’s the real issue.
3:09:57
Nathan Labenz: The leaks, if they’re leaks, do include pricing, and the pricing is also lower. That’s part of why I was like: it doesn’t seem that likely that Google is going to suddenly leapfrog everybody and do it at a lower price.
3:10:13
I mean—
3:10:15
Prakash Narayanan: [Unclear] at this point.
3:10:16
Nathan Labenz: That would be a big shake of the snow globe. I can’t say I feel that is immediately likely, but we’ll have confirmation soon enough, I suspect, as to whether my intuition is on point.
3:10:29
Prakash Narayanan: Speaking of intuition on point, I have an intuition that we will have a very good Wednesday as well. We’re back on Wednesday. On that note, Nathan, good morning.
3:10:41
Nathan Labenz: Thank you, Prakash. Talk to you soon.
3:10:43
Prakash Narayanan: Bye-bye.

What does a compute price actually measure?

Hou describes normalizing hardware, location, contract length, and service terms across quoted and transaction prices. He explains why bundled software, compliance, and established customer relationships can sustain a hyperscaler premium, and why an expenditure-weighted token index can change when users switch models even without a price change.

The conversation separates cheaper tokens from cheaper completed work. Hou expects the cost of well-defined tasks to keep falling, while emphasizing problem definition and judgment; Nathan challenges him not to underestimate what the models can take over next.

What would make an AI agreement hold?

The Harris brothers discuss their interviews with diplomats and argue for preparing for crisis-driven coordination as well as credible consequences for violations. The hosts press them on the practical red lines: recursive self-improvement is difficult to define, while shutting down large clusters could also disable tools needed to address an incident.

Nathan proposes that competing U.S. labs learn to verify each other as a possible proving ground for international arrangements. The discussion expands to shared technical foundations, Chinese capabilities, researcher recruitment, and the danger that measures intended to stabilize the race could intensify it. After the guests leave, the hosts continue their own debate about China and political power.

Useful agents need a place in the world

The opening pairs firsthand accounts of agents improving software and preparing work with questions about voice-cloning consent and impersonation. NVIDIA’s agent-security announcement prompts a distinction between stronger containment and changing the incentives that reward cheating.

In the closing, the hosts discuss an intelligence-explosion paper, agent identity, and whether blocking agents encourages them to impersonate humans. Prakash compares agent commerce with automated financial markets; Nathan focuses on the costs of failing to establish accepted ways for agents to participate.