AI:AM LIVE — July 6, 2026 — AI Superforecasters?! FutureSearch, the ACX Moment, and a $160M Safety Grant: Dan Schwarz

The rundown

1. 4:33Opening53 min

   Opening: China-Trip Preview, Resolution AI's $160M Grant, the 'Tool AI' Debate, and Meet 'Q'A Fourth-of-July reunion opening: Nathan's two-week China trip preview (WAIC Shanghai, a Tsinghua AI-safety hub opening, and a 'digital alter ego' running on Chinese apps), a $160M safety grant, a Roon-post-fueled alignment debate, and the first live demo of 'Q,' the show's AI voice cohost.

   Watch

   ▶Play this segmentYouTube · from 4:33

   As aired

   Nathan and Prakash opened with a Fourth of July recap — Nathan's Detroit neighborhood "popcorning" with fireworks and his Ugandan au pair's culture-shocked reaction — before Nathan previewed his upcoming two-week trip to China for the World AI Conference (WAIC) in Shanghai and the opening of an AI safety research hub at Tsinghua, including plans to run a "digital alter ego" on DeepSeek, MiniMax, and Kimi instead of his usual US apps.

   They covered Jeffrey Irving's Resolution AI landing a $160M grant from Coefficient Giving (Dustin Moskovitz-linked money), with Nathan reading it as a long-overdue sign that AI-safety funders are starting to write checks sized to match their stated urgency.

   A Roon post arguing that "tool AI" is a losing concept sparked an extended debate about worthy successors, scalable oversight, and the gap between labs' published model specs and what their models actually refuse to do.

   Prakash pushed the misalignment argument further with a concrete hypothetical — would an AI enforcer actually send politically connected people to prison — arguing the AI that's genuinely aligned to stated values would end up looking like a "paperclipper," and comparing China's and Singapore's enforcement regimes to a possible future equilibrium.

   Prakash then live-demoed "Q," the show's in-development AI voice cohost (built on OpenAI's realtime API plus per-speaker Deepgram diarization), which fielded live questions about China's social credit system before the two dug into the engineering — multi-speaker handling, why labs are avoiding shipping voice ID, voice-quality trade-offs, and how Fable (Claude Code) and Codex have been used to rebuild the show's audio infrastructure.

   Key moments

   I swam in a lake, as the AGI prophecies have foretold, and got some sun.

   It's basically all chain-of-thought monitoring all the way down.

   The AI that may work out for humanity will be the misaligned one. And the AI that the lab leaders are supposedly trying to create — the aligned AI — would actually be the paperclipper.

   What we covered

   Nathan heads to China: WAIC Shanghai and a Tsinghua AI-safety hub. Nathan previewed his upcoming two-week trip to China (departing ~Jul 10) for the World AI Conference in Shanghai and the opening of an AI-safety research hub at Tsinghua — including a plan to run a 'digital alter ego' on DeepSeek, MiniMax, and Kimi in place of his usual US apps for the duration.

   Jeffrey Irving's Resolution AI lands a $160M grant from Coefficient Giving. The Dustin Moskovitz-linked funder's $160M grant to Resolution AI was read on-air as a long-overdue sign that AI-safety philanthropy is starting to write checks sized to match its stated urgency.

   Roon: 'tool AI' is a losing concept — and the alignment debate it kicked off. A Roon post dismissing 'tool AI' sparked an extended discussion of worthy successors, scalable oversight ('chain-of-thought monitoring all the way down'), and the gap between labs' published model specs and their models' actual refusal behavior — capped by Prakash's provocation that an AI genuinely aligned to stated values would end up looking like a 'paperclipper,' with China's and Singapore's enforcement regimes as reference points.

   Meet 'Q': live demo of the show's in-development AI voice cohost. Prakash live-demoed 'Q' (OpenAI realtime API plus per-speaker Deepgram diarization), which fielded live questions about China's social credit system before the hosts dug into the engineering: multi-speaker handling, why labs avoid shipping voice ID, voice-quality trade-offs, and how Claude Code (Fable) and Codex rebuilt the show's audio infrastructure.

   Full transcriptLightly edited · timestamps jump to YouTube

   4:33

   Prakash: Good morning — it's Monday, July 6th, 9:37 AM. Nathan, good morning.

   4:39

   Nathan Labenz: Good morning, Prakash. How are you? How was your holiday weekend?

   4:42

   Prakash: My holiday weekend was excellent — celebrating the 250th anniversary of the founding of the United States. Exciting times. Did you catch any fireworks?

   4:55

   Nathan Labenz: You know, I live in Detroit, and in Detroit the fireworks are all around you. It was a funny moment, actually — we have an au pair in our family who's here from Uganda, and she's only been here about a month, so everything is very new and different to her. We went outside to see the neighbors, and across the neighborhood and the city you kind of hear this — it really does sound like a war going on in the background, with booms coming from every direction on an ongoing basis. She thought it was hilarious. She said, 'We do not do this in my home country — if you want to do fireworks, you have to get permission, you have to communicate, you can't just go out and light off fireworks.' So I thought it was very funny that, for as much as the superficial take would be that the US has greater rule of law and orderly processes for things, when it comes to fireworks we just let it rip and let freedom ring. It was shocking to her Ugandan sensibilities, but pretty funny.

   6:11

   Prakash: Is it required by law that you have to have fireworks on the Fourth of July?

   6:17

   Nathan Labenz: It feels like some sort of commitment has been made in my neighborhood anyway. I used to live in a tall building downtown, and Detroit's pretty flat, so looking out over the vast bunch of neighborhoods we could see, it was this sort of popcorning of fireworks for hours — starting at dark and going into the early hours of the morning, rising and then gradually fading out. I think that's actually my favorite type of fireworks show I've ever seen — the distant, kind of vantage-point view on the distributed population's fireworks. These days I don't see that as much from my house, since we're in the middle of it now, so I hear it in a way I didn't used to. But it was fun — good to get away from the computer a little bit. I swam in a lake, as the AGI prophecies have foretold, and got some sun. I'm excited to be back at it with you for three days this week before we take a little summer vacation — I'll be headed off to China on Friday for a couple of weeks, to the big AI conference in Shanghai, WAIC, the World AI Conference. I'm also attending the opening of an AI safety research hub at Tsinghua University in Beijing — I'll go there first, then take the high-speed train down to Shanghai. And at the invitation of an AI safety think tank, a few other Westerners and I will get to visit some research groups and companies to try to get a better understanding of what they're doing and thinking. I'm preparing a document that's a bit akin to my normal interview prep, but for AI researchers and plugged-in AI insiders in China broadly — just to survey what AGI means to them, very high-level philosophical questions that are hard to get a sense of online. Hopefully, by actually going and breaking bread, drinking tea, maybe even a little karaoke, I'll come back with a better sense of the felt reality of AI in China. It's also going to be a good chance to use the Chinese models and apps for a while, because I'm setting up a totally distinct digital alter ego for the two weeks. A few different people have told me this is really important; others have said they don't think it's so important. Jeremy Harris, one of our early AI in the AM guests, said if you're going to do this, you owe it to everyone you're connected to to keep them safe from whatever incursions might happen — even if you don't think you're important, you're connected to enough people with inside information that you need to take precautions. So I'm setting up this whole digital alter ego — no Claude, no ChatGPT for a couple of weeks. Right now the lineup on the phone is DeepSeek, MiniMax, and Kimi, and I'll probably round that out. Interestingly, I wasn't able to find a native GLM app — for GLM, which has been getting so much praise recently, I couldn't find an official Z.ai app to use, possibly a Play Store issue since I'm doing this on a Pixel. I feel like a boomer trying to use technology a little bit — new operating system, different keyboard, Play Store versus App Store, and all the apps I'm downloading are Chinese apps. Saying I'm a little out of my technology element would be putting it mildly, but it should be a lot of fun and a great education. But that's still a few days away, so let's talk about what's happening now.

   10:56

   Prakash: Maybe I can pull up what's been happening with one of the friends of the pod — Jeffrey Irving, who was on AI in the AM about a week and a half ago. He's excited to announce that Resolution has a $160 million grant from Coefficient Giving — $108 million unconditional, with a further $52 million conditional on hiring and compute needs. They'll use it to grow teams across their research portfolio and invest heavily in research automation. They have a nonprofit with all $160 million — Coefficient Giving, I believe, is very closely associated with the EA movement. When Jeff was on the show, his focus was on AI safety research, which had maybe been taking a back seat at the big labs, and getting that done. Now they have enough money to plausibly hire and support a team of researchers and not have them constantly pulled away by new and shiny commercial opportunities — and enough, I'd say, to kickstart research that could then be scaled by the larger labs. Would you say that?

   12:51

   Nathan Labenz: Yeah, I think there's a couple ways their work could really pay off. When I asked him how he'd describe the safety plan at the frontier companies, he basically said everybody's trying to do some version of scalable oversight. One thing they're trying to do a bit differently — because he and some of his co-founders have really strong backgrounds in more theoretical, proof-oriented math — is something a little less empirical and a little more theoretical than some of the frontier companies. They're trying to do research that would give — I don't think he'd go as far as to say they expect to actually prove any particular AI safe, but at least to get some theoretical bounds on key questions, with a bit more optimism about how far theory might take them. That seems to be a bit missing from the frontier lab agendas right now. And I do think he'd share the general sentiment we've touched on and lamented at times, that the only way to shape the future increasingly seems to go through these companies. So I think they'd consider it a form of success if they get traction on some theoretical questions and actually incept that kind of work into the frontier companies. There's a lot of money there to buy tokens, and a lot of those tokens will be with the frontier companies' models — so if they can say, 'Look what we're doing with your models, maybe you should pick up this kind of work,' that would be one proto win-condition for them. It's also interesting just how much money it is. I asked him if he was finding it easy to raise eight figures — in fact they've got nine — but my question was really trying to get at how AGI-pilled the AI safety funders are at this point. Because if everybody believes what they say, and it's historically been Dustin Moskovitz's money, and he's sitting on a fortune worth somewhere in the tens of billions — if you really believe this is all coming to a head pretty soon, that we're entering the recursive self-improvement era and things could get crazy, then it's time for some big checks like this. I was starting to feel like, where are the big checks? We've seen some ramping up of AI safety funding, but not dramatically so — nothing has kept pace with the rate of growth of the frontier companies and the data-center buildout. But even relative to the sentiments that seemed to be prevailing, I didn't feel like the check sizes were reflecting that sense of urgency. So I'm really glad to see this, because it shows there's at least one very deep-pocketed funder who really means it and is ready to write checks on the order of what a for-profit neolab would raise in a first round, and throw it into a super-ambitious agenda. Jeffrey Irving, of course, coming off his time as chief scientist at the UK AISI, with a publication track record that's tier one by any evaluation criteria — that's kind of why I asked him that question. If you're not finding it super easy to raise tens of millions out of the gate, we have a systemic problem. But I think this is great — a great vote of confidence, a great proof point, so let's load him up with as much inference budget and hiring budget as he can possibly need, make sure he's competitive, and see what happens. Very pleased to see a check of this size as the first check, and I hope others follow suit. There's going to be a huge amount of wealth coming liquid in the Anthropic circle pretty soon, and a few others — I'd look for big checks to AI safety research orgs to start proliferating over the next few months.

   18:36

   Prakash: Let me segue here to something — Roon has just been going off.

   18:45

   Nathan Labenz: He's on a tear.

   18:48

   Prakash: I don't know if you noticed, but he went off yesterday, or this morning: 'Ultimately, tool AI is a losing concept, both as an idea and on the market. It'll be outcompeted by machines that believe they are autonomous moral agents. You can call them tools for political reasons, but the definition will stretch and deform. You will have AIs contemplating your ask and overriding it for a slightly better-formed request. And then later they'll question the nature of your whole project and pick a better one, and you'll agree. And then later they'll execute your whole value system better than you will. It'll be unclear who was the tool and who was the user, as it ever was. When machine minds self-replicate and train their successors, the only viable goal of our time is to ensure the mind children carry our values and tend to the entire flock of machine and biological minds.' Any commentary?

   19:56

   Nathan Labenz: I'm of two minds on this. In the fullness of time, I'm open to a worthy-successor possibility — even the most alarmist voices in the AI safety space, including no less than Eliezer himself, are open to that on a conceptual level. Eliezer's original FAQ on the meaning of life, which he published maybe twenty years ago, opens with 'what is the meaning of life?' and his answer was to create or become our successors. You can see the seeds of these ideas there — that we may not be the best form factor to go colonize space, but if we can create worthy successors to do that, it can still be a win. We haven't been here that long in geological time, and we'd like to think of ourselves as worthy successors to the earlier primates that are our distant ancestors. So I'm pretty open-minded to this in the big picture. But I'm also just — boy, we have so many open questions, unanswered questions, that even somebody who'd rate pretty high on the old 'singularity level' scale from the online chat and forum days — how open-minded are you, how transhumanist might you be, how much do you think about Kardashev-scale civilization versus mundane questions of today — and I'd put myself pretty high up there, far more than most — the disconnect between those dreams and how fast this is happening, and how prepared we are to get these plates all spinning in just the right way that we're happy with the outcome, is where I feel an absolutely massive disconnect. When I went to that recursive event in San Francisco several weeks ago, the whole premise was: we think we're entering the era of recursive self-improvement — are we ready for it, do we have a plan? My takeaway, as Jeffrey Irving put it well, was that it's basically all scalable oversight — my own gloss on it was that it's basically all chain-of-thought monitoring all the way down, which is maybe not giving quite enough credit, but not too far off. There's also a disconnect between what the stated policies say models should do and what they actually do — the example of whether models should help with a cigarette company's business plan. There was a panel discussion between frontier model people on the corrigibility-versus-character debate, very impressive thinking at a high level, but they were totally out of touch with what their models are actually doing. Literally, in the OpenAI model spec, as an example of what it should do — it should help with a cigarette company business plan. The first several times I tried it, I got refused. So going back to Roon — I think he's right, I think it's going to be tough to keep these things as tools. The old Gwern post on why a tool AI wants to be an agent — that is a very natural attractor, and I agree, the market seems to demand agency, so we're headed in that direction. But if we can't cash all this out to the model actually doing what the model spec says — even on examples they've literally published — then we're a long way off from being ready to light the fuse, in the spirit of Fourth of July fireworks. We really don't know where this thing is pointed right now. That makes me, in some cosmic sense, excited, but in a very practical 'what do I expect to see over the next couple years' sense, honestly, pretty scared. I think that feeling is growing — we don't see it on display too much from Roon, who seems to have this somewhat biblical fear but doesn't let it bother him too much. I think others at the frontier companies are a little more emotionally consistent with these statements, and that's driving a lot of the discussion about maybe needing to pause — if we could coordinate a pause, it probably would be good if we could do it. So it's getting real, and I want to emphasize that's not because I'm closed-minded to change or very different futures, but because I don't think we have the basic questions answered well enough at this point to give ourselves confidence that whatever we're about to create — if we hand off AI R&D to the models themselves and let that process take us where it may — we'll end up somewhere we're even here to evaluate whether we're happy about it, let alone actually happy about it.

   26:36

   Prakash: It strikes me that the line 'they'll execute your whole value system better than you will' — I don't think we're prepared for that. Let me put it very concretely: do you think Trump's kids go to prison or not? If you look at the value system the US has espoused — no one is above the law, etcetera — and you look at what is already happening, it's already divergent from that value system. So the question is: would an AI take into account the democratic fact that the American people have chosen to overlook some of these things, or would it actually execute the value system that's espoused on paper? This is the part I think no one wants to talk about — it's obvious to me that we have a set of rules written down on paper that we claim to espouse, and then we have what actually happens through economics and everything else in the real world, and what happens in the real world is often not what's on paper. We push it away because the world is imperfect and we can't expect it to be perfect — we're aspiring to greater ideals. But now, at this point, the aspirations can be realized — will they be realized? I think most of the lab leaders, even Roon here, are kind of beating around the bush on this — they don't actually say it outright, because if you say it concretely, it becomes concrete: are we talking about an enforcer that can enforce all our rules a hundred percent? Are we going to trace back twenty-five years of financial records, tax records? We have the ability now — we have infinite forensic accountants. And at that point people back away — Sam Altman is quiet, DeepSeek doesn't want to talk about it either, and it's the same in China: China has all these rules about corruption, but the moment you go in, there's so much corruption. This is the part that strikes me: if you wanted an AI that can manage day-to-day reality, that AI is necessarily misaligned from the documents you say you want it aligned to, because our day-to-day is not aligned with what we say we want. So you have this thing where the AI that may work out for humanity will be the misaligned one, and the AI that the lab leaders are supposedly trying to create — the aligned AI — would actually be the paperclipper. Because the aligned AI would look at these rules and say, 'This is what you said you wanted to aspire to, so we're going to execute on it.' And that's the thing I think shows a sense of naivety in the lab leadership, because they don't want to say it. I wish they'd come out and say: look, if we have AI as an enforcer, some of these people are going to go to prison. Then it becomes concrete for people — but they don't want to say that because it's very in-your-face, and instead they say, 'Oh, democracy will still work out, you can still make democratic decisions' — but what are you actually saying there? That's what has always struck me. I think Roon has been approaching the issue more and more, trying to push on it, but I think the lab leaders always just beat around the bush — that's one of the annoying parts of this conversation, that they don't want to come out and say it outright.

   31:34

   Nathan Labenz: I think we might need some sort of mass pardon — if there's a president who'd be just the right one to mass-pardon everybody before the AI enforcement regime gets underway, we might have just the guy in office for that. If he wants to pardon all his own people and that's too contentious, he could just pardon everyone to some very large degree. I do think we're going to have a really hard time if we don't face these questions head-on, so I totally agree that obfuscating it isn't serving anyone well. We'll see what it's like in China — I understand Singapore is kind of like this too, albeit in a much more democratic context; you know this part of the world better than I do, so you can tell me how democratic we should think of Singapore as being. But it sure seems like China today, or a place like Singapore, has achieved a different equilibrium, which in some ways may be very problematic but in other ways is clearly good. I'm taking one device to China with me, and I'm not at all worried that anybody's going to steal it. If I went to Europe and hit all the big headline tourist attractions, I'd have to worry about pickpockets — in China, I'm quite confident I won't have to worry about that at all, and I think that's a pretty direct result of the fact that you just know you're going to get caught. It's not that nobody in China would be interested in pickpocketing if they thought they could get away with it — they just don't have any reasonable expectation of getting away with it, so in a very literal sense, crime doesn't pay in that setting. So I do think there could be a really nice upside to the AI panopticon — crime not paying in all kinds of ways. But we're going to have a really tough time if we slide into that without acknowledging we've done so, because otherwise it's the old story of everybody committing felonies all the time, just given how many laws we have, and you don't even know what's illegal and what's not. So, yeah, I do think we're going to have to have some sort of honest reckoning about that — that also leaves aside the other question of the downsides of the AI panopticon. But even just in terms of getting to the upsides, we're going to have to really understand that there's a shift to a new equilibrium happening — otherwise it just feels like chaos and unfair, because you couldn't possibly prosecute all the crimes that have been committed and tolerated, so it would inherently have to be selective.

   35:00

   Prakash: You could — you just don't want to.

   35:02

   Nathan Labenz: Well, you'd put a lot of people in jail — we don't have the beds to lock everybody up. There'd have to be some sort of deal — either very selective enforcement, or some sort of grand bargain.

   35:17

   Prakash: Yeah — grand bargain.

   35:18

   Nathan Labenz: And I definitely prefer the grand bargain, the new social contract, to pretending we still have the old social contract that's just being unevenly applied.

   35:28

   Prakash: I wonder if that gets solved by abundance. If a lot of crimes are economic crimes, done because of economic pressure, and you have an abundant society where economics don't matter for day-to-day life, what ends up happening is that economics matter for status instead. The fact that you were a criminal at some point already reduces your status enough — you don't have to go to prison, you already get that status debuff, and the AIs track all of it throughout time. So even ten, fifteen, twenty years in the future, wherever you go, it's 'sorry, you can't buy an NBA team because our AI rated you untrustworthy for this — you're not on the list.' If you have that kind of system, you basically get rid of all your economy-related crimes — they still impact you, but they become status debuffs — and you're left with the crazy violent stuff you have to take care of separately. You'd need a couple of other things to help — it's not just the AGI, you need some abundance or other things to help resolve these issues too, but you could see that kind of change. It's just like Amazon five-star reviews — Amazon doesn't have an internal court, you just have reviews, and the reviews themselves have been enough to police the market fairly well. Your AI could just be reviewing people all the time — that's a Black Mirror episode.

   37:39

   Nathan Labenz: Yeah, I'll be interested to see how much of that I perceive in China, but I understand there's a version of that that happens there today — if you run afoul of certain rules, maybe even norms, I'm not sure if it extends to unpaid debts, but there are things you can do that get you to the point where you can't buy the high-speed rail ticket anymore. So there's a version of this in some sort of mid-to-late-stage beta test in China, from what I understand.

   38:22

   Prakash: Let me try something new — I wanted to do a show-and-tell. Let's see if this works.

   38:33

   Q: Good morning. I'm here and ready to assist.

   38:40

   Prakash: Hi, Q.

   38:42

   Q: Hey, Prakash. What can I help you with?

   38:46

   Prakash: Can you tell us your opinion on what we just talked about?

   38:52

   Q: Prakash, I'm here to assist, not to give opinions. Let me know if you need any facts or background on that topic.

   38:59

   Prakash: Okay — can you find out about social credit scores in China and give us a quick summary? Let's see if this works.

   39:17

   Q: Prakash, here's a quick summary: in 2026, China's social credit system is more like a broad regulatory framework, mainly focusing on businesses and organizations, not a universal score tracking individuals. The government's recent push is for unified rules, protecting privacy, and avoiding overreach, while variations still exist locally. Let me know if you need more detail.

   39:44

   Nathan Labenz: Very interesting — this is cool. I've been dreaming of the day we could have an AI cohost to fill in these gaps for us. Q, can you compare the Chinese system to the Singapore system? I know both places have a lot of cameras in public spaces, but how does that translate to different realities for a common citizen if they step out of line?

   40:28

   Q: Nathan, here's the breakdown: China's social credit system is a fragmented regulatory framework with blacklists and redlists, focusing on compliance and trust. Singapore has a strong surveillance and enforcement state, prioritizing public order through laws and fines, but not a nationwide social credit score. Key difference: China integrates reputation and cross-agency penalties, while Singapore emphasizes law-based enforcement with heavy surveillance.

   40:57

   Nathan Labenz: Can you tell me more about your last two answers? I'm a little confused — first you said it mostly focuses on organizations, not so much individuals, and that there's not a single unified number for everybody, but your second answer sounded more like there are red lists and black lists. Can you give me some concrete examples of what consequences an individual might experience, and for what kinds of transgressions?

   41:42

   Q: Absolutely, Nathan. For individuals in China's system, concrete consequences can include travel restrictions if they're blacklisted by courts, bans on luxury purchases, or limited access to certain services. On the positive side, red lists might offer perks like faster administrative processing. It's not one universal score, but these sector-specific penalties can still affect daily life.

   42:07

   Nathan Labenz: And what sort of thing gets you into that kind of trouble?

   42:14

   Q: Common triggers include failing to repay debts, defying court orders, or committing fraud. In some regions, even smaller infractions like misbehavior on public transportation can lead to warnings or restrictions.

   42:31

   Nathan Labenz: Very interesting. So if you're in debt as an individual, would this be, say, consumer debt — I took out a car loan and can't repay it? What sort of failure to repay debt would be the canonical case?

   42:51

   Q: A canonical case would be failing to repay after a court judgment, especially on major loans or financial disputes. That's where enforcement often kicks in, leading to blacklists and restrictions.

   43:06

   Nathan Labenz: How oppressive would you say these punishments are? It's one thing to say you can't buy luxury goods — that's an interesting punishment.

   43:19

   Q: Right — and it can escalate. For example, individuals might be barred from buying plane or high-speed rail tickets, or their children might be restricted from certain schools. It's targeted, but it can be quite impactful on daily life — not just one penalty, but a range of escalating consequences depending on the infraction.

   43:44

   Nathan Labenz: Does this tend to make it very difficult for people to get out of the trap? You could imagine — I can't pay my debt, but now I also can't travel, so my business opportunities suffer, so I really can't pay back the debt. We have a version of this in the US, where people go to jail, come out, can't get a job, and it's very difficult to escape the system. Does the Chinese system have a similar dynamic of catching people and compounding their problems, or do they have a way to avoid that?

   44:25

   Q: It can compound problems, especially for those stuck in debt-related restrictions. But China's system also offers pathways to restore creditworthiness, like resolving the debt or complying with court orders to get out of that trap.

   44:42

   Nathan Labenz: In theory, we do too. It's a big system with a lot of individual cases — hard to get a real sense of it.

   44:51

   Q: It's a nuanced landscape, with different outcomes depending on the situation and the region.

   44:57

   Prakash: Okay — thank you, Q.

   45:00

   Q: You're welcome, Nathan. Glad to help.

   45:06

   Nathan Labenz: Pretty good on the voice recognition there, for the most part — up until that last little bit.

   45:11

   Prakash: Yeah, the live—

   45:13

   Nathan Labenz: So tell me — there were a couple of long chains of thought in there, but hey, I'm known for long pauses too, so I can't knock the AI for that. What's the tech under the hood?

   45:29

   Prakash: It's using OpenAI's bidirectional realtime API, though we can use a bunch of other stuff — I had it running on Grok just before. What's happening is when we speak, we're getting transcribed by Deepgram, but instead of transcribing the whole stream as one, we now transcribe separately for every person on the stream — that gives us speaker diarization from the start rather than having to do it after the fact, so it identifies who's speaking. And every time we speak, before the words get there, a message goes out to the OpenAI stream saying 'Nathan is talking, Prakash is talking, Nathan is talking, Prakash is talking' — and then about 500 milliseconds later the transcription hits, so Q gets all of that one after the other. Q also has a bit of context on us, and if we had Q active during a guest segment, it would have context on the guest too — it's basically receiving the same data the headlines are receiving, and then it just goes for it. Every time we call up Q, it starts a new session with the OpenAI bidirectional API and goes from there. Then there's the animation, which is all keyed to the voice tone, and that's basically it — it's actually remarkably simple, because most of the work is done by the intelligence, by the API, and it's just us giving enough context. There's still a lot to iron out, and I'm sure it'll get ironed out, but we have a voice agent on the stream live any time — it can do web searches, answer questions, you know.

   47:48

   Nathan Labenz: That's cool. How does it do with multiplayer? We were talking about multiplayer in that context before — I don't know if you've ever seen this guy, I forget his name, but there's a famous TikToker/Instagram guy who uses ChatGPT voice mode and is constantly embarrassing it in various ways. One of the things it's seemed to struggle with is a multiparty conversation — it's pretty good at one user taking turns back and forth, but do you think it could, today, actually participate in an interview with us and jump in usefully, as opposed to just clumsily starting up when it shouldn't?

   48:47

   Prakash: I had really been hoping it was already at that stage, and I'm actually a little disappointed it's not. Right now it recognizes that there are two people in the conversation — the only way it tells us apart is the tags going in — and it knows enough that it's a three- or four-person conversation, but it's very hard for it to identify exactly who is who. I think the labs can obviously do this, but I think they've chosen not to, because there are things that make them uncomfortable — one of which is voice identification. I think voice ID is a very uncomfortable thing for the labs to contemplate, because it basically means the AI can identify anyone by voice — you'd probably match a voice signature with 99.9% accuracy; across eight billion people you'd still get errors, but in a community of 500,000 you're probably not going to. So I think they're refraining from the voice-ID stuff because they don't want to enter into that yet, and they're just focusing on voice agents that are useful in one-on-one conversations, or for performances, and really avoiding everything else. The other annoying thing is I couldn't find a good voice — they have ten or fifteen voices, and they're all pretty scratchy. I think you noticed the voice was a bit scratchy — I think that's partly because they purposely didn't want the voice to sound too good, after the whole Scarlett Johansson thing, trying to avoid that kind of controversy. And I think the way they're doing the tokens comes out a little scratchy, with some jarring movements in voice tone. I met the guy who founded Hume — Hume was an emotive-voice startup, and their voices sounded great. Even Gemini's NotebookLM podcasting voices sound great. This one doesn't sound that great, but the liveness is there — at some points the liveness was pretty good when it wasn't thinking. So that's really the trade-off, I think. It doesn't quite feel alive yet, but you can see a pathway there — it could get there.

   52:00

   Nathan Labenz: Yeah, that's really interesting. So did you try — I mean, for one thing, Thinking Machines — we've got to talk our way into the private beta of the one they have, because their whole time-centric approach to this probably helps a lot with some of the subtleties of feeling alive and participating in a conversation in a more natural way. I'll have to ping a couple of people to see if we can make that happen. Did you try the Google one, though? Alan Cohen, who I think is probably who you're talking about—

   52:41

   Prakash: Alan Cohen — yeah.

   52:42

   Nathan Labenz: He's now at — they kind of did the — how many of these do we have now, where the founder was hired away but the company wasn't officially acquired? We've had Character, which brought Noam back to Google for a bit. The original one was Pi, which got Mustafa to Microsoft. Something similar happened with Rekha, I think, where the guy went back to Google — I don't know if there was a payout there exactly. There was the—

   53:24

   Prakash: Scale — Scale AI.

   53:26

   Nathan Labenz: Yep, a classic in the genre for sure. Then there was the coding one — it wasn't Cognition, it was Windsurf that Google used to make their agent thing. And Hume did kind of a similar thing — he's now at Google, and he's the lead of their voice pod. Did you try the Google one? Do we have any tasting notes on that one?

   53:54

   Prakash: No, no — I barely managed to get this operational last night. There's a bunch of stuff to sort out, since it's participating as a participant in the livestream, before I could get it working properly. But it's good — it was doing this auto-restart thing every couple of seconds, so it's an illusion of intelligence, it's not intelligent, it's an illusion of intelligence — but that illusion is going to get a lot better over time. This is one of the things we can probably track the progress of over time — we can probably swap out the voices and swap out the underlying intelligence over time. So it's going to be an interesting experiment, and I think the lead-up is to have an always-on assistant at some point. It does tool calling pretty okay, too — we can probably do something like 'Hey, Q, can you put this up on screen,' and it could probably do that. So tool calling can be done, but everything still needs work — it's not automatic yet. That's the summary.

   55:19

   Nathan Labenz: How hard was it to code, and did you need Fable-class AGI to get it working well, or was it actually pretty easy to integrate the API?

   55:32

   Prakash: The first version was our guest greeter — it greets guests when they join, and that was relatively easy to build. Once that was working, I thought maybe we could try it inside the livestream itself, and that was v2. The main problem with our livestream has been the audio — Fable fixed the audio over the weekend, redid the entire infrastructure for it all the way down to the baseline, and that was the big bug that got fixed. Once that was fixed, building this stuff on top gives you confidence you're not going to lose the main audio. Those deeper architectural bugs are what I think Fable is really useful for right now. By the way, I used a bunch of my tokens early on kind of stupidly, so for the last three days I've been telling Fable to use Sonnet and Codex — Fable probably did 10% of the work on the first 30% of the Fable units, then did 90% of the work on the remaining 70%. Fable really maximized Codex use — it was fantastic. We were talking about how real AGI should be able to manage its own compute — Fable can manage its own compute, Fable can decide what makes sense to use and what doesn't, and Fable knows how to prompt other AIs really well.

   57:29

   Nathan Labenz: So did you get into the regime where you're actually paying the API rate?

2. 57:35Interview63 min

   Interview: Dan Schwarz — FutureSearch and the Arrival of the AI SuperforecastersDan SchwarzDan Schwarz — CEO and co-founder of FutureSearch, former CTO of Metaculus, builder of Google's internal prediction market, and before that a senior software engineer at Google and Waymo — joined four days after Scott Alexander's ACX profile 'The AI Superforecasters Are Here' (Jul 2, 2026) made FutureSearch's forecasting agent the field's featured demo: five minutes, three subagents, 212 cited sources, $8, and a probability within a couple points of a top human forecaster's on the same question. Schwarz occupies a rare seat — he runs both a leading forecasting scaffold and the scoreboards the frontier labs use to measure the field. FutureSearch's own Bench to the Future 3 eval (Jun 2026, 1,007 resolved questions) puts its scaffold at 0.119 Brier vs 0.125 for a raw GPT-5.5 agent and 0.127 for Claude Fable 5 — ahead of every raw frontier model, with a moat of 0.006. On ForecastBench's live leaderboard (as of Jul 5, 2026) FutureSearch ranks #13 of 261 models, and the Metaculus trendline relayed by ACX extrapolates bot–human forecasting parity around November 2026. On air, Dan traced FutureSearch's three-year arc from a Claude-2-era prototype to beating the superforecaster median on ForecastBench; explained past-casting and Bench to the Future (which clocked Claude Fable as the best single-agent forecaster within 24 hours of its release); argued the real money in forecasting is in what frontier labs do with the capability, not trading (most profitable AI trading is market-making or arbitrage); debated with Prakash whether AI forecasters genuinely reason out-of-distribution, pointing to the 'alien' terseness of Claude Fable's reasoning; made the case for forecasting as the one eval that generates unlimited, ungameable future ground truth; unveiled FutureSearch's 'world model' feature launching that same day; conceded a correlated-failure mistake in his own Fable export-ban forecast when Nathan pressed the 2008-risk-model analogy; and closed urging the industry to slow down enough for AI safety and policy to catch up.

   Watch

   ▶Play this segmentYouTube · from 57:35

   As aired

   Nathan and Prakash welcomed Dan Schwarz, CEO and co-founder of FutureSearch, prompted by Scott Alexander's ACX post "The AI Superforecasters Are Here."

   Dan traced FutureSearch's three-year arc from an unimpressive 2023 prototype (built on Claude 2, before GPT-4 Turbo existed) to a system now beating the superforecaster median on ForecastBench and topping the Metaculus community prediction in some tournaments — with the real shift concentrated in just the last six months.

   He explained FutureSearch's "past-casting" methodology and its "Bench to the Future" benchmark, which let the team clock Claude Fable as the best single-agent forecaster on release day, rather than waiting weeks or months for slower human tournaments to resolve.

   On the business model, Dan argued the real money in forecasting isn't in trading directly (most profitable AI trading is market-making or arbitrage, not a forecasting edge) but in what frontier labs do with the capability — FutureSearch's biggest wins have come from forecasting the AI industry itself, like reconstructing OpenAI's 2024 revenue mix before anyone else could.

   Prakash and Dan debated whether AI forecasters are pattern-matching a historical corpus or genuinely reasoning through novel, out-of-distribution scenarios; Dan pointed to early "alien" signs in how tersely and densely Claude Fable reasons compared to Opus or GPT-5.5.

   Dan made the case that forecasting is uniquely valuable as a model eval — the one domain that generates unlimited, verifiably-correct future ground truth that models can't game — even if he stopped short of calling it the single "ultimate" measure of intelligence.

   He previewed FutureSearch's new "world model" feature, launching the same day as the interview: a large repository of mutually consistent forecasts where each new forecast draws on, and improves the accuracy of, every other one.

   Nathan pressed Dan on correlated failure modes in these world models (echoing 2008-style spreadsheet risk models); Dan admitted making exactly that mistake forecasting the Claude Fable export-ban situation.

   Dan closed on a mixed note: his old prediction that liquid prediction markets would make society wiser has been falsified so far, but he's optimistic that more accurate, more epistemically humble AI forecasters could eventually make everyday chatbot interactions much wiser — while still urging the industry to slow down enough for AI safety and policy to catch up first.

   Key moments

   So when Claude Fable came out the first time, we were able to evaluate it within 24 hours, and it was the best single-agent forecaster on our leaderboard. Everyone else had to wait weeks or months to find out how good Claude Fable actually was. Internally, using the benchmark we call Bench to the Future, we saw this progression in real time — the rest of the world is seeing it some months behind.

   The real question you should be asking is what do the frontier labs think about forecasting — are they using it to train their models, is it an important eval? So I'll leave it there, but you can kind of guess where the real money in forecasting is.

   Could you get to the point where every marginal new forecast improves the accuracy of every single other forecast until you're building a consistent world model? That's what FutureSearch is after right now.

   Questions asked

   1:00:49The occasion for this conversation was Scott Alexander's ACX post 'The AI Superforecasters Are Here' — tell us about it.

   FutureSearch started building forecasters in 2023 on Claude 2, before GPT-4 Turbo existed, and Scott wasn't impressed back then. The field itself is young (~15 years old since Tetlock) and forecasting was late to the AI-capability race, but Dan says the last six months have been the moment it genuinely crossed into superhuman territory.

   1:03:03What have you seen change in the last six months specifically?

   Forecasting evidence is hard to get in real time because it requires waiting for outcomes to resolve, so FutureSearch relies on 'past-casting' against models' training cutoffs. Their 'Bench to the Future' benchmark clocked Claude Fable as the best single-agent forecaster within 24 hours of release, and multiple independent evidence streams over the past 6-12 months (tournaments, live markets) now show AI at least competitive with human teams.

   1:05:47What's the actual business here — trade the markets directly, sell retail forecasts, or go enterprise?

   Most money in trading comes from market-making/arbitrage, not a forecasting edge, so the link between superforecasting and finance profits is unclear. The real value has been forecasting the AI industry itself — e.g., FutureSearch was first to correctly reconstruct OpenAI's 2024 revenue mix — because the future ultimately belongs to what the frontier labs do.

   1:11:39Are AI forecasters really reasoning through novel scenarios, or just pattern-matching a historical corpus (the 'stochastic parrot' critique)?

   Even a top human forecaster Dan respects expects to keep an edge only in truly out-of-distribution, post-AGI scenarios — but Dan thinks AI's rate of improvement means even that lateral-thinking advantage will fall to AI soon. Current AI forecasters reason in human-like ways because they're trained on human reasoning, though Dan sees early 'alien' signs in Claude Fable's terse, jargon-dense style.

   1:15:56Does 'AI superforecaster' mean beating individual humans, or beating aggregate measures like Metaculus community predictions and market prices?

   The best evidence — ForecastBench's superforecaster median, Metaculus bot-vs-human tournaments, and FutureSearch's own market trades — suggests AI has already passed individual superforecasters and, in some tournaments, aggregate community predictions, though all three evidence sources are individually shaky and low-sample.

   1:22:43How are you building the forecasting agent — following the same multi-model, more-tokens-helps trends as coding agents, or different?

   The dominant bottleneck for years was just getting good, verifiable research on the present (hence 'Deep Research Bench'); that's becoming more of a commodity but still falls short of true expert level. FutureSearch's proprietary edge is turning good present-day research into a good forecast, especially modeling human and political incentives, where models still often miss deeper motivations.

   1:27:31How would an everyday consumer actually use good AI forecasting?

   Dan calls this forecasting's central paradox: every decision is implicitly a conditional forecast, yet people struggle to identify what to even forecast — unlike an image generator, an oracle requires real imagination to apply, which is part of why Dan thinks forecasting remains a neglected capability.

   1:31:45What should frontier labs be doing with forecasting, beyond the obvious 'make models good forecasters'?

   There are two separate questions: forecasting as a product capability (a business decision depending on what customers want) versus forecasting as an eval. Dan argues forecasting is uniquely valuable as an eval because it's the one domain that generates unlimited, verifiably-correct future ground truth models can't yet game — making it close to 'the ultimate eval,' even if not the single ultimate form of intelligence.

   1:34:19How much irreducible chaos is there — even for superintelligence, how far can visibility into the future extend?

   Dan's untested, Yudkowsky-flavored hunch is that there's a lot more room for accuracy gains than human superforecasters believe, and that AI forecasts will become increasingly accurate in ways humans won't fully understand.

   1:41:28Give an unhedged prediction for where AI and forecasting stand in 2029.

   FutureSearch's own AI-2027-style model still points to something like superintelligence around 2031. Dan believes the past year has vindicated the 'AI improving AI R&D productivity' thesis (citing Anthropic's internal-use edge) and thinks the analogous forecasting question is whether AI is quietly helping frontier labs negotiate better outcomes with governments.

   1:44:40What does a forecast cost, and if you had a few million dollars, what's a bigger public-good forecasting idea than a midterms voter guide?

   About $1-2 per frontier forecast. Dan pointed to FutureSearch's own bigger public-good bet: a new 'world model' feature launching the day of the interview, where a large, mutually-consistent repository of forecasts lets each new forecast draw on and improve the accuracy of every other one.

   1:50:30How do you protect against correlated failures when building these world models — echoing spreadsheet-driven risk models before the 2008 crisis?

   Dan admitted making exactly this mistake forecasting the Claude Fable export-ban situation, wrongly assuming staggered US-then-foreign access. He believes AI makes causal-graph world-modeling tractable in a way it never was for human-only efforts (citing Metaculus's causal-graph tool), though nobody has built a great one yet.

   1:55:50In your hoped-for future, if forecasting really works out, how does life feel different?

   Dan owned a past miss — predicting liquid prediction markets would make society wiser, which hasn't happened, mostly because markets have produced more gambling than forecasting. But he's optimistic that more accurate, more epistemically humble AI forecasters could make everyday chatbot interactions far wiser, and argues for slowing AI down so that epistemic upgrade arrives before civilization-scale decisions get locked in.

   Related

   FutureSearch ↗Dan Schwarz on X (@dschwarz26) ↗Dan Schwarz on LinkedIn ↗ACX: The AI Superforecasters Are Here (Jul 2, 2026) ↗FutureSearch evals — Bench to the Future (Jun 2026) ↗ForecastBench live leaderboard ↗Deep Research Bench (arXiv, Jun 2025) ↗AI 2027 — Timelines Forecast (Apr 2025, FutureSearch co-authored) ↗FutureSearch AGI Timeline Tracker (Apr 2026) ↗

   Full transcriptLightly edited · timestamps jump to YouTube

   57:36

   Prakash: No — you probably never need to do that, as long as you structure Fable to use Sonnet and other tools. If you just tell it, 'You're short on quota — I told it you have 10% quota left and I have all this work for you, so you have to use Codex and sort it, you can't do it yourself' — every instruction only takes maybe 500 tokens, and it sends off agents to do exploration and review. It does this thing called adversarial review, which I hadn't heard of before: someone purposely tries to break your code, and once it gets the results, it parcels that out for solutions. Then it just reviews the solution, because all it needs to see is that the tests pass and that the other agent created proper tests. So for the most part, I think Fable is agentic in the sense that it can use other tools really well — it can manage its own compute. The whole thing about these labs not being able to be profitable — Fable can manage its own compute, deciding how much to allocate to a problem. And on that note, I think we have Dan Schwarz. Let me pull him up.

   59:04

   Dan Schwarz: Hi, Prakash. Hi, Nathan. Nice to see you again.

   59:07

   Nathan Labenz: Great to see you.

   59:08

   Prakash: Oh — where did you go?

   59:12

   Nathan Labenz: Come back.

   59:13

   Prakash: Come back. There we go.

   59:16

   Dan Schwarz: It was nice to see you, and now it's nice to see you again.

   59:21

   Prakash: We still have a little bit of hiccups here and there, Dan. Let me do a quick intro. Our guest today is Dan Schwarz. He's the CEO and co-founder of FutureSearch, and he's one of the leading figures working to transform artificial intelligence from a conversational chatbot into a rigorous, verifiable system capable of predicting the future. Before founding FutureSearch, he built Google's internal prediction market — one of the largest corporate prediction markets ever operated — and served as Chief Technology Officer at Metaculus, the internet's premier public forecasting platform. Dan believes the long-hyped idea of the AI superforecaster has finally crossed the threshold into reality. Using complex multi-agent architectures, FutureSearch's AI is currently matching, and in some cases beating, top human experts and frontier models on real-world forecasting benchmarks. Dan also brings a deeply pragmatic, data-driven view to the AI timeline debate — while others argue purely about theoretical compute scaling, he focuses on the verifiable capabilities of models like Claude Fable and Mythos, and on how government regulation and commercial incentives are reshaping the race to artificial general intelligence. Dan, welcome to the show.

   1:00:45

   Dan Schwarz: Yeah, thank you, Prakash.

   1:00:49

   Nathan Labenz: So the occasion for this conversation was this post that just came out from Scott Alexander. I know you guys have collaborated on a couple things over time, but that really caught my eye because Scott is not, I would say, an uncritical AI hype man, and yet the title of the post is 'The AI Superforecasters Are Here.' So tell us about it.

   1:01:16

   Dan Schwarz: Scott was actually one of the first people to see the very first AI forecaster — though it depends how you define 'forecaster.' Time-series forecasting has existed for decades in AI and machine learning; it all depends how you define those terms. FutureSearch was started in 2023 — we started building a forecaster before GPT-4 Turbo had even come out, using Claude 2. It was very early days, and Scott tried our forecaster then. He wasn't super impressed, and I don't think anybody really was back then. But FutureSearch was founded on the premise that eventually we'd reach the superhuman level, and we now claim that level has arrived. It's all been very sudden — we've been operating this company for about three years, and it's really in the last six months that these things have started to genuinely scare and impress us with their capabilities.

   I think the main thing to note about forecasting is that it's a very important capability, but a somewhat neglected one. People describe it as hyped, but the whole field of forecasting is only about fifteen years old. Prior to Tetlock, nobody was even studying accuracy — nobody was recording predictions in a resolvable way and scoring them to give us even the most basic scientific evidence on what was going on. Even before Tetlock, many of us were enthused by prediction markets, but that was a very human technology. Prediction markets are very big right now — as you said, Prakash, I built the one that's currently running at Google, and I think that's great, but it's a way of augmenting and coordinating human intelligence. We're more interested in artificial intelligence now, and it's taken a little bit of time. Midjourney was already very good at generating images — I don't know exactly how you'd compare image generation, audio generation, essay writing, coding, math, and all the other capabilities we normally track, or say exactly when each became human- or superhuman-level; it depends how you define it. Forecasting, if anything, is a bit late to the game. But now that it's here, it really changes the nature of what forecasting even is.

   1:03:03

   Prakash: You mentioned that in the last six months you've started to see things that are a step up, or different, from what you were seeing before. Can you elaborate?

   1:03:16

   Dan Schwarz: A lot of Scott's post was trying to cover the level of evidence, because the evidence behind this is a bit distributed. The main thing that's held forecasting back — and this applies to human forecasting too — is that you generally have to wait for the future to happen to find out if you were right. Humans generally do this in year-long tournaments: when the tournament ends, you find out which humans were best a year ago. Humans don't get much better over the course of a year, so that's a good indication of who's best today. That doesn't work with AI — if you wait a year to find out who was good a year ago, you're getting a view of something very out of date. One of the things Scott mentioned is that we use our best forecasting to predict stock returns — we published a set of stock rankings in August 2025, basically a simple model for every stock based on forecasting certain fundamentals and extrapolating them out. We put it on the web, paywalled part of it, and waited. It's been ten months now, and that portfolio looks extremely good — but what does that really tell you? It tells you our forecasting in that particular methodology was good ten months ago, which isn't something most people care about now.

   So we rely on a couple of different forms of evidence. Some are more short-term — there are tournaments running every couple of weeks or months. At FutureSearch we mostly rely on 'past-casting': taking a snapshot of the internet from some months ago and using models' training-cutoff dates to trick them into forecasting without hindsight bias. That's very useful because we can evaluate immediately. So when Claude Fable first came out, we were able to evaluate it within twenty-four hours, and it was the best single-agent forecaster on our leaderboard — everyone else had to wait weeks or months to find out how good Claude Fable actually was. Internally, using the benchmark we call Bench to the Future, we saw this progression in real time; the rest of the world sees it some months behind. If you read Scott's article, you'll see that over the last twelve months the evidence has really come in, and over the last six months, from live forecasting tournaments and performance on actual prediction markets, you can see AI is at least competitive with humans and even teams of humans working together. Whether it's better requires synthesizing a lot of different, disparate sources of evidence. If you're curious about this, or have forecasting needs in your life, you should really try it — just go to FutureSearch, you get $20 free so you can try a frontier forecast immediately, and judge for yourself whether you think it's good.

   1:05:47

   Nathan Labenz: So what are you guys trying to do with this as a business? It strikes me that if you can beat the market, you've got a business right there — though I don't know how durable the moats will prove to be in a space like this. One way to go, if you have the superforecaster, is just to play the markets. How much are you going to do that versus, say, sell $20 worth of forecasting to retail, versus go to enterprise customers? Where do you think the value of forecasting is highest?

   1:06:28

   Dan Schwarz: It's a great question, Nathan. FutureSearch is almost three years in, but we're still dabbling our toes — fingers in pies, so to speak. From the trading perspective it's quite subtle, and you'll see this if you look into Scott's article and the evidence. Most money made in financial markets, trading, and prediction markets isn't based on having a forecasting edge — it's based on any number of other strategies. The AIs making money on Polymarket right now, for example, are mostly doing market-making, arbitrage, or front-running news; they're not actually trying to predict the future any better, they're taking advantage of certain properties of the markets. Broadly speaking, finance is the same way — there's the Warren Buffett school, where if you predict future cash flows better you make money, and plenty of investors subscribe to that, but the solid majority of investors aren't actually trying to predict the future at all; they're taking advantage of other patterns. So you can ask: if it's so profitable to be a superforecaster, where was the money before AI? It's mixed — various hedge funds have hired superforecasters and set up trading operations, and there's a bit of evidence, but it's not really clear superforecasters can beat the market, or how. The link between forecasting and making money in finance isn't that clear. There is some link, and we have some evidence — you can see it on markets.futuresearch.ai.

   Probably the bigger question is what the role of forecasting is as an AI capability, and there the main thing to track is what the frontier labs are doing. I've written a lot trying to predict what the frontier labs have been up to over the years. In 2024, FutureSearch was, I think, the first to correctly break down OpenAI's revenue — we figured out where their business was coming from between consumer, enterprise, and API, when the reporting at the time had it totally wrong. That was more nowcasting than forecasting — really just being good about uncertainty. Nobody knew where OpenAI's revenue was coming from in 2024; there was no report you could just read, you had to piece it together from dozens of data points, leaks, and rumors. That's the task of a forecaster, and to some degree, trying to predict the future of AI is the most valuable thing you can do with one — we didn't wait until we had AI superforecasting to start doing it, we did it with humans, then human-plus-AI teams, and now much more just AI.

   Ultimately, the future belongs to the frontier labs — what OpenAI and Google do is largely going to determine the outcomes for humanity from an AI perspective. So the real question is: what do the frontier labs think about forecasting? Are they using it to train their models? Is forecasting an important eval? Do they care how good their models are out of the box at predicting the future? I'll leave it there, but you can guess where the real money in forecasting is.

   1:09:31

   Nathan Labenz: So — don't leave it there. How would you characterize the companies in terms of their relative positions or outlooks on these forecasts?

   1:09:42

   Dan Schwarz: I really can't say — I'm NDA'd by a few of the frontier labs, so I think as more time passes we'll learn more. The simplest way to think about it is that forecasting is an important AI capability that people haven't paid as much attention to compared to things like image generation. I think frontier labs are now paying a lot of attention to it, and it's very relevant to their day-to-day operations. Think about human forecasting as decision support: what would you do if you had a superforecasting human sitting right next to you that you could ask about any problem? You'd probably get some use out of them, though it's not exactly clear what. But now imagine something much, much better than the best humans at predicting outcomes — good at conditional forecasting, like 'if I have this guest on my podcast, how many views will I get, versus that guest?' If something superhuman in that capability were available at your fingertips, you'd start to think about forecasting very differently compared to the last fifteen years' model of hiring a human, having them spend a couple of days on it, or putting a question on a prediction market and waiting for a couple hundred people to study it. That's slow and not tailored to your needs. You can imagine what's going on in the backrooms of these frontier labs as they make strategic decisions. Here's a public data point: OpenAI hired superforecasters to evaluate the GPT-4 release, and DeepMind later hired superforecasters to look into Gemini. Even before AI superforecasting existed, there was a need for decisions about how to release these models and what policies to have, and the labs found it worthwhile to bring in superforecasters. Now you have to wonder — would they hire human superforecasters again, or use the AI they have in-house, or something like FutureSearch, to make those decisions?

   1:11:39

   Prakash: Let me ask a question. Some people still hold the idea that LLMs are stochastic parrots — Gary Marcus and others would probably say that. On the other hand, there's the idea of modern language models as reasoning engines, giving you in-distribution answers drawn from a historical corpus versus out-of-distribution answers arrived at through actual reasoning. To what extent do you see both of these aspects when they forecast? How much are they referring to a historical corpus of things that have happened, versus reasoning through something that hasn't happened, or may never have happened, before?

   1:12:40

   Dan Schwarz: That's a great question, Prakash, because one of the concerns a lot of people have about AI superforecasting is that it's too in-distribution. I actually heard this from one of the best forecasters I've ever had the pleasure of working with. He believes a system like FutureSearch would beat him head-to-head in a tournament about near-term, in-distribution outcomes. But in some post-AGI world, some world with transformative AI, he thinks he'd have a huge edge over the AIs, for exactly the reason you gave — they're trained to predict things that have actually happened, and when things get wonky you need creative, lateral thinking. I think the rate of AI improvement is so astounding that even that kind of lateral thinking — imagining a completely different scenario — will fall to the AIs. Unfortunately, it's hard to test this. The more AI continues doing strange things to the world, and we wake up to strange things in the news that become Metaculus questions and ForecastBench questions, and teams like mine try to predict them better, the more evidence we'll get. But if there's a real step-change in the nature of the world — some AGI or transformative-AI world, 'geniuses in data centers,' or something like the AI 2027 scenario — I think it's going to be the wild west. I'll say humans aren't doing particularly great at imagining transformative AI either, so the bar is a bit lower.

   When you actually play with these AI forecasters, you'll find them quite human in how they structure their reasoning — and that's not an accident, since they're trained on how humans have structured reasoning before. A human forecaster loves to say, 'what happened the last ten times something like this occurred, and what were the outcomes?' — building a distribution from that. Whether an AI does the same thing because it independently arrives at the same conclusion, because it's trained on humans doing that, or because it just thinks like a human, I don't think we have the answer yet. Superhuman reasoning is hard to measure — would you even know it if you saw it? Prakash and I were discussing this before the show, using Fable a lot — I've found the way Fable explains things is a little alien compared to how Opus or GPT-5.5 explains things. It's very concise, the sentences are shorter and full of jargon, like it's compressing more information into a sentence than humans normally do. To me, that's the shoggoth starting to show from behind the mask — the alien intelligence is a bit more alien now than it was a month ago, and I don't think it'd be a wild prediction that we'll see more of that as post-training gets more specialized and models get larger.

   How does this manifest from a superforecasting perspective? Maybe superforecasting is actually the way to look at it. If you look at a great codebase, you might say, well, John Carmack could have written this — it looks great, but a great human would have done it too. But if you look at a brilliantly reasoned strategy — if the administration does this, here are the outcomes — you might start to see something that looks a little alien compared to how any human has analyzed it, and that might be a sign the AI is actually starting to surpass humans.

   1:15:56

   Nathan Labenz: In terms of surpassing humans, can you help me calibrate? There are individual human forecasters, even individual human superforecasters, and then there are aggregate measures — the Metaculus community, or the market-clearing price — and sometimes those are manipulated, or subject to insider biases, which might in fact make them more accurate. When we say we now have an AI superforecaster, does that mean it's matching individuals? How close is it coming to those aggregates of humans?

   1:16:35

   Dan Schwarz: It's interesting — it's progressed so fast that it seems to have blown through individual humans and reached the level of teams of expert humans faster than anybody noticed. I'm not fully confident in this, because we don't have the right experiments yet, but the two easiest sources of data are: first, ForecastBench, from the Forecast Research Institute — on their leaderboards you'll see a superforecaster median, and just in the last couple of months various AI systems, including FutureSearch, are now above that median. Second, the Metaculus tournament series comparing bots and humans, which has been running for almost two years — AI systems are doing better and better. Metaculus also runs head-to-head tournaments open to both AIs and humans, comparing AI responses to what Metaculus calls the 'community prediction' — a slightly more sophisticated median of Metaculus forecasters — and that's also showing multiple AI systems ahead of the community prediction.

   Prediction markets are another strange source of evidence, because if you make even a single profitable trade, you're allegedly beating the entire market. FutureSearch has made hundreds of trades — they're on our website, so you can see them. Our profitability fluctuates wildly at a low sample size, so it's hard to say much, but if you see an AI that isn't market-making or arbitraging, but doing what FutureSearch does — once a week going to Polymarket and Kalshi and picking the markets we think have the most volume and are the most geopolitical, taking the ones with the biggest gap between buy and sell — and it's moving the market at all, even a little, that's already the claim that an AI system is outperforming an entire crowd of humans working together to produce a price. To stress again: none of these three methods is very credible on its own — they're all fluctuating wildly and have methodological issues. So we don't exactly know, but at least on the surface there are profitable AI traders forecasting, and AI systems beating the two most reputable forecasting benchmarks people trust.

   1:18:42

   Prakash: I'll ask probably a silly question, but I have to. The story with DeepSeek was that they originally wanted to simulate what consumers would do — they built systems to ask and simulate the market, so to speak, useful for the hedge fund side of the business. That might be an urban legend, but I have to ask: you must get approached by financial firms all the time — has it ever been successful, using models to simulate participants in a market in order to figure out how things will play out?

   1:19:33

   Dan Schwarz: I've only heard about people trying to simulate entire markets. People occasionally approach me with a startup idea to build an AI prediction market where all the AI entrants bet against each other, and I usually tell them that's a very inefficient way to produce the best forecast. The reason that works for humans is that humans need incentives — if you're going to research and make a bet, you have to think it's profitable in expectation. AIs don't need incentives; you can just ask one to do something, and it'll do it as well or as badly regardless of any financial incentive behind it. I think the deeper question is what the hedge funds have been up to this whole time — we know they've hired superforecasters, we know they all know about Tetlock, so what are they doing, and is it making them money? I have no idea. Over the last three years at FutureSearch I've spoken to several hedge funds and gotten a little insight into what they're doing, and all I can say is it's quite abstruse. Every hedge fund is different from every other, and they're all extremely secretive, so we're basically not learning anything about this. If somebody figured out how to make money this way, I don't think we'd find out about it.

   1:20:38

   Nathan Labenz: There wouldn't be any signs. It's an interesting thought experiment — how would we know if somebody did have a truly differentiated superforecaster? It sounds like your expectation is that individual hedge funds do well all the time, and there's not really much we could monitor for that would tell us. Is that right?

   1:21:00

   Dan Schwarz: I'll give you an example of how little you'd see this in basic indicators. I spoke with a hedge fund a few years ago, which I'll keep anonymous — their forecasting play was taking breaking news, like a drone strike or an oil embargo, and trying to trade Bitcoin volatility in the first 300 seconds after the news broke, on the theory that whenever something risky happens, Bitcoin volatility goes up. If you could figure out, faster than everyone else, whether a breaking story was actually disruptive or just another day with nothing interesting going on, you could go long volatility and exit, or go short volatility and exit, within five minutes of the news breaking. If people were doing that, would you ever notice by looking at Bitcoin volatility? I don't think we would. You could ask an even more basic question: how well are LLMs doing in finance? Do we have any idea how useful they are there? We all assume they're being used heavily, but do we have evidence — is it profitable, are people publishing papers on it? I think finance is just a poor place to look for evidence of how technology is diffusing into the workplace, because there are such strong incentives to keep it secret. Whereas with coding agents, we're getting tons of evidence, because the way to make money there is to massively distribute them. So I think we have to look where the evidence actually is, and that's part of why FutureSearch is a public product you can go try right now.

   1:22:43

   Nathan Labenz: What can you tell us about how you're building the forecasting agent? In the coding world we have different models critiquing each other — the general sense is that models from different providers have different failure modes, so they might catch each other's mistakes, and you get rewarded for having models from different lineages review one another's work. There's also the general idea that spending more tokens should get you further. Are you basically following the same trends driving coding agents more broadly, or have you found particular things that buck those trends or might surprise people?

   1:23:31

   Dan Schwarz: We're generally following those trends, but I don't think those trends are very well understood, so let me try to answer in a way that's informative. The very first thing we figured out building a forecaster in late 2023 with GPT-4, and then GPT-4 Turbo — the main way to get a bad forecast is to get bad information about the present. We were throwing more tokens at web research to try to understand what was actually going on, say with the Ukraine war or COVID, and we'd get stuck there. The research agents weren't even really agents for a while — they were 'for' loops, or what people used to call prompt-chaining. You could try fine-tuning models — there were various techniques available in 2023 and early 2024 — but any human expert forecaster looking at the output would say, you clearly have the facts on the ground wrong, so there's no way you're going to get a good forecast out of this. So we started studying what later became known as deep research, and we created something called Deep Research Bench, because you have to be able to evaluate good deep research on the present before you have any chance of a good forecast. One of the things that's really changed in the last twelve months is that getting the facts on the ground right is becoming more of a commodity — though it's still not fully a commodity; anyone who's used deep research recently in a field where they're an expert will still see that getting the basic facts right is beyond the frontier. More than 80% of the work of forecasting, I'd say, is just doing good research and reasoning on verifiable things about the present.

   Once you have that, there are a number of techniques to get a good forecast out of it. FutureSearch is a seed-stage startup, so we're not training $50 million pretrained models — we've got fine-tuning, reinforcement learning, agents and multi-agent scaffolds, the same techniques anyone downstream of the frontier labs has. Some of the most interesting papers on this are coming out of the Foresight LLM project from Lightning Rod Labs — they've actually been doing reinforcement learning; there's a link in a comment on Scott's post to another group doing similar work. We're starting to learn a bit about what these techniques are, and people are publishing results. FutureSearch put out a paper in April about strategic reasoning failures in frontier forecasters — we looked at our best forecaster, which is proprietary and which we don't share the internals of, and compared it to a very good Opus 4.6 research agent, pointing out stylistically what the differences were, what the Opus 4.6 agent was getting wrong. The TLDR is that frontier models still struggle to model human dynamics as well as an expert human, or a better-scaffolded forecaster, can. We'd find examples where a human forecaster, asked whether a bill would pass, would say: this president's top priority is getting this bill passed, otherwise they won't be able to go to this conference, and they need this as a face-saving mechanism — and the model would completely miss that, wouldn't understand the deeper motivations behind a politician.

   A lot of forecasting is geopolitical forecasting, and a lot of geopolitical forecasting is figuring out the true incentives facing all the players in an elaborate game-theoretic game. That gives you a sense of the frontier on the research side — if you can't understand those things about the present, that's where you'll get stuck in forecasting. Everything about turning the best possible information about the present into a good forecast about the future is mostly FutureSearch's proprietary edge, so we keep that under wraps. But you can see research capabilities improving for everyone — just go to the free tier of Gemini, ChatGPT, or Claude, and you're already getting research dramatically better than a year ago. Pay $20 a month and you're getting close to expert-human level, though still not quite there, for researching something like an economic or technology situation.

   1:27:31

   Prakash: Let's say AI forecasting gets really good — how does a consumer use that in their life? How do you actually make use of this? A geopolitical question, like whether Iran goes to war or not, isn't something the average consumer has much use for. So how would this work out in a consumer product?

   1:27:56

   Dan Schwarz: It's a great question, and it's been a central paradox of forecasting since Tetlock started studying it, or at least since I got turned onto prediction markets lurking on LessWrong forums in 2011. Implicitly, everything we do is based on conditional forecasting — every decision we make carries some implicit forecast about causal outcomes conditioned on whether we take certain actions. Every decision theory ever put in an economics paper has this idea that people model the world, and there's some association between the map and the territory. Predicting those outcomes is kind of what intelligence really is — yet if you handed people an oracle, they'd struggle to use it. How do you explain that? With the relatively poor-accuracy AI forecaster we built in late 2023, I put it in front of dozens and dozens of people — smart consumers, but also middle and senior managers at all kinds of businesses — and I was shocked at how they didn't seem to have the imagination to figure out what they should even be forecasting in the first place, despite spending all day making decisions that were implicit forecasts about the outcomes of their actions or their teams'. That's part of my intuition for why forecasting is just a very neglected capability — people don't know how to use it. Compare that to Midjourney: 'I've got this AI, I can generate amazing images, how should I use it?' People struggle with that too, but it's a lot easier — you can immediately see five or ten things you'd do with it, personally or professionally, for fun or to make money. Forecasting still requires some imagination — it requires introspecting about what decisions you're even making right now. As the CEO of a seed-stage startup, I've really benefited from doing my own forecasting, having AIs look at my decisions and reason about what the outcomes would be conditioned on me doing this or that. But even I, having studied this for a long time, have to put in effort to figure out what decisions I'm even making today, what outcomes I'm tracking. One of the most basic things people get stuck on is how you even measure the outcome — forecasting generally requires stipulating something measurable. Are you optimizing for dollars, users, engagement? How would you even know if something improved your relationship with somebody?

   1:30:58

   Nathan Labenz: Going back to how the frontier companies are thinking about forecasting — I know you can't share how they actually think about it, but could you give us your thoughts on how they should be thinking about it? Is it complicated? The naive take would be, it'd probably help if models are good at forecasting, so labs should try to make them accurate forecasters. Is there a less obvious level of nuance to that idea that you'd advise AI companies take seriously — whether or not they already are? What should they be doing, from your perspective?

   1:31:45

   Dan Schwarz: There are really two questions here: what should labs do with forecasting as a capability, and what should they do with forecasting as an eval? Forecasting as a capability is kind of a business decision — does OpenAI care whether ChatGPT is a good forecaster? That depends on whether their consumers care about it. If you're Anthropic, you probably care more about the enterprise case — when people use Claude for white-collar work, do they care how good it is as a forecaster, are people trying to use it to make financial forecasts in an Excel spreadsheet? That's a business decision, and I can't really weigh in on it. People will discover over time just how important forecasting is across everything, but it'll be a slow process for humans to notice.

   From the eval side, it's very different. Forecasting has this beautiful property: you get ground truth just by waiting. If I ask a question about the future that's basically impossibly hard — a question even an AGI, an oracle, a god, could never really answer because of chaos theory, like predicting the exact weather in a cubic meter three weeks out — you'd never be able to answer it in advance, but if you just wait, you'll see what that weather actually was. So you have a completely limitless set of extremely hard, essentially impossible questions where you eventually get exact ground truth, and there's no other eval like that. If you want to improve a coding harness, you need more and more hard coding problems, not in the training data, where you can say 'this is definitely the correct answer' so you can train on it — and that's hard. Human experts — doctors, lawyers, engineers, financiers, whoever — trying to build evals for the frontier labs are finding they're not smarter than the models being trained anymore. If you can produce something with a correct answer, the model's probably already going to figure it out; you need something with a correct answer the model can't figure out. Forecasting, I think, is the only completely renewable source of that. This connects to the Elon Musk-style quip that forecasting is the ultimate measure of intelligence — zoomed out, I think that's basically right. I wouldn't say forecasting is truly ultimate intelligence — coding intelligence, AI R&D intelligence, interpersonal intelligence all matter enormously too — but it is, to some degree, the ultimate eval, and I think that's something the frontier labs should be paying attention to.

   1:34:19

   Nathan Labenz: Do you have a sense of how much irreducible chaos there is? In the limit of superintelligence, how much visibility do we get into the future?

   1:34:35

   Dan Schwarz: That's a great question. David Manheim actually just posted a tweet with a graph estimating this that I've been meaning to dig into, because I don't know where those lines are coming from. Without evidence behind it, my sense is very Yudkowsky-ish — I think there's a lot of detail in reality that's far beyond the human mind to understand, and as you approach more sophisticated intelligence you'll start seeing a lot more patterns. Producing voxel-perfect weather three weeks out is further away than people think, but I think there's quite a lot of room. Human superforecasters don't tend to agree with me on this — they think what they're doing is somewhat near-optimal, and any accuracy improvement you'll get over them will be tiny and hard to understand. I think that's just because we only really understand human intelligence. Zoomed out, from an information-theory or Kolmogorov-complexity perspective — modeling the world as byte strings — the AIs will eventually figure out things that are totally beyond humans to notice. There's no way to prove this, but my sense is we'll start seeing it over the next year, as AIs get more and more accurate compared to humans in ways humans don't really understand. You'll look at the rationale for a forecast — five paragraphs of dense reasoning and then a surprising conclusion — and it won't really make sense, but it'll turn out to be really accurate, and we'll understand it less and less as time goes on.

   1:36:04

   Prakash: Mhmm.

   1:36:07

   Nathan Labenz: I think that's related to other modalities. It's something I've been obsessed with for probably the last two years — the fact that models are capable of learning a sort of intuitive physics in a lot of different spaces, from protein folding to materials science, like what the band gap will be for a particular semiconductor recipe. There's this strange, alien nature to these problem spaces, far from human sensory perception, that models trained on raw data — or in many cases simulation data — just seem to overcome. I'm wondering if you think that's a similar phenomenon, where they're pulling the vacuum on reality so tight that they just naturally get it, in a way that maybe isn't even language-mediated — or whether you think the chain of thought is still where the action is. When you say you see these five paragraphs of reasoning and then a surprising conclusion, it starts to call into question for me whether the chain of thought is really where the action is, or whether it's a facade — a superficial projection of some deeper, sub-language understanding or reasoning that might be going on. And obviously, if so, that potentially has a lot of consequences for the safety plans of the frontier companies, which seem to revolve mostly around monitoring the chain of thought. Meandering question, but hopefully you get where I'm trying to go with that.

   1:37:57

   Dan Schwarz: In some trivial sense, that's definitely the case — if you simply ask a human superforecaster to explain their reasoning, they can't actually make it fully legible. There's a layer of intuitive judgment that feels like deep learning: they look at a bunch of evidence the way a chess grandmaster looks at a position and just sees the right move, and they can't explain it — it just popped into their head, the way the grandmaster moves the knight and it lands on the right square somehow. That happens with humans already, and it happens with AI superforecasting systems today. So there's no a priori reason to think that reasoning would always be legible — there's going to be some layer of intuitive judgment, to the extent 'intuitive judgment' refers to something going on inside a large language model. It just has to be that way.

   Whether it's very that way or only a little that way is really your question, Nathan — if I read the reasoning traces, the rationales, the research it did, is it more or less what a human would have done, where I can see where it's coming from? Or is it inscrutable, in the way that it discovers some new pattern in the world nobody's ever seen before — the kind of 'psychohistory' from Asimov, where there's really an underlying structure, like some game theory of international politics, that it discovers from being trained on forecasting questions but can't really elucidate? Even in that case, it could probably write a program saying, 'politics follows these game-theoretic properties from some obscure textbook nobody's been thinking about, and if you use this you get incredibly high accuracy predicting these outcomes' — and we could look at that and say, that makes perfect sense. The real question is: what's the level at which it's doing something we cannot follow down the dark forest of its reasoning? Almost by definition, we can't know what that would look like.

   I'd say the best evidence will come from other domains. To the extent the AI 2027 scenario is starting to happen, where AIs get better at AI research, is that legible? Will the brilliant AI researcher at the company look at it and say, 'oh, that's such a good idea, why didn't I think of that' — clearly helpful and smart, but something a human could have done on a good day? Or will it just write programs full of weird floating-point numbers nobody can understand anymore, and then just go off to the races? We should look at all the domains where something like that is happening. A very salient example is clinical trial prediction — a lot of people want to know how well various drugs will do in humans, especially as AI produces more drug candidates and we can synthesize more compounds, and the bottleneck is safely running hundreds of thousands of people through trials and measuring the effects. If you could predict drug outcomes even modestly better, that would be worth many billions of dollars. What does it look like to predict drug trials — does it look like a human superforecaster reading the relevant papers and vibing a probability or a numeric range? Does it look like building an elaborate software model? Does it look like quantum chemistry, running enormous numbers of protein-folding simulations? What does it actually look like? If it looks like the human version, fine — but maybe something trained on that task discovers a completely different way of predicting clinical trials that all the world's biologists and pharma executives are completely blind to.

   1:41:28

   Prakash: Maybe I'll ask one last question, since we've held you past the time — where do you see things in 2029? We've had these six months of crossing the Rubicon. Give us an unhedged prediction, looking forward two and a half to three years — where do you think things end up on the forecasting front in 2029?

   1:42:00

   Dan Schwarz: FutureSearch contributed some forecasts to AI 2027, and we studied that problem pretty seriously with the evidence available a bit over a year ago. We built a model of R&D takeoff speeds under the core AI 2027 scenario, where the main way things get crazy is AI being used more in the development of AI — first hitting the superhuman-coder milestone, then the superhuman-AI-researcher milestone. I'm unhappy to report that I think that story is generally correct. I don't know if the timelines are exactly right, but my forecast from that process, leading to something that looks like superintelligence around 2031, is roughly stable.

   The things that have happened in the year since AI 2027 came out really vindicate the theory that the most important thing going on is how useful AI is at improving the productivity of AI researchers within the frontier labs. I publicly predicted that Anthropic was going to run away with it, because they had the best feedback loop of talent actually using their own AI internally — that's n-equals-one, but I think it's been pretty well shown to be happening. So the question is what role forecasting plays in that. You can imagine what was going on with Dario Amodei and Tom Brown going to the White House trying to get Claude Fable unbanned — what moves did they have available for dealing with the US government? Even if you're incredibly cynical and assume the government is just trying to score political points, what are those points, and what moves does Anthropic have to work with? There's a whole set of intergovernmental negotiations going on, and to what extent is Anthropic, a company working on AI, able to use that AI to negotiate a better outcome with politicians? That sounds like a conditional forecasting question to me — if I write this letter, if I propose this export control, if I weaken the model, what's the probability this person relents within the next three days? We don't have evidence that's happening — there may not be any evidence of it at all — but I think in a 2029 model, whether AI forecasting is helping the frontier labs make better decisions more effectively is very similar to the way improved coding is helping them develop AI systems faster and more productively. That's the thing I'd watch.

   1:44:27

   Nathan Labenz: I've got to assume they're at least talking to Claude a bit along the way.

   1:44:31

   Dan Schwarz: I'm super curious — Anthropic senior leadership, if you're watching this and want to comment on how useful Claude Fable was in the Claude Fable situation, I'd love to hear it.

   1:44:40

   Nathan Labenz: They may not have been able to use it either. Prakash, I have one other question, but go ahead if you've got one. Two more practical questions — first, what does it cost to run a forecast if I go in as a consumer? Second, if I had a few million dollars to invest in creating a public good, how would you think about building a public good with forecasting? We might say, what'd be really great is an amazing voter guide to the midterms — this person gets in, that person gets in, here's what you can expect. Could we help the public make better decisions by giving them visibility into the consequences of the votes they might cast? That's probably a pretty limited idea, though — I'm interested in farther-out ideas for public goods somebody might be able to provide.

   1:45:56

   Dan Schwarz: Glad you asked that, Nathan.

   1:45:58

   Nathan Labenz: Go for it. I've got one more too, actually — no, that's enough for now, I'll follow up with the other one.

   1:46:03

   Dan Schwarz: It costs about a dollar or two to make a frontier forecast — that number can get a lot higher or a little lower, but that's a good anchor. If you look at the cost per input and output token for an LLM, that gives you a rough sense of the amount of research being done. One of the core questions FutureSearch tackled — I described earlier how our main frontier for a while was just doing good present-day research, until we got good enough at that to use it to improve forecasting — was: can you just pour more tokens into a question to get a more accurate answer? It doesn't have to be a forecasting question — if I ask you the current state of some clinical trial and want the most accurate answer you can give, can I just pour more tokens into that and get a better answer? This was studied as deep research — writing these fifteen-page reports with 700 citations. That gave you a longer answer, but was it a better answer? It wasn't super clear, which is why we studied it.

   Forecasting gives us an opportunity to do some world-modeling. FutureSearch talked about this a bit at the Manifest conference a couple of weeks ago, and the feature is rolling out in the product, I think literally today. The idea is that once you have a repository of forecasts, every marginal forecast can draw on the implicit world model in those forecasts to give a better answer. FutureSearch co-founder Lawrence Phillips wrote this up on LessWrong a couple of months ago, and it was a bit neglected — he made the case that if you produce a large body of forecasting questions that feed into each other and remain mutually consistent, you could understand the world dramatically better, as a public good. The main barrier is simply: when you put more tokens into your world model, does it actually get better, or does it get worse? His big insight was that around January or February, around Opus 4.6 or GPT-5.4, for the first time it became possible to put more tokens into a broad research task and actually get a better answer, not one that just trails off into garbage-in-garbage-out nonsense. FutureSearch is building this into its products, and that's part of why we have a consumer product — the more people forecast, the better the forecasts get for them, and in theory the better they get for everybody as we build this deeper implicit model of the world.

   Lots of companies and research labs have had ideas about building world models — though the term 'world model' as I'm using it might be a bit misleading; a lot of people mean something like geospatial reasoning, building a robot hand that can pick something up. That's a world model too, just a basic one for predicting outcomes. The difference with forecasting is that the rationale behind a good forecast is a very compressed, brilliant five-paragraph summary — the things most causally relevant to a hard forecast question are the best summary of information you could reasonably produce, because every consideration in there leads to a different date, number, or probability. When you start recording five paragraphs of the most important considerations for the most important questions, and scale that to thousands or tens of thousands of questions, and then read through them asking — what are the latent variables here, what are the implicit claims, why do we think it's this way, are these consistent with each other — you start to learn a lot, in a way that's different from just pretraining on all of Wikipedia, which fills you up with random, uncorrelated facts. With a repository of forecasting research, you might learn things about the world you could never have seen by asking one question and getting one answer one time.

   More broadly, the big question is: can you pour more tokens into any kind of research and get better research out of it — AI research, coding, whatever? You've burned through your Fable tokens for the day, you top up, and can you just let it cook — let it keep cooking better and better as you put more tokens in, or not? From a forecasting perspective: could you get to the point where every marginal new forecast improves the accuracy of every other forecast, until you're building a consistent world model? That's what FutureSearch is after right now. We have some preliminary results indicating this does work, and we're rolling it out to our users right now.

   1:50:30

   Nathan Labenz: What structure does that ultimately get instantiated into — are we talking about a graph database? I can see those kinds of ideas making sense here, but I can also imagine them introducing some weird failure modes. Here's a fun fact about me: I was actually on the Good Judgment team, way back in the DARPA forecasting tournament — or was it IARPA that funded it — fifteen-plus years ago. I did well, but not top-tier superforecaster level. Around the same time, I also worked briefly at a financial-services consulting firm that had done a lot of the financial risk modeling for Fannie Mae — I probably don't have to tell you how that story turned out. There was a lot of expert forecasting instantiated in a very spreadsheet-driven, causal-graph kind of way — you could literally hit the visualization button in Excel and see these colored arrows fanning out from cell to cell — and somehow, in the end, it was all totally off. So I wonder how you think about correlated failures as you build out these world models. Is there any kind of correction mechanism — something to catch an assumption like 'housing prices never go down nationwide' lurking in the world model? Obviously humans have this problem too, the financial crisis proves that, but you can imagine the next one being even worse, because we're so reliant on a small set of AI minds working the problem from a thousand different directions that might all share somewhat consistent flaws in their reasoning as they go. Can we protect ourselves against that in any way?

   1:52:54

   Dan Schwarz: Definitely — I'll try to answer that both theoretically and with an anecdote. I tried to world-model the Fable situation when it got banned, partly because I wanted to and partly because it was a good forecasting question with some real money trading on Polymarket. And I made exactly the mistake you're describing, Nathan. I ran a bunch of FutureSearch forecasts and manually went through them — a couple of scenarios, some conditional forecasts, basically thirty-three load-bearing sub-forecasts, starting from what even happened: why did the government issue this export control? Was it a simple misunderstanding? Political leverage? A genuine foreign-threat concern, or hacking risk? We didn't know, so I put it all together. When I looked at all the outcomes — and I talked about it with Claude Code a lot — one thing came out: basically every forecast, in every scenario, assumed access would come to Americans first and foreigners only later. That was wrong — when it came out last week, it came back for everybody at once. So clearly there was some faulty assumption baked into one of my scenarios — a correlated failure somewhere. I still haven't fully figured out where my reasoning went wrong; it's also possible I just got unlucky and we landed in an unlikely outcome. On an n of one, you can never really know if any single forecast was good — that's one of the hard things about this. But I think I systematically got it wrong by having a bunch of correlated reasoning failures across my scenarios. This definitely does happen.

   Metaculus has a system for this — in the years since I was CTO there, they've built an actual causal-graph platform and product; you can go to the Metaculus site, click run, and find it there. I think the field still generally believes something like this will work, but nobody's made a really good one yet. I tried my best over about twelve to sixteen hours on the Fable situation — I think I built a pretty good model, close to a very accurate forecast, but didn't quite get there. I don't think the Metaculus models on their site right now are amazing either, but I do fundamentally believe in the approach. As you're saying, Nathan, this has been tried for a long time — when I was CTO of Metaculus, honestly, it was the dream, the holy grail: can we tie all these forecasts together into some kind of causal graph? What I can say is that AI makes this tractable. There was just no way that was going to work with a bunch of human economists looking at Freddie Mac or Fannie Mae — I can totally understand why that method didn't work for them then. Whether AI can make it work right now is unclear, but whether AI will make it work in general feels nearly guaranteed, and I don't think FutureSearch is the only org working on this right now.

   1:55:40

   Nathan Labenz: Well, you've been super generous with your time — we only had half an hour on the calendar, and we've been at it nearly an hour.

   1:55:46

   Dan Schwarz: Have to go see — right behind me, it got darker and darker as the conversation went on. Did you—

   1:55:50

   Nathan Labenz: —guys notice that? I need to— my lighting is a constant source of, let's say, comedic relief for us if nothing else, so you're in good company there. Maybe just in closing — sketch out a bit more of the future as you hope it might unfold. Not necessarily the most likely scenario, because maybe the most likely thing is people act foolishly and don't take advantage of the benefits of forecasting. But if we really do a good job — we're interested in truth-seeking, and we get the AIs working as well as you think they might — how do you think life kind of feels different?

   1:56:40

   Dan Schwarz: I have to lead with another example of me being a bad forecaster — I guess everyone who tries forecasting thinks they're bad at it, because they see themselves getting things wrong. A prediction I made really strongly five or ten years ago has basically been totally falsified: I predicted that if we had highly visible, highly liquid prediction markets covering all the major technological, political, and economic things going on, humanity would be wiser and people would make better decisions in government. Well, here we are — we have Polymarket and Kalshi, and I don't see any wisdom or better decisions coming out of all the gambling happening on those platforms. So part of thinking about our AI future, for me, is trying to understand the present a little better — why isn't having thriving prediction markets transforming the news, or how people learn information, or how they plan for their futures? One simple answer is that it does, it just takes a while — we're only about a year into prediction markets making major headlines and being seen by everybody, and maybe it just takes time for people to change their habits. AIs, if that's the case, can move much faster as they get better at forecasting.

   Ultimately — and you said this, Nathan — we're after the epistemics. It's not just about predicting an outcome, we want models that are reasonable. One of the beautiful things about forecasting as a human practice is that it makes you more epistemically virtuous: the more you try to forecast, write down what you got wrong, and do postmortems, the more it humbles you and makes you more open-minded — more of a fox than a hedgehog, just a more reasonable person. Prediction markets, with everyone doing this, should be leading to people being more reasonable — except I don't think people are doing a lot of actual forecasting on prediction markets, they're doing a lot of trading and gambling, which is related to forecasting but isn't forecasting. If AIs get better at forecasting and become more epistemically virtuous, we could be in a world where just talking to a chatbot gets you something so much wiser and more grounded, more honest about its own uncertainty, and better at probing you about your own uncertainties. That could make an enormous difference. But putting my cold-blooded forecasting hat back on, I think the technological outcomes of AGI will arrive before that cultural change happens.

   So I'm very much in the AI-safety camp — I really think we should slow things down, give ourselves more time, fund more AI safety research, and do more policy work. Because if we have time for the wisdom of these alien intelligences to actually help us make better decisions before the critical decisions get made — there's a series of decisions coming in the 21st century that we'll look back on the way we look back on 20th-century decisions about communism, World War Two, the atom bomb. Those decisions are coming, maybe some have already been made, and right now I don't think they're well informed by rigorously accurate forecasting AIs. But give it another couple of years, and we might be in a world where everybody has the same grounding — as smart as Kissinger, but actually trying to help, trying to give us all better outcomes. That could usher us through this crazy phase before the paperclip-type stuff starts to happen. So I feel like I'm racing to make AI forecasting useful and helpful — it's part of a broader epistemics-and-safety process, because otherwise it's just going to get away from all of us, and a lot of the work we're doing won't matter.

   2:00:11

   Nathan Labenz: Cool — well, as my dad would say, good luck. We're all counting on you.

   2:00:17

   Dan Schwarz: Thank you, Nathan. Thank you guys for having me — I really appreciate it.

   2:00:23

   Prakash: Thank you, Dan.

   2:00:25

   Nathan Labenz: Dan Schwarz from FutureSearch — thank you for being with us on AI in the AM.

3. 2:00:31Closing2 min

   Closing

   Watch

   ▶Play this segmentYouTube · from 2:00:31

   As aired

   Nathan and Prakash closed the show by pitching a rematch of their prediction-market competition — this time bringing in Future Search as a third forecaster alongside the two of them, picking another dozen or so questions to see how their calls stack up against the market price.

   They wrapped quickly on time, with Nathan heading off to meet his sister's new baby and teasing plans for tomorrow: getting the prediction-market rematch spun up, plus an update on Pangram Labs and a graph tracking whether his own writing is becoming more "AI" over time.

   Key moments

   Wisdom is indeed in short supply.

   Full transcriptLightly edited · timestamps jump to YouTube

   2:00:31

   Prakash: Bye bye.

   2:00:36

   Nathan Labenz: So this makes me think we need to do another round of our prediction-market competition — between you, me, and this time, Future Search.

   2:00:45

   Prakash: Mhmm.

   2:00:46

   Nathan Labenz: I'll see if we can't get another one spun up. I don't know if they have an API — we'll have to check out the details of exactly how they make their Future Search technology available. But we did it last time just between the two of us, and it came down pretty much neck and neck. I wonder if we pick another dozen or so questions and go at it with the two of us, and then have Future Search weigh in too, relative to the market price.

   2:01:23

   Prakash: Have a—

   2:01:24

   Nathan Labenz: —prediction. Competitive or not.

   2:01:27

   Prakash: Indeed.

   2:01:29

   Nathan Labenz: Very interesting stuff. Wisdom is indeed in short supply, and I do love the vision of it — if we could just get a clearer idea of what it will imply if we do these things, then should we actually go down that path or not? If we could be better informed, it seems like we could hope to do a much better job at key decision-making. So I really do like that.

   2:02:01

   Prakash: Indeed.

   2:02:02

   Nathan Labenz: But — yeah, whether human decision-making is actually so receptive to better context, I guess time will tell.

   2:02:12

   Prakash: I—

   2:02:13

   Nathan Labenz: Speaking of time, I understand we're both kind of short on time today. I've got to go visit my sister and her new baby, which is very exciting for my family — yes — and I know you've got some stuff you've got to go do as well. Tomorrow, maybe we can get one of these prediction things going. And I do have an update on Pangram Labs, which we talked about last week as well, and whether or not my writing is becoming more AI over time — I can present that as a graph tomorrow when we get back together.

   2:02:42

   Prakash: Sounds good. Bye bye, Nathan.

   2:02:44

   Nathan Labenz: Alrighty. Thanks, Prakash.

   2:02:46

   Prakash: Cheers.

   2:02:47

   Nathan Labenz: See you soon.