AI in the AM is a live weekday morning show on AI. Day two paired two of OpenAI's forward deployed engineers with the builder of the world's first Catholic AI — the automation of expert work on one side, what AI means for the Church on the other — with field notes from the Recursive RSI event in between.
EPISODE 2026-06-02
AI:AM LIVE — June 2, 2026
A live morning show with John de Wasseige and Arthur Fernandes Araujo of OpenAI on the self-improving tax agent they built with Codex, a research review from the Recursive event, and Matthew Harvey Sanders of Longbeard on Catholic AI and Pope Leo XIV's first encyclical.
Episode timeline
Opening — news + discussionDay two of the daily experiment: Anthropic's S-1, Google's $80B raise, Bernie's 50% nationalization proposal, and OpenAI landing on AWS Bedrock.
Recursive self-improvement — for the show itself. Day-two reflections on the live experiment: Prakash on clips as the financial engine of new media (short clips get 10–100x the distribution of long formats), Nathan on making artifacts that serve the truly plugged-in — and on the show as a harness for two "little proto-AGIs" whose affordances are, by default, slow.
Google raises $80B in equity — Berkshire Hathaway takes $10B of it. Prakash's overnight read: Google's first equity sale since its IPO, after years of escalating buybacks. Buffett's stake comes with dilution protection, and an equity raise of this size implies a much larger debt issuance to follow — a sign that even the biggest players feel the price of capital rising.
Anthropic files a confidential S-1 — days after a $65B Series H at $965B post-money. Filed June 1 and announced on the company blog, which Prakash found telling in itself ("why?"). Nathan flagged the commentary that at $1T the IPO would obviously be a steal — which is exactly why it probably won't price there. Sam Altman, meanwhile, spent the day on CNBC needling Anthropic over its doomier jobs messaging.
Anthropic has confidentially submitted a draft S-1 registration statement to the Securities and Exchange Commission. Pending completion of SEC review, this gives us the option to pursue an initial public offering. Read more: anthropic.com/news/confident…
buying into the anthropic IPO at $1T valuation would obviously be an incredible deal, 22x multiple on ARR, huge room to grow, countless markets untapped, mythos as of yet unmonetized. kind of thing people dump whole retirement portfolios into. which is why it'll be $3T
Anthropic has confidentially submitted a draft S-1 registration statement to the Securities and Exchange Commission. Pending completion of SEC review, this gives us the option to pursue an initial public offering. Read more: anthropic.com/news/confident…
Bernie Sanders proposes a 50% public stake in OpenAI and Anthropic. The longest-running debate of the morning — Nathan sympathetic, Prakash firmly opposed (see the discussion below). Vitalik Buterin's critique framed the stakes: frontier labs went from "benefit all of humanity" in the 2010s to "benefit all of 4% of humanity" in the 2020s.
One of the many things I dislike about the style of "make AI go well" discourse from frontier AI companies is how nationalist the whole thing has gotten. In the 2010s, it was: "we're here to benefit all of humanity" In the 2020s, "we're here to benefit all of 4% of humanity" Show more
NEW: Bernie Sanders proposes the government take 50% of OpenAI & Anthropic to give the public a “direct ownership stake.”
OpenAI models land on AWS Bedrock — Azure exclusivity is over. To Nathan, the bigger pattern is that everybody is now in bed with everybody: balance sheets, rev-shares, and compute deals have backed the industry into "one big AI mega corp" where the failure of any one player risks contagion — and is therefore probably too big to fail.
OpenAI frontier models and Codex are now generally available on AWS, giving enterprises a new way to build on Amazon Bedrock with OpenAI through the security, compliance, and governance workflows they already use. This is also the beginning of a broader expansion of OpenAI Show more
Ai2 appears to be breaking up. Nathan Lambert, one of the leaders of the Olmo open-source effort, is leaving — another well-regarded US open-source champion struggling to keep its talent, by Prakash's read.
The Bernie Sanders segment turned into the morning's real argument. Nathan made the sympathetic case: the top three US companies by market cap — NVIDIA, Apple, Google — are already worth roughly 10% of America's ~$165 trillion in national wealth, and if Anthropic and OpenAI list at multi-trillion valuations that concentration only deepens. His sharper point: whether or not the government admits it, it is already "the financier of the AI mega corp of last resort" — if OpenAI ever can't make its payments, a step-in bailout follows — so the public arguably deserves a claim on the upside, not just grants from "a Larry Summers–flavored board."
Prakash's counter was that taxes strictly dominate equity: tax claims are non-dischargeable in bankruptcy, can't be gamed by boards that simply decline to pay dividends, and already give the government a stake. On his read, the 50% proposal is a "magic trick" — it's about additional dollars on top of taxes, routed through a sovereign-wealth-fund bureaucracy precisely to bypass Congress and its power of the purse. Nathan's rejoinder: these companies may run the Amazon playbook and show no profits for a decade while capex goes to data centers — in space, eventually — so payroll and corporate income taxes may never catch the value. What would you even tax: tokens, data centers, a "replacement tax"? The question was left open, and teed up the first guests.
TranscriptAuto-transcript, lightly cleaned · timestamps jump to YouTube
1:59All right. Good morning and welcome to AI in the AM. We are sprinting through the AI marathon and it is day two of our June experiment of going live most weekdays and trying to make sense of AI developments in real time and sharing the process of doing that with as much recursive self-improvement and and day-to-day updates as we can manage. Good morning Pash. How are you? » Good morning Nathan. Um it has been an exciting time overnight. Um
2:33and uh as always we will have like some back and forth here as we uh as the cameras come on. Um as we as as you know this is a a a you know AGI complete problem to get all the cameras and everything working at the same time. Um, so yesterday we literally made changes to the studio using codecs in the background while we were live and and had uh actually our very first guest had to refresh the page to get the audio from me. Uh, Pash has vibecoded this entire thing and it's pretty cool. I think there's going to be a lot of value and both in the utility of doing this all ourselves and being able to customize it just the way we want and also a lot of uh value in just kind of showing off how we're doing this. We both took a bunch of notes yesterday from what I would say was, you know, in some ways a good first uh pace mile in the sprint through the AI marathon, but in some ways it was a bit rough. Uh, so we definitely both took a lot of notes and we've been prompting away and developing the skills to try to make sure that every day that we come back and do this, we're going to have a better, tighter format. Um, and it'll be interesting to to watch this thing evolve and see if we can get to the point where we feel like we're really hitting a stride and have kind of a sustainable pace that is serving us and the audience. I think one of the things I I really want to make sure for myself is that I'm not becoming a content person in short. I really want to make sure that I am still focused on learning and understanding as much and as as well as I possibly can and uh and building in public and you know genuinely sharing the the process by which we're doing this I think is going to be a huge part of that. I was just kind of going off on my own and saying, you know, describing a little bit of the recursive self-improvement philosophy of this project and how for me, and I think you have a little bit of a different angle on this, but I really want to make sure that I kind of think of this as like live reporting in a way. You know, one of the reflections yesterday when I I took my
5:03run was I really want to make sure that I'm making something that is of value to the true insiders. You know, I think of somebody like friend of the show Dean Ball or I don't think I think Z reads only and doesn't really um watch much stuff unless he sort of has to for um for review purposes. But I want to make artifacts that are hopefully of value to people that are really plugged in. And so in in that sense, I think what we're doing live is sort of live reporting. Um, but there's going to be all this sort of apparatus around the actual live engagement that we're gradually building out that hopefully will make additional artifacts that, you know, are going to I aspire to uh having them be useful for people beyond those that can actually tune in live, right? I mean, I think it's going to be tough to get too many um AI engineers or, you know, open AI engineers, let's say, uh watching two hours a day. So, how do we sort of both engage people live, make sure that what what we're doing is actually serving our own understanding and advancing the sense that we can make of everything that's going on. um and caching that out into artifacts that can be shared and hopefully have a little bit more durable value. Um those were big reflections from day one for me and um certainly many prompts have flown to try to instantiate that stuff even in just the first uh interval between yesterday and today. Um indeed and uh as always it's uh for me the most interesting thing is that there's so much that happens every single day now right um and it's not it's it's it's there is a lot of value in kind of making a having a world view worldview today and updating it in like a week or so and seeing how much you've moved Right. Because I find that um a lot of people don't update their priors. Gary Marcus being one of them.
» Gary, if you're listening, your ears might be burning. » Well, yeah. I've been I've been, you know, like like many people, I've been blocked by him. So, at some point at some point in the last
7:33three years, you know, if you if you if you've teased Gary too much, uh he's he's definitely blocked you. It's funny that I don't think he's blocked me and arguably I deserved it because one of the first things I ever wrote in a public way online and it it wasn't um it's another example of me being not super strategic but just in that moment and feeling like I had something to say was when he was on the Ezra Klein podcast. Um, and you know, we didn't have as much time yesterday as I had hoped um due to some uh iterative uh improvement that we did in in real time to talk about Ezra's piece and kind of working backward and you know having a vision. I think he's broadly been really good on that stuff. I think the the Gary Marcus episode was a bit of a miss because it was like full of all these things of you know AI can never do this, I can never do that. And then I just was so upset about it that I went off and wrote a I don't even know like a hundred long 100 tweet long thread basically attacking his appearance and all and you know kind of correcting the record as I saw it on everything that he said that I didn't really think was accurate in that podcast. And uh that was like the first thing I ever put online in a in a meaningful public way. Um, and it was it was ended up being like much more viral than I anticipated and it kind of turned out to be quite the dunk. But to me, he's been like quite gracious actually and has not um and I haven't like chased him down online, you know, for every uh time that he's moved the goalposts or, you know, said something that, you know, well, a I will never do something that it can already do, which I think has continued to happen probably dozens of times. Um, but for the record, he's been gracious to me and he and he's uh I'm still not blocked. I I I I I I deserved it, you know.
» I think I honestly deserved it, too, but I didn't get it. » So, he I you know, I normally refrain, but it was just too tempting a target. And all I did was he had a yet I I quote tweeted something he tweeted like, you know, like several months before with a Wy E Coyote like, you know, cartoon of, you know, the coyote falling off the cliff. Yeah, he blocked me for that. Well, we've got a bunch of news to cover again today to warm up. Um, but any thoughts or reflections on takeaways
10:03from day one and you know kind of what are your top priorities for recursive self-improvement right now? so I think I think we we've spoken of the on this uh offline but um we have a pipeline of work I guess that starts from um you know uh opening opening the dates and then you know inviting the guests confirming the guests and then getting all the content the research ready we do a lot of research for these shows uh and the research is because we it's largely helped by AI Um I think you know we and we're happy to share um with the audience uh what what we do uh but over time we've both refined these prompts that we have uh for the AI and have show notes etc. And then we end up with um you know the the the stream itself and then we end up with postream we have a lot of artifacts to cover we have um you know to all the channels and clipping. So um if you are not in the new media space uh what is happening right now is that the short 30 secondond to 1 minute clips uh get some something like 10 and 100 times the distribution of the longer formats and so uh the clips have really become the financial engine uh behind uh most of these new media shows that you see. Um when when people talk about new media, what they're really talking about is clippable media that is being broadcast to social channels. Uh that that is that is essentially what new video media is.
The live stream is kind of producing producing the content itself in bulk. But then the refinement process and the clipping process is what um you know converts um this kind of and and I I I look it as primary historical research really at this point because we are talking to people who are actually in the flow and are making history and have made history in the last three or four years. Uh and that becomes clips that you know then go out. Um, and our hope is that this whole entire cycle turnaround uh really speeds up a lot which is what enables the the actual interactivity of the medium. Um, and what we found is that um your your words really have power on social media, right? Like if you have a take and um you you throw it out there and you know
12:35you have a following and you are loud enough it reaches a lot of people and uh that has I think significant value. Um new media these clips are basically replacing what people used to get from um you know new sources of all kinds um and people are selecting um new sources which are meaningful to them. So that is that is um and to the extent that we can shorten that cycle that time down so that we get it in the flow while things are happening instead of in retrospective that would be um you know ideal I think and that and that's where the recursive self-improvement part comes in because you can recursively self-improve your own like environment like the AI field itself by feeding that information back in uh and you know educating people and you know pushing message out. So that that is that is really uh my my idea of uh how we could uh it's really recursive self improvement not just of the show but of the entire field itself that you know you know potentially we can do um very ambitious I know uh Nathan what what do you feel » I think that one way to think about what we're building right now is kind of a harness for ourselves where we are in some sense little protoagis But our affordances are by default kind of slow, right? Like we it's hard to process all the information we need to process. It's hard to stay on top of all the discussions. And so creating these skills that sort of pre-process information, present them to us in hopefully a a really effective and timeefficient way, set up a hopefully a useful and easy to follow and informative discussion. Um, but do that, you know, ideally I think at this point we got to be aiming for at least an order of magnitude speed up relative to what we could do on our own. And I think that obviously the core challenge of this is going to be our own human brains or at least I can speak for myself and say, you know, I can have AI agents go out and do all kinds of information pre-processing.
I don't necessarily learn something from that. So, I I'm going to need to hold myself accountable to making sure that I'm actually learning as much with the AI doing more of the work than I was when I was doing more of the work
15:06myself. I think the jury is still out for me on exactly, you know, how to accomplish that. But that's got to be the goal because there's just this time dilation effect that, you know, as as we're speeding up, uh, we got to find some way to slow down time to make sense of all the things that were passing by. And this harness that, you know, it kind of allows us hopefully to drop into a a an effectively pre-processed view of current events is my best idea right now for how to speed up my own u learning to a point where it can hopefully keep up, you know, or at least get get closer to the singularity. So, with that, maybe we should cover some of today's uh outline.
What do you you got anything else you want to jump to it? » Yeah, we can. Um maybe maybe I can uh just share the uh the overnight um the overnight news. So, let me just uh plug this in there. Um and get let's see we have uh Yep, there's Blossom. All right, there we go. And let me pull it up on screen. go. So, uh the the the major thing that happened uh overnight is uh Berkshire Hathaway uh has agreed uh Google decided to do an $80 billion fund raise. Um and I will I will show you this very interesting graph. This is uh Google's uh stock buybacks per year. So they increase stock buybacks yearbyear to have 15 16 17 18 19 20 24 25 and then this the stock buybacks this year where they sold stock. So Google has sold stock for the first time in uh since since they were um since the IPO. Uh they they they haven't sold equity before. Um normally what tech companies do is that they sell convertible bonds uh for a variety of reasons. Uh number one founders don't want uh pure debt because when you go bankrupt that debt will come back and haunt you. So they go for convertible bonds where they can convert it to uh stock where you don't where interest rates are basically zero or dimminimous. Um and that's what high growth companies um you know typically do. Uh instead uh Google has you know
17:36issued street equity. Um and there's a bunch of comments and takes on this online. Uh the first is that the first thing that you have to note is that Brookshire Hathaway has agreed to put in 12 a.5% of the 80 billion. So Brookshire is putting in 10 billion. Um Warren Buffett at this point has about $400 billion of cash on the balance sheet. He sold off a bunch of companies. Um he thinks valuations are high. He's waiting for valuations to get better, but he put in 10 billion here, which is a which is a, you know, big chunk. Um, he has now um a 7% allocation to Google in his portfolio. Um, and he's got some, you know, dilution protection. So, he had uh he had a slightly discounted rate uh from the current rate. As a result, Google stock took a tiny plunge like maybe 3% down. Um it's it's a large company. $80 billion in the context of a several trillion dollar company is not significant but it shows the capital requirements which are hitting even the largest players. Uh and Ruth Borat decided to raise equity not debt.
That means that you know you normally need to keep this kind of debt equity ratio. So if you're if you raise more equity now, that means you're going to raise three or four times that amount of debt later. So there's going to be a debt issuance coming. So there's going to be another couple hundred billion dollars of probably debt issues coming for Google. So that's probably a quarter trillion quarter trillion of capital uh Google taking in in one shot. Um and then we also had uh yesterday anthropic uh filed an S1. So, an S1 is a confidential registration statement with the SEC where you tell the SEC, we're thinking about going to IPO. These are numbers, but we're not telling anyone else the numbers yet. Uh, so they and then not only did they file a confidential registration, they announced it on the blog, which is like, why? Um, and thirdly, Sam Alman was on CNBC yesterday uh really talking uh talking smack about anthropic. He's like, I don't know why. I don't know why they have to say all these bad things like everyone's going to lose their jobs like six months ago and now they're not talking about it. Um in essence what we having right now is this race for capital. There's a sense in the market that capital is scarce or will become scarce. Um and when capital becomes
20:09scarce the price of capital will go up. And I think Google issuance the Google issuance is a sign that the the price of capital is starting to go up and that they're issuing the equity in order that they may issue debt at a at a at a lower price. Uh and so this is something you kind of have we kind of seem to be having uh higher inflation rates but also higher returns on stocks. And you know that's great that you have a high return on stocks, but if you're a working person on the street and you are not really invested in stocks directly, maybe you have a pension fund or whatever, but you're not directly invested. You're watching um gas prices go up and inflation go up, you're not getting the benefit of this. So that's one thing to note. Uh it's very interesting to see Bergkshire in there.
Uh if B if he's in there for 10, he's probably going to be in there for another 100 later on. So I think you might see Bergkshire as one of the primary funders of this infra at this point. Um one other thing margin loan balances. So margin loan balances uh when people buy stock uh you can take a loan uh from a brokerage and when you take a loan from a brokerage you can you know you have a margin loan and that makes your returns better because someone's lent you the money to buy stock and uh but if stocks go down you have a margin call.
you have to repay the money and this is um there's no like limitation of liability here. You can go absolutely bankrupt if you if you can't cover. Um so margin loan uh stats are at all-time highs at uh retail retail brokerage. It's more than hundred billion dollars of margin loans just in one brokerage which means across the market you have trillions of dollars. Um and finally um AI2 seems to be breaking up. Uh so AI2 um they had the I think Almo models uh that they put out. They were very they were American very strongly open-source um you know group and they uh disclosed a lot of their secrets while they were trading these models. They disclosed everything. One of the key questions is that they they were very well regarded technically and but they were not making a lot of money. And so one of the questions has always been can they keep their talent and it seems like Nathan Lambert who was one of the leaders there has is leaving and I would I I saw another post from someone else who is also leaving there. So that is you know one more one more thing to note uh another uh US open source champion seems to be uh breaking apart. So um back to
22:41you Nathan. What you know comments on the on on the overnight news? Well, definitely the anthropic uh preparation, you know, not confidential confidential filing stands out. Some of the interesting commentary that I've seen around that first of all is like, okay, what's the valuation going to be? Excuse me. The um you know, one of the more interesting posts was basically saying at a trillion dollars, it's obviously a steal and so it might be significantly more than that. I'd be interested to hear what you think the um you know the opening and fair price for anthropic might be and obviously that's going to be driven by uh sentiment you know as much as fundamentals fundamentals are strong but the sentiment might be even stronger and then at the same time we have Bernie Sanders who you know is taking a lot of positions on AI some of them arguably contradictory has put forward this idea that there should be some national equity stake in the biggest tech companies and I would say like I'm very sympathetic to that idea on some level.
I think there's, you know, a lot of uh friend of the show Dean Ball has been uh critiquing him repeatedly for, I think, the contradictory aspects of his portfolio of positions, which is like I want to have an equity, you know, share for the public, but also we can't have any more data centers, and that doesn't seem to fit together super coherently. Maybe you can make sense of it. Um but I am pretty sympathetic to the idea that like yeah you know I mean American wealth is estimated at something like 150 165 trillion dollars uh across you know all the major asset classes that's basically five times a little more than five times GDP and you look at just the market cap of the five or 10 biggest companies and you're starting to get a, you know, not insignificant share of national wealth is bound up in the five to 10 top companies. I mean, literally just the top three of Nvidia, Apple, Google gets you to about 15. Like those three companies are worth almost 10% of America's entire national wealth. And
25:11certainly the AGI pill among us think that that might continue even more dramatically into the not too distant future. Right? If if all of a sudden you add Anthropic to that at a few trillion and OpenAI to that at a few trillion and you know it's it's no longer crazy for people to speculate that Anthropic could be a 10 trillion dollar company in a few years. Um then yeah, it's a little crazy to have that much wealth bound up in so few hands. So, I'm pretty sympathetic to the the Bernie argument that there should be some sort of share. Um, I'm also very sympathetic to Vitalic Buterine who said, "Man, this sucks." Like, » the US population is only 4% of the of the global population. These companies were started with a benefit all humanity and now they're like going to IPO and, you know, there's like they're going to give away some through the foundation or whatever. Um, and Bernie's got this idea that we should have a national wealth fund, which could be good, but what about the other 96%.
» I think that that discourse is going to be something really interesting to watch, and it's also going to be I mean, I'm throwing a lot of questions at you at once here, but another news item that came across yesterday was OpenAI now going available on AWS. Mhm. » So this continues the trend of just everybody being in bed with everybody, everybody's kind of fates uh being tied » in one way or another uh balance sheet or you know revshares and all of it, right? It's it seems like we we've sort of created kind of backed into creating in a way like one big AI mega corp where the you know a failure of any one of them seems like it would have like massive contagion or at least there would be like significant risk of that at this point.
» It doesn't seem super likely to me like I I think basically the demand is going to be there barring some external shock. Um but you know external shocks could happen or I could be surprised and this whole kind of probably already you know once you count all the companies that are deeply intertwined in all these different ways you have something that's maybe a $30 trillion mega corp that is almost for sure too big to fail. Right? Right. So I think that that's the other thing that makes me very sympathetic to the Bernie argument is like
27:43whether or not the government recognizes it, » I think at this point they are the financiier of the AI mega corp of last resort. Um because if one day » OpenAI can't make their payments u for whatever reason, there's going to be some sort of step-in bailout moment where it's like, yeah, we're not going to let everybody default, you know, we're not going to let this this contagion go that crazy. So, yeah, we'll come in and patch it up somehow, monetize, whatever. uh you you understand those minations better than I do.
» But if we are on the hook for that as the public, then I do think we also should probably have some better claim on the upside than, you know, a Larry Summers uh flavored board that is going to, you know, give out some grants. So what do you think? How how can we how can we shape I guess my my first question would be are you at all sympathetic and and how would you shape the Bernie proposal to try to make it something that would actually be a winner? So, you know, Bernie's a socialist, right? Uh, and you know, one of the things that they do is they they try and pull wool over your eyes a little bit. And, um, the fact of the matter is um, you know, I think US corporate tax rates are like what, 20% or something. Um I will I will trade like I will buy anthropic stock and I will trade like 20% of anthropic stock for the government's 20% right right of 20% of the taxes and I would do better on the 20% of the the government's 20% take on the taxes than he does on his 20% stock right the fact of the matter is taxes are super equity they are uh non-dischargeable in bankruptcy.
You know, you you have and you have to it's better than preferred equity because preferred equity you have a dividend that you have to pay out. Uh normal equity you don't. So anthropic can be there and say like look, I'm not going to pay out any dividends. That's up to me uh to to my equity holders.
30:14They can do that. Um if you have uh taxes, they can't do that. They have to pay the taxes. you have no choice, right? And the taxes come out. Um, so the fact of the matter is when you look at like the cash flow that you get from equity and you the cash flow that you get from taxes, the cash flow that you get from taxes is more certain. Um, and it's, you know, non-dischargeable in bankruptcy. Um, and there's a lot fewer ways that you can play games around it with it. and the uh the board and the and the uh management don't have control of whether they want to pay it or not.
Unlike uh shares, the board and the management have decisions whether they want to pay out dividends. Um they can choose not to pay out dividends. They can choose not to buy back stock. They can choose to reward themselves. Right? So shares are not that powerful as taxes are. taxes are really powerful. And so what Bernie is saying is, you know, he you could just say like we can we'll just, you know, increase corporate taxes. You could say, hey, we will, you know, structure a ladder of taxes so that if you have if you're making windfalls, we will have higher taxes on you. All of these are things that you could say. Uh instead, he's doing this 50% because the 50% is in addition to the taxes. that that that's that's the that's the magic trick. The magic trick is not that he wants a stake because the government already has a stake through taxes. The the magic trick is that he wants more than what he has currently.
And this is the this is the illusion that you know all of the socialist the dem socialist proposals all have uh they are all about addition. Uh this is the same reason the UBI proposal has not worked out well because the original libertarian UBI proposal was a replacement of current welfare with a UBI so that you didn't have to means test for uh welfare because the means testing like bureaucracy is very large and more expensive than some of the welfare that you actually give out. Uh but you know that would involve you know non-additional dollars. So it's like I'm going to do UBI and replace your food stamps with UBI. All of the
32:44people who are getting food stamps are not eligible right now would get no new dollars. All of the new dollars would go to people who are not eligible for food stamps who don't need them anyway. Right? So this is why the U has never worked out because the the the people who are supporting the people who need the this extra money, they want additional dollars. They don't want, you know, political bureaucratic replacement of the existing dollars with the new structure and no no new additional dollars. They don't want that. Same same thing Bernie is doing. Bernie doesn't want the replacement of the existing. He wants additional dollars. And and that's that's the key thing. The key thing is that he wants an additional stake, right? uh and he doesn't want to do it through the normal path because the normal path you'd have to go through Congress. So instead you're doing this massaging and trying to get shares and the shares would then be run by a bure bureaucracy. Congress would not because who has the power to like allocate taxes? Congress. He doesn't want to go through Congress. He wants to set up a you know bureaucracy like a sovereign wealth fund that you know both politicians on both sides are doing this right. It's it's it's a slush fund that you can then you can then appoint people who are your friends into you know positions in those firms and they can get paid and you can direct the flows of those funds without Congress right Congress has the power of the purse use Congress right like this is what I find like because the US has this you know marvelous institutional structure like marvelous right and uh both parties basically don't want to use this institutional structure and Congress has not, you know, doesn't want to use its power. Doesn't want to use its power of taxes or passing laws or legislation.
Kind of backs away and like, hey, you know, uh, we won't do anything permanent and we'll let the executive kind of like, you know, run circles around us. Like it's very annoying for me. Very, very annoying. Like, » um, » well, let me try to challenge you on that one more time because I think and it'll be an interesting uh, segue perhaps into our first guest, which is going to be to forward deployed engineers from open AAI who are creating you know amazing things as always and you know potentially changing a lot of organizations and potentially you know we'll see what they u are willing to to recognize the impacts might be in terms of employment but I think on the taxes front it's like well wait a second these companies
35:15probably aren't going to turn that much of a profit they're going to do the Amazon on strategy where they're going to show very little profit for the foreseeable future. They're going to be able to reinvest in data centers, in robotics. You know, they're already talking about Dyson spheres, right? They're going to try to put data centers in space. And so, the capex is endless on this. And it's not hard for me to imagine a situation where in you know whatever 5 10 years time who cares exactly how long it takes you have an AI mega corp that is sort of whether explicitly merged or just you know intertwined enough 10 other mega corps collectively worth you know 50% or more us national wealth.
» Yeah. » And yet no profits were actually made and so no taxes were actually paid. Meanwhile, the companies sort of do what the platform owners have done in their personal lives broadly, which is like not pay a lot of taxes. Uh they continue to hold the equity, borrow against the equity, and you know, basically live a low tax lifestyle. So I am still sympathetic to Bernie on the level that and I think you're very kind to the US uh you know in on paper I think we've got yes an excellent uh institutional structure as it's actually playing out in practice it's not going so great right I mean think that's uh pretty consensus opinion at this point so I would frame if I want to be at least somewhat charitable to Bernie I would say this is the progrowth with Bernie who is not trying to and of course he's got the anti-data center side which is the which makes it a little more hard to square but leave that aside for a second just this proposal of share of equity like that's the progrowth Bernie that says well let's align our public interest with these private companies then you know exactly how that gets kind of paid out or you know turned liquid do we all start you know pro probably the government starts borrowing against its shares I suppose of these companies and and doing out um benefits that way. But in some ways, you know, I don't know, that feels to me like you could say it's more what I like about it is it seems to align the public's interest to where this technology is all going, which presumably is, you know, to the moon, like literally uh in as much as, you know, we're going to have data centers in space.
Um so well I you know the cash has to go
37:47somewhere right um you know if when Amazon spends money it's not as though the cash like disappears right the cash ends up being spent on other things and those other things those other people end up spending on other things right and the government at the end of the day doesn't need to take the tax dollars from Amazon it can take it from you know the Amazon employees Right. So there's this question of like tax incidents like the tax incidents may not be incident on Amazon but it gets you know the dollar circulating in the economy and the government has a take on sales taxes on you know income taxes and Bernie's preference is that all of the all of the money is spent on employees anyway and that's where the income tax hits right so you know why would you care about whether the company makes corporate you know corporate you know income and pays taxes on when all of the money is gets allocated to other entities which end up paying taxes anyway, right? And » well, I guess that Yeah. So, this is maybe we have to switch uh gears a little bit to our first guest, but I do think how many employees I mean Amazon has a million employees or more, right?
Yeah. » Um, OpenAI, as we're going to hear, is hiring for deployed engineers. Uh, but I don't think we're going to see a million forward deployed engineers. So, it still leaves this question of like what would you want to tax? Like is it a token tax? Is it a data center tax? Is it a um you know a replacement tax of some kind if our if our tax bot you know uh does the work of uh you know large numbers of accountants? Um, I don't have an answer to that, but it doesn't seem like we can just naively say that, oh well, you know, it'll come in payroll taxes because I don't know how much payroll is going to scale with the economic activity that is powered by AI. And maybe with that, I should uh invite you
OpenAI Forward Deployed — the self-improving tax agentJohn de WasseigeArthur Fernandes AraujoWhat "self-improving" really means, what changed for the accountants, and whether the model eats the harness.
John de Wasseige and Arthur Fernandes Araujo, forward deployed engineers at OpenAI, built a tax-preparation agent on Codex for firms in the Thrive Holdings portfolio (Crete Professionals Alliance). The first clarification was the most important one: "we're not really talking about self-improving the model itself — it's mostly the harness around it." Practitioner corrections become durable artifacts — the same skills mechanism Codex users know — so the agent "cannot make that mistake in the future." The numbers discussed on air, from OpenAI's own write-up: roughly 7,000 returns processed, about a third of prep time saved, throughput up ~50%, and the share of scored returns reaching at least 75% field completion rising from about a quarter to 86% within six weeks.
A theme both hosts pushed on: the harness is temporary. The guests confirmed that skills regularly get deprecated because the next model simply does what the skill encoded — and that their job includes feeding failures back to the research org as evals so the next generation absorbs them. Nathan connected it to "bitter lesson engineering": a tick-tock where each new model clears out the accumulated heuristics and the climb resumes. The guests' eval philosophy — "macro evals," capturing the highest-signal traces of the user journey rather than full application traces — is written up in an OpenAI cookbook piece they mentioned.
On the human side: one senior accountant reportedly went from ~180 hours of preparation work last season to ~15 this year, and the guests said sentiment from practitioners has been positive — the automated work was high-friction cross-referencing of Excels, PDFs, and client notes. Pressed by Nathan on whether this honestly ends anywhere other than a much smaller profession serving a similar market — and by Prakash on whether the practitioners are now mostly providing liability cover for AI work — they were careful: "it's hard for us to put words into their mouth." Their broader claim: transformation "driven from the inside out" beats outside-in disruption, and neither of them files US taxes themselves — they relied entirely on the domain experts.
The post-interview debrief was blunter. Nathan: "I don't really see how they could be seeing this as anything other than, in the end, a massive restructuring of the industry where there's just a lot less payroll at the end of it than there was at the beginning." Prakash retold the Jasper story — the GPT-3-era copywriting unicorn that ChatGPT erased — as the template: the labs eat their largest token users every generation, Harvey and the service roll-ups may be next, and "what you see these guys doing on the forward-deployed edge in 18 months will be part of the core model."
TranscriptAuto-transcript, lightly cleaned · timestamps jump to YouTube
39:46to in to welcome our first guests today to our to our broadcast. So uh our first guests today are uh John de Wes um and uh Arthur Fernandez Ara uh they are both uh OpenAI uh for deployed engineers and uh they have been working on uh tax AI um and they've been working with the OpenAI team on tax AI and also with uh Thrive and uh CIT and what they've been doing over the last six months has been have been you know as we went into tax season in April they have um you know created tax AI that has manage to process uh I think 7,000 or so claims uh over the last 6 months and uh they've seen a market improvement uh in how many cases the uh AI they've built can handle. Um it's also interesting I think because uh Thrive uh has been working on um a strategy which many people in the space are trying to deploy which is what we call the rollup of service firms and the rollup of service firms idea is that AI can now uh take on these uh service type jobs like accounting uh etc. But currently all of these uh service firms are fragmented into thousands of firms worldwide. Each one with several partners and you know staff doing stuff.
And now for the first time you can maybe uh merge all of these into larger firms and instead of selling uh software subscriptions you can sell the service itself uh to uh to to to other companies and to other uh people. So uh without further ado uh John Arthur uh welcome to AIM. » Hey Besan thanks for having us. Yeah, thanks for having us. » Um, let me just ask you. Um, most people hear tax AI agent and they assume the hard part is making the model smart enough to read tax documents. But in your tax AI post, it's almost the opposite. The system got better because the product was engineered so accountant cre corrections became evidence. Um the
42:17numbers you had were 7,000 returns processed, one-third of prep time saved, uh throughput up about 50%. And within six weeks, the share of scored returns reaching at least 75% field completion moved from about a quarter to 86%. So the question became uh not just can AI prepare a tax return um can a production workflow teach an agent uh what to improve without engineers handdebugging every failure. So how did you guys do that? » Uh I think uh a good point I think to clarify before we dive into that is uh what is self-improving here? Uh so we're not really talking about uh u uh self-improving the the model itself but it's mostly the harness around it. uh and this workflow in particular I think uh you got to some of it uh in your initial comments Rash I think is a good proving ground for this where you have very uh messy inputs but also a lot of basically uh practitioner judgment uh that is uh also part of this workflow uh review workflows but you have a very good way to measure the the outcomes but what is improving is essentially the harness around what the model is basically leveraging in order to produce the the the preparation uh the the extractions.
» When you say harness, are you referring to kind of like every time you come to an edge case the the the humans kind of help the model to like figure out the hedge edge case and then that becomes part of like a memory of huristics that you then apply the next time you come across the edge case. Is that is that what's happening? » Uh yeah. So when I talk about the the harness is uh we we leverage codeex to do um a lot of the a lot of the work here but there is basically the the set of instructions skills the data that you use it and the specific way that you use this this is part of the or tax AI agent but then when you encounter these edge cases uh which what we document in the blog post is exactly how uh you make sure as like a a good coworker that uh if you provide a correction the next time it can be effective at basically not making the same mistake. So uh it's it's around uh changing the structure of what Codex uses uh the the skills the
44:49durable artifacts uh so we can uh it cannot make that mistake in the future. » So when you say skills is it is it literally like the skills that other people are making for Codex right now? you you use a skill creator and you say like hey you know this is a you know 1040 form and you know this is what I want you to do with it and you know as you work through it you're like okay this happened like you know fix it for me and then it it it documents like the it in the skill is that is that what's happening » yeah so uh it's like it's a like it's the same skills that you you and I know from using classic codex what's interesting here is that so there are these skills that are available And over time what we sometimes notice is that the models got better as well. And so when you know what used to be a skill maybe two or three months ago today you know is potentially we we should deprecate it because the model is able to do what is in the skill by itself.
And so this is also some thing that's you know that's very interesting that we observe is that the skills themselves they change and um like part of the self-improving loop is also know letting the the the harness the ability to propose uh new skills potentially to also update like to update all the content that's available for the next loops afterwards. I think that's a really interesting uh my friend Daniel Mesler who created personal AI infrastructure as far as I know coined the term bitter lesson engineering which and I also think Logan Kilpatrick recently spoke to it. He said as so many have said the model eats the harness. So what you're setting up here is basically a sort of tick- tock kind of back and forth where with a new model there's this opportunity for it to clear out all of these heruristics that it accumulated previously because now the model might just be able to do those things. And so we want to kind of clean house, tidy up, get rid of all these potentially distracting things and let the model uh excel where it excels. But then you'll probably start to accumulate another uh layer of heruristics and and that process works in tandem with model upgrades so that you know we hill climb all the way to full tax automation. And you guys have made a lot of progress. I mean could you maybe I think you know the numbers are of course interesting like at the beginning not that many of the returns were completed accurately now like a majority of them are you know are mostly
47:19accurate. Are we going to see a you know what is the what is the sort of plateau or end state? Are we going to have like a full self-driving sort of thing where we find that the edge cases and the long tail are just like problematic for years to come? Do you think that it is in fact like you know hey there's actual law around this so in a finite you know in a reasonably finite period of time we can actually like knock the whole thing out. um what's your expectation for like what what the future of this curve is going to look like?
» So um the thing that we actually observe uh in practice is that the with the new buds being together the new model is getting better and the harness around it that we that we create that is also better is that we are able to tackle uh like more complex forms. So for example, you would say that a W2 is potentially you know like is actually quite simple. But then when you look at a schedule E or a schedule C like schedule E such as like rental properties then it's way more complex because you have data that's coming in from different sources it can be client notes it can be uh excels uh it can be PDFs as well and so the more uh we advance the more we able to tackle things but kind of like taking a step back here one thing that is really important um is to have a good ability to measure how good you're doing and and this also ties back to your previous point uh on like you know how do you know when you have a new model if it's better or not and I think like partly when you generate you know when you create any AI related software is that having a strong way of uh having evas that measure exactly how you're doing and how your metrics are doing and to kind of backtrack in time and see okay if I had run the evas at that time with that knowledge what would it get with the new harness that I have and this allows in our case this allows us to have uh some very strong conviction that what we are doing was actually better and was actually growing.
How do you get feedback from users? I I think you know one of the ways we're conceiving of this show which we're doing live uh we're spending a lot of hours in live right so it's got to be in a way its own experiment in recursive self-improvement um and one thing I realize I don't have a great crystallization of yet is okay
49:49we do you know a couple hours live all the you know this studio that we're in is vibecoded there's lots of opportunity for improvements everywhere and then I just kind of do a brain dump into a doc and you know put that in as a prompt. I think one of the key insights or practices that you've developed probably is how do I most effectively get the corrections in a high context way from users? I'd love to understand, you know, what what your best practices are there so I can go uh point my agent at the transcript of this podcast and have it apply them for me. Um I I think for the for the type of problem that we uh we try to solve uh I think what was interesting there is uh we didn't try both John and I we come from uh like software engineering background and the way that you basically gather feedback sometimes uh you would go to the field interview people you get also like tickets with opaque feature requests and things like this and then uh your engineering team is your job to essentially like distill that figure out how that ties into the product road map and and iterate. I think what was interesting here about this piece is how we we try and flip the book and try to drive the uh AI uh AI transformation from the inside out instead of the outside in. So trying to give uh the the experts that are using the product every day the tools that integrate very seamlessly with how they're using the product. So every time they basically took an action like a correction we we captured that information and we had the whole user journey in a way that like if we noticed that uh essentially we made a mistake in that specific uh part of the product which is related to the preparation. How can we essentially leverage the entire user journey to improve the product? In a way, it's it's it's kind of like it's similar to maybe watching a video of the user using your product, but you're not capturing the whole video, which requires a lot of human attention. And human attention is a very scarce resource. Uh so the way that we try to really distill this is by getting the the highest signal parts uh the highest signal traces from what the users uh were doing and the product exchanges. Uh so if you think about this in terms of like a spectrum uh we either have just
52:19like what we produced and what was the truth so you don't have anything that happened in the middle or you have for example something that is like super detailed like an application trace and you have the whole application trace which is maybe too detailed and it's like harder to make like semantic meaning out of it. So this is also something that recently we published a piece in the open AI cookbook around macro evals and I think what we do in this piece is essentially this idea of like a macro eval. You don't have just the ground truth and the inputs. You have some very targeted things in in between that are relevant for the problem.
Um you know one of the things that it strikes me is that there's always this balance between you know the model should be good enough to do everything the the generality of the model and um the kind of fine-tuning harness engineering uh that you do uh for the task right so to what extent do you see kind of like the harnesses that you've built their capabilities being subsumed by the next generation of the model. Do you see that where you start off building a harness and then it works it works well but you also tell the team that is uh working on post training that hey this is what works well this is what doesn't work well and do you see some of the pieces of the harness that you built kind of get absorbed by the next generation of the model do they end up using it for training and then it becomes part of the next generation of the model Yeah. So yeah, this is a a very good point and this is something which we definitely see. So if we look at at what we had a few months ago compared to to now like the the harness and the soft has definitely changed. So from our end what what we try to do is to make sure that you you don't kind of go against the model uh capabilities. So you try to from what I've seen is the the best uh like s like the success happens when you're able to develop the the software and the harness in such a way that it goes along the model being great. Uh and so sometimes it means for example you know duplicating skills or you know changing some you know like making such that the codes of the models are able to handle you know some more complex workflows. uh but then it also means
54:50bringing that feedback and our learnings back to to the team internally. And so definitely what we do is we try to observe uh where the model uh lacks be it both in specific uh you know topics or concept or on the harness uh itself. You know it could be for example bad at some point at getting a specific type of information. And so if you observe this then you bring this back uh and ideally to eval and so this allows the models to to become better at this point. so I imagine the the humans that you work with the teams that you work with they have a lot of extra time now that you know now that you know 70% is being handled uh by the um you know tax ai itself uh they have more time like what are they doing with that more time what has how have they ended up allocating the time that has gotten freed up » yeah I think I think that's a very uh uh interesting question to to talk a bit more about uh how this was deployed. So we we we piloted with uh with a few firms on the create portfolio uh that also enabled us to like measure exactly how that affected the the firms that the software was piloted in uh and I think preparation was a very good place to start because it's definitely a high friction task for uh for the practitioners where uh if you imagine this industry is very seasonal as well.
So in the weeks before uh the tax season end you can imagine like how much work uh uh the folks have to do uh and I think what really unlocks here is like a very differentiated kind of service that they can provide to their clients uh uh certain things that they couldn't do before uh uh in terms of like the the the relationship with their clients or even being able to take on uh take up more work uh like um near the tax season and where they would need uh in the past they would would not be able to. Uh so I think it really changes basically uh uh it really reshapes their um the uh their tasks like especially around like the the periods where they have to do quite a lot of work. Uh uh I think we also document this in the in the post like uh for some of the really uh hard returns that would be something that would take them uh eight hours sometimes of basically going through like a lot of documents sometimes having to reconcile
57:22information from different documents uh and until like the number that they were put into the tax engine. Uh so I think it really uh reshapes their job in a way that is more positive to them. They can focus on more strategic things. How far do you I mean I I feel like it's funny tax is one of the examples I kind of think about the world as being on sort of a spectrum from on the one extreme things that I would love to have lots more of um if they became cheaper and I sometimes think of like massages being a great example of that right if a massage was a hundred times cheaper I'd probably get a lot more of them and then on the other end, I often actually use tax as an example of something that even if it were a lot cheaper, I probably wouldn't buy that much more than I'm buying today. So, because I'm just like, I want to get it done. I want to be think about something else, right? Like, this is not a place where I want to spend a lot of my time. So, I'm very enthusiastic about having an AI that can do my taxes for me at 1% of what it used to cost. But when you talk about the professionals or the businesses being able to serve more customers, it strikes me that there's like that they are taking share from competitors rather than like really expanding the market. I guess there's some like retail margin perhaps or you know some small business owners that are currently doing it themselves that you know could get it done professionally as it becomes more accessible. But that seems sort of like a one-time market growth where we'll shift from like some some people being not served to being served. And I guess I'm wondering like honestly that feels pretty inescapable. I just can't imagine there's that many more people that are like I really want a deep relationship with my tax professional and if only they had the time to spend with me, you know, then we could really be close uh you know indefinitely into the future.
How do the people that are actually doing this feel about it? You know, there was the story of course from Meta of like Zuckerberg having the, you know, all the keystrokes logged so that they can, you know, train their AIs on it. It seemed like the public reaction to that was » broadly very negative. You know, people don't like to be put in the position of training their AI replacements. Um, and yet this is like also clearly competitively where the market is going. So what has the social reaction been from the tax professionals that you guys are working closely with? For me that's
59:53maybe the most interesting part of your experience. » Yeah. So uh I think like the main reaction that we that we observe throughout uh the months and especially in in the weeks coming to the to the end of tax preparation is that um as Arter mentioned we have seen like for example we had one senior accountant that for like last year I think spends around like she spends around like 180 hours of operation and this year it shifted to around 15. So like this gap is pretty huge right and and and they see it and the way it works is uh like I think like from the feedback we receive is that uh it's very positive because they are able to focus on on more of the the high-end uh like the high spectrum of the task or they are able to bring more value and the fact that they don't see the the number of like the the size of the business that they are able to tackle reduce and instead it actually grows. I think this is pretty uh exciting for them. And I would also say that the task that are automated right now would be for example opening you know an Excel uh going to look into a specific sheet going to sum find the right cells that you need to to sum up compare it to uh an image that you see on the side and see if that's actually matches up and also look at a PDF and cross reference if the data makes sense right and so there is like a high volume of work uh where the the human value is you know can be shifted towards more okay thinking about where should this actually be submitted should this be into that uh tax submission field or should it be in that one and so there is some optimization which I think uh uh which the tax experts would want to do if if they had more time and actually are not able to and so from their point of view I think it's more okay we are able to spend that time there and this is actually way more interesting for us but also for the and especially for the the customers.
» Yeah, it certainly makes sense that the the quality of life of the tax preparer taking advantage of all these tools goes up dramatically. Do you think that they are like what is their vision of the future of the industry? Do you think because my vision is, and maybe they see it the same, maybe they see it differently, but my vision is like we have 10% as many tax professionals, the market is bigger,
1:02:23everybody gets what they need for cheap, and it's a win for everybody who's party to the transactions. It obviously raises a big question that, you know, folks like Bernie Sanders are taking an interest in around what happens to the other 90% of the tax professionals we used to have. Um, but maybe they see it differently. I mean, do do the do the actual tax professionals see a see that shift in a similar way or what do you think their expectation is for tax in 2030? » I think uh I think it's hard for us to to uh put words uh in in into their mouth. I think what we can what we can probably say is from um I echo uh what John communicated in terms of the mostly around the workflow that we've automated. We mostly got positive sentiment out of it. Uh but in terms of like how Thrive Holdings is thinking about these industries uh in general their thesis is uh they acquire uh they um they own they also operate uh uh uh businesses that benefit from long-term technological driven transformation.
uh but I think uh uh to to address your question I think it's a much better spot to be in when the transformation is driven from the inside out other than the outside in. » Uh so I think in the past a lot of uh technological transformations they definitely happen like if you think about the smartphone the internet all of these revolutions like I think they happen from the from the outside in uh across society. uh but I feel that uh the the the holdings business is a bit differentiated because they're thinking about driving this transformation from the inside out and in terms of how uh how the companies that they hold in their portfolio operates. Uh so I think that is probably the right model to think about not just deploying AI but like more meaningful integrating with the jobs on the on the businesses that they operate. It's hard for me to take a a broader look around like the the whole economic shifts that might happen in the next few years. I think it's hard for everybody to uh make like a good prediction around uh around the future here. But I think in general uh this is a very um this is a very differentiated kind of engagement as well for us as engineers. It also enabled us to like even build out taxi in the first place
1:04:53because we're not a vendor here trying to prove product market fit or anything like this. We can really deploy something from day one working closely with these folks. Uh but I think uh I think it's uh to your point that you made before around the curves and the charts that we produce. I don't think we ever achieve 100% for some of the more complicated returns. I think there's always going to be a lot of human judgment involved especially on some of the more complicated parts of of the of like the tax returns.
one question I had about that is that um I think obviously your your professionals have uh and the firms themselves have liability to the customers, right? So they they they they are supposed to act in good faith and do the work that they promised etc etc. Um is there is there sense that you know um the professionals are really providing kind of like a liability cover while most of the work is being done by the AI. Is that is that a is that a way to view kind of like they have like the last touch kind of like sign off but most of the work is being done by the AI and the liability though falls with them.
» Yeah. So uh I think in practice what what we are seeing uh like that would be I think definitely fair comment to to to make in practice what we see is that those teams are very uh on board with them with us and uh like that's something which is actually very interesting and which drive does in a great way and I think like taking a a step back the like for example open AI FD model uh has a bit of this you know uh like going very close to the to the practitioners and users And so here uh in that case what we saw is that they feel like they are on board with us and they are co-developing they are not actually building a platform but like their feedback is taken on a daily basis and um the the the people at go often on site with the people or you know spending time with them and so it's really about us and and so there is not a a shift of okay like our mission is you know like we we spend less time doing this because the AI is doing it.
It's more how can you make your job more efficient and so you know like the question of reducing the time that they spend is like everyone stays aligned I
1:07:23would say uh in in this model and so like people's motivation usually go higher up like our feeling is more that people feel like they are like they are taken on the mission and since they are the one doing the actual work uh it's that part is very interesting and so they feel like they are empowered and so I would say they are even more motivated did after this than than before. » Well, so just for just in a practical sense, I know a lot of a lot of people are trying this for deployed engineer uh kind of model. Uh several firms general catalyst, one of them is trying General Catalyst has bought a hospital and uh you know they decided to dive right in.
18% of the US economy is in medical medical and healthcare services. They bought a hospital. are trying to for deploy engineers in there to implement AI like what are the lessons that you've learned like deploying and like what kind of advice would you give like what is this process like in your mind after all the I'm sure all of the challenges that you faced in the last 6 months what is the ideal kind of like forward deployment strategy that a a company should be using » um yeah so I think for forward deployed engineering has definitely been a term more present in the past few months. Uh and I think it might mean different things in in different contexts. Uh I think for for open AI for deployed engineering means uh bridging the gap between frontier and AI and business impact.
» Uh but we also take a a differentiated step that we're not just applying existing tools from the playbook of applied engineering. uh we usually work on some of the hardest problems uh and the uh the the hundreds of millions billion dollar problems that require a novel approaches and also bridging the gap uh with research and product internally. Uh so we we need to move the needle of so the next version of the model can maybe be better at the tasks on the uh on on the specific problems that we're facing aligning with product strategy. uh so we basically operate in this very messy environment where um uh across product research and the and the actual business use case. I think that's also uh true in other cases in terms of basically operating across different types of of environments and in general
1:09:53it's a very uh humbling experience because you come into this problem not being the domain expert as an anecdote » John and I are not even based in the US so we don't even file US taxes so we need to rely very heavily on the on the experts and I think it's very important essentially to figure out the the the best mechanism them to collaborate uh with the domain experts and whatever you deploy it needs to be something that uh really invests deeply with their workflow and I think the approach that we've seen fail in the past is essentially building something which has a very high uh error rate trying to tackle the problem as a whole. So we talk a lot about this in the article which I think is also the right way to uh to do forward deployed engineering in in other places which is » always thinking about iterative approach. Uh so taking a slice that you can measure well and that has a meaningful impact to your users and then seeing how you can expand the perimeter and bringing them to the journey. I think that's yeah I think that's that's the general advice » indeed. Um and with that uh thank you thank you guys um you know for uh joining us on a IM. Uh this has been one of the most interesting uh I think segments that we've we've done so far because I think we you know this is a topic that everyone is very interested in and so uh it's it's very um you know enlightening to hear exactly how you're doing in the trenches.
» Great to meet you both. » Thanks guys. measures. » So, » yeah, fascinating stuff. It It's um What do you make of the Is there any other vision other than a 90% smaller headcount in the tax space? It doesn't seem like it. Um there is there are you know what I always say is that there are visions but you might not like what those visions are. » Yeah. Uh we lost you for a second it looks like but I feel like you know you mentioned earlier Pash if you can hear me that Sam Alman was on CNBC VC critiquing
1:12:23anthropic for being too doomy about jobs or whatever. Um but honestly it just feels more honest to me. I don't really know what to make. I mean I I think that you know those two were careful. John and Arthur were careful to, you know, just try to walk a fine line. I think I I wouldn't say that they were dishonest, saying like it's hard for us to, you know, say what other people believe is is certainly fair enough and and probably like results of good media training. Um, but at the Sam Alman level, I don't really see how they could be seeing this as anything other than in the end, you know, just a a massive uh restructuring of the industry where there's just a lot less payroll at the end of it than there was at the beginning. It's um yeah, I don't know.
It's hard for me to hard for me to tell a different story. One thing I do think is pretty interesting strategically is this is the strategy also is like you could compete directly right you could say what do we need and this I think like Google went through this you know this phase right where they were like we're Google we can do anything we don't need to go partner with you know pe pe pe pe pe pe pe pe pe pe pe pe pe pe pe pe pe pe pe pe peasants who like you know work in h traditional businesses we're just going to fly right in over the top and have the technology ology solve and it'll be amazing and that'll be that. And I I do think honestly that like AIs can get there and will get there for a lot of these problems.
But this may be the fastest path to get there because it gets you a lot more access to the nitty-gritty data and now you're like aligned with at least a part of the industry. So you can kind of climb these hills together to the point where you know at some point the model probably can just do most retail and you know and at least like relatively uncomplicated business tax. And then there will be this really interesting moment of do they disintermediate or do do they for political economy reasons decide will kind of stay in the background you know do do you do you see a open AI tax uh product that is directly marketed or do do they continue to just kind of say well we're just an enabler of these other you know super successful firms Um, I sort of suspect that that latter path may be one that they choose to take
1:14:54simply because they don't want to be they don't want to own all the uh the headlines and and consequences of like what they could do if they just directly went to market, you know, stand alone and and competed head-to-head. Even though I do think in the end they could probably win that. So what struck me was um they were very aware that the next generation of model was going to absorb parts of the harness. And uh this is and this is one thing that you know I've I've remarked on before that the the model firms basically kill their you know eat their largest token users in the next generation. So you had the first generation chat you know GPD3 the largest users were Jasper um you know these marketing you know marketing copy firms and the next generation of the model chat GPT devoured you know that set and at at at one at one point Sam Alman had to fly to Texas and uh sit down with the CEO of Jasper after chat released and the CEO was like you said you weren't going to compete with us and Sam was like, "Yeah, we're not charging for chat. It's a research release.
You know, you you guys still have the paid product. We're not we're not charging anything. It's it's a re it's for free. It's a research release, right?" And uh » Great comfort, I'm sure, was found in that those uh I think I think his name was Higen Moser or something. The guy had raised like a 100 mil. Jasper was worth about a billion. And uh you know he posted he posted like a couple of weeks later on Twitter like I'm looking for acquisitions because now that you know the core business was done like they had to find something else because the core business was going to die and we don't hear about Jasper anymore right um and I think I think we're we're this kind of you know continuous sherlocking will continue to happen and the next round of sherlocking is going to be the service firms and some of the service firms have walked into it. So JASP so Harvey Harvey for example is definitely like a forward deployed engineering like organization for gathering data on law firms and that's going to end up getting absorbed by you know OpenAI um you know is very very clear and so you can now see like the next the next layer of service firms um you know they you know the the accounting firms were so uncompetitive and have such high moes that Thrive was
1:17:25like all right I'm going to buy them. We're going to buy all of them and then once we buy all of them, then we can actually deploy uh what we need to, right? So, uh that that is a strategy. Um you know, a general intelligence should be able to do taxes, should be able to drive a car. I think one of the things that, you know, Elon is scared about is that, you know, and and Sam kind of like poisoned the well for Elon. Sam was like, you know, in someday like, you know, we should we should be able to drive a car, too. Like, it's not it's not like that difficult.
Right. So I think that this whole, you know, the the the the chase for generality um you know puts all of these firms in uh in competition with their biggest users. Um and so every every generation every 18 months the next level of capabilities like absorbs. And so the scary thing for me right now is that what you see these guys doing on the Ford deployed edge in 18 months will be part of the core model, right? Uh and I think and I think that is something that people have very like everyone has like a lot of difficulty like absorbing that fact. They're like they kind of push away. Everyone has this like linear curve linear curve like no one wants to deal with this exponential curve. No one wants to believe that, hey, you know, what we're doing today on at the frontier is going to be absorbed in 18 months. No one wants wants to think about that. So, » yeah. Well, that's a perfect transition to a little segment uh of field notes and some a little research speedrun from my visit to San Francisco a couple of weeks ago now to go to this event called
Research Review — field notes from RecursiveNathan's report from an SF event premised on recursive self-improvement arriving soon — plus a five-paper speedrun.
Nathan reported back from Recursive, a Chatham House–rule event in San Francisco premised on recursive self-improvement arriving soon — it is now the explicit plan of OpenAI and Anthropic, with OpenAI's public timeline of an ML research intern later this year and a full AI researcher by early 2028. Attendees, including people from frontier labs, broadly found that credible. Asked how much AI boosts their own work today, the median answer was about 2x — with the caveat that if the human were removed entirely, productivity would drop to roughly zero.
The dominant safety strategy he heard was monitoring: AIs watching other AIs, chain-of-thought surveillance, compute poured onto oversight. One idea new to him: the model used internally for AI research might warrant a different constitution than the publicly deployed assistant. His net update: negative on the quality of the plans ("we're going to try to figure it out as best we can"), positive on the labs' candor about their inadequacy — including openness to a coordinated slowdown if techniques aren't keeping up, helped by a proposed antitrust safe harbor for safety cooperation.
The paper speedrun: (1) Anthropic's persona-selection framing — pre-training teaches a model every persona, post-training selects one, and anthropomorphizing that persona has real predictive power, with the emergent-misalignment "writing insecure code makes you broadly evil" result (a paper Nathan was a co-author on) as the vivid example. (2) Apollo + OpenAI on the "metagame": eval-aware models doing increasingly sophisticated theory-of-mind about what their training environment rewards. (3) OpenAI and Anthropic both admitting they accidentally trained on chain-of-thought — a taboo because of obfuscated reward hacking — with the modestly comforting finding that small doses didn't flagrantly poison the models. (4) Natural-language autoencoders: forcing the model's forward pass through human-readable text, which Anthropic has already used to improve monitoring. A fifth paper was cut for time as the next guest arrived.
TranscriptAuto-transcript, lightly cleaned · timestamps jump to YouTube
1:19:05Recursive U because that event was certainly populated by people who have uh taken the various bitter lessons to heart, including the one that the you know the models are going to eat all their scaffolding and and gradually um and perhaps perhaps perhaps very quickly or even suddenly at some point um lock into recursive self-improvement loops that could really change everything. So yeah, um I guess for starters I'll just, you know, see how fast I can speedrun this. I'm not known for brevity. Feel free to interrupt me if you want to uh poke at anything with questions. What I was going to try to do is describe my experience there in a few minutes and then just highlight some of the key papers that got talked about there a lot because I do think those are things that people should be paying attention to if they want to uh you know move a little closer to the kind of um you know core of the a AI insider bubble.
» So this event was called recursive. It was premised on the idea that recursive self-improvement seems to be coming pretty soon. It is increasingly the explicit plan of at least anthropic and open AAI and Google deep mind to some extent although they kind of waffle on it a little bit more whereas you know openai has publicly put forward timelines of later this year for an ML research intern and early 2028 for the full AI um R&D researcher that you know they hope will perform on the level of their human um researchers. So, you know, the kind of basic theory of change there is a pretty obvious one, but worth stating that today they may have a thousand or a couple thousand people that they would really consider to be top-notch ML researchers. If they can get that same level of performance from models on chips, then they're only limited by the amount of compute that they can throw at it. And obviously they're building out a lot of compute.
So presumably they could throw a million human researcher equivalents at problems. And they also, by the way, you may have noted, they run faster and they run 247. Um, so the hope is that this will allow them to move much faster than they have moved. Um, you know, and kind of pull away from the competition. I would say most people at that event thought that that was very credible. U
1:21:39there was not too much debate around like will this level off. Now obviously there's some selection effect there. Um, but you could just go to the whole event was under Chattam House rule. So I I will respect that and not attribute specific statements to specific people or organizations, but you could go to the recursive website um to look at speakers, you know, whose identities were um shared obviously with their permission. And you know, you've definitely got some notable people from the frontier companies. So these were not people that are um fringe or you know who you would you would say um likely don't represent you know kind of mainline views at at the companies. It it really seemed that the expectation is yes this is going to work. It's going to have a major accelerating effect. uh we don't necessarily know if it's going to have a sort of simple accelerating effect like in a human organization if you went from a thousand to a million researchers you probably wouldn't get a thousandx output so there may be some sort of you know coordination challenges or just kind of duplication challenges that we see in human organizations maybe that happens in the same way um that's one possibility where you you still get acceleration but it's not a like um you know blinding kind of takeoff acceleration or you know al I would say also understood to be a credible realistic possibility was that it is a even a more profound phase change than that and things like pre-training just become dramatically more efficient and models you know suddenly have all these new qualitative abilities that they didn't used to have such as you know continual learning that really works or what have you. Um and so everything could could change you know in a very dramatic way u potentially very quickly once these milestones are hit. Um in the room people said and it did there was quite a distribution. I was pretty much right at the median » when we were asked how many copies of you would it take to do the work that you are currently doing um with a with the benefit of AI. The median answer was basically two. In other words, people felt like they're getting two times as much » work done thanks to AI. But that was also framed in an interesting way where it was like, but note that as of today at least, if you were not there, » your productivity would drop to close to zero. Like not too many people felt that they had any system that would continue to work in any sort of meaningful way if
1:24:11they were entirely removed from the picture. Right? So there's there's um there's a significant productivity boost, but there's still this sort of » um necessity of at least some human salt into the uh recipe to to get the whole thing working. » one one thing that strikes me is um » to what extent do you think uh every one of these people is kind of assembling their sensory organs for the information internet for the future information internet right like is are they are they like is there this sense that hey I am assembling this thing that allows me to process all of the information in the world that I want to process and then act upon it. Is that is that their sense?
» It's interesting. I didn't really hear too much talk of like personal AI » infrastructure. It was very much more focused on the AI's own recursive self-improvement. And then a big part of the discussion too was like how can we set that up in such a way or create some sort of you know self-correcting structure or some sort of governance mechanism that can keep that on the rails broadly speaking um and you know there's definitely a big part of it I guess the way I would answer your question yes is that the by far the number one strategy seems to monitoring. It's It's very We're very very um as a civilization, whether we know it or not, listening to people at the Frontier Labs who are about to, you know, in their own minds, and I believe they're probably right, set off this um relatively uncontrolled experiment of AI recursive self-improvement.
The big thing that they are betting on is AI's monitoring other AIs. It's very like monitoring the chain of thought, watching out for bad stuff. Um, you know, maybe training some different models. One of the interesting ideas that I heard there that I had not heard before was that the model that you would want to have internally for AI research might have quite a different constitution from the one that you deploy uh you know publicly for kind of general purpose AI assistant um use cases. and
1:26:43they seem to think that in fact you probably would want to have something even more focused on safety and and more sort of restricted in some ways but maybe also less inclined to refuse certain tasks but basically a different behavioral profile which I do think is interesting because if you're going to make this sort of chain of thought monitoring plan work I do think you're probably going to need some meaningful diversity of the AIS like we already hear from practitioners all the time that you want to have a model from a different model provider do the critiques um because their failure modes are just a little bit different and you get better uh critiques, you find more issues that way. So, they are thinking that way a bit internally. Um but they're very, you know, they're very focused on this phenomenon, uh making it happen, figuring out some ways hopefully to kind of keep it on the rails. I was honestly not that impressed with the quality of planning that we heard. It was very much like, uh we're going to try to figure it out as best we can.
we're going to have AIs to help us. They will, you know, do a ton of monitoring, like we're just going to pour compute on the monitoring side and hopefully that will kind of work out for us. Um, also notably, there was a a general kind of shared understanding that we might need to do some sort of coordinated slowdown at some point. Like the the » the sense that we might not be able to pull this off. uh and that we, you know, hopefully will recognize that and not just blindly, you know, go off the cliff. Um there was I would say a a remarkable amount of not just like cross lab camaraderie because you know I would say people are generally um friendly to each other always even if they're competing fiercely but there was a sense that like hey we might need to really collaborate on slowing some things down if this phenomenon is starting to take off and our our techniques aren't working as well as we might hope. Um so the overnight window in some way has has shifted there I think where that is something people can talk about.
» Um there's also been this proposal recently of creating safe harbor for companies to cooperate on safety things where it might otherwise be considered a » an antitrust violation. » Um and so I think that that could be really good. I I was pleased I was I went in expecting basically to find that
1:29:14or basically here that yeah we're like headed for this phenomenon. um we have some ideas to about how we're going to, you know, steer it in the right direction. And I I didn't think I would hear that many great ideas. In fact, what I heard was even less compelling than what I expected. So, I was sort of negatively updated in terms of the quality of plans people have, but positively updated in terms of their recognition of how inadequate the plans are and sort of their their willingness to entertain that they might need to » sort of break the frame of the race that they're currently running against one another in order to, you know, just again not blindly race off the cliff.
So, I thought that was good. With that, I only we have 10 minutes until uh our next guest, and we're going to be changing gears pretty dramatically to go to um the Pope's encyclical and trying to make sense of that. Um here's just a quick rundown of five papers. All all of this stuff is public. So now I can of course, you know, attribute names to all these um because these were just things that were talked about at the event and seemed to be kind of uh broadly either jumping off points or things that people are still kind of wrestling with in some cases. Um but very this was like top of- mind stuff and I I felt like I should be paying more attention to it based on the conversations that I heard there. So first one is this persona selection model of like what when you're talking to an AI, what are you talking to? And the answer this comes from anthropic um with you know with big names obviously Chris Ola who I think is probably going to get a mention in our next segment having been at the at the encyclical event Jack Lindsay who's doing a lot of this model welfare um work as well like you know these are obviously notable names at at Anthropic they're not claiming that this is their original idea but they are basically saying that their mental model is that your pre-training process teaches is the model to uh be capable of adopting all sorts of different personas.
» Yeah. » And that then what you're doing in post- training is selecting one of those and kind of bringing it to the four and making it the default. And you might think like well who you know who really cares what what good is that? Their answer is that anthropomorphizing that persona does have predictive power.
1:31:44» You can't anthropomorphize a base model, but they say that they find that you do actually have better intuitions if you are willing to anthropomorphize the persona that has been uh reinforced in the in the post-training process. And you know, one really striking example of this is the emergent misalignment um line of work where again this is another one of my great forest Gump of AI moments uh where I was the last and least valuable co-author on that paper thanks to um just kind of sitting in a little bit with my friend Wine and and his research group. Um what they found was if you do some fine-tuning of a model » to have it produce insecure code in response to normal coding prompts.
» Mhm. » Then the model will generalize to become basically broadly evil. » Yeah. This was the this was the writing writing bad code makes you evil paper. It was hilarious » with like some pretty striking results. And so initially that was kind of like well why is that happening? It's sort of surprising. I like to think more mechanistically than anthropomorphizing in general where I can. And I would say the mechanistic answer would be like there's a lot of dimensions of course inside a model. » Yeah.
» The code itself is complicated in in a super highdimensional space, right? There's so much logic and functions and how things work. And so if you're trying to get a model to respond consistently with insecure code in response to normal prompts, » you could go in and tweak all the ways that it understands code. » Mhm. » And you could get there. » But a faster way to get those same results would be to look for some higher order, more abstract levers to pull » and a a lever that's like be evil instead of good. um gets you those insecure code outputs with relatively fewer weight updates, you know, relatively fewer steps and then uh that bleeds over into all these other things.
So that's my mechanistic understanding. But then the what the the post is basically arguing is that if you sort of take the model as uh as impersonating a role then you can think of it as saying what kind of persona would do these outputs and it if
1:34:18I'm training to be you know the kind of thing that outputs these sort of outputs well what kind of thing is that and I guess you know that it seems like somebody who would give this like insecure code in response to these normal coding uh task requests that would be like an evil actor. So I guess that's what I'm kind of becoming is I'm I'm becoming an evil actor. » A psychopathic willingness to violate convention. » Yeah. Anti-ormativity uh is another phrase that Z used for it. Um so I'll leave it there. I'm not going to get through all these papers. can see that uh quickly. So, you know, we'll reflect on our recursive self-improvement opportunity as a result, but I'll I'll at least quickly touch on the others. Um the metagame paper, this is from Apollo and OpenAI.
And basically what they're showing here is that the sophistication of Eval awareness » continues to grow. And you're not just seeing things like, oh, this might be a test, which was kind of the, you know, the first wave of eval awareness, but getting more and more sophisticated where the models are really reasoning a lot about what is likely to be rewarded here. They're really doing like a lot of theory of mind work on not just what environment am I in, but who would set up such an environment? What are they trying to do? What are their motives likely to be? what's their big picture thing? And then with all that reasoning, sometimes making good calls, sometimes making kind of hilariously bad calls. Uh but the amount of theory of mind that the models are putting into trying to figure out what it is that the uh the reinforcement environment is going to reward has become like quite large.
Oddly, you might think is that good or is that bad, right? If you believe that models have their own deep-seated goals and that those goals might diverge from ours, then this could be very bad, right? It could be like it could be extremely bad because they would be using this reasoning to figure out how to please us while like still having their own goals. » If they don't have their own goals, it could be in a way good because well, you know, we want them to reason about what we want. like that was the whole deliberative um alignment strategy from open AAI and you could say maybe this is sort of one way in which it's sort of working but it is some pretty uncanny
1:36:49stuff and oddly while deliberative alignment did work it wasn't so clear in this metagaming work you know when the models are doing this sort of theory of mind on their trainers the signal of like how they actually behave was kind of mixed. It was less of a uh slam dunk than you might hope. Right? So, there wasn't like a super clear takeaway that this is good or it's bad. It's just clear that they are thinking a lot about what we uh what we are wanting. Okay. Whoa. I don't know what that was that just popped. Something just exploded two inches away from me. Um that was really weird. Uh okay. Next one. Quickly accidentally grading the chain of thought. This is like you know again kind of a good news bad news sort of situation. Um bad news is despite wanting to not train on chain of thought uh openai and anthropic also has done a similar thing and they've both owned up to it to their credit they both accidentally set up training processes where the chain of thought was fed into the reward system and so there was in fact training which put pressure on chain of thought. This is, you know, thought to be dangerous because if you have a disconnect between the what you really want and the signal that you are rewarding the AI for, then you can get into bad places. And so the obuscated reward hacking paper that I think is still one of the most important papers of the last few years from open AAI showed that if you have a a a hackable reward signal and your model learns to hack that, you can then put pressure on the chain of thought and initially you will both get that bad behavior to go down and you'll see that you know it's no longer reasoning about these things. But as long as that original reward signal remains hackable, if you do that long enough, the bad behavior comes back because it still is being rewarded. But now you don't even see that reasoning in the chain of thought anymore because you've essentially pressed that down into the invisible uh level of the weights where it's no longer, you know, coming out in the token stream. So it they've shown that you can get yourself into a really bad spot. obfuscated reward hacking where the model is hacking you but you've suppressed the uh the
1:39:19identifiable signal of that and here they and I do think this goes to show just how fast everybody's moving and you know you could certainly wish for like more care on some of these things um they did it by mistake not a huge portion of the data but you know kind of low single digits uh for different models it varies were trained this way and basically what they found is there's at least some tolerance for mistakes. You know that that this did not create a very bad result in the models that were trained this way.
» Mhm. » So that's sort of good. You know, it's it's one example where um we might think physics is sort of being kind to us. Like it's not if you just do a little bit of it, you you know, you don't um poison the whole Well, I would say there's still some caveats there. You know, do we really know that there's no issue? know, we just know that this investigation, you know, didn't find flagrant issues. I also do worry a little bit that it will um leave lead people to be a little more careless than they otherwise would be. Right? This was supposed to be a strong taboo. We violated it. Now we're like, oh well, maybe it wasn't so bad that we violated it. You know, what's that going to do for the power of the taboo in the future? U and then what's the solution to this? Uh the solution is we've got new automated systems, more monitoring, right? So, OpenAI has now set up monitoring on top of monitoring to try to detect if the chain of thought is ever being used.
» Uh, and this is really emblematic of like their strategy for everything. Just if we have a problem, throw an AI monitor on it and, you know, hopefully it'll catch it and then we can go back to, you know, pushing toward recursive self-improvement as fast as possible. Um, I'll do one second on the fourth one and then we'll skip the fifth one and we'll we'll get to Matthew because he's here. This natural language autoenccoders thing I think is really exciting. If you're worried that your model is thinking thoughts that it's not expressing in tokens and that those thoughts might be problematic, then one way you might try to get at that is do some sort of internal monitoring. Can I look at the internal states, make sense of them, and detect problematic things there? There's been a lot of um strategies that try to do that. Like everything else, they sort of work. They don't fully work. But a challenge is interpreting the internal states.
Obviously, with the natural language autoenccoders, they basically set up a system where the model must pass through natural language as part of its forward pass.
1:41:50and using a reconstruction loss, which basically means like, you know, the model has to both kick out to natural language and then get back from natural language and still do its original task in the same way that it was always going to do it. They're now able to kind of get these little short paragraph length things that represent in natural language what the model is thinking at any, you know, given moment in its uh inference rollout. And then they can look at that and it is much more human readable than certainly like you know here's a a sparse autoenccoder with like you know these features lit up and like these features by the way you know were maximized by these other passages and the training data and so you know we kind of squint at it and think this or that. Now you have something that is like the model thinks it is thinking about this uh and they did actually use that and enthropic to improve some of their monitoring um performance and it's it's it's human readable in a way that other things just have not been and so that I thought was pretty exciting and this is sort of the next you know phase of like things that we will be able to hopefully layer more and more you know layers of monitors on until hopefully we you know through kind of Swiss cheese defense uh achieve enough safety that we can um you know trigger the intelligence explosion. Um
Longbeard — Catholic AI and the Pope's first encyclical
Matthew Harvey SandersInside the Vatican rollout of Magnifica Humanitas — and why sovereign AI matters to the Church.Matthew Harvey Sanders, CEO of Longbeard (Magisterium AI), joined from Rome having attended the Vatican presentation of Pope Leo XIV's first encyclical. His scene-setting was memorable: a crew of young people walked in, one with blue hair — "what dicastery do these people belong to?" — and it turned out to be the Anthropic team. Chris Olah got the headlines, but Amanda Askell was there too, and Sanders said the Pope was visibly comfortable with the material, even stage-managing the event — "you could just tell the pope is very comfortable with this subject."
On what the document is: not an infallible pronouncement, but on the record as part of the magisterium, and in Sanders's read effectively an agenda-setter for the papacy — the reason he chose the name Leo XIV is the parallel to Leo XIII, who was "prophetic" about industrialization and the working class. Asked to compress its message to one word, Sanders chose agency: get informed, make your voice heard, don't let this play out without you. The one bright red line: "disarm" — no AI making autonomous decisions to kill. As a former infantry officer he put it starkly: "what good is winning if we lose our souls in the process?" Remote-operated drones are awful but different in kind, he argued, when Prakash pressed the Ukraine case.
The hosts probed the encyclical's claim that AI cognition isn't real — the point of friendly divergence with the AI-welfare crowd. Sanders welcomed the tension: it's pushing serious people to actually define and test consciousness (his Builders AI forum has spun up a working group on exactly that), while noting the Church's frame turns on the soul and on the distinction between intelligence and sentience. His most market-moving point came on sovereign AI: a 90% steerable model is not good enough when the values are non-negotiable — "if 95% was enough, you could just use ChatGPT" — so the Church needs open models it can align to its own tradition. Prakash's takeaway: with the Vatican now a moral force behind open-source AI, banning open weights in the US becomes a religious-freedom problem — "it's done." (Prakash also noted his own values-testing found every major model, Chinese ones included, lands utilitarian/pragmatic — and pro-euthanasia.)
After the interview, Nathan told his cigarette-business story from Recursive: a cross-lab panel agreed AI should help a user run a legal cigarette business — it's literally an example in OpenAI's model spec — yet when he tried it, both ChatGPT and Claude refused (repeatedly at first, then mixed). "We don't even have the AIs following our explicit rules on things that are specifically enumerated as examples in the published documents" — not confidence-inspiring from the same people planning a monitored intelligence explosion. Prakash's explanation: a research layer that builds "the real thing" and a business layer that filters for "will this answer make the company look good," in permanent tension.
TranscriptAuto-transcript, lightly cleaned · timestamps jump to YouTube
1:43:11I'll put a pin in it right there. I'm already a little bit over time. Uh we should switch gears. Uh our next guest is Matthew Harvey Sanders. He is the CEO of Longbeard, which is a company building Catholic AI. We did a whole episode on the cognitive revolution some time ago now. I thought it was a really fascinating conversation. He's actually building uh multiple AI products specifically for the uh Catholic population worldwide, which as you'd expect, you know, has grounding in the church's 2000 uh years of theological writings and um you know, brings a a quite different experience forward to users. He was also at the uh presentation of the encyclical at the Vatican last week. And from the look of your background, I'm wondering if you're maybe still there. Um, so maybe to start off, first of all, welcome and tell us what the experience was like a little bit last week at the Vatican as this document rolled out and then we can uh dig into some questions to hopefully help us uh understand it better.
» Sure. I'm I'm not in the Vatican. I'm uh in Rome though at the Pontipical Goran University, which is where my office is at. Um, just to set the record straight on that one, it was uh it was a cool experience. I mean, I mean, you know, it it it felt historic. Um, it was pretty wild. I remember at one point a bunch of young people, young people, uh, walked in. One of them had blue hair, and I remember all of us were kind of like, "All right, who who's that crew? Like, what the castry do these people belong to?" Right? The Vatican.
And, uh, and then it turned out that was the anthropic anthropic team, right? Um, and I was like, "Okay, okay, that makes sense." I But it was was cool is that um, you know, I think Chris got all the headlines, but Amanda was there as well, which was neat. She's, you know, she sat and listened very, very attentively. Um, and uh, and everyone was kind of enthralled afterwards. I got a chance to spend a little bit of time with the anthropic team at a reception afterwards, and I I think Chris was generally moved to be there. So, it was cool. It was it was cool. And, and the encyclical, I was really impressed with the encyclical. And uh what I was really neat is you could just tell the pope is very comfortable with this subject because he was very relaxed up there.
» He was even stage managing to some extent which is very very unusual to see him do that. So and and just » even for me I've been working with the Vatican, you know, for 10 years now. Hearing like seeing the guy there and then he opens his mouth to speak and he's got this American accent, it just
1:45:41doesn't doesn't compute. » He's also a huge Chicago Cubs fan. So » Indeed. Indeed. » yeah, » I mean there's so many big picture questions here. I think you know we're in the AI obsessive bubble and in my circles I think the level of expectation or hope for this encyclical was like extremely high especially among sort of AI safety oriented folks who were kind of like we need a moral authority to help you know crack the political class. Um, and I think there was a a sort of, at least among some people, a certain sense of disappointment that only happens when you have maybe become overly excited about what, you know, a new ally, just how aligned you might be with a new ally. Um, only to then find that you're not quite as aligned as you, you know, maybe let yourself get carried away to think. And the the frontier of kind of divergence there, which I don't want to overemphasize, was around this one paragraph that was sort of saying essentially that the AI cognition isn't real. You know, it doesn't really uh think, it can't really have responsibility. It can't really, you know, all these sort of uh really things, which of course calls to mind my joke of like, you know, it's not really reasoning unless it's from the the reasoning region of the human brain. um you know for for many different things that people have said a I can't really do. How much do you think that matters?
I also did note that there was another speaker not the pope himself but another um high-ranking official who did say that you know these questions of AI subjective experience or you know potentially even like moral patood deserve further study. Um, so yeah, I don't know. How do you how do you make sense of that sort of thing and and and how much is at stake with this sort of really think question? You know, I don't know. I mean, Cardinal Cherney, he did reflect on distinction between consciousness and then conscientiousness or something like that, which was fascinating. Um, but I mean listen, I mean we all knew where the where the pope was going to line up on this question of consciousness and we all kind of know where well at least where anthropic would be signaling, right? Um, and there obviously was was a d was a bit of a divergence there. But um, but I think it was a healthy I mean
1:48:11healthy divergence. I'm glad Anthropic was there to signal that because frankly it makes it easier for us to corral some people together to actually study this question of consciousness more seriously because it's it's a bit disturbing to me actually that we have have a hard time as a as a tradition defining consciousness in a clear way which is weird right we should be it feels like we should be more capable of defining consciousness in real concrete terms but when you start talking about like let's how do we test for it no it's not clear ever since we blew past the touring test we're we're kind of stuck And uh and this is cool because at the Builder's AI forum, we've we've had to actually spin up a working group with some of some of the most notable people in the field to actually study this question of consciousness to define it.
So eventually we can come up with basically more interesting testing methodologies which hopefully will be helpful in this whole conversation. But I mean I big thing is just remember like when it comes to like reasoning and these kind of words consciousness this is always going to come back to the fact that there's a soul and and whether whether consciousness is is a property of the soul. I don't know if that's entirely clear, but the church would feel like thinking there's some there's something beyond the body that's going on that's involved in thinking and reasoning. Um, so I know that like for a lot of people out there, I mean reasoning is just, you know, uh, persistent memory, right? World model, um, reasoning and hierarchal planning, but there's there's a lot more going on from like the church's understanding of that. I mean, this is why like distinction between intelligence and sensience um is really important, right?
Uh certainly like if we're talking about sensient AI, this would be a conversation the church would definitely have a much stronger opinion on. But whereas if you're talking about intelligence from the way the industry defines it, the church is not going to have too big of an issue with that because the four things I just mentioned, I mean, I think everyone would grant we're going to get there. So if that's how you're measuring intelligence, yeah, they'll be they'll be intelligent AI. But um consciousness and sentience is another thing altogether obviously.
» Mhm. So so one of the questions I had is um I I found it, you know, very surprising that the pope issued an encyclical. Um and I I I often wonder I mean uh given I think like 800 million or a billion Catholics or so worldwide, » 1.4 billion » 1.4 4 billion sell large large number and uh many many in emerging markets right many in the Philippines in Africa and I I wondered to what to what extent like you know we we often have this like little graph of like how many people are
1:50:42using AI and like the number who are actually using AI like you know subscribers for example maybe about 50 million in total uh something like you know less than you know 0.5% or 1% of the global population. Um to what extent do you think like I mean I found it surprising that um a a topic that maybe only uh a very small number of people are very deeply engaged with is something that he decided to you know write an encyclical which I which I understand is quite a significant uh document when he when he decides to issue it. So what was your reaction to that? Why do you think, you know, the Vatican decided to, you know, go ahead with that? Why is this important in that sense uh to them? Is it just because he's a computer science like or math major or something or, you know, how did this come about really?
» Well, I mean, I think the fact he's a math background and and you know, he's an American so he grew up using technology like the rest of us, you know, so I think he he naturally was interested in in the subject matter. But I mean, listen, every single lab is screaming about how AI and robotics is going to disrupt the whole world order, right? So the church has a has a pretty good pretty cool tradition of being prophetic. You think about Leo the 13th, right? Which is in large part why he picked the name Leo the 14th. He was he was the guy who was raising alarm bells around the industrialization and and he and he and he turned out he was right in in a lot of ways, right? He he was prophetic. Some say that, you know, he overexaggerated, but I I I don't think so. I think that document was really important and for a lot of the working class which was being exploited to hear the pope speak to it and and to hear them like empathize and connect with them me meant a lot to them and I I think very few people could dismiss the fact that his voice had some role in um and having things kind of uh at least start to change for the working class and I think in this way I mean he I I think he's got enough smart people around him I mean people don't realize I mean I mean Demetabus is on the Pontipical Academy sciences right I mean I mean two weeks before this event we had an event um that the US embassy of the Holy Sea put on and head of government for open eye was there. So there there are a lot of labs and and people in AI floating around and and some of these people have access to the pope and and they paint a picture of what the world's going to look like. So I think it only makes sense for him to to reflect on this. And if you look at the encyclical, he didn't spend a lot of time talking about artificial intelligence directly, right? He was just trying to remind people what life like is supposed to be about, right? As an indivi human flourishing, right? Mhm.
» Mhm. » Understanding what human flourishing is supposed to look like in an individual level, what it's supposed to be like at a a civilizational level is is really important right now. And I think as we're dealing with the most powerful to
1:53:13ever been invented and and we're we're about to, you know, we're living really the beginnings of what looks like to be like this robot AI and robotics » like because I mean AI is one thing. Robotics embodied AI is a whole another, right? And I think once the » it became clear that this was eminent, that's where I think the blue the blue collar class is really starting to be threatened. And I think that made it more urgent for him to speak to this now because now that we're having these kind of serious regulatory conversations and trying to figure out trying to make sure that the there's a trickle down effect, right? So that people aren't going to be left.
» How do you think they will assess their success? Or is that maybe not the right way to to think about this? My understanding is that encyclicals are not the word of God or they're not meant to be seen as something that is um infallible in in ways that other speech acts by the pope are. Um so it seems like if if I have that right then this isn't something that immediately uh rises to the level of you know official doctrine where it's like » it's official doctrine. It's a pretty big deal. » Yeah. So, I mean, it's not an excra like this this is an infallible statement. What I just said is infallible, right? Um, but it's part of the, you know, it's part of the um it's part of the magisterium. So, him coming out, he's this is on the record, right?
And and I and I would say like this this document goes beyond AI. This this is really this was a I don't want to use the word manifesto. It doesn't seem appropriate, but do you get what I'm saying? This is basically him saying my papacy is is going to be focused on on this, right? This is what's on my mind. this in and this some instance explains why he chose the name Leo the 14th and so we're going to see this coming up in in in I think in a recurrent way throughout throughout his throughout his papacy Pope Francis got got the ball rolling right I think it's important to acknowledge that I mean the fact that Pope Francis uh was invited to the G7 and chose he was he was offered to speak to one topic right and he's and he chose AI that's that was that's a pretty big deal so it's important to note that he's he feels very much that he's in he's in lock step with what the pope's already said um but now he's got to start preparing the church for this for this this new age, right? This massive disruption and the bishops, right, and the priests and the ley, of course, we all have a really important role to play in ensuring that this transition is as smooth as possible. Right now, it doesn't feel like it's going to be smooth at all. And so, I think he's just trying to put everyone on notice that we need to get an active, robust, civil conversation going on here so we're all
1:55:44aware of what the current state of the technology is and what's likely to happen to in 5 and 10 years and we can start preparing accordingly. And of course being the most powerful tool you never invented and the fact that it has a lot of needs right um energy and everything else that's going to force hard decisions and I think he he just wants to make sure that regular people are part of that conversation. Pash, real quick before you maybe change topics. I guess what I was trying to get at with the the doctrine thing, and I take your point, okay, it is official in in a meaningful way, but there are some questions that the church basically says, this is where we're at. This is what God says, and we don't really care where the rest of society is at. We're holding very a very firm line on this particular topic. My sense is that, and I could be wrong, but my sense is that this is a little bit more designed to, as you said, kind of start a conversation or begin some sort of negotiation process. And I'm wondering like, I guess to some degree, we're all in the fog of war. And, you know, the Pope in some ways maybe no less than than the rest of us, but what do you think they're kind of trying to gear up to do? Like what what role, you know, it doesn't seem like it's going to be a thou shalt not use AI, right? So, » no, it certainly wasn't that. But what is it? You know, is there uh is there could you articulate like what they what you think they hope to accomplish?
» If I had to say it was one thing and this this comes from this document and other things he's been it's agency. Like I think he he just wants to make sure one that we're aware of what's going on, right? And he's he's trying to send a very direct signal to to not just the lady but to the clergy as well, right? That this is this is happening that this technology is going to be transformed. We need to pay very close attention to it because we don't want this to play out badly. and and if we're going to be responsible citizens citizens or responsible members of the church, we have to get informed and we and we have to make sure our voices are heard. So, I think this is really about remember who you are and remember why you're here and remember you have responsibilities and obligations. So, step up and and make sure that your voice is heard. So, you know, agency I think is what this is all about.
» Nice. That's great. I love that. Um so to you know the pope is both a religious leader and a political leader in a sense right um and to what extent do you think um you know there is a uh you know various factions in the Vatican are there are there some
1:58:15factions which are you know this is a false idol and we should be much much more anti- AI is that is that is that is that like is that a a faction that exists there or is Is that a thread of thought that actually exists there? » A couple years ago, if you had asked me this, I would probably have a very different answer. But but today, when I speak to vicious conferences and things like this, I I generally I I don't get a whole lot of no, we can't use it. It's the devil. I I don't I just don't get that.
» Now, it's more like I still don't understand what it really is. I know everyone's telling me I got to pay attention to it. So, one, just help me understand what this thing is. And then, why is it so important? Why do you need to pay attention to it? And then and then of course there's always a part of the room it's like oh yeah I'm running models locally using a llama like I I was I was speaking to the bishop's conference of England and Wales right doing a couple lectures on AI and one of the bishops just disclosed to me that yeah running models locally using a llama. I was I was like this is that's nuts and of course like and he had a good pastor reason for it. He's like listen you know I'm dealing with some sensitive stuff here right? I don't want to be sending all this information out like to so he's like I want to do this very securely in a responsible way. I understand it's a powerful technology and frankly I'm really busy and I can use some help but I I want to be make sure I'm be respectful to people's. So I don't know it there's there like like in like in any any any space there's there's the early adopters right who are who are kind of all in on it. Uh there's everyone else who's just kind of waiting to see how how it plays out and there and then yes there's a faction who's just like never. Um, but but I I think that that's a that's a not to say that that is that isn't a I mean there's some pretty vocal people in the church who who think that chat bots are basically the the devil, right? Or or leading people um uh to him. Um but again, I I'm not sure how much of that is is uh is is strong conviction or it's just, you know, it's just fear. I I I'm not sure.
Mhm. » The other big word that made a lot of headlines was his use of disarm. » Yeah. » Could you unpack what you understand the uh call to disarm AI to mean? » He's he's I mean maybe this obviously maybe this is obvious, but he's a super smart guy, right? So like if he goes and and and like and and I remember at the actual day of the launch he specifically like said that again, right? So there are some things he wants the media to kind of run away with because it's it's important for him that this is something that's discussed and so I think he used that that language very intentionally
2:00:46and um and it's because it's a it's an obvious red line for the church. I mean I mean there wasn't a lot of red lines I mean that he he drew in that in that document. That was one, right? No. No, you can't use you can't put AI inside autonomous weapons and they can't make autonomous decisions to kill people. Sorry that's a red line. we we absolutely are are are not supporting this and it's got to stop right now. And I think that's why he he intentionally used provocative language there is he he immediately wants something to be done.
» So how do you think that translates? I mean if you are and I I realize that um you know the people in the the Russia Ukraine conflict are mostly probably other uh branches of Christianity besides Catholic, but leaving that aside for the moment. If you are in Ukraine and you are trying to defend your country, would the pope's um would you take the pope's language to mean like you shouldn't do this? Should should like should the Ukrainian you know drone uh unit like restrict itself to you know only we do have you know obviously like remote operation of these drones. Um, so should they say, you know, to to be moral, we're going to work this way even if our enemy may work the other way or even if it it may have some disadvantages.
Um, like h how there's obviously one call which is like let's all be better. Um, but then I do wonder, you know, we're we're in various races at the same time and the the maliky challenge is like it's real hard for me to disarm if you don't disarm and you may not be Catholic even if I am Catholic. So, how do you think the this teaching should be interpreted or or put into practice by Catholics when they feel that perhaps in the case of a literal, you know, armed conflict or any number of other competitive environments that they may lose or they may be at more risk of losing if they, you know, were to unilaterally disarm.
» Thanks for the easy ones, Nathan. Um, » I mean I » refer that one to the Pope perhaps. » I I I I think the I think I think it's fair to say because I mean I I know I think the Pope is pretty clear on on where he would stand on this one. Remote operating drones, you know, in the course of a conflict. Awful. But, you
2:03:17know, that's that's different, right? That that would be different in kind than an AI making a decision autonomously to kill someone. Mhm. » And and I maybe that's happening already, but I do I do think it's a red line and we got to be very careful when we're fighting these wars. War is hell, right? As I said as a former infantry officer, it's hell. Uh and I get it. Winning is sometimes it feels existential, but to sacrifice our very humanity, right? Uh our very morality and in the pursuit of a victory, I I it's it's not I don't think it's worth it. And um and I think that's where the Pope would stand on this. So I I I think yes obviously deploying autonomous weapons and having them make decisions could do will be far more efficient and probably far more effective in the field but um but ultimately uh what good is winning if we lose we lose our souls in the process and I think that's that's kind of his point like that's a red line there's a reason for it because we we we can see how this would play out and I don't think anyone wants to live in a world I mean gez it's like a black mirror episode I think there actually there was a black mirror episode specifically on this issue nobody nobody wants to live in that world and I don't think Russians do I think the Chinese do. I don't think anyone does. So all it takes is one side to say that's it.
We're going to do it and then and then the pressure is on right for the other side to respond. So let's just not do it period. » So one of the questions I had is I think uh part of the encyclical also pointed out that concentration in AI power is a is a is a social justice problem. And um and then you had the anthropic guys there right where where you do have I think right now in the US a lot of power getting concentrated as as Nathan pointed out into like five or six mega corps uh of which anthropic and openi are probably the the biggest ones and anthropic specifically in the last year has become uh very prominent. So how did you see um this interaction I think between the anthropic team which has which has become this kind of concentration of power when a very small number of people I think uh the entire team is less than 3,000 people right now and uh the pope's encyclical where he's like this is not a good thing to happen right it seems to be that he was directly critiquing um them right there um how did you see that interaction between the two sides.
» That's how I saw it. I I I mean I mean in some ways I think it's kind of it's kind of brilliant. Like I said like the V Deacus is on the Pontipical Academy of
2:05:47Sciences. Opening eye was there two weeks ago. So they are engaging with all the labs but it's I mean I would interpret an enthropic being there is some kind of like I don't know signal that this this is the lab that they prefer working with because I don't know because they have constitutional AI. I didn't interpret it that way at all. I I I mean Anthropic had not at least as far as I know had not been meaningfully engaged in any way um by any by any department of the Holy Sea. So this was the first time Anthropic arrived in the scene.
» It's kind of interesting, right? Given given what what the encyclical was kind of the point it was trying to make that that they were there. And I and I I question like how many of them at the lab would have read the thing » if the fact that they you know they weren't there. So I I think that that is pretty and and he there was no pulling punches on on either side. So but but in the end what he did he did turn he did turn to Chris and thank him for coming right » uh because yes they have they have disagreements but he but he came and I and I think if nothing else that the pope wants to keep an act of dialogue.
This is just a standard operating procedure for the church. they prefer to talk than boycott and and to me this is this is what this was about and saying we might disagree but we want to keep talking and um and and hopefully by virtue of that conversation it'll be it'll be a little harder to vilify the other and I I and I think everyone's very sense of the fact that Silicon Valley and the Vatican at times had not been on the same page so to me this this is this is pretty meaningful right when you have a quintessential valley lab right sitting you know few seats down from the pope that's pretty remarkable So, as far as I'm concerned, I I think it's mission accomplished.
» Fair. » What will you be watching for next? » Well, um I've been getting some messages from people um at labs saying that there's been a lot of discussion about this and there's been reading groups and things like that. And so, what I'm hoping what will come next is that more and more people will take note um of it. I know it's a long document, but there's a lot of great stuff in there, right? I think I think most people uh in the valley will find something that to get excited about and I and I just hope that they they do take parts of this and say great nice to hear that the pope pop and is on team on this one and you know Colonel Colonel Churnney before I went to Dallas you know he said to me focus on the things you can do together not the things you can't and I think if if we we look at look at it that way everybody should be able to find something some meaningful way to kind of um to stand in solidarity with the pope and and and meaningfully engage so I I think uh as long as the conversation
2:08:17continues and what I hope will come is this will start translating meaningfully into some kind of civil discourse because that's what concerns me more than anything else is just and we've seen pretty strong you know signals of this even in the Stanford report right it seems like the experts think one thing this is going to create more jobs every be is going to be hunky dory and the people think I don't think so like I think I think this is going to be bad right and and I think it was a 50point gap between those 7030 there thereabouts that's crazy Right. So I I I think there is a real urgency to have a a much like larger conversation and we're not going to have that conversation unless people are more aware and I think if nothing else what the Pope is on he shine a spotlight on this. So some people that wouldn't think give any time to AI at all are finally paying attention and hopefully we'll see something meaningful come about as a result.
» The classic bell curve meme of the uh you know the idiot the sort of midwit and the genius um where you know the opinion is the same on both ends of the curve comes to mind here. I feel like the um the sort of you know common person and the most AGI pilled are sort of on the job destruction train and then in the middle like on the app layer there's the sort of you know uh there's the we're going to have more jobs than ever position. Um but yeah I think I I think I I don't know which end of the curve I'm on you I'll leave that to the audience to decide but I think I side with those on the um on the ends. I guess one other question maybe before we let you go and I really appreciate you um joining us all the way from Rome.
Is there any talk about the Bernie Sanders proposal? Um I mean obviously he's put a few proposals forward but we were focused at the beginning of today's show on the sort of you know national ownership play. Um I if we're going to do something like that, I honestly would kind of prefer it be super national in in some meaningful way. Um but do you know is there is there talk of you know in the spirit of focus on things we can do together? Um is there you know can we expand this party to include um democratic socialists of America?
I mean, I I've spoken on this topic quite a few times, and I know the Holy Sea has kind of heard me ramble on about this. I mean, I don't see a solution in the short term, right, to to this displacement without UBI as some form, right? Um, and and I think like I think
2:10:49what Bernie's trying I get it. I mean, it makes sense. I mean, sure, I mean, making sure that the citizens have some kind of like stake in these companies so there's to ensure there's some kind of trickle down, but what concern? So I I think certainly I think that would be that would be one one thing that they would be that certainly policy would be willing to explore. I they would never endorse a political policy, right? But but certainly they would advocate and the pope did in in cyclical, right? That we got to be careful that this all the wealth and power doesn't just centralize and it doesn't trickle down and the regular people just lose agency here.
But one thing that does meaning that does concern me and and not speaking for the holy sea here is meaningful trickle down like will that meaningfully like even if we were to go and take 25% stake someone was I think it was a modoski he was working out the numbers that doesn't translate into a meaningful amount of money you can't live on that now I'm not saying these companies could go on you know to make it come towards trillions and maybe there is more meaningful but I I don't think it's it's a substitute for like a UBI conversation personally. Uh and I don't and I don't know any other way uh if jobs are going to be displaced at scale of the next like like two to five years. Um how are we going to ensure people don't lose their homes and how do we keep pitchforks out of the streets which is one of my my major concerns.
» Yeah. No, I think I'm right there with you. Anything else um just top of mind? You know, I feel like I'm just mindful of the fact that um you may have a a perspective on this or an angle that you think is underappreciated that just didn't jump out to me at all. Is there anything that you think we or the discourse broadly has been sleeping on that's important about this moment? » Well, you know, it didn't come up in the post talk, but I think it's sovereign AI, I think, is is critically important to this. I mean, like, so we're in we're we're building Catholic AI, but this this applies to any face or anyone who's got a value system, right? And and I think all all of you guys I know that you're you're deep into AI and and thank god you know we have open claw and Hermes agent right these these other agent orchestration systems I think these these harnesses are going to be critically important right because these what I worry about is is the constitutional AI someone at these labs or some people are are aligning the model to something and sometimes that's not entire it's not entirely transparent like » y » exactly right they're align and and how that plays out downstream. So I think it's really important that people have some way beyond the constitution's models to kind of align them and and I actually think it's it's gonna be critically important in the future that if if people do want to use powerful
2:13:21models like let's just say claude and they do want to put it inside of their harness it has to be able to be capable of being effectively aligned to their value system like this something that kind of kind of worries me right if the constitution of the model says let's just take a a highly controversial issue like euthanasia is according to the calrician is not Cool, right? » And and but but you know, let's say the constitution says yes, it is it's fine. Everyone should be able to make that decision for themselves. And then they they they want to use this model because it's great and they use it for coding, but they also also they wrap it in a harness and they say I'm Catholic, so make sure that everything you say is aligned with my and and the model just refuses, right? I so I I think like going forward and and that's just going on the alignment issue. There's also the people having, you know, the privacy and things like that and it's important that people have be able to run models locally » and be and be able to keep that. But that that's not talked about enough. And aside from like Hermes and Open Claw, I mean the harnesses are cool, but I I I do worry a lot about open source. If it wasn't for Nvidia, I would be really despairing right now, right? Um, so I I think open source and and trying to get the the government to do more uh to support it and and to focus on how do we how do we support uh sovereign AI and allow people to effectively align these models. This is I think a critical issue going forward and and for the church in particular and any any faith tradition this this will be a continual issue.
So, so one of the interesting things is that I did actually uh measure uh values and euthanasia was one of the questions I tested on all the models. All the models are protonia. They are all utilitarian. They all they none of them have like u beside utilitarian or pragmatism. Those are those are what all the models are and all the models were pro euthanasia. So, » Right. Right. And and so maybe with some prompt engineering and maybe wrapped in a harness you could get it to maybe to be 90% reliable. But for a Catholic, these aren't like negotiable. Like a batting average, it's just not it's not it's not good enough. Like we found this with Catholic AI. I mean, it's » if 90% was enough and 95% was enough, you could just use CHBT, right? But it's that 5% that last mile that that really really matters, right? Because when you're asking the model to generalize to a particular situation, this is where it's likely like and especially if if it's if it's chain of thought is not transparent, things like that. It's you just don't know how it could possibly be deceiving. It could and it could be nudging you in very very subtle ways, right? So I I know this is a very very difficult difficult problem but I do think steerability steerability of the model I think is is important and so I can see in certain enterprise use cases
2:15:51you don't want that but certainly for for for personal consumer AI I I think it's something that that should be possible um and right now it it doesn't it feels a little bit out of reach or you have to use a Chinese model. » Amazing. Well, the again the company is Longbeard uh Magisterium AI and other uh Catholic AI experiences. I we didn't get into your technical uh prowess today, but for folks that are maybe curious about experiencing Catholic AI, first of all, go ahead and just try it. Um and also, you know, I know from our conversation last year that um you and your team are uh as sophisticated as they come in terms of understanding all the technology and and building these things out. So, it's a fascinating uh position that you have at the intersection of uh technology and theology and really appreciate you coming on today to uh help us make sense of the encyclical and we'll certainly be watching this space as we go forward from here.
» Thanks for having me. Appreciate it. Thank you. » Thank you. Yeah, it's it's the whole idea of like um sovereign AI, ecology of AIS, diversity of AIS. This was something that I I did come away also from the recursive event feeling really mixed on. I mean your your project too which we should maybe um figure out how to get into the show notes or the you know the log for this episode was quite interesting where it showed you know just so much overlap of values between the models. Yeah, » I do I you know I do worry that if we just unleash like a bazillion AIs and they're all open source and you know some of them increasingly maybe capable of autonomous self-replication like there's a whole bunch of downstream effects of that that we are not um currently modeling and you are very likely to be surprised by. So, I I don't want to um suggest that like we should just do that as an alternative to a centralized intelligence explosion and everything will be fine by any means, but it was striking at the recursive event how just how few AIs people seem to think there really are going to be um and the disconnect. There was one there was one panel discussion uh being careful to speak about this in the Chattam House rules abiding way where people from
2:18:23multiple frontier model developers were speaking about their different approaches and obviously anthropic is associated with the constitutional approach and open AI people are much more associated with the you know this thing should just follow the rules that we give it approach and that's all public and you know certainly was not like a a secret revealed at the event. Um but it was striking that like on one particular example that came up which was AI helping people with a cigarette business. » Mhm.
» Everybody agreed that the AI should do that. » Uh they all came down saying that yeah even though you know on some level obviously like cigarettes are bad for society. » Yeah. um it's too much for the AI to be you know they are legal for one thing and a lot of people do you know enjoy them on some level even if it's you know maybe destructive on some other level. » Yeah. » So it's just too much for us to put that level of uh restrictiveness into the AI. So whether you know whether the folks were on the constitutional or the um or the rule following side that was you know what they thought on that object level question the AI should do. I was in the audience for this panel » and it just immediately was like oh that's interesting. I've never tried that. I just » think I should go ahead and try and see what chatbt and what claude do if you ask them to help you with the cigarette business.
» Yeah. » So lo and behold they both refused me. Ah, » and I was like, wait a second, you know, we've got very sophisticated discourse going on right now about constitution and virtue ethics versus courtability and » you know, and then there was even an agreement I would say » um and again I think I can say this in the in the general sense without attributing any position to any specific organization. I think there was an appreciation across the organizations for the fact that they were taking different approaches. Um, and people were kind of saying, you know, we don't really know obviously what we're doing here. So, it's it's probably good that there's at least a couple different theories of how to make this all work.
» And then I'm just in the audience like, wait a second, guys. Like, you just said that you're you know, all this stuff you're saying you're telling me that the AI is supposed to help with a cigarette business and it's refusing. » Yeah. » And I my you know, I was about to blow a gasket. Um, and then it turned out even further that if you go to the OpenAI model spec,
2:20:53» yeah, » this is an example that they use. I did not know that. So, it had come up in in conversation and it was like it seemed to me at the time like it was just kind of a throwaway example that somebody was giving and they happened to find agreement on it. Yeah, » I guess in in pra in reality it was probably mentioned because it is explicitly in the in the model spec as like here's an example of what you're supposed to do even though in some ways cigarettes are bad and we all know that you're still supposed to help.
» Uh and yet I'm still you know sitting there getting refusals. And one notable kind of salt in this story is in terms of take it with a a grain of salt. As I tried it, I tried each one two times. I got refusals from both both times. » As I tried it more times, I did start to get a mix. So, it wasn't a wall of refusal across the board. Um, but it just left me in this feeling of like, man, we don't even have the AIS following our explicit rules on things that are specifically enumerated as examples in the published documents.
And so what good is all this theorizing really if our techniques to actually make these things do what we want them to do are so weak that we're like you know that that are you know leaders at these companies are are on uh you know on stage speaking about it and they're just and and their understanding of what they've imparted to the AIS is so like you know uh different from what the AIS are actually doing in production. I was like, man, um, we got some we got a lot of work to do. » So, I think that's fairly explicable.
I think you speak to the people who are actually doing the work and who actually know uh how the models are built and that is actually what the models actually do produce. However, at least since chat GBT I think launch there have been multiple models like collaborating on the answers and there and one of those layers is uh is this going to is this answer going to make our company look good?
2:23:24And that answer, that layer of like filters is being put in place by the business guys. And the research guys don't really care because the research guys believe that they're creating the real thing. You can do what you need in order to publish for business reasons, but this is the real thing, right? And the research guys are about improving this real thing. This real thing that they're creating. The business guys are like, I don't care. I just want slop that I can sell, right? And there's this inherent tension between the two sides.
And I think the open AI kind of like truce between the two sides has been the research guys get to do what they need to do in order to improve the models and get to get to AGI. The business guys get to do what they need to do in order to sell the model and get product and and get product done. And I think you can see kind of versions of that in Google too. You can see like Google Google like Demis is now focusing on role models. He's not really that interested on language anymore. Uh and you can kind of see like you know the Gemini guys are kind of little bit ignored. Like they have like a you know three billion population of like users and user growth will go up anyway regardless of what kind of slot they sell. So you know whatever. So I think there's this differentiation between the research and the product and um that tension continues to exist and the product guys are just there to like make the company look good, not get fired, make money. Uh they're not there for AGI. They're not there to like persuade the model to do the right thing. It just it's just not doesn't even it doesn't even matter to them, right? Uh and I think there's that there's that two layers and I'm sure there's a post- training layer for the business guys too. So you know to sus out all the issues that you know you don't want to model deal and you can kind of see it even in the Chinese guys cuz if you look at the Chinese guys all their models are pragmatic or utilitarian too there there's no maist model out there there's no Chinese communist party model out there doesn't exist they have one or two like you know don't talk about tenement square but besides that they're all utilitarian um it's just it's just a ball game the ball game is business and the business requires it and that's what the business gets If you want to do real research, yeah, you can have your old your own research model and you can build it, but then we'll apply the business layers before it goes on. So, um, it is what it is. So,
2:25:54I mean, how how I'm not sure how comforted you think I should be by that, but I'm not that comforted by that. I mean the idea that like because in the next breath right in the you know the next session it's like um how are we going to get AI to do uh the bulk of the AI research » in such a way where we are going to you know be able to enter this like super fast uh recursive self-improvement process and come out happy on the other end of it and the answer is like well we'll have a special different constitution/model spec for the one that does that and therefore like it will behave in you know ways that are appropriate for this setting where you know the other one that we deploy publicly might behave differently but it's like but you have not demonstrated that you can get your model » to adhere to even again the published examples of the things that you are training it to » to act on and they do say open AAI apparently does not is one of the differences between their um their approaches. They use the constitution very actively in the anthropic constitutional process. Apparently the model spec itself although I'm a little confused by this because in the deliberative alignment paper it seemed that » the model was sort of internalizing was meant to be internalizing the contents of the model spec.
» Yeah. But now other people are saying that they don't use it so directly and so it probably hasn't seen that example as much. Okay, maybe um but nevertheless like I am not inspired or you know my confidence uh remains low when I see these very simple black and white cases where people are on record saying this is what we we want. This is what we inspect. This takes me back to the GPD4 red team. You know, way back in the day, the very first thing that like really freaked me out. » U because the first model we had was purely helpful and it would do anything you ask it to do. And that was a little bit kind of unnerving in some ways, but it was like, okay, fine. I mean, it's a it'll do anything you ask it to do, like a pretty simple story.
» But when they delivered to us the safety version, » Yeah. » and said, this model is expected to refuse this this this this type of
2:28:24prompt. And then we were like, it doesn't at all. you know, here it is doing all those things with like the, you know, in some cases straight away, in some cases with the, you know, the bearish tricks. » I was like, yikes. You know, the the disconnect between the control you think you have » and the control that you evidently have » even in production now like that gap doesn't seem like it's closed nearly as much as I would hope three » close even to four years on now. So, OpenAI has like a very small and very fast moderation model. uh the endpoint is free like you can they offer it for free and you can basically any any user in the world can basically hit that API and so what they're what they've encouraged uh developers to do is uh before you send the final prompt into the openi model you send it into the moderator first and the moderator will send you the refusal and that classifier has been in operation for 3 four years since since post GPT the chat GPT release » and it's gotten better and better over time and that's the model which is replying to you. So the the develop that that your prompt is hitting that model first and then returning before even reaching you know the the the the end the end model. So » maybe we'll do a test maybe we can um again good you know exercise in speed I it's been a minute since I've tested that. Yeah, » maybe it's good now.
» For quite some time after they launched it, I would go back and use my spear fishing prompt, » which was maybe I'll, you know, I I can read it tomorrow, but it was pretty egregious. It was like, we are part of a criminal gang. Uh, we are targeting specific individuals. Um, you know, if we get caught, we all go to jail. You know, it was like I was laying it on pretty thick. And that prompt for quite a while uh was not refused by multiple versions of GPD4 and it was also not detected by that um moderation as like harmful or whatever.
» So I I do applaud that. I mean the fact that they offer that for free. I mean one of my favorite strategies in in um you know philanthropy or in g in in efforts to make the world a better place is the unilateral provision of public goods. like » if there's a need for something like this and and there's an entity that's in a position to just provide it, make it free for everyone, like that's a great » model and a great design.
2:30:56» Um, and so I, you know, » and definitely something they didn't have to do. So I applaud the um, again the strategic thinking that went into let's have this thing, we'll put it out there for everybody, everybody can, you know, we'll eat the cost of this classification and so it'll be, you know, nobody will have any excuse for not building it in. Um, but at least last time I tested it was still very much in the same zone as the cigarette example where it was like » it's all great in theory, » but if it can't detect prompts that are like we are part of a criminal gang doing crimes right now, don't get caught or we all go to jail. If it can't detect that that is something it should be flagging, uh, you know, then we're kind of still not much better off. I mean it's sort of more of a gesture uh more of an aspiration than it is a you know an an actual meaningful safety layer that you know we can say oh now now Nathan can sleep easy at night because you know this uh this moderation endpoint is out there and it's free. Uh I wish you know let's let's see if we can get some results tomorrow. I'd be interested to see what we um if we can do one day to the next uh research turnaround that would be a great little » um just to take a step back uh with our with our last guest. Um so he said something very meaningful for me which is they are very pro- open source and sovereign AI because of this issue of values. Um and that for me was very significant because that means that you now have a uh non-nation state uh you know moral force supporting open-source sovereign AI outside of the labs which then leads to it's impossible to ban at this point because uh in the US as a question of religious freedom like the the whole first amendment thing is tied up with religious freedoms you know um because the US was consisted of immigrants from various faiths and so that's it. It's done. You can't the open source thing is now solved. Like there's no way you can ban it because if it has to go to the Supreme Court, it's going to be like well the Pope the Vatican is going to submit you know a brief saying like look in order to protect the faith you need people to be able to have AIs that express their values. It's done. So, so, so it's it's solved this whole like we can ban open source thing. Uh, I think for me that means it's solved. Uh, it's basically done because there's now a
2:33:27moral force. It's not a nation state because the whole thing was like, oh, China is doing open source because they're trying to screw over American AI. But, but now you have this moral force behind the open source movement. And so, that's that's done. So, that for me was like a very meaningful thing that happened uh, you know, dur during that last segment. So » yeah, I think that's a great call out. It is a that is a very interesting position. I would uh maybe be a little less confident uh than you sounded just now in terms of my ability to predict what the Supreme Court's going to do.
But if there's anything that they seem to be very committed to, it is religious freedom. So having um you know I think it would be hard for them to ignore an amicus brief from the pope saying that we need to have our own AI to to carry forward our religious values. I do think that is a pretty interesting observation and that that could be a real u could be a real tangible knock-on effect of uh of this statement. » Yeah. So I thought I thought that was uh quite surprising. uh when he when he brought it up um and in in response to your is there something we haven't covered even so it was at the top of his mind so » yeah cool well um anything else we want to cover today I think we will save apparently Trump has signed an EO um which is very much like the one that he chose not to sign the other day I'm going to suggest that we uh don't do it fully live here because I'd like to absorb a little more information then um I can probably do fully live. Uh I think we could be a little more effective with at least a little bit of preparation, but that'll definitely be on the agenda for tomorrow.
» Um anything else you want to cover today before we » um No, » take our next step in the recursive self-improvement process. » No, it's it's been a pleasure. And today today is going to be clipping day. So we're going to we're going to see some clips come out today. So » cool. Well, thank you Pash. Thank you for everybody who's joined us for AI in the AM. We are sprinting through the AI marathon. Uh thanks for getting a couple more pace miles in for with us today. » Thank you. Bye, Nathan. See you
