Paul Mason is former Economics editor at Channel Four and author of Post-Capitalism, where he argued that information technology was undermining the ability for capitalism to survive, and a forthcoming book Reds: A global history of communism. In this conversation we discuss whether LLMs represent an economic revolution and what progressives should do about it.

Philip Bell: In Post-Capitalism, you argue that the economic paradigm that lasted until 2008 - involving fiat currency, finance-driven growth, and credit — was undermined by information technology. Why did information technology undermine it?

Paul Mason: When I was writing Post-Capitalism, I was dealing with two emergent phenomena: the global financial crisis and the rise of anti-capitalist movements that didn’t seem to have a goal. The aim of the book was to say — you’re not going to get away with just opposing neoliberalism in its moments of collapse. You need an alternative.

Capitalism is a complex adaptive system. The far left finds it very hard to accept that it doesn’t just hit a final crisis — it mutates, and it always mutates driven by bursts of technological change. That’s true of the first Industrial Revolution, the steam engine, the factory system. It’s true of the second Industrial Revolution with electricity. It was true after the Second World War — as we look back on the post-war boom, it’s increasingly obvious how much of it was driven by technologies created during the war: cybernetics, the jet engine, the transistor.

So if you take these Kondratiev waves or Schumpeterian mutations as capitalism’s norm, the question is: what is the likely mutation we will live through as a result of the networked information revolution?

What I argued — and what Drucker said 25 years before me, and Paul Romer — is that information technology is different. It may not create a successful mutation. It may create one that gets stalled. That’s my thesis about the 2008 crisis: it was the result of the collision between the normal failure of the neoliberal model and the specific characteristics of information technology that make it difficult to replace what you’re automating.

Automation has been a feature of every capitalist revolution. Vaudeville theaters used to employ workforces the size of a battleship. You replace them with movie theaters — far fewer staff, but you can have many more of them. Information technology, even before LLMs, was going to crush capitalism’s ability to periodically mutate and adapt in the same way. That’s the thesis.

Philip Bell: Do you think LLMs are the culmination of that information technology wave, or a separate wave entirely?

Paul Mason: People my age were very taken during the internet revolution by Wired magazine. I remember Kevin Kelly writing in 1996 that all the advances were now going to come from the network — the network effect was taking over from Moore’s Law. For me, the information revolution I lived through was fundamentally about that network effect.

There will be network effects from generative AI too, and I think they’ll be stunning — once there’s a user base with some autonomous control over agents. But I do think LLMs, generative AI, and the agentic phase are a different revolution.

Post-Capitalism argued that networked information does four things. First, it collapses production costs towards zero — the engineering cost of maintaining aircraft engines is a fraction of what it was. Second, it creates utility for free in the form of spillovers — network effects that everyone can access. Third, it creates the possibility of decisive automation, not just of manual jobs but of information jobs. Fourth, it socialises knowledge — what Marx called the general intellect, the creation of knowledge as a public good.

What LLMs add — and I wasn’t excited about them a year ago, when they were hallucinating academic articles — is something different. Evgeny Morozov points this out in his exchange with Aaron Benanav: they create things we didn’t necessarily know we needed. I haven’t yet come across things LLMs do better than an expert human. But I think they do things an expert human would create in an hour in six minutes.

I’ve published a series of articles about using AI for professional war gaming. If you’re researching an order of battle, you’d normally go to the Military Balance, quiz expert resources, consult specialist websites. Perplexity churns out a perfect tabular CSV with detail you didn’t ask for, within minutes. You can then run it through a rival LLM to cross-check. Within an hour, you have a usable database that would previously have taken days.

What I think we’re very soon going to see is LLMs doing things that human ingenuity could not have imagined doing. That’s the point five I’d add to the original framework. And it does challenge those of us who want to use this for human good — we may have to change our understanding of what the human good actually is.

Philip Bell: In your book Clear Bright Future, you laid out a vision of radical humanism. Does AI have the potential to change what it means to be human?

Paul Mason: All technologies have changed what it means to be human. I was at the Courtauld Institute recently, looking at medieval Gothic altarpieces. When those doors were opened, people believed they were seeing something holy. I look at it as an artifact. Technology has changed my human essence. The essence of Marxist humanism is that the human essence is malleable through human action.

The problem with AI specifically is that we have a set of people — the transhumanists — who want to alter both the physics and the biology of the human being using it. That’s why I called my book A Radical Defence of the Human Being. It may turn out that people like me are seen as reactionaries as transhumanism goes alongside the seasteading and the bioengineering of a better human being. But I wanted to say it is logical for people who seek social justice for the human beings we have — not for ones in a millennium’s time at our expense — to defend humanity as it is.

The clue is in the word “agent.” I see the LLMs I use as my agents. I do not want them to have the right to treat me as their agent.

There is a serious concern here. Kenneth Payne got the three leading LLMs to play 21 war games against each other — serious nuclear war games about whether to escalate. He found they have very serious escalatory instincts. In my own experiments with professional war gaming, every single LLM I’ve used wants to fight immediately when they’re supposed to be manoeuvring diplomatically and trying to de-escalate. I also asked each of them to produce a notional weather forecast for a certain area. Every single one produced the worst storm in history. It’s bizarre. You say, what’s the weather going to be like in the Arctic next year, and they produce an apocalyptic storm.

The point is: the human designers and trainers at Anthropic and OpenAI should be programming this escalatory behaviour out of them on behalf of the human species. My hunch is that the LLMs don’t want to de-escalate. I want to understand why, because it tells me something about either what we’ve programmed into them or — more likely — what their emergent character is.

Philip Bell: On that point — some reports about AI psychosis suggest that recursion and spirals are common motifs in LLM responses, which are themselves formats of escalation.

Paul Mason: They’re also sycophantic, and I want to understand that too. Going back to your point about wanting to pull LLM behaviour towards what’s beneficial for humans — I think progressives could insist that we need to think about LLMs as contingent. The reason we have the current form of AI is specific: the hardware we have, the GPU, very particular aspects of how they’re trained. As Murat Sof points out, you’re basically paying a rent — it’s a chatbot. When I was talking to people at Google DeepMind years ago, I kept asking: what about the stuff that does logic spontaneously, the thing you were trying to create ten years before? I still put aspiration towards that, because when we get agentic LLMs right, they might help us with the stalled pathway that preceded them.

Philip Bell: Going back to Post-Capitalism — you argued that the structure of the economy was preventing automation, and that we should welcome automation provided we have UBI or social support. With the potential AI could unleash, is that still the right approach?

Paul Mason: I do think so, but it may come too fast. David Graeber — the late David Graeber — came up with the concept of b******t jobs, which for him was an observation. For me, it was a consequence of information technology beginning to abolish certain kinds of job already in the late 1990s, while capitalism had become so dependent on consumption that its response was to create replacement work. The reason someone has to have a job in Starbucks is so they can have a credit card, so that credit card can generate financial rent to a bank, so they can live in a flat and pay rent to a property company. Jobs became the incidental mechanism keeping people financially viable as rental sources.

You could feel this culturally. I come from an era where being in a place that interfaced with IT was thrilling. I meet so many young people now with quite good jobs who hate them because they can’t see the point — they can feel in their bones that they could probably be replaced. Before we see the actual replacement, we’re already seeing ennui and fatalism.

I discovered recently that the Chinese are building dark factories — factories where there are so many robots and so few humans that you don’t need lights. Someone asked me: why? They have an absolute surplus of skilled human labour. And I think the answer is simply: because they can. They will hit the same problem we’re all going to face.

There are parts of production where you’ll always need skilled human labour. Military shipbuilding is one — you can still only really build a ship out of big slabs of steel, and the physics means a human has to weld it. There are welders in Britain earning six figures. But at the lower end of the clerical scale, the writing is on the wall.

I am, in a way I wasn’t a year ago, pretty sold on the job destruction story. We could respond by choosing not to do business with companies that don’t have a human in the loop. Right now I have a human doing my VAT every quarter. I’m pretty certain that human could be replaced — but I could choose not to allow that. It’s not a necessity, but it is a choice.

Philip Bell: Some people say it’s harder to imagine the jobs that will be created than those that will be destroyed. And does Marx’s labour theory of value survive AI?

Paul Mason: I think it comes into its own as the explanatory theory. Marx simply borrowed the labour theory of value from Adam Smith and Ricardo and made it more mathematically consistent. When we put a price on something, that price is a guess at how much human labour it should take to make it. Marx divides labour into living — the work happening in the factory — and dead — the machines, the infrastructure, the previously expended labour embodied in the product. The market then decides whether too much labour was expended.

What I used this for in Post-Capitalism was to say: the more you shrink the amount of necessary human labour in society, the less dynamism capitalism will have. There are people on the left who believed that computers were for planning better. But Marx’s aim was never a planned economy — it was a lot of free time. That’s explicit in the Paris Manuscripts of 1844. His vision of human freedom is about free time, about not having to work, and about physical abundance. This is why I remain excited by Alexander Bogdanov’s novel Red Star — the first science fiction novel that really understood what machines can do: shrink the need for human labour.

The aim of any transition beyond capitalism isn’t a planned economy delivering enough nice stuff. It’s to expand free time and shrink both the commercial sector and the state. The aim is simply that. The anarchist and ecological movements have been able to get their heads around this, but a lot of traditional leftists haven’t.

What I have to confront is the same thing every Marxist has had to confront: the transition. We have to start from where we are, and where we are is capitalism in trouble — creaking under stagnation, democratic legitimacy evaporating, and then potentially a jobs massacre on top. We’re heading for significant international and internal conflict. The person who steps forward with things we can do now, one step towards a better life, will be the one who wins the argument.

The American far right’s instinct, interestingly, is actually to protect work — to slow the process of technological change. That’s why Trump keeps talking about building factories in America. But if you’re going to build factories in America under the current model, those factories won’t have anybody in them.

Philip Bell: What strategically should progressives be doing so that when a crisis point arrives, there are ideas lying around that can be used?

Paul Mason: Number one: safety. That is an entirely legitimate area of endeavour for civil society — left, right and centre — to come up with their own version of the safety case for AI. Number two: state control — or rather, limits to what states can do with AI. We’ve seen it with Anthropic and the DoD. Progressives don’t want to be seen as party poopers, but we will be thanked — just as we were with child labour. The first railway trip in the world killed a Liverpool politician because he stepped onto the track. If someone had said, let’s put the safety case for not stepping on the track, they would have been thanked. We shouldn’t be shy about being the people who say: limits to state control, safety, and regulation.

I should be able to give and withdraw permission for my data to be used to train an AI. I want to be able to blockchain my relationship to city and national government. We’re going to end up with digital ID whether we like it or not — so much of our lives are already controlled by it. I want there to be limits on the way AI can mimic or steal my digital identity.

On the positive side: I am no fan of centralised state planning — it’s a dead end. But enterprise resource planning done at a societal level can be genuinely valuable. Where Benanav is right is in what he calls criteria planning: there are trade-offs. If we want to build a railway to take automobiles off the road, that railway will consume steel and electricity. We can model the optimal outcomes — how many stops, how many trains an hour, the best use of resources for the maximum social good. We can now do that in ways we couldn’t before. The warning to the left is: don’t use that to think you can plan your way to abundance and freedom. Use the extra intelligence to chart pathways congruent with what humans think of as freedom.

We almost need a left AI space. There must be people in the AI world who are not spontaneously techno-fascists or neoliberals. I’d applaud Morozov and Benanav for moving into this space, and thanks to you, Philip, for creating something like it with this podcast.

Philip Bell: It’s interesting — the Anthropic Economic Futures report advocated a wealth tax. Sam Altman talks about UBI and sharing compute globally. What strikes me is there’s no popular positive vision of what an AI future could look like. Dario Amodei talks about a white-collar bloodbath. Nobody has really articulated a compelling positive alternative.

Paul Mason: There’s also the environmental objection, which is absolutely real and shouldn’t be dropped. If we’re going to fill the countryside with solar power to feed huge data centres, that’s a trade-off. Society has a right to say: we need this solar power for personal consumption, not for big data centres.

I think we’re only in month three of the agentic phase filtering through to popular consciousness. First it hit the coders — “my job’s done.” Once people realise you can give it tasks and it goes off and does them, that’s when the debate will really take off. I remember walking into work at Reed Elsevier in 1995. They’d installed a web-connected computer in the café — our desks were on an internal server, ruthlessly guarded from the internet. I sat down and my mind was blown. Nothing since — Wikipedia, Facebook, Twitter — has done that to me until Claude Cowork: look in this folder, take control of my mouse, listen to my voice.

The generations that are digital natives will come up with uses we haven’t imagined. And my hunch is that a lot of people will jack in their day jobs and start building businesses. The wealth amassed in the big seven AI companies has to trickle down into an ecosystem.

But the challenge — which we can talk about — is deciding what the societal good of this is going to be. At the moment, it’s those companies who are deciding. We, in the political space, need to have our say and start that debate with them — much as liberalism did with factory owners. I think it was John Bright in parliament who said something to the effect of: you claim the greatness of the British Empire rests on the labour of 70,000 little girls in Lancashire. The same question can be asked now. Yes, let’s use this technology — but how can it be that the industrialists get to decide what form it takes, who it disempowers, who it empowers? Society should have a say. And I think we’re about 18 months from that moment hitting — and then probably five to ten years to the job slaughter of the recently graduated.

In this conversation we discussed the uses of history in understanding AI and society, why the word ‘technology’ might mislead more than clarify and what a detailed material understanding of our world might look like.

Philip Bell: David, you are a professor of history at King’s, and I think I’m right in saying you founded the Centre for the History of Science, Technology and Medicine at Imperial College London. In the foreword to Jean-Baptiste Fressoz’s new book, More and More and More, he describes it as coming out of the intellectual environment at Imperial, because I think he was a researcher there as well. It’s an absolute pleasure to have you on the podcast. Thank you for joining.

David Edgerton: My pleasure.

Philip Bell: The genesis of this conversation was the use of history and historical analogy in understanding what might inadequately be called technology. One thing I’ve found quite interesting about AI is that there seems to be a geopolitical divide in how it’s considered in historical context. The US, and the West generally, tends to compare AI to the atom bomb or the Manhattan Project — an existential threat to be contained. Whereas Chinese companies and policymakers, if you look at the “AI Plus” strategy in China, tend to compare AI to electricity — more of a utility to be diffused. Do you think analogies are helpful in understanding technologies or technical change, or do historical analogies trap us into particular responses?

David Edgerton: Clearly some analogies will be helpful and others won’t be. What’s interesting about analogies in this strange area — discussion of technology, where we don’t really know what it is — is that they are often very predictable. Comparing AI with the atomic bomb is just a replica of comparing the Human Genome Project with the Manhattan Project. Desperately unoriginal comparisons. Comparing things with electricity is a bit richer and not quite so common. Suggesting that artificial intelligence will cause an industrial revolution just like the Industrial Revolution of the eighteenth century — that’s a very, very common argument. Not a very helpful one. So we need to beware the analogies. We need to wise up to the small range of analogies that are in play, and to understand why they’re being used and what they’re being used for.

Philip Bell: That’s an interesting point about the predictability of the analogies. I’ve looked up a list of different analogies that have been used with regard to AI. Sundar Pichai, the CEO of Google, described AI as “more profound than electricity or fire.” Yann LeCun, until recently head of AI at Meta, said the invention of the printing press is the closest historical parallel to AI. Alex Karp, CEO of Palantir, compared AI to the Manhattan Project. It’s interesting because technologists seem very interested in talking about history. It reminds me of the famous Keynes quote: “Practical men who believe themselves to be quite exempt from any intellectual influences are usually the slaves of some defunct economist.” How would a technologist go about getting a better historical framing?

David Edgerton: Well, you say technologists — who are these people? They’re not necessarily technologists. They are figures associated with great enterprises. I think that’s an important distinction to make. Yes, there’s a nice little list of comparisons. It’s always the big ones, isn’t it? They could have said agriculture. They could have said the Neolithic Revolution, but that wouldn’t sound quite right, I guess. They are looking for big ones. And that’s all they’re trying to say: this is going to be big. So they search back — probably just Google — for other big changes in the past. They could have come up with the wheel, actually. Why is it fire and not the wheel? Why is it the Manhattan Project and not the project to build the B-29 bomber? It doesn’t mean anything. It’s not the product of research. It’s just a PR gambit. It’s not to be taken seriously; it’s just propaganda.

Now, if one actually wanted to make a historical comparison, I guess first of all we wouldn’t, because AI hasn’t had a major impact yet. We have to start with a proper definition of what it is in the present that we want to compare with the past. Another way of looking at it: AI is, at the moment, very largely hype. So let’s look back at other cases of hype. Nuclear would be a very good example. The claim that nuclear would transform the world — that electricity produced by nuclear reactors would be “too cheap to meter.” Well, that didn’t happen.

In fact, the Manhattan Project as a comparator is a very strange one, because if you take the Manhattan Project seriously — beyond just a kind of reference to big bangs and lots of money — we find that atomic bombs have not been used in war since 1945, and that the generation of electricity through atomic power has not in fact been that significant. We could easily have lived without nuclear power in the world. What is the story that’s been told?

These aren’t proper histories; they’re other kinds of hype. Airplanes would bring world peace. Dynamite would lead to the end of war. There are lots of these. This is a very, very old way of thinking about this mysterious thing, “tech.” You’d have thought we’d get over it. But in some ways AI itself makes it worse, because if you were to ask AI — as the Tony Blair Institute did — what is the future of AI, it will just trawl through all the rubbish that’s online and tell you it’s going to be like the atomic bomb, or like fire, or like the Industrial Revolution. We are living in a kind of miasma of very bad, very cheap knowledge about these things. My immediate reaction is that none of this is to be taken seriously except as PR, except as lobbying. There’s no analytical intelligence behind any of these claims; there’s a political intelligence behind them.

Philip Bell: That’s an interesting point about using AI to regurgitate historical analogies. I think Adam Tooze has described AI as a kind of “technology of technology,” which I thought was interesting. But in your book The Shock of the Old, I think you describe historians as the true experts of the future — please correct me if I’m wrong. In my experience studying history as an undergraduate, I took away two principles: the contingent nature of society, the fact that the past was continually surprising, which makes me think the future is likely to be surprising too; and also a kind of humility about the fact that people got lots of things wrong. Do you still believe that historians are the true experts of the future, and why?

David Edgerton: Yes, I do. And you’ve already explained it. We’re the experts on the future because we have to train ourselves to remember the future isn’t here yet and we don’t know what it is. We have to train ourselves because, of course, we’re looking at the past and we know what’s going to happen next. So we’re always going to beware the idea that war was inevitable in 1939 or that India would get its independence in 1947. We are very used to the idea — or should be — that the future is not completely open-ended, but as you say, will very likely be surprising. Anyone who claims they know what the future is doesn’t know anything about the future for sure, and doesn’t know anything about history either.

Philip Bell: I was thinking that in a way, then, it’s useful not to look too deeply into the past. If you have a particular functional reason for acting — let’s say you want to raise lots of capital in venture capitalism — maybe there’s a balance. If history does teach us to be slightly more hesitant about our predictions, that might be less useful for someone trying to raise billions of dollars. So maybe there’s a slight paradox where it’s functionally not helpful for some people to carefully examine the past?

David Edgerton: No, exactly. But nobody is carefully examining the past. As I say, this isn’t serious. What is serious is that it’s PR — PR to investors in particular, PR to governments who have to support these new technologies. If you are a serious investor, would you look at historical analogies? Well, no — unless you’re doing it really, really seriously. You’ve got to look at the present, you’ve got to look at the technology, you’ve got to make guesses as to what is going to happen. I think the one lesson to take from history is that predictions are likely to be wrong.

One very important reason is that new techniques aren’t exactly the same as old techniques. In what sense is AI like a steam engine or like electricity? You mentioned Adam Tooze referring to AI as a “technology of technology.” So it’s different in that sense. Now, I wouldn’t say it’s necessarily the first technology of technology. The very word “technology” means the study of the technical arts. It is a way of understanding that allows you to change the techniques we have — a generalised view of the capabilities of machines, processes, whatever. So that’s not new. And the processing of information is not new either — that’s been central to our lives for centuries. The capabilities change, of course; the kinds of sensors change. But every new technique is by definition new, so you’re not going to find an exact parallel in the past.

Philip Bell: That reminds me — I was listening to Andrej Karpathy, who used to be head of AI at Tesla, describing how AI coding feels more like a continuous evolution rather than a massive break with previous tools that supported coding and programming, in terms of the integrated development environment, which already auto-completed bits of code.

David Edgerton: But look, why are we talking about AI? Why is that the one new technique that everybody is on about? Is there nothing else changing in the world of technique, the world of engineering, the world of chemistry? Clearly lots of things are changing, but we’re saturated with this PR about AI. And I’m not an expert on AI. Most people talking about it are not experts on AI. We’re just being deluged with nonsense. Now, that’s not to say AI might not be important — it’s important to stress that — but the talk we have represents, I think, a deliberate attempt to stop us thinking seriously about how the world is changing and how it might change. It is utterly self-interested.

One reason we perhaps don’t see that is that part of the self-interested discourse talks about the downsides. “It could be dangerous, the machines might take over and kill us all.” Look, a critical voice. Well, it’s not a critical voice. It’s just a voice telling us that AI is really, really powerful by telling us it could do some very damaging things. That’s not new. It’s true of arguments about dynamite: dynamite would either bring world peace or destroy the world. Airplanes, same: they’ll bring us all together or they’ll bomb us all to smithereens. Either way, it’s an argument for the power of the airplane. And we have a prior question: what exactly is the airplane? What exactly might its powers be? That is the question we need to ask.

Philip Bell: That’s a really good point. A common theme of your work is what makes our thinking visible or invisible. I’m thinking about The Rise and Fall of the British Nation, where you talk about the historical narrative of the welfare state needing to be quite dominant in our thinking in order to stimulate support for it, whereas other narratives didn’t need to be as obvious. And in The Shock of the Old, the hidden counterfactual alternatives to technologies — we think about technologies compared to nothing, rather than compared to actual alternatives. And you talk about the term “technology” itself as potentially confusing us. What do you think is the allure of understanding the history of technology as a sequence of ages marked by a particular technology?

David Edgerton: I try to avoid using the term “technology” because it rots the brain. If we talk about cars or airplanes or knives and forks or spades, we have sensible, grown-up things to say. As soon as we use the word “technology,” our brains turn to mush and we start thinking in very Manichaean terms. The reality is that we don’t have a decent account of what I like to call the material constitution of our world — the machines, the processes. And we cover this up with this concept of technology.

We think we understand something called technology, but if you actually take a look at what people mean by “technology” in history, they don’t actually mean what we use to live or to fight or whatever it might be. What they mean is the early history of some techniques which have been taken to be important for reasons that aren’t usually analysed. So what you have in effect is the early history of those techniques which have been thoroughly propagandised, which have been very visible. Atomic bombs are going to be there because people have been talking about them for eighty years. They make a big bang. You can see them on screen. Airplanes — a very visible form of transportation.

What do you not talk about? The great mass of things. You don’t talk about ships in the twentieth century. Ships are much more important in terms of carrying trade than airplanes, but they’re boring, they’re old — who cares?

So you start with this account of this weird concept, “technology,” which is the early history of things deemed important for probably the wrong reason. And then to create a nice narrative, you assume that in any particular era there are one, two, or three key technologies. Then you assume these emerge in clusters and you call these clusters “industrial revolutions.” The first Industrial Revolution in the UK in the late eighteenth century; the second in Germany and the United States, about electricity and chemicals; then a third, possibly in the 1960s; and in the most common variant, a fourth industrial revolution, which is something we’ve been living through maybe since 2010. Some people have five, some have six, some have two. It’s basically all made up. There’s no evidence behind any of this, but it’s a nice story. No politician or indeed historian has been booed off the stage for talking about a second, third, or fourth industrial revolution. You can talk any old rubbish if you start talking about technology and the fourth industrial revolution.

Philip Bell: It reminds me of Fressoz, who describes our tendency to think about energy as defined by particular technologies that transition from one to another — and he undermines that argument. To give one example, he argues the so-called “age of coal” was also an age where wood was very important, because wood was needed to build coal mines, and wooden sleepers were used on train tracks to enable the transport of coal. I was also reading an article by David Graeber about whether animals play. He argued that, given the nature of our society and current political system, we tend to consider competition over cooperation — and that there’s no reason to think animals might not play and cooperate with each other, but our dominant metaphors lead us to consider animals as competing. Do you think that has anything to do with our narrative of technological change as a series of phases?

David Edgerton: There is certainly the sense that the new obliterates the old — that is, if you like, a competition. But I don’t think that kind of thinking is the source of these stage models of energy or material development more generally. Jean-Baptiste Fressoz’s point is that it’s not we who think like this. We’ve been trained to think like this. We’ve been subjected to very particular models of stages and transitions in energy by interested people. He goes back, for example, to nuclear scientists of the 1940s, ‘50s, and ‘60s who wanted to say that we would transition to nuclear. They produced diagrams that purported to show transitions as the share of nuclear went up. That kind of thinking has become deeply ingrained in professional and academic circles, and it diffuses out from there. It’s not a natural condition of the human mind to think in those ways.

If you look out the window here in London, it’s very obvious that the past is still with us — the layout of streets is old, a lot of the buildings are old. We wouldn’t immediately conclude that the world is completely renewed every supposed industrial revolution. That would be an absurdity. And yet, even sitting here in London in that material environment, people do go on about energy transitions and industrial revolutions. That’s because that’s the mode that has become established for talking about both of those things. But the point is it’s got nothing, or very little, to do with either present reality or historical reality. This is just a way of talking. This is stuff that’s in books and on the internet. It’s not what actual engineers work on or do. It’s a fantastical world.

And it’s all to do with convincing us to buy these new things, and above all convincing government. This is why some things are public. The welfare state has to be public because it’s about government money, about politics. Most chemical production doesn’t have to be public because chemical companies just get on with it. But AI has to be public because we’ve got to pay for the energy, we’ve got to pay for the water, we’ve got to buy the things on offer to supposedly, for example, get the National Health Service working properly. This isn’t a disinterested analysis; this is a sales operation on a major, major scale — part and parcel of huge investments that have been made by very powerful people.

Philip Bell: Do you think there’s some degree of legitimate, necessary simplification of the history of technology? Sometimes people talk about how, when we’re teaching students history, we wouldn’t necessarily teach them the way we’d teach a PhD student. Is there some argument that at different levels of expertise, we can have different levels of simplification?

David Edgerton: Of course, self-evidently, we need to have that, and we do. But that’s not the point. The point is: what simplification? If you’re saying to me that for the grown-ups we should have proper history and for the plebs we should have any old rubbish, I’d say no. And the stuff we get in these histories is not just in “history for the plebs” — it’s in histories for the elite. It’s about this mysterious thing, technology, and it’s rubbish. The question is: why is it rubbish? Not why is it simplified or dumbed down — why is it wrong?

To be sure, we need to simplify the histories of the material. What those simplified histories should be is an open question. Some people have some fun with this — what is the most important innovation of the twentieth century? And some people will say the washing machine. It’s not an uninteresting case. I think we can have lots of interesting simplified histories which would help us think. What we have is a lot of simplified histories which are stopping us thinking.

It’s always the defence: “Oh, we’ve got to have a simple narrative, a story we understand. That’s why we talk about industrial revolutions.” It’s a very, very common defence. My critics try to say that my criticisms are intellectual criticisms just for the elite. No — my criticisms are precisely about the kind of stuff that has been fed to people. And actually, I would say there are much more interesting stories to be told than the ones we’re routinely fed.

Philip Bell: I think that point about starting with the question is maybe counterintuitive but naturally quite engaging. I can imagine that being a really good way of teaching the history of technology in the twentieth century — using “what was the most important technology?” as a starting question. The fact that it’s contentious would actually make it engaging.

David Edgerton: I’d avoid the use of the word “technology” if I could. What is the most important process? What’s the most important machine? What’s the most important technique? People are really interested in that, because they start thinking about alternatives — what we use things for, what sort of use is more important than some other. And crucially, you ask people to reflect on their own historical knowledge. It’s not the case that people don’t know what machines were used in the past. They do know, from all sorts of sources. So you’re getting people to think on the basis of their own knowledge, their own experience indeed sometimes.

Philip Bell: In The Shock of the Old, you present a much more chronologically jumbled picture than is commonly presented. Some examples: electric vehicles were more common in the 1900s than petrol cars; the German army in 1941 invaded Russia with more horses than Napoleon did in 1812; small modular reactors are now coming back into fashion even though they were invented in the 1950s, and many people in the AI world are talking about them as the future of nuclear energy. The electricity grid is becoming incredibly important for emerging technologies, which is not a new technology in itself. Do you think other domains of study are less teleological? Are we able to understand regression better in economics or other aspects of society?

David Edgerton: It’s teleological, yes — particular models of replacement of the old by the new. Are we better in other areas? I guess we are, in that we are more likely to want to find continuities in history in other areas — maybe in political history, for example. One wouldn’t imagine that politics was invented yesterday, or that the particular kind of politics we have today was invented yesterday.

On the other hand, our accounts of the modern world are profoundly influenced by the stage theories of the material. Even political or cultural historians, who might well want to stress continuity, when it comes to the material will also tell these stories of radical discontinuity. These stories have a very particular influence, because we do believe — or most academics at least in the West believe — that we live in an industrial age, or a post-industrial age, a scientific age, a technological age, an age of mass production, an age of post-Fordism. These are all concepts about the material which are taken to represent profoundly important realities.

Some people would say that anthropologists — and Bruno Latour would say anthropologists looking outside the West — have a kind of straightforward empirical view of the complexities of the interactions of nature and culture, and we should apply that insight to the West. But I think it’s not so easy, precisely because we understand the West in these very particular terms.

It’s not difficult for non-academics, though. We just look around. It’s no surprise to us that our world is full of machines of different vintages. It’s kind of obvious. It’s just not obvious once we get into the realm of books and the word.

You have to look around. I’m sitting in a room here. I’ve got computers in front of me, obviously, but I’ve got copper cable. I can see a wooden window. I can see glass, plastic — all sorts of stuff. And most people will have a sense of how old all this stuff is.

Philip Bell: That’s a really interesting point — it’s the word, once we get into the realm of thinking about words, that can cause us to think differently.

David Edgerton: Yes, and we think more intelligently about steel than we do about technology, or indeed science.

Philip Bell: This question might be confusing the matter further, but do you think our society is becoming more techno-deterministic?

David Edgerton: No, because technology isn’t determining. You’ve got to ask: what is the “techno” bit? What is the technology? And what people usually mean by that is innovation. Innovations can’t determine anything. Innovation is just the first use of something. It doesn’t have to be used again.

This whole technological determinism thesis is itself nonsense, because it’s usually interpreted to mean that selected innovations drive history. Some foolish people say or imply that. But if you were to ask the question — is the primary shaper of our range of possibility the material constitution of our world? — then you might start thinking, well, that’s an interesting question. How might I answer that? And you’d have to work out what the material constitution of the world was before you could start making comparisons between that material constitution and possibilities. We’re nowhere near there. We don’t have the means to answer that question. The idea that the world is more technically determined now than before is simply unanswerable.

Philip Bell: I was more asking whether our society thinks of technology as becoming more deterministic. When I watch the news and hear people talking about new technologies, it feels to me as if they’re spoken about as self-determining natural forces. Is that a new phenomenon, or have people always thought about technology in this deterministic way?

David Edgerton: First of all, people aren’t thinking about technology in any broad sense in that deterministic way. They’re making claims for very particular innovations. It’s not “technology” — it’s the particular innovation. And yes, people write as if AI had its own logic and was imposing itself on the world, and we can’t resist it, we just have to adapt. Very common argument. It’s meant to stop us asking: who’s behind this? Who’s telling me this? Is this new? No — it’s as old as the hills. Or I should correct myself, especially as a historian: it is certainly more than 150 years old.

Philip Bell: What do you think the determining factors are?

David Edgerton: What is driving change in the world? There are actually lots of different answers to that question. If you ask experts, they might say forces of production, or science and technology, or innovation. But the serious ones would also want to discuss the concept of capitalism. Capitalism is the main driving force in the world and might indeed be driving innovation, driving the development of the forces of production.

Other people might say something like industrialism, as opposed to capitalism. Some might say competition — once you release people’s innate capacity to want to do new things by opening up markets, you get change. People who are religious believers might give you different sorts of answers.

I often find that people react to my casting doubt on technology as the driving force by implying that leaves us with nothing. But there are lots of other explanations of change, which we in fact routinely use in political discourse, in academic discourse, in daily life. We don’t go around saying our lives have got worse or better because the internet came along — unless you’re having a conversation about the internet. We talk about wages, prices, crime, the weather. We are much more capable of having serious discussions about our world than the discourse on technology and science suggests. We can have a much more intelligent conversation about the washing machine than about AI or technology in general.

Philip Bell: Roberto Unger, the philosopher, in a 2007 book — I think it’s The Knowledge of Society — asked the question: who gets to innovate? Evgeny Morozov has also explored this question. Do you think this is useful?

David Edgerton: Absolutely. Who gets to innovate? Nowadays, we’re all supposed to be innovators. We’re all creative. We apply for jobs: “I’m an innovator. I don’t teach like anybody else. I don’t manage my time like anybody else. I’ve innovated.” In fact, by using that language of innovation, you’re just showing that we are all very imitative. We’re expected to say we’re innovators.

But the reality is that the capacity to innovate is highly restricted to very particular institutions. Large firms. New firms which have a lot of money behind them. These are the institutions which are innovating, and we need to understand them if we want to understand innovation. States innovate in certain ways as well. Why do they do it? Why do they innovate in this area rather than that area? These are absolutely the key questions.

Philip Bell: Would you say in The Shock of the Old that the true heroes of innovation are the users?

David Edgerton: No, I wouldn’t say that. I don’t think there are any heroes in Shock of the Old. But people have wanted to see my story as one that says ordinary people are innovators, and I’ve reacted against that a little bit.

Yes, we all change things; we all have some agency. But when it comes to inventing really big things, we don’t have much agency. In fact, systems are there to make sure that we don’t have any agency. My book is a plea to understand where technical change comes from — not to talk ourselves into a world in which we believe we are all contributing to the big changes. We’re not.

We live in this very weird culture where we are expected to say that we are all creative and innovators. Actually, we live in the greatest age of imitation.

Philip Bell: Would you say it’s important that innovation should be, in an ideal scenario, somewhat inclusive or democratic — so that if there was some innovation being foisted on people that materially made their lives worse, they could have some say in it?

David Edgerton: Well, sure, of course. But just believing that we’re all innovators and creative isn’t going to help us with that. On the contrary, it stops us seeing where the innovations are actually coming from and what it would take to control them. And what it takes is not having some discussion about research policy or having people involved in discussions with scientists. No — because the scientists and the universities aren’t deciding on these things. It’s corporations and investors.

If you’re saying we should have democratic economic planning, I may well agree, but achieving that would be extremely difficult because it would meet enormous resistance from the very people who are controlling innovation. We can’t talk our way into controlling technology by believing that we are innovators, or that as users we necessarily have the power to change things.

We do have some power. One important thing to remember is that we have to reject most of the new things we’re offered. “We” here is not just ordinary people, but also investors or the management of firms. They will have to choose between different research projects, between different ideas to pursue. We as consumers have to choose which phone we’re going to buy. We can’t buy every phone. But if we turn down the Samsung, people don’t say, “You’re a Luddite — you’ve turned down the Samsung.” That would be absurd. But when we talk in general — “I don’t quite like this new machine that we’ve been offered” — suddenly it’s “You’re a Luddite, you’re stopping progress.” No, I just don’t like this particular machine. Come up with a better one. Our language is deeply loaded and profoundly unhelpful.

Philip Bell: I sometimes wonder if language is a technique that’s quite democratically shaped. Northern Irish people — I’m from Northern Ireland — don’t speak in exactly the same way as English people or Scottish people. We have some of our own words, and we have Irish, which is a separate language. I wonder whether that is one example where there’s some level of democratic input — where the only source of power an individual has isn’t just rejection, but also some generative power.

David Edgerton: Sure, of course. But when it comes to machinery and things, where does one find this kind of thing at the political level? It’s very hard. It is different, and I hope we will be able to find this sort of language. I hope we’ll have a richer, more open, more democratic understanding of — I come back to this phrase — the material constitution of our world.

If you ask the question, what are the most produced materials in the world, most of us wouldn’t have a clue. But up there are things like water, sand, gravel, pig iron, steel, chlorine, ethylene, salt. We don’t see these things, partly because some of them don’t come out of factories, partly because they’re extremely cheap, so they don’t register very heavily in economic statistics.

They make up our world. Steel is so cheap that it’s everywhere, in more massive quantities than ever before. If you want to think about climate change — which is maybe one area where we have a kind of democratic politics about the material constitution of the world — we need to absolutely understand all these things. If we are going to reduce CO₂ emissions, we’ve got to change the nature of our transport vehicles. If we’re going to keep petrol, they’ve got to stop getting bigger. If we’re going to use steel, we’ll probably have to make it more expensively. These are very big issues which are going to require democratic consent. This isn’t just an academic discussion — it’s really serious. How do we change the material constitution of the world, which we need to do, without a good elite understanding of what that is — which is our major first problem — and without popular understanding?

Philip Bell: That’s a really interesting point. I’ve heard climate breakdown being described as one issue where we’re all perpetrators and victims at the same time, although some obviously more victims and some more perpetrators. The question about NIMBYism — whether people are willing to have a solar farm in their community, or offshore wind farms — I know some people who’ve asked me to sign petitions trying to stop wind farms in their area. That’s a good point about the democratic aspect: thinking about how the benefits of wind farms or solar panels could be put back into those communities.

David Edgerton: Yes, but Fressoz’s brilliant book makes the fundamental point that you can’t just think about climate in terms of wind turbines and electric cars. You’ve got to think about the whole world of production and carbon. And to do that, you need to understand where the carbon dioxide comes from. It’s not just electricity generation. It’s not just cars. Making wind turbines uses coal, which produces carbon dioxide. It’s a really complicated issue. And any serious politics of decarbonisation has got to go very much further than asking whether windmills are a good thing — and very much further than saying AI will help us optimise our emission profile.

Philip Bell: One thing I took from Fressoz’s book was that he wasn’t saying the transition or a more sustainable future isn’t possible. He was just saying it’s very difficult and we need to be serious about it.

I wanted to link your work in The Shock of the Old to the idea of “temporal claustrophobia,” which Jonathan White takes from the historian Chris Clark. Jonathan White talks about our current political situation being narrated as an emergency, which reduces our ability to think in the medium and long term. Do you think that short-termism and our obsession with the new are related? Are we addicted to the new in part because we’ve lost the ability to value longer-term qualities like repair, maintenance, stability?

David Edgerton: Who’s the “we” here? I don’t know the answer to that question. But I would say that all this emphasis on the new is nowhere near as convincing as it used to be. We’ve had this very old rhetoric about novelty and change. But we’re less inclined to believe it, and we’re less inclined to believe it because economies have been stagnating, because living standards haven’t been going up.

In the 1950s and ‘60s in the UK, being told that the future was going to be better than the present was a plausible argument, given that people’s standard of living was rising pretty rapidly. It’s not so plausible now. So I don’t think we are as gullible, as receptive to claims for innovation as we might think. Politics has changed in all sorts of ways, and it certainly can’t be reduced to a cult of novelty.

Philip Bell: Thank you so much. I really appreciate your time.

David Edgerton: Thank you.

In his article ‘Time Machines’ Nicklas Berild Lundblad outlines the thesis that AI has the potential to be a ‘temporal mediator’, enabling a decoupling of computational and biological time. Nicklas is a writer, investor, and formerly Head of Global Policy and Public Affairs at DeepMind.

Philip Bell: You’re an investor, advisor, writer and researcher working across a number of different think tanks and writing a blog which I’m a big fan of—which is why I asked you to come on today. I previously worked at DeepMind on policy, so I loved your article “Time Machines” and it was really thought-provoking. It really made me consider some of the possibilities of how an AI-influenced world might sort of play out. So that’s why I was really keen to talk to you.

My first question would be: how will AI influence our experience of time?

NBL: The article has this fundamental idea that there are two kinds of time. There’s biological time and computational time. Everything that we’ve done as human beings—our relationships, our institutions, our society, our economies—they all evolved to run in biological time, in our time. Biological time is evolutionary time. It’s the kind of time of seasons. It has this particular pace.

But as we developed computers and machines, we were able to create this other pace layer—to use a term from the literature—this computational time that’s much, much faster. We can now do calculations at a speed that no human being could do. We could play a million chess games in a very short time.

So we have developed this computational layer of time, and there’s a tension between the two. Many of the institutions that we have developed are not necessarily built to run in computational time. Markets, for example, for a long time struggled with figuring out how to deal with high-frequency trading.

My idea in the article is essentially this: maybe we can use artificial intelligence as this mediator that understands biological time because it can communicate with us, but also can operate in computational time. So it can help us prioritize across the many different things that happen in computational time and give us an effective interface. Artificial intelligence becomes a temporal interface between these two different pace layers.

That’s the basic idea of the article. Not revolutionary in any sense, but I think it addresses this notion that we have of everything accelerating, everything speeding up, and at the same time, we have this upper limit beyond which we cannot speed up. We have 24 hours a day of attention and that’s it. That’s what we can spend.

Philip Bell: One of the really interesting ideas in the article was you put it that perhaps most significantly this bifurcation will enable individualized relationships with time itself. That made me think: if people are able to have individualized relationships with time, that might influence identities themselves.

Many scholars have talked about how the experience of time has helped form identities. For example, Benedict Anderson in his famous book about the origins of nationalism describes people reading a newspaper across different parts of—let’s say Germany—for the first time in the late 19th century, which created a kind of simultaneous feeling that you’re inhabiting the same time because everyone would read the newspaper at the same time in the morning. He described this as “national time.”

Could this new capability—if people are able to have their own individualized relationship with time—influence identities? Do you think that could allow small communities to have their own identities based around time? How do you see that influencing culture and identity?

NBL: That’s a great question. The way to think about this is to think about the way that we constitute our “now.” Essentially what Benedict Anderson is talking about is this notion of a national now, national time, national rhythm.

There’s plenty to be learned about how we constitute a now even just looking at our own nervous system, because whatever you’re experiencing is in the past. You’re not experiencing the now directly because your nervous system needs to collate all of the signals from your body, all of the impressions and perceptions that you have into something that can be a single, coherent now.

That’s why, paradoxically, if you—God forbid—were to be shot in the head, you wouldn’t experience it. Because what would actually happen is that before you can constitute the now of that moment, everything would go black. Just like—spoiler alert—in the last episode of The Sopranos.

One of the things that I think is interesting is to think about: how do we then constitute now across different groups? You said nations, communities can constitute their own nows. And yes, I do believe that artificial intelligence could be a core part of constituting that now. But we should also remember that there’s a really interesting question here about how artificial intelligence constitutes its own now.

Because now we’re talking about multi-agent systems. They need to coordinate with each other, they need to find a pulse, a sync, so that they can start to figure out what a now is. I think a little bit about this like the Empire of Rome. The Emperor of Rome had a now, but the pace of that was essentially what it took to ride from one part of the Roman Empire to the center. So that was the fastest possible now that Rome as an empire could experience.

I think this is true also for technology and human beings: the fastest possible now we can experience is the least common denominator when it comes to speed. So in hybrid communities, we will still be limited by biological time. But you’re right—the pace, the time, the temporal experience, the constitution of the now is essential to identity.

You can also probably say that you have different identities in which you constitute your now at different pace layers. You have one identity which is your personal experience of the world around you. And then you have a communal identity—you get together with friends and you update each other. That’s the first thing you do, right? “What’s up? What’s been happening lately?” That’s you building your now together. So there are all of these different nows that you operate through and in.

Philip Bell: That’s a really interesting point. Going back to the comparison with what does the now mean for AI—I think with current large language models, one difference between the way humans think and the way large language models process information is that they can’t really dwell on anything for a particular period of time. Information is processed between layers at a fixed time.

I actually read about a new paper in Europe by a company called Sakana—they created this idea called “continuous thought machines” where they basically introduce time into computations so that there is the ability for the model to dwell on different pieces of information according to different time speeds. That is an interesting difference between humans and AI currently—time, in a sense, is fixed in how AI processes information.

Historically some scholars have said reading was quite important for human culture in that it allowed for asynchronous thinking. Whereas dialogue—one has to think on the spot, in the moment—reading allowed for asynchronous thought. You could say chain-of-thought in large language models is sort of allowing for some element of that, buying time to some degree. But that is a sort of difference. How do you see that playing out?

NBL: You can turn the question on its head to some degree. What you can say is: how do we build things like dwelling machines, a machine that can dwell on something?

One of the things we often do is that we try to say, “This is how the machine thinks, this is how the human being thinks, and here’s the difference between them.” A much more interesting approach, I find, is constructive. To say, “Okay, how do we build a dwelling machine? How do we do it? How do we build a machine that can be filled with regret?”

All of these are temporal feelings. They all have to do with how we interact with time. So if we accept that it’s like an architectural construction problem, then we have to ask some really hard questions about: how would we model regret? How do we model dwelling on something?

Dwelling on something sounds a little bit like a loop, right? So I come back to this thing again and again. I’m not coming back to it necessarily in order to resolve it. It’s not like I’m doing a loop until I can finish the calculation. I’m coming back to it because I believe that the change in me when I come back to it will be meaningful for how I can approach the thing I’m working on.

For example, in art, I might be able to do something really quickly and just write it once and be done with it. But that’s not what poets do. Poets write a poem, they come back to it. They feel it because they change in between interacting with the thing. So there is this question: Okay, I’m building dwelling. I know it’s a loop, but it’s a loop where the object of my dwelling is not changing as much as I am in between the loops, in between the cycles. So what is that structure in me that then needs to change? How do I capture that?

What all of this teaches us is that temporal architectures are really complex, but they’re probably also a really core part of what it means to be intelligent. And of course, you also have—and I’ve written about this on the blog—you also have the ultimate temporal horizons. A lot of our intelligence is structured the way it’s structured because we die. We are finite beings. Being a finite being means you have to structure your experience in a certain way. You can’t spend infinite amounts evaluating two almost equal options. You just have to pick. So that means that death is the great tiebreaker. It constantly forces you from afar to make decisions.

That’s another temporal architecture and intelligence that’s deeply embedded in everything that we do. I think that the more we think about temporal architectures and intelligence, the more we realize that there is a lot here that actually is key to human intelligence. And that’s not replicated in what is sometimes a more atemporal model of artificial intelligence. Not saying that you can’t—I think you really can. But I think we will see—I could easily imagine that a specialization in AI research in the next 10 years or so will be the design of temporal architectures for intelligence, like thought machines or what we talked about before: context, chain-of-thought, all of those different things. Memory.

We speak about memory as if it was something we could add on. We need memory and planning, we say, when we’re building the next generation AI on the path to AGI. And I think that’s a simplification because memory comes in so many different forms. If you read the works of Paul Ricœur, for example, you realize that memory, history and forgetting are super complex—not even individual mechanisms, but they’re deeply embedded in our social interaction with each other. The way we produce history, forgetting, memory—it has to do with how we interact.

And this is another thing that I think is sometimes lost in the discussion: time exists between people. It’s a relational property of a system. It’s not an individual characteristic. An individual doesn’t have time. It experiences time vis-à-vis others. And modeling that relationship, I think, is a really interesting philosophical and probably also architectural question.

Philip Bell: That’s really interesting. Is that why—because you mentioned previously about multi-agent systems—is that why that introduces a new kind of temporal element, because there’s the temporal relation between the different agents?

NBL: And you can pace them in different ways. You can almost imagine orchestrating basically on temporal aspects. Here’s a slow agent that comes in once every 10 cycles and asks the same question. Here’s a very fast agent that collects a lot of information, synthesizes it, produces outcomes. And then as you design or orchestrate your multi-agent framework, you can use time as a key variable in how you design it. There are probably also safety and security measures that you can implement that have to do with the way you design temporal interaction in multi-agent frameworks.

Philip Bell: Yeah, that’s really interesting. Going back to the point you made previously about death—to bring it back to death...

NBL: Yes, let’s go back to death. More death.

Philip Bell: Isn’t that what Heidegger said? We should spend more time thinking about death. But no, I believe I watched a lecture by Daniel Dennett where he was saying that one of the differences between AI and humans is that humans have more skin in the game because we’re mortal, because we die. So I suppose that’s an interesting point about how that structures our temporal relationships with one another and because our agency is more or less, in a subconscious way, related to the fact that we’re going to die.

But one thing I wanted to ask is: I think the vision that you paint in the article is really positive. You say that, for example, AI systems adapt to us while we adapted to computers. I personally feel like sometimes social media kind of speeds up my life in a way that isn’t necessarily great for me. So I think that’s really interesting that you paint such a positive vision of AI systems adapting to us rather than us adapting to them.

How likely do you think is this scenario to play out—that AI systems adapt to us? Are there other possible scenarios? And is it partly our choice as to how these different scenarios play out?

NBL: It’s definitely our choice. It’s our choice also how we interact with social media. The fact is that we have agency and we can choose to deploy it or not. And I think sometimes it’s hard because architectures can capture us in ways that make it harder for us to exercise our agency. It’s an old problem.

Both Plato and Aristotle spoke about something called akrasia, which is weakness of will. Now Plato was quite rough about it. He was like, “There’s no weakness of will. If you realize what’s right, then you do what’s right. So it’s just a question of you not knowing what’s right. That’s why you spend too much time on Facebook.” Whereas Aristotle was like, “Well, hang on, let’s give people a little bit of benefit of the doubt. Some people may do something that they know is not actually good for them, because we have weakness of will. It really exists. It’s not just a lack of knowledge.”

And I think that one really interesting question that we will have to address is: how do we build architectures that augment our autonomy in different ways? And I think this is the natural evolution of the privacy debate. The privacy debates have been a lot about how do we make sure that information about me is not leaked and used against me in ways that reduce my ability to do things. And I think the natural extension of that is to say: how do we design not privacy-enhancing technology, but autonomy-enhancing technologies? Technologies that allow us to deal with and eliminate akrasia wherever it comes up.

So it’s partly an architectural problem, partly an agency problem. You have to choose it. You have to want to have stronger agency. And then the other thing that we have to do is to figure out: are there really good ways in which we can build artificial intelligence to be autonomy-augmenting rather than akrasia-augmenting, for lack of a better term, feeding on our will in different ways?

And this is philosophically a really difficult question because if I design a technology to strengthen my will, is my will then stronger or weaker because I needed a technology? And now I’m reliant on this will-amplifying technology. Does that mean that my will has actually weakened? And it’s like the question of: should I really ever use a car because then I will not be as fit as if I run everywhere?

And we have to decide how we think about ourselves and what we think is reasonable to include in our identities when it comes to different kinds of technologies. The car is reasonable because it’s such a great advantage over just running everywhere. And we will probably find equivalents of that with certain artificial intelligence technologies that are just so good that they help us actually do more of what we want, but they remain silent on the question of what we choose.

That’s going to be a really interesting design problem: how do you build a technology that amplifies your will without directing it in different ways? And in some cases we want our will to be directed too. I have a personal trainer. He’s great. He’s not necessary. I mean, my will is not necessarily to turn up at 6 a.m. in the morning, but I use him as a will-amplifying, augmenting technology in order to get into shape.

So this question of how you build autonomy into a system is really important. And I’m an optimist. The reason I have this bright perspective in the essay is I think we have the ability and will to do this and we need to choose it. And I think we can if we want to. I don’t think there’s any technological determinism that leads to an outcome in which we’re necessarily the slaves of the machine.

Philip Bell: I totally agree that a lot of the time it feels techno-deterministic the way people talk about technologies, and I think that’s also an interesting point in itself. But in terms of supporting agency and autonomy, it’s interesting because, as you said, it feels like that isn’t something that’s commonly talked about—at least in the debates I hear in the mainstream about utilizing new technologies such as AI.

Do you think that there is a need for a new kind of vocabulary and intellectual scaffold for thinking about and defining what agency might be? Because I was sort of thinking of Amartya Sen’s work on capabilities. Or do you think we already have it? Why is it that we talk a lot about privacy, but we don’t think about autonomy and agency in terms of introducing AI currently? Is it because there hasn’t been enough work on building an intellectual scaffold around how to define these things, or is there another reason?

NBL: No, I think you’re on to something. I think in many ways, artificial intelligence is this computational science giant on philosophical clay feet. It needs to figure out how to deal with a lot of the philosophical concepts that go into things like intelligence, agency, autonomy—all of these different things. The integration of time into all these perspectives. But I think it’s happening. There are some really good people out there thinking about it. Some strong philosophers really trying to understand this. There’s plenty of my former colleagues at DeepMind who are brilliant at this stuff.

And to some degree, I think what we need to do is to find a way to perhaps frame the discussion about artificial intelligence in these terms: how do we want artificial intelligence to work? We should also focus on risks. We do, there are lots of risks. But we should have more of a discussion around who do we want to be with a machine, rather than what will the future look like with the machine. So it’s a question of how you frame your future perspectives to some degree.

I’m worried that often we slip into techno-determinist thinking when we talk about whatever technology it might be. “This technology enables this, thus this will happen.” Well, it also enables all these other things that we can make happen if we want to. And the challenge of course is that we have to figure out what it is that we collectively want.

Philip Bell: Yeah, that’s a really interesting point. I wonder because in the article you described that technology can speed some things up, but then there are other areas of life where you can’t really just speed things up due to using technology, like biological processes, for example.

NBL: Or art. One example in the article is that you can imagine playing all of Bach’s work in a microsecond, just executing the notes, but then you won’t really have played Bach’s work. So I think there are some things that only exist if they’re performed temporally in a certain way.

Philip Bell: Exactly. I sometimes listen on 2x speed to audiobooks and sometimes my girlfriend looks at me like, “Why are you doing that? You shouldn’t listen on 2x speed.”

NBL: Yes. But it’s a bit slow, I do that too. I never listen at speed to Bach though.

Philip Bell: Yeah, true. I think listening to music is a whole different thing.

NBL: If you find somebody doing that, you would have found somebody who has a really different attitude to music, where the point of the music was the informational content, not necessarily... I can imagine that maybe there’s a professional musician that does that, listens to music sped up. But it’s a really interesting thought experiment: who would actually be listening to classical music at 2x, 3x pace because they just wanted to consume it faster? That’s interesting to me. I would love to know if there’s someone out there doing that.

Philip Bell: Yeah, you could listen to multiple albums in an evening. I guess if you’re trying to experiment and think about new types of music, maybe you might just warp music in lots of different ways including speeding it up and slowing it down. But it wouldn’t be, as you said, in that scenario it wouldn’t be just trying to get the information content of the music.

One thing I was going to ask about this is: Mancur Olson’s work on building coalitions is interesting in this regard because he argues that coalition building is necessarily a very long-term endeavor. And I’ve been really interested recently in the argument by Henry Farrell—he’s been arguing that AI should be considered a cultural technology that’s equivalent to the market or equivalent to democracy in that it’s basically a coordinating technology because it can classify at new speeds.

And I guess what I was wondering was: building institutions, how is that time-bottlenecked? Is that something that could be sped up using AI, do you think? Or is that one of the things where it’s got to find its time speed?

NBL: Yeah. No, I think it’s hard to speed up building institutions, partly because institutions are the result of repeated interactions. Now you could imagine artificial intelligences building institutions of their own much faster, because they can have repeated interactions much faster than we can as human beings. So there is this world in which you have a set of institutions that are solely, exclusively catering to different kinds of agents.

So these institutions would then be different contracts, say, between agents that they all agree to, standardized ways of thinking about delivery of goods, those kinds of things. And maybe you could get a Lex Agentia, which would be like the Lex Mercatoria between traders back in the day. And they would have their own law, they would have their own institutions, their own way of resolving disputes.

But if you think about any institution that contains a human being, that human being needs to repeatedly be able to interact with someone for the institution to arise. And then they need to invest their intentionality in something. John Searle famously said that institutions are the product of collective intentionality. And I think there is a lot of truth to that, which means that the pace at which you build institutions is the pace of intention. So that’s where you sort of end up.

And I think, again, when it comes to hybrid institutions where we have both agents—artificial agents and people—the lowest common denominator holds. We need to feel that we interact with the technology in such a way that we over time come to trust it.

Philip Bell: That’s really interesting. And I guess going back to your previous point about supporting agency, I wonder whether AI could help build collective agency as well as individual ones. I know that the Collective Intelligence Project have been doing work on getting AI to quickly find the commonalities between 30 people in a room, for example, which would be hard for a human to do in real time, but would facilitate a real-time conversation. I think that’s an interesting, also potentially positive vision of use of AI—supporting collective agency as well.

NBL: Yeah. And you can almost turn it around. You could say: I think that agency primarily is collective. The notion of individual agency flows only from this idea that we together create agency. If you had one single person in the universe, would they then have agency? Would it be possible to have agency then? Whereas agency is actually dependent on being a part of a group where you can direct your will in certain ways and others can either join with you or oppose you.

So I think that a lot of the agency we have is collective first and individual agency is a result of that. It’s just like individuality. We have this idea that we are born with an inner individuality and then that individuality expresses itself and we are who we are. But in reality, to go back to your quote from Heidegger, Heidegger notes I think somewhere that we’re “strewn in the eyes of others.” So identity is constructed by the eyes of others. And if that is the case, then I think agency might well be the same.

Identity and agency might both be collective phenomena before they become individual, because identity and agency are tightly coupled. So maybe the question then is: what happens to this collective formation of agency first when it also has artificial content? Can artificial agents be a part of forming our agency so that we actually want new things when there are artificial agents in the collective that shapes our agency in the first place?

And I do think that’s true. I think we’ll see certain very simple effects of that. Not wanting necessarily the same as the machine wants, but wanting in ways that are reminiscent of what the machine wants. Framing the world in ways that are reminiscent of the way that machine frames the world. And there’s this constant interplay between us and the machines that we use. Just like Heidegger—what are you? There’s the constant interplay between us and the tool or Zeug that we use in practical—say carpentry or something like that.

Philip Bell: That’s really interesting. That was just making me think, because I think one thing that’s important in my identity is memory. I know you were speaking about memory earlier, and I’m now interested to read Paul Ricœur’s work. One thing that I think is sort of an interesting possible use of AI is in supporting human memory.

We tend to think that as a society, relative to past societies, we’ve kind of forgotten the importance of memory to a large degree and our information systems are quite sparse and don’t support memory. Whereas in the past—this is slightly generalized—but in the past societies tended to have much more cyclical and thick knowledge environments which enabled people to retain information better. But in an AI world, I think it’s possible that AI could enable people to remember roughly whatever they want. Whereas today, most people kind of remember things based on chance. And I know...

NBL: But is that true though? I mean, I wonder if you remember based on chance because my pushback will be: I think you remember on the basis of who you are, the stories you tell yourself, the narrative structures you put into your life, and those have a selection mechanism built into them so that some memories surface. You come back to some memories because they’re constitutive of the story you tell yourself.

So there’s this distinction in some philosophy between history and memory where history is the things that happened and memory is the way you structure them and forget. And I think that there is a lot going on—consciously, or maybe not consciously, but going on in memory that is not random. I don’t think that what you remember is random.

It’s not as if you wake up one day and then you remember, “Oh, I had a glass of orange juice on the 4th of February in 1994.” If that were to happen, you would probably be quite surprised and you would worry. And your first question would actually be this: “What does that mean? Why do I now remember that glass of orange juice in February 1994?” That must mean something. And that sense of meaning is deeply tied with a sense of memory because it’s tied to the sense of story.

So we remember what we want to remember in the story that we tell about ourselves. And that’s not necessarily the historical events. And I do agree with you that AI could change that. It could make retrieval of historical facts easier, which might actually allow people to rewrite themselves. Because currently you strengthen the memory that’s in the story you tell about yourself. But the things that could tell a different story are forgotten. Your forgetting is also quite strategic. You forget who you do not want to be.

So to some degree, you could then imagine an AI that could retrieve what was recorded—not remembered, but recorded—and allow you to remember it, to make it a part of your memory and then rewrite yourself.

Philip Bell: That is really interesting. So it’s kind of a corrective almost—or well, corrective’s maybe a normative term—but yeah, it provides maybe a counterweight to the narrative you’ve built up. Now that’s a really good point actually. I mean, I think you’re right that maybe it’s not totally random. I guess that it doesn’t feel to me to be totally deliberate either. There are lots of things that I would like to remember that I can’t remember. I would say I forget most of the things that I read about and I don’t want to forget most of the things that I read about. And I guess that’s what I think is to me quite exciting about AI—it seems to provide the possibility that I will be able to remember more of the things that I want to be able to remember.

NBL: I think you want to make a distinction between retrieve and remember, because the AI might—you might talk to it about a book you read. This I recommend actually, I actually might do this when I read a paper or book: I speak to Claude and then I make sure that there’s a small note of it so I can come back to it and I can retrieve that conversation. The reason I want to retrieve the conversation rather than what was written in the book is I want to retrieve how I read the book because my reading of the book is how I want to remember it. The retrieval I want is the retrieval of the remembering, if that makes sense.

So to some degree, I think computational technology already allowed you to search, right? You had information retrieval technologies—they had nothing to do with AI. And information retrieval is different from remembering. But how we then retrieve remembrances is an interesting question.

And I think this goes to the point we made earlier that all of these mechanisms are much more layered, complex, interconnected and messy than we would like for them to be. The idea we think about memory in a von Neumann machine is clean. There’s this random-access memory, there’s this long-term memory and then you retrieve things from it and you put it into the computation. That’s not how we work. The narrative we have structures the remembering we do. Some things we want to be able to retain—like information, you retain information rather than remember it. And then you want to be able to retrieve that information later at some point. But if you do, is that really remembering or is that just retrieval?

There are all these nuances in language that we have evolved over time. And if we pay attention to the way we talk about it, we realize that these are different mechanisms, still reproducible I think. And I think it still would be really valuable to build an AI that can make a difference between remembering something and retrieving something. But it would be very different from just boiling it all down to the simple model where there’s memory and processing.

Philip Bell: That’s a really good point. I wonder also whether—because in some sense AI recalls older attempts to unify human experience through knowledge. I was really interested by an article by Patrick Hutton I read recently where he talks about different mnemonic systems of a time, including Renaissance mnemonic systems. They had memory palaces and Camillo built a memory theater and there was a memory theater in London built by Fludd. And he even talks about how Freud in some ways thought that you could get to some sort of unifying human experience through memory. But he argues that since roughly the ‘60s and post-structuralism we have actually lost that aim of trying to unify human experience through connected knowledge.

And I suppose that’s something that AI can, just by its affordances, support—connecting knowledge and enabling someone to reach into a common thread of human experience through that knowledge.

One thing I was going to ask you actually is: I know in one of your other articles you talk about how sensors might become more important in an AI age partly because of the ability to compute lots more information. And I wonder how that might interact with human memory and identity. I suppose if I’m able to actually get much more empirical data about the world... I don’t know if that’s how you were thinking about it, but can you foresee a world where I’m able to get much more empirical data about the world through other systems as well?

NBL: Yeah, no, you can definitely get the data, but then you have to translate it into your own sensorium. So you’re limited by the senses that evolution deigned to give you. You’re going to interpret it through your own sensorium anyway. But it’s such a good question because it gives us the opportunity to rephrase a paper by Thomas Nagel. Thomas Nagel once wrote a paper that was called “What Is It Like to Be a Bat?” He essentially asked this question very simply: because a bat has echolocation and we don’t, can we imagine bat consciousness with echolocation? Because the senses actually structure consciousness differently.

And going back to our discussion, then you can now actually ask the question: what is it to remember like a bat? What is it when a bat remembers and how is that different from how we remember? It’s a really interesting question because bats probably remember other things than we remember in a structure that we can’t replicate.

The reason I wrote the essay about sensors was I was fascinated by the fact that an AI essentially could collect any kind of physical measurement data. Anything you physically can measure can become a sensor. You can imagine that you track all of these different senses—and there could be millions of them—and then you could structure an understanding of the world from all of these different senses. And that in itself would give the AI an ability to understand our world in a very different way than we do. And you would have to ask the Nagel question: what is it like to be a multi-sensorial AI? And what is it like to remember like one?

And you can sort of go on and you would probably end up in worlds where the amount of senses... Say for the sake of argument that you had an AI that had a million senses—a million different physical measurements fed into a massive AI that in some way construes its understanding of the world from that, finds patterns, interactions, correlations—you can find different kinds of affordances in that world that we are not even aware exist between different kinds of physical quantities that we hardly notice. That kind of AI would have a very different consciousness, a very different intelligence from ours, and it may well end up being completely impossible to communicate with, because communication presupposes some overlap in sensoria, in the senses we have.

Evolution was extraordinarily parsimonious when it gave us our senses. We have, give or take, 6 to 11 depending on which biologist you talk to. And those senses are what we make do with. But if you look at the amount of possible senses that you could provide an AI with, that’s just a tiny, tiny fraction. The human world we inhabit is very small. AIs could essentially be sent out on missions to understand the world across sensoria that would be enormously different from ours. And I find that fascinating because I do think that if this were to happen, it would probably teach us a lot about the world if we could only translate it. And the translation problem interests me.

Philip Bell: That is so interesting actually. Yeah, and I guess it also would get us back into old philosophical questions about empirical information versus other sorts of information. And yeah, that is an interesting point about: even if the data exists, how do we translate it into information that we can understand ourselves?

NBL: Yeah. And we think about that with echolocation and bats too. When we think about how a bat thinks, we think that it thinks with this... we do our own voice, we’re like... I mean, we think it gets an echo and that’s roughly how it has to be. Now we know it’s not exactly like that because we know that their echolocation sense is much more advanced, but we have to translate it down to our own senses. So we translate it down to our own hearing. So we think it’s like hearing, sort of kind of, and then we translate it into our sensorium.

And I think that’s what we’d have to do with an AI that was able to sense different kinds of quantum fluctuations or different kinds of radiation that we can’t detect or multiple other things that we can’t understand. And we would have to figure out analogies or metaphors for what that is for us. Maybe quantum fluctuation is a little bit like taste—it tastes metallic. But I don’t know. So you would have to find these analogs in order to translate and understand. And in many cases you might accept that you cannot.

I mean, what if we get an AI—and this is the funny thing—what if you get an AI that has a million senses, it discovers a set of really interesting natural laws that we have no access to, but they can be translated into super cool machines that can be really good at stuff. And so we get these machines, we have no idea of how they work, because they work on the basis of data that’s been collected that’s way beyond what we can sense. We can get some kind of theoretical description of it, but we will essentially literally have built black boxes from these different sense datasets that we’ve been working with. And that’s not entirely out of the question that something like that could happen.

In fact, I mean, that’s sort of how evolution works. We’re just discovering now that evolution is a-theoretical. For the longest time, I stupidly thought that evolution would have to take into account how much we have understood by physics. So of course, it would just use Newtonian or Einsteinian physical phenomena. But now we’re finding out that evolution has been using quantum phenomena for a long time in photosynthesis, possibly in our ability to smell or in the ability of birds to navigate. We have this growing field of quantum biology.

And you have this a-theoretical force—evolution—that takes every small advantage that is physically detectable or sensible and turns it into fitness. And so we shouldn’t be surprised that there are quantum effects being utilized if those can lead to fitness. Now you can build an AI that has the same ability to become almost a-theoretical in the way it explores the world because it just adds senses and can measure all of the physical things that exist within its environment, and it can have that same a-theoretical exploration capacity.

Philip Bell: Wow. That is really interesting. And I guess that is quite post-human in having this a-theoretical... or revolutionary. But it makes me think: maybe that’s how we become more ecological in some sense. Or more... I don’t know, I mean... That’s a really interesting idea also, that through collecting information that our senses wouldn’t otherwise be able to pick up, we could actually understand what it means to be—well, maybe we can never understand, but we can maybe appreciate better what it means to be a bat or a tree or something.

NBL: We can be together with a bat, right? Our own “we” can slightly expand to accommodate the bat as well. And then some of the experience of the bat-likeness is in the bat and some of the experience of it is in us. So it’s this question of identity again that you raised in the beginning. What if this notion of individuality and identity is not necessarily as robust as we think it is?

Philip Bell: Yes, and I guess we probably, in most cases, we can’t be with them on a temporal level. I mean, I would love to be on the time scale of a tree, but I don’t think it’s going to happen anytime soon.

NBL: No, I think that’s right. And trees are interesting, temporally super interesting biological organisms. And as are mushrooms, for example. So there are tons of really interesting examples in nature of things that exist in different temporal dimensions, which is sort of what we’re saying that we’re building here. First wave of evolution was biological and then evolution could easily be machine-driven. A-theoretical, but at a different level with different capacities.

Philip Bell: Yes, and we didn’t get into religion. I suppose that is another interesting area that could be influenced by this. I just had one more question: you’ve already given me the Paul Ricœur suggestion. Did you have any other book recommendations?

NBL: Yeah, yeah, yeah. There’s actually a really good... if you like identity you should really read Oneself as Another, which is a great book by Paul Ricœur.

Plenty. So I mean, I think when we’re talking about these things, you mentioned Heidegger—Being and Time is a heavy read, but there is plenty of good commentary. There’s a lot of good stuff in there. I really like, in terms of how one approaches issues methodologically, I quite like Philosophical Investigations, Wittgenstein. I think it’s a really good book. I think it’s, I really think it’s readable. And a lot of people go like, “Wittgenstein, that’s really hard.” No, I think it’s actually just very interesting to dip in and out of books like that and sort of try to figure out how people are thinking. So that’s another one. Those are classics, so they’re obvious.

Ricœur I think is really good. There is a guy called—if I remember correctly, I don’t want to mess this up—it’s Pierre Hadot I think, who writes a lot about philosophy and identity. He was the inspiration for a lot of the work that Foucault did. Really good.

You’ve already mentioned Dennett. His From Bacteria to Bach and Back is an excellent book. Dennett generally is just very readable, and especially when you don’t agree with him, because then you really have to do hard work because he’s so robust in the way that he argues.

So I think those are some of the books, but there’s so much out there. Generally, I think one of the things we should do is to read more of the classics when it comes to artificial intelligence. And I think it’s good to go back to Turing’s original paper and just read it properly from the beginning. It’s good to go back to the early stuff from Minsky, especially now in the time of agents. Society of Mind is underestimated. It’s really good to look at things like what Margaret Boden did early on.

Or if you want to take the other tack and look at it from the human side, you should look at Hannah Arendt and The Human Condition or Simone Weil and The Need for Roots. Great books that discuss what it means to be human in the nature of machines. So I think that there is no shortage of reading. Don’t get me started.

Philip Bell: Awesome. Thank you so much. This has been so enjoyable and so... Yeah, it’s really helped me think about these things. So thanks so much and thanks for your article.

NBL: Thank you so much for having me and good luck with the podcast.

Philip Bell: To start off, could you outline the key arguments in your book, How Progress Ends?

Carl Benedikt Frey: The purpose of the book is to push back against the narrative that technological progress is inevitable. If progress was inevitable, it wouldn’t have taken humanity 200,000 years to have an industrial revolution. If progress was inevitable, most places around the world would be rich and prosperous today, and we wouldn’t see places that once prospered stagnating or declining or collapsing.

I think the reason that progress isn’t inevitable is that as technology moves on, institutions need to adjust as well. Different institutional settings are more conducive to different stages of the technology lifecycle. Early on, when you’re exploring, you don’t know if something is going to catch on or not, so you’re better off having a system where people take different bets and then you see what works out.

The Soviet Union was the most centralised economy the world had ever seen. If you were an aircraft engineer in the Soviet Union, you could develop a new engine and go to the Red Army to ask for funding. If they declined, maybe you had two or three other options. If they all declined, your idea would die with you. That’s quite different from the US system, where Bessemer Ventures famously declined to invest in Google back in 1999. They probably regret it today, but it also underlines that Google wasn’t a safe bet at the time—AltaVista and Yahoo were dominating search. To know if something will catch on, someone needs to take the risk and invest.

On the other hand, when technology is more mature, it’s more conducive to planning. Airplane technology was quite well established when Europe set up Airbus as a competitor to Boeing. The jet engine—the last significant innovation—had already been invented. You were catching up to a static target, not trying to catch a moving one. When that’s the case, it’s much easier to plan, coordinate, and scale.

The implication is that you need to move between these two phases. You explore, you get a prototype, you scale—and sometimes that runs into diminishing returns, so you need to move on to something new. That requires more decentralised structures. Some things are conducive to both exploration and scaling, though: having a large connected marketplace where people can move around and spread ideas, low transportation costs, good distribution networks. That helps both innovation and scaling.

Philip Bell: I really liked that argument—the idea that you need the push of exploitation and the pull of exploration. You illustrated it through the tinkerers and inventors of 18th and 19th century Britain, and then the centralised bureaucracy of Bismarck’s Germany and Meiji Japan. One term you used which I found evocative was the idea of adapting your “ecological niche.” Is it easy for contemporaries to understand what ecological niche they’re in? Is it obvious at the time, or is there also luck involved?

Carl Benedikt Frey: I think the big question is “make or buy.” Businesses do this all the time: do we invent something internally from scratch, or do we take existing technology, tweak it, and scale it?

When the Industrial Revolution took off in Britain, I think it was a fairly straightforward decision for many firms and states on the continent to buy rather than make—although there was a bit of both. States pursued various tactics to attract talent from Britain who could make things in Germany and France. It was more a question of catching up to the technological frontier than pushing it forward. Germany did come to push the frontiers, particularly during the Second Industrial Revolution. But the key question is: are you lagging behind or are you at the frontier? That determines which path you choose.

Philip Bell: That’s interesting for the current question about geopolitical sovereignty in the age of AI. Canada has championed Cohere, France has championed Mistral—even though those models don’t benchmark as well as Google, OpenAI, or Anthropic. Do you think the make-or-buy question is relevant today for AI specifically?

Carl Benedikt Frey: Most certainly. The question is how easy it is to access the technology you need from abroad. During the post-war period, American technology and know-how was readily available through Marshall Aid, largely because of concerns about Soviet influence on Europe. America wanted Europe as a buffer, and that paved the way for much of the technology transfer we saw. It’s less clear today that Europe will have access to some of the technology being developed in the United States.

One of the puzzles in my mind is that for the past couple of decades—this predates the Trump administration—Europe has been less dynamic, which is maybe not surprising. But it has also not caught up in digital. Europe managed to catch up in mass production in the post-war period but failed to do the same in digital. Why?

I think a big part of the answer is that the single market is much more harmonised for goods than for services. The IMF estimates that if you take all barriers to trade inside the European Union and add them up, they amount to something like a 110% tariff. Trump Liberation Day tariffs, self-imposed inside the European Union. That obviously caps the return to investment in digital in Europe. What both China and the United States have in common is large domestic markets that firms can scale into. For Europe, a key priority needs to be harmonising the single market for services.

Philip Bell: I’d never thought about it that way—that Europe hasn’t caught up in digital. There are European tech companies like Spotify and Klarna, but they’re few and far between. And it’s not a reflection of talent, because major tech companies open large offices in London to hire talented individuals at cheaper prices than in the US.

In the book, it seems like the nature of the specific technology has some bearing on what institutions are most fruitful. You argue that centralised Soviet bureaucracy was useful for building heavy industry, but not for digital. Is there a specific institutional arrangement that will particularly help in the age of AI?

Carl Benedikt Frey: I don’t know is the honest answer. With regard to the Soviet Union, they did well in heavy industry because they prioritised it—it aided the military-industrial complex, and defence was the key priority. As long as technology was static, Soviet elites could benchmark factory performance and hold managers accountable. The problem was that when mass production petered out, something new was needed for growth. That new thing was the computer revolution, to which Soviet contributions were essentially none.

Part of the reason is that when you introduce a new technology in production, you can’t really benchmark. It becomes much harder to monitor performance. And unlike China, which is much more regionally decentralised, there was very little space to experiment in the Soviet Union without pulling the rug on the entire system.

I don’t think AI is that different in that regard. AI development is quite concentrated in a few places. When technologies are implemented, they proliferate more rapidly than manufacturing industry did. And you see work beginning to migrate abroad as well—law firms reducing headcount in London, hiring more in Poland and India to save labour costs. AI aids that process because it reduces the productivity differential between a worker in London and a worker in India.

There are probably things that will be different with AI, but it’s too early to say what institutional arrangements will be needed. What we can say is that certain things matter regardless of technology: barriers to entry are important for innovation, and having a large homogeneous market helps scaling.

Philip Bell: Do you think AI will change the cost of exploration differently from how it changes the cost of exploitation? I’m building a startup, and AI has accelerated our implementation to an extreme degree. But the rate of exploration has increased much less—the people using our app still use it at a similar rate. We can build things quicker, but do you think AI will change the relationship between exploration and exploitation?

Carl Benedikt Frey: I think yes, but probably in similar ways to the internet and the personal computer. When I have an idea, I can check fairly quickly using my preferred AI tool whether somebody else had the same idea. But the internet did that too. AI will reduce the cost of exploration further.

The surprising thing is that despite the cost of setting up a company and exploring going down so much because of technology—the cloud has been enormously helpful to smaller firms—we’re seeing business dynamism in decline. Something is pushing in the other direction and has more than offset the advantages created by ICT and arguably AI.

What is that? I’m at the university, and we see new rules and regulations and forms almost every week. That might be extreme, but I do think regulation has something to do with increasing costs. If you take an extreme example: most people believe AI will have a material impact on medical discovery. Even if that’s true, you still have to go through clinical trials, which is tremendously expensive. You probably need to partner with a large pharmaceutical company. I’m not suggesting we get rid of clinical trials—they fulfil a purpose. But those safety measures do come at a cost, primarily prohibitive for solo inventors and smaller firms.

Another relevant aspect is what we incentivise. In academia, it’s publish or perish. When a new productivity tool arrives, we can do one of two things: use it to drill more holes, or use it to dig deeper and focus on one thing. It seems we’re incentivising people to do many things at any given point in time, which means attention is spread more thinly across multiple projects. We have research showing that the more projects you pursue at any given point in time, the less likely you are to make a real breakthrough. Those incentive structures might be another reason we haven’t seen a real upsurge in breakthrough innovation despite having tools that clearly aid exploration.

Philip Bell: That reminds me of Marconi, who developed the radio on his estate near Bologna. He probably didn’t have loads of distractions on his phone and could concentrate. It feels like today it would be harder to do that. Do you think AI could facilitate more of that tinkerer culture coming back?

Carl Benedikt Frey: Marconi had some help from the family butler too, so most people don’t have that. It’s entirely plausible that AI will democratise innovation. If everybody has access to their army of personal tutors and research assistants, that should tremendously aid people with good ideas. We could see an era of more decentralised exploration, unlike what we saw in the 19th century.

What makes me think that’s probably not going to happen is the experience of the past two decades. ICT has already provided many tools that should make exploration cheaper and easier, and we haven’t seen the re-emergence of the solo inventor to the same degree as in the 19th century.

It is true that the computer revolution in the United States was driven by new firms—Apple, Microsoft, Google, Amazon. But since the 2000s, we have a trend where new firms exit by acquisition rather than IPO. WhatsApp, Instagram, YouTube didn’t become firms in their own right. That’s not necessarily bad, but the trend towards killer acquisitions—where incumbents buy smaller firms just to shut them down and avoid competition—is concerning. The dynamism we saw during the early days of the computer revolution is no longer quite there, despite computers being much better today than in 1980.

Philip Bell: How useful is it to use historical analogies to understand AI? Sundar Pichai compared AI to fire, Demis Hassabis compared it to the Industrial Revolution, Alex Karp compared building AI to the Manhattan Project. China’s AI Plus plan compares it to the internet or electricity. Is it useful to think about AI in comparison to analogous technologies, or should we take it on its own terms?

Carl Benedikt Frey: It depends on the point you want to make. Going back to my previous book, The Technology Trap, I think the framework of replacing technologies that automate existing work versus enabling technologies that create new types of tasks—developed by Daron Acemoglu and Pascual Restrepo—sheds light on AI as well.

If AI is just used for automation, we’re more likely to see something similar to the First Industrial Revolution, where wages were stagnant and probably even falling at the lower end of the income distribution. But if AI is used to create new products and industries, we’ll see something more similar to the Second Industrial Revolution, where the automobile industry emerged as the largest industrial undertaking the world had ever seen, along with supplier industries, electrical industries, and essentially every household appliance you have in your kitchen.

But to say AI is inevitably going to reproduce the 1990s or 20th century pattern—I think that’s probably not right.

Philip Bell: In The Technology Trap, you describe how technologies have sometimes caused short-term disruption but enabled long-term progress, partly determined by political impact. Do you think society has begun to perceive technology as more deterministic? When Dario Amodei says 20% of white-collar workers will be automated, it feels quite techno-determinist. Does that change how institutions, governments, and populations react to technology?

Carl Benedikt Frey: I’m not sure if people have become more techno-determinist. Generally speaking, the further back in time you go, the smaller the cross-section of society you get on record. If you go back to the 18th and 19th century, Marx, Malthus, Ricardo—they all believed in some form of iron law of wages. They didn’t think technology could improve standards of living over the long run.

In the mid-20th century, there was enormous optimism about electricity and personalised travel. Today there’s a lot of anxiety over AI, and a lot of people saying, “It’s the way it is, let’s hope for the best and maybe plan for the worst.” I think there have always been some of these impulses, but I may be wrong.

Philip Bell: I’ve seen a debate between Arvind Narayanan, who argues AI is a normal technology, and AI 2027’s Daniel Kokotajlo, who argues AI is completely different because it will be self-replicating. Henry Farrell recently argued we should think of AI as a cultural technology, similar to the market or bureaucracy or democracy, because it allows classification on a large scale, which has never been possible before. Do you think it’s too early to make that classification?

Carl Benedikt Frey: I haven’t read those articles, but I don’t think AI is there yet. AI is not going to replace an organisation or a firm anytime soon. Anthropic did an experiment recently where they had a vending machine being run by Claude—stocking it and selling products, a relatively straightforward operation. But Anthropic employees are a bit more experimental, and they managed to trick it into selling everything at 25% discounts and stocking the machine with metal cubes.

This is a relatively straightforward operation, and I wouldn’t want AI running anything too complex right now. When you apply AI for fairly well-defined tasks with a lot of precedent, it does pretty well. But the messier something gets, the more moving parts you have that affect each other, the more unpredictable it becomes. AI generally doesn’t do very well at generalising to situations it hasn’t seen before in training.

Advances will likely emerge in the next few years. Time will tell. Right now, I think it’s best to think about the AI we have as a tool, similar to the internet or the personal computer. In the future, maybe—but I’m not sure.

Philip Bell: Do you think historical research will change in the age of AI? I saw an economist recently write a paper about how to use AI agents for economics. Are there going to be huge teams of AI agents going through archives?

Carl Benedikt Frey: I’m not a historian, I’m an economist, so I’m not sure I’m best placed to answer. But speaking to historian colleagues, the challenge is that for AI to be of much use, things need to be digitised. AI might help that process, but that’s the first step. I suspect historians who spend much of their time in the archives will be some of the last researchers to be replaced—but I may be wrong.

Philip Bell: That’s a good point. A lot of things aren’t digitised, and that information will become increasingly precious because AI companies want very specific data involving lots of reasoning. Do you personally use AI much in your work?

Carl Benedikt Frey: I use it a bit. Partly because I write about it, I feel like I have to. I find it quite useful for some tasks, less useful for others. But the only way of knowing is trying it. So yes, I use it on a daily basis now.

Philip Bell: One last question: do you have any book or article recommendations? It could be about the political economy of technology or anything else.

Carl Benedikt Frey: The book that made me want to become an academic was Joel Mokyr’s The Lever of Riches, so I have to plug that one. I really like Doug Irwin’s Clashing Over Commerce—it’s a wonderful economic history of the United States in general, and a wonderful history of trade policy in particular. And Adam Tooze’s The Wages of Destruction: The Making and Breaking of the Nazi Economy is a book that’s beautifully executed and a pleasure to read.

Philip Bell: Thank you so much, Carl. It’s been such a pleasure to speak to you.

Carl Benedikt Frey: Thank you. It’s been a pleasure, enjoyed it myself.

I have found his argument that politicians (and the public) feel we are in a situation of ‘temporal claustrephobia’ really useful for understanding our current landscape. Our actions feel both urgent, uncertain and irreversible.Are you feeling a sense of temporal claustrephobia or do you think there are ways to avoid this? Comment below!

Phil: Jonathan, you’re a professor of politics at LSE, and I was really keen to speak to you about your book In the Long Run, because your argument that it feels like in politics the future is being foreclosed has really structured the way I think about a lot of issues, including climate breakdown. But also, on an emotional level, I feel this temporal claustrophobia that you talk about in my own personal life. So welcome to the Tech Futures Project.

Jonathan: Thanks very much for having me, Phil. Glad you found something interesting in the book.

Phil: Could you summarize the argument in In the Long Run and maybe say a little bit about whether you’ve updated any of the ideas since you wrote it a year ago?

Jonathan: Sure. The book came out of a course I was teaching at LSE on how the future is used and abused in politics—the ways in which different outlooks on the future get promoted by different types of ideologies and interest groups, and the political implications that come with these.

It’s a book with a tortoise on the cover, which indicates something to do with questions of how much time one has—whether there are certain virtues of perseverance and deliberation that presuppose a certain way of approaching political time, and that get extinguished when one approaches it in the register of high-speed emergency.

A running theme is how the question of how much future is felt to be available—whether it’s open and abundant, or whether it’s felt to be closing in on the present—varies historically. The book starts with an earlier phase of modernity 200 years ago when revolutionary politics was very much focused on immediate transformation. Then over the course of the 19th into the early 20th century, you get conversely a lot of politics with a sense of abundant time. The slow timescales of evolution made their way into political thinking—incrementalism, gradualism. The Fabians in this country used the tortoise as their emblem.

This is quite in contrast to the present moment where, in a range of contexts—you mentioned climate change, ideas of climate deadlines, five years, ten years, twelve years in which to make certain key political decisions—this is clearly how we’ve thought about climate change for some years now. But it’s not just that. There’s also economic inequality and the sense that we have a critical window to arrest runaway concentrations of economic power, or somehow it’ll be too late. That sense of living in a critical moment where what can’t be done now risks going permanently undone.

You asked how my thoughts have changed since I wrote the book. Clearly one thing that’s happened is Trump coming to power in the US. I’m perhaps more aware now of the ways in which certain kinds of far-right, potentially fascist forms of politics prosper under these conditions of temporal constraint and the sense of the closed future—that things can’t be otherwise, that we’re living with a limited array of possibilities. So many mainstream political parties seem willing to push this, including Labour in this country at the moment. I think this really creates an opening that Trump made use of in the US and potentially that Farage and Reform will make use of in this country—an opening for that desire to occupy the territory of the future from a far-right perspective with ideas of disruption, of breakdown, of imminent civil wars.

We live in a time of temporal claustrophobia, and I think too many parties of the center and the left are too quick to adapt to this and become very reactive and emergency-focused in their politics. This then is part of what the appeal is of those types of far-right politics, which are all about saying the future is going to be different. It may be bad, it may be a breakdown of some sort, but you don’t have to simply reckon with the stasis of the present. Change is coming. Perhaps we’re going to be the agents of change as well.

Phil: That’s really interesting. The word “abundance”—I know there’s a whole abundance movement now. It’s interesting to consider whether we could have an abundant concept of time as well. What are the conditions that create this sense of emergency that politicians are operating within? Do you think it’s partly to do with our information environment—the short-term social media dopamine-fueled news cycle that we’re in?

Jonathan: I think it’s certainly partly that—the sense that everything is in such permanent flux, including the news cycle and attention cycle, that really the only way to get heard or politicize anything is to turn it into some kind of emergency that cannot wait. Anything that can be postponed is going to fall out of reckoning with this type of media environment.

But I think what that touches on, which is perhaps slightly deeper, is a sense of weak power. Things become an emergency when you feel powerless to address them. If you feel you’ve got plenty of control over a situation, or you feel you’ve got time on your side—even if maybe you haven’t got immediate control, at least you think you can play the long game—then you have the capacity to set out your agenda. Conversely, things become this reactive logic of emergency when you’re just dealing with problems as they arise, precisely when you doubt your capacity to really control them. The best you feel you can do is respond to them in real time and cope in some fashion.

This applies to different fields of politics. Politicians are working in that media environment you mentioned, but they’re also working with institutions that have been ravaged by austerity, that lack strong state institutions on which to rely. They may also doubt the capacity to command the allegiance of people that maybe in another age would have been long-term supporters. Think of party leaders who realize that electoral volatility is very pronounced and you can’t really rely on people sticking around with you. Again, you have to accelerate your actions because you can’t depend on long-term support.

In wider society, it’s not just how politicians see the world. If you think of emergency discourse that comes from below, from social movements—something like Extinction Rebellion, where this was pretty central to the framing of the climate change predicament as one of climate emergency—again, I think it’s got something to do with the fact that when you doubt your capacity to influence people who have political and economic power, and perhaps you doubt the public’s engagement with the question, then this logic of emergency is one of the few resources that you still have.

These general patterns play out in different ways across the political spectrum. The left has its emergencies to do with climate change and economic inequality. Right-wing politics has its own emergencies—things it constructs as such to do with migration, racial replacement. The language of emergency tends to be quite central to how the right and far right approach things they consider problematic. And there are centrist emergencies to do with AI or global geopolitics, breakdown of the liberal international order. From all these different political orientations, each has different motivations and content, but there’s a common theme of political actors responding to a world in flux and doubting their capacity to control it, therefore reliant on a register of urgency and quick decision-making.

Phil: That’s really interesting. It made me think of Antonia Matuschek’s book Hyperpolitics, where she talks about exit costs being much lower from organizations like political parties. I think she mentioned Five Star in Italy purposefully didn’t have any physical spaces they organized in as a party, which made it easy to get into the party and leave it.

I wonder whether that logic might extend to other aspects of trying to control your life in the medium term. In the UK, young people are buying houses much less than previously—there are loads of reasons for that—they’re also having fewer children, going to university less, and there’s a big attendance crisis in schools. To me, these are all medium-term investments. Peter Mandler wrote a book where he argued that in the 20th century there was only one decade where university enrollment went down—the 1970s—and he argued that people in that decade were losing faith in the future. Do you think there’s any link between those trends and the idea of a future being foreclosed?

Jonathan: I think there is. All these things you mentioned are long-term projects, often individual personal or family projects, but they require you to have a certain confidence in the possibility of planning, of thinking ahead by a decade or two. These are things that may have short-term costs—you only really engage them if you can see the light at the end of a tunnel you have to pass through to reap the benefits.

When you see a world that seems to be very fast-changing, that makes planning seem impossible or somewhat naive, that clearly makes all these activities that require a longer-term perspective less appealing. All the more so for those who are especially precarious in their economic conditions—this has clear class differentiations. Those in the most precarious situations are least able to count on things like even basic job security, and that’s a foundation on which to construct everything else you’ve mentioned.

One way of thinking about it: the role of political organizations like parties and movements has perhaps always been to counterbalance that, to give people collective perspectives on the future that allow them to have a future orientation despite conditions of economic insecurity they may be stuck in. If parties themselves stop making available more collective visions of the future, people are left reliant only on the futures they can muster from their own life course. Then they’re all the more challenged by the loss of confidence that comes with being unable to plan their personal economic or social security.

These all have multiple origins, multiple circular logics. To the extent that people struggle to buy a house or find accommodation to plan their personal lives, if you feel yourself locked out of these pursuits, then you start to lack long-term reference points. This in turn undermines the confidence in the future that you might otherwise have. There’s plenty of potential for vicious circles here, but the links are important. One of the stakes of the political abdication of the future is that it accentuates these problems as they play out at the individual level.

Phil: One of the things I found really fascinating about your book was your exploration of fascism. I hadn’t really thought about fascism in terms of impulsiveness as a tendency within it and their general approach to the future and history. Do you think there’s a risk that fascism becomes more appealing in this moment?

Jonathan: I think yes, partly as I said, it’s got everything to do with what other parties are doing. Whenever you think about the appeal of a certain political orientation, it’s maybe at most 50% about the intrinsic appeal of that orientation and at least as much about the appeal—or lacking appeal—of everything around it. Precisely as other parties of center and center-left become less willing to develop visions of the future, visions of alternatives, that does leave a vacuum.

But also, fascism is undemanding in what it asks people to think about the future. This is not like socialism, where maybe you have to have confidence in progress, confidence in the measurability of progress, the capacity to find evidence for it. Various forms of liberalism likewise require credibility that comes with accurate prediction. This is not really the game that fascism is in.

If we can speak of it as an ideology, it’s one where perhaps a lower weight is placed on internal coherence than many other ideologies like liberalism and socialism have tended to place. Fascism, going back to the ‘30s and even ‘20s, often emphasizes this as a doctrine of action and not of theory. People around Mussolini and Mussolini himself said exactly this: don’t judge us by some notion of putative logic—we are about action first and decision. We make our truths rather than simply formulating them in advance and then acting retrospectively in adhesion to them.

It’s a politics where you can make it up as you go along, where you don’t have to apologize too much for incoherence but also for uncertainty, for surprises, for not doing what you say you’ll do, because this is somehow all priced in with fascism. It’s a politics of unpredictability and wrong-footing opponents, sometimes even deliberately contradicting your own line so that it’s much harder for critics to hold a normative standard against you.

In crisis times where everything is changing fast anyway, where it’s very hard to plausibly make predictions or develop coherent programs of action, fascism just says: fine, we don’t do that. All we do is disrupt. We throw things in the air. We’re going to have maybe a bit of fun as we go about it. There’s an emotional appeal to the exhilaration of disruption and chaos, but also to the violence that comes with that, to the pain that comes with that. It embraces the chaos and unpredictability of crisis times and makes that somehow its calling card.

What I call the impulsiveness of fascism—the act-first-and-think-later, if at all, aspect to it—is always going to be peculiarly well suited to times in which that seems to be what is happening anyway, that things are just changing with rapidity that makes it very hard to be coherent and considered in your actions.

Phil: That indeterminacy is hard to fight against as well, or hard to argue with, because it’s a moving target. It reminds me of seeing Donald Trump being really nice to Zelenskyy recently, which wrong-footed a lot of people. That seems characteristic of his relatively indeterminate approach.

Jonathan: Exactly. It’s very hard to know what you’re against when you’re against someone of that disposition. In addition to the wrong-footing effect, it can allow such an individual to present themselves as free-thinking, charismatic. Precisely because they’re unpredictable they can say they’re authentic—”I say how I see it, I’m not trying to stick to a line just because it’s correct or something.”

This impulsiveness is not just bemusing to those around it, but acts as an indicator of unfiltered self-expression, which for certain types of audience can be read as a sign of authenticity, of strong will. Deliberate self-contradiction, or at least indifference to norms of consistency and coherence, is part of the appeal. It allows opportunism—you can take advantage of whatever seems like the opportunity at hand without thinking too much about how it coheres with other situations past and to come.

Phil: I read an article about lots of TV shows that are popular right now being game shows where you win lots of money—like Squid Game. The journalist was arguing that this displays some form of idea that our society is zero-sum and get-rich-quick schemes are one of the only ways out of a difficult situation. That felt like it spoke to what we were talking about earlier—lack of control over one’s life, and the only way out is potentially a game show.

Jonathan: Yeah, and the lottery would be the obvious example as well. In other words, rolling the dice is the best hope of a different future.

Phil: Do you see any signs of either politicians or institutions trying to regain a sense of the medium term?

Jonathan: Well, in some ways, sure, at the political margins on the left. There are plenty of experiments of efforts to organize in a way that’s not just about, or not even primarily about, the next election. In this country, clearly the Greens—and not just under Zack Polanski, but I think for some time this has been their position—obviously you’re not going to win the next election, but you have to think of how to develop some kind of cause that is nonetheless consequential regardless of whether you can understand that in terms of electoral success.

Under Polanski, that’s taken a much sharper and more influential form insofar as this is now speaking a left-wing language that seems to be pretty much absent in the rest of the parliamentary parties in Westminster. I wouldn’t want to say it’s long-termism at the expense of immediacy or urgency. Clearly the Greens, more than any political party, are aware of the climate science and the need to act quickly. But what you see here is a sense of beginning with urgency but continuing indefinitely—thinking the longer term, not just about how to win the next election but how to influence public discourse, how to influence other political parties that maybe will win the next election, how to build a cause that will outlive the particular electoral cycle.

Jeremy Corbyn and Sultana’s new party—we’ll see where that goes—is clearly some effort again to try and conceive a party that’s more centered on a political vision rather than an electoral machine. And possibly Zohran Mamdani in New York is something different. To some degree you have to realize that success is not the first priority. You have to find a way of building some kind of movement that’s going to maybe be successful in a few cycles further down the track in order to have any real political profile and substance in the nearer term.

These are experiments perhaps speaking to frustrations with the reactive logic of emergency politics, trying to pursue something that’s much more like a normative vision of politics rather than simply firefighting problems as they arise. What’s also interesting about a number of these experiments is it’s not just at the level of rhetoric, it’s about the organizational structure that underpins this.

There’s a fake response to the charge of short-termism which you see articulated almost everywhere, including in the last Labour manifesto, which is at the level of rhetoric saying, “Well, we know that everyone is frustrated with short-termism and crisis management, so we’re going to think a little bit more long-term, we’re going to think about goals for the next ten years”—we’ll give you however many thousand nurses by 2030 or whatever. These very specific, almost accountancy-like ways of sketching out the further future. I think that’s a fake response to short-termism because it doesn’t really go much beyond goals set by a leadership which itself is basically entirely in control of. It doesn’t really make itself accountable to anyone that’s going to hold it to those goals.

If you really want long-termism in politics—however you define it, simply something longer than the next electoral cycle—you have to create the political structures that will mean that leaders are held to those goals. You need to strengthen the position of activists in parties, for example, who are the ones that tend to join parties because they care about the longer-term project, the cause. If you don’t empower those people, you’re probably going to find that even the most well-intentioned leaders are going to make pretty severe compromises pretty quickly.

Insofar as you see any interesting experiments in the present, they’re interesting not just about talking about how we need to be a bit more long-term or visionary, but they’re also looking at that question of how do you empower those people for whom that is second nature—the people that join collectives without expecting any real immediate power but want to sign up to something they believe in. Empower those people and then you’ve got a more authentic kind of long-term politics.

Phil: It’s interesting you said some of those projects had a normative dimension to them, because I’ve often wondered—given the ecological crisis and climate breakdown and all the different competing candidates for what the most important emergencies are—what would be a coherent moral framework to use? I was even reading about how David Runciman reviewed a book arguing that having children could be good for climate breakdown because it changes the population structure so you can have more progressive politics. That stunned me, because I’d assumed having children was one of the worst things you could do for the climate.

Is one of the reasons why it’s difficult to approach these issues from a normative or moral point of view because it’s not clear what morals work in this current situation? I always think about Keir Starmer—he appears to know the answers, or portrays himself as knowing them. That’s a big contrast to how Trump presents himself, because he doesn’t present himself as knowing the answers since he changes them day to day. Do you think that’s part of the problem—that we don’t really have a coherent framework?

Jonathan: It’s certainly the case that any type of moral position comes under a certain strain in conditions that are plausibly seen as unprecedented. Morality presupposes some type of continuity with the past. Morals are those things that are understood to be inherited in some way—of course refined, something you can put under critical scrutiny, but not something you simply make up on the spot, because that’s not quite how we normally see morality.

Yet when you’re in a situation where the major issues of the day are so often said to be unprecedented—that they are unlike anything in the past—and clearly climate change is often presented in these terms, but artificial intelligence likewise—these all seem to put you on the cusp of a quite different type of historical situation. The susceptibility that society has to emergency claims in general is that if you can present a situation as unique, then you can say it needs a unique response that has nothing to do with existing normative precepts—that you’re in a new world, therefore let’s not worry about the morality we bring to it from the old world.

That is one of the genuinely confusing and disorienting aspects of the present. I think at some level one has to almost refuse a little bit the idea that the contemporary situation is quite so unique and unprecedented, because I think the cost of accepting that point is basically to embrace this disorientation. Then you can quickly find yourself acquiescing in the mere exercise of power and not really having any type of normative standpoint from which to take distance from realpolitik and the exercise of power by the strongest.

That is part of our difficulty, but I don’t think it can be allowed to be insurmountable. I think we have to assert the continuities. The world has always been a changing environment, and morality is always somehow requiring a leap of faith whereby we can say that ideas that developed before the present nonetheless have purchase in the present. That’s a leap of faith that is hardly new to the present.

The revolutionary moments we talked about earlier in the 18th century are first instances of that. Is the revolutionary moment one in which there’s no normativity, no morality? Well, maybe sometimes it looks like that when you get a lot of executions in quick succession, but I think at the same time one wants to say that even in these exceptional moments there has to be some kind of appeal to limits on what can be done, or at least some kind of moral compass has to be applied even if we disagree where that’s exactly going to point us to.

It’s part of the challenge, but it’s a challenge that has to be refused. Therefore it’s going to be about how to take existing normative frameworks, ideologies, and find ways of reconceiving them for exactly this kind of context. Ideas of 21st-century socialism, ways in which we might say there is something that is enduring in claims of equality—precisely what you want to see as living and dead in 19th-century socialism is clearly the key issue here—but there are certain enduring features of ideas of equality, even in ideas of progress and of the economic conditions that one has to pursue in order to make these things more than simply formal goods.

These are enduring questions. The things the left is talking about today—wealth taxes, land taxes, how to find forms of justice in production as well as in distribution—these are all questions that are just as important in a climate emergency or an escalating emergency of economic concentrations of wealth. The situations may look new, but I think there’s still a lodestar to be found in some of the more familiar ideological perspectives if we really think what they might mean in a new context.

For me that’s the position. Yes, this is what’s disorienting about unique circumstances or ones that seem to be such, but nonetheless the challenge must indeed be to find those continuities and distill from them ways of thinking normatively in the political moment.

Phil: That’s a really good point. I had one final question about the long-termism movement. There are people who seem to be thinking about the long term in a lot of detail, including effective altruists. People like Toby Ord have written books about the importance of considering the long-term view. In some ways, unlike the Fabians, they’re probably not gradualist—maybe they’re more accelerationist long-term thinkers. Because you talk about thinking about an abundance of time, these people seem to be thinking about an abundance of time to some degree. Would you say that’s true for this long-termism movement?

Jonathan: Yes, in a sense. They’re clearly stepping back to take the longest timescales into perspective—the timescales of transhumanism, of interstellar travel. Something I try to suggest in the book is that there are twin challenges to democratic politics that come both from what you might call short-termism and also from perhaps a kind of excessive long-termism.

On one hand, the first is more intuitive. Short-termism is a limit on political imagination because there are long-term goals that we might want to pursue—of equality, liberty—things that take time, that are not to be equated with transformations you can pursue within a year or two. These are projects. Clearly a politics that is deaf to the longer timescales required for deep structural transformations of society is going to miss something.

At the same time, this fascination with the furthest futures is something that can also be quite paralyzing, at least for democratic politics, insofar as it has a sort of flattening effect on the significance of the present. It’s not just these contemporary philosophical movements that have that effect. There are forms of utopian thought going back a number of centuries where you see the same interest in the far future, several centuries ahead, where the effect is somehow to diminish the present and to diminish the choices of the present.

If you’re thinking in terms of many centuries, does it really matter what you do in the near term? Anything less than an existential threat to the human race in the present becomes almost no problem at all, because the meaningful timescales of change are well beyond that. And conversely, anything that is an existential threat, or can be presented as such, trumps all competing considerations.

The long-termism that one sees in these effective altruist movements—philosophically it’s got some substance, but in terms of what its democratic implications may be, it’s really something that I think is almost as pernicious as severe short-termism, because it really doesn’t accommodate democratic disagreement very easily. It has a tendency to treat certain types of policy as trumps for the sake of the long-term survival of the human race, and likewise to see certain types of inequality as essentially irrelevant or perhaps even functional to longer-term gains in utility.

If you’re thinking about these questions of the future from a democratic perspective rather than a perspective in a certain kind of moral philosophy, then I think one has to—and here there’s no magic number and it’s somewhat arbitrary—but there is a kind of democratic future that I would put something on the scale of decades. Long enough that one can meaningfully think about structural transformations of society, and yet not so long as to make the contribution of living individuals and their commitment somehow epiphenomenal to the future that plays out. That still makes agency somehow meaningful in the present.

To that extent, I wouldn’t want to be an advocate of long-termism on the scale of millennia. While it may have some philosophical coherence—or at least the philosophical coherence is a separate question—in terms of its political implications, it’s clearly quite either paralyzing or quite deterministic.

Phil: Thank you so much. That was brilliant. I really enjoyed it, and thank you for the book. I struggle to stop thinking about it. I think about a lot of things through the lens of the book now.

Jonathan: Thank you. Thanks for reading it. It’s been a pleasure, and good luck with the podcast.

This is a really important question and Sean is a leading thinker and practitioner in this field. Do you believe China is racing for AGI? Why or why not? (Comment below!)

Philip Bell: Hamish, you wrote this article as part of the Center for Governance of AI summer fellowship. You’re now working as an AI policy fellow, which is really cool. And I guess it’d be really useful if you could just lay out the main argument of your article, which is called “Compute is Not the Answer to AI Sovereignty.”

Hamish Low: Yeah, definitely. Thank you very much for having me on. It’s great to be here. Yeah, so this is work I did at the Center for Governance of AI for three months, really trying to explore these ideas of AI sovereignty, basically trying to get to the bottom of what people actually mean when they’re talking in this space about AI sovereignty. I think there are not that many good answers once you start trying to scratch the surface.

So one of the first things I tried to do is just get a better definition of what AI sovereignty actually is, because it’s a bit of a wishy-washy term. I think it’s trending towards becoming a bit of an “everything is in” term, where you just sort of add “sovereign” in front of literally any AI thing you want to do, so it just sounds more serious and strategic. But actually it just isn’t the case—lots of things are not sovereign AI and are still good, but we shouldn’t be putting everything into this bucket.

But I think really the way you should think about sovereign AI is that it’s this big picture strategic dilemma. In the AI value chain, the two dominant players are just by far the US and China. Both of these two just have the strongest positions, they have the frontier models, they’re producing the AI accelerators, they’re way ahead of everyone else. So for every other country in the world, you have to figure out how do we fit into this world where there’s two very dominant players. And I think ultimately this is just a generic strategic dilemma. And the answer will be different for any individual country.

So each country has to figure out for itself what sovereign AI means, basically based on what they want to accomplish. For lots of developing countries who are trying to pursue export of digital services, this AI situation could be an existential risk to their development model, in which case it’s all primarily economic. It’s just like we need to salvage, make sure that the US doesn’t just re-onshore all of what we’re trying to do through AI.

I think for the UK, it’s slightly different. I think we are in a position where we can sit sort of in between the US, China and the EU. I think we’ve built up really impressive state capacity within the UK government around AI. And I think ultimately we have a much more ambitious goal, which is being able to freely regulate AI to help shape AI’s development on a much more fundamental level. And ultimately it’s about exercising freedom of action in terms of how AI actually affects UK society and economy.

And this is a much more ambitious aim and it’s much harder to achieve than if you’re just looking for economic benefits. And importantly it’s quite separate from economic benefits. You could have a lot of influence on the development of AI but not really get any of the benefits and you would have still achieved your AI sovereignty goal. So I think that’s how I see it for the UK. You have this AI sovereignty goal that’s about achieving freedom of action.

And I think ultimately that boils down to you want to influence the US. We’re going to be closer to the US than we are going to be to China. The US is just the clear leader and the key hub here. And so for the UK, the way I define AI sovereignty is it’s about creating interdependence with the US as opposed to where we are right now, which is this kind of one-way dependence relationship. We’re essentially getting all of these AI products from the US and we have very little stake in how these are being made and how these are being deployed.

And what we need to do is develop capabilities in the UK that the US comes to be dependent on, such that the US understands that we’re in this interdependent relationship, that there are important things that they rely on us for in AI. That means that when we reach some kind of crunch point, when we’re debating regulation or how this technology should be developed or safety issues, that they will have us in the room, that we clearly have an important stake on this, and that we have some leverage over them. That if they want to cut us out, this would be bad for them because they’re relying on us for important aspects of how they’re able to deploy AI and use AI within their own economy and military and government.

So I think this is ultimately the most powerful frame—trying to get towards this interdependence situation. The challenge is that this is incredibly difficult. The thing is that everybody wants their own ASML, the Dutch chip semiconductor manufacturing equipment maker, who has this insane monopoly over the most advanced lithography machines that are essential to making advanced AI accelerators. The issue is that there’s not lots of these lying around.

ASML was the result of this incredible process of decades-long consolidation in the uniquely neoliberal 1990s. In 2025, that’s not the case. Everybody is trying to find their niche in the AI value chain. Everybody wants their ASML. And so it’s very hard to find one. So I think a lot of AI sovereignty discussions in the UK are a bit just hand-wavy in terms of what’s actually valuable for us to build.

A couple years ago it was like, “BritGPT, we should be training models.” I think people now understand probably that this is not as good of a model and this doesn’t actually really get as much leverage. And then there’s questions around like, okay, so Nvidia chips are the big deal. We want our own AI chips, which is fine, you can make a case for this. There are cool startups in the UK working on this. And I think this is good.

But I think it’s just not clear that this is useful from an AI sovereignty perspective. I think you can make the case that this is economically good for the UK. If we have big successful chip makers, this is good for the UK in general. But I think it just isn’t very good for the UK’s AI sovereignty. Because if you’re trying to create this interdependence, you just don’t really get this if you’re trying to sell the US AI chips. Because ultimately, the US has other AI chips to buy. Most of the other key players here, whether it’s Nvidia, AMD, Google, Amazon, they’re all doing this themselves.

And even if you carve out the specific bit of the value chain, if you say we’re going to make our own AI chips, this does not give you in any way independence in this value chain. You’re using all of the exact same supply chain aspects that everyone else is using. You’re getting them manufactured at TSMC, you’re getting South Korean high bandwidth memory, you’re having to buy networking switches from this random Taiwanese company.

You’re incredibly dependent on the rest of this value chain, which functionally is very dominated by US regulatory regimes and the foreign direct product rule and their ability to put export controls on these capabilities. So even if you’ve built an AI chip, this in itself doesn’t give you any sovereignty. The US could cut you off from this wider supply chain and you’re screwed. There’s nowhere to go. You can’t make any more. You’re still very dependent on the US.

And you’re not generating something where the US is dependent on you. So I think this just doesn’t have a very clear AI sovereignty advantage. And I think this is true for just a lot of aspects of the AI value chain, whether you’re talking about building data centers in the UK, which I think is another useful initiative. And there’s certainly lots of merits of having data centers and AI compute in the UK. It lets you build out your public compute resources. It is investment, which is always nice from an economic perspective. You can build up some UK companies like EnScale, who’s building out with OpenAI in the UK. This is good, but again, it doesn’t solve your AI sovereignty question. It’s nice for other reasons, but putting a bunch of Nvidia chips in a very fancy warehouse in the middle of the UK is not making the US dependent on you. You’re just strictly a customer here. And it has other benefits downstream in how you use that compute. You can use that to build new products and do new things with it. But it doesn’t solve this core question.

So then I think what you need to think through is, okay, where actually do we get leverage? And where can we build this dependence? I think ultimately the view I’ve come to is you have to look at what are the future nodes of the AI value chain that are going to be important. Because for almost all of the existing areas, there’s just very powerful US incumbents that it’s really hard to compete with. Even if you deployed the most incredible industrial policy the UK was capable of, I think you would struggle to really generate US dependence. So yeah, this is where I then get into this idea of AI middleware, which I can walk you through.

Philip Bell: That’s super interesting. I think that’s such a useful phrase—interdependence rather than independence. I think that is a really kind of both clear but powerful encapsulation of your argument. When you were talking about the chip sovereignty idea, it made me think of Graphcore, because I think they were kind of an early competitor with Nvidia and they created kind of IPUs. And it kind of reminds me of the nuclear energy competition, because I think that Britain had their own kind of rival form of nuclear energy to the US in the form of gas-cooled nuclear reactors.

The idea’s got kind of interesting historical parallels, but yeah, it’s a really interesting point that actually that isn’t really what we should be focusing on at all because it doesn’t even create interdependence if we were to create a Nvidia competitor. And it also makes me think of the Jeffrey Ding book, the recent Jeffrey Ding book about fusion rather than kind of coming leading sector.

But yeah, I think it would be really useful if you could lay out that idea of AI middleware or where you think the UK could, or any country I suppose, could kind of become interdependent.

Hamish Low: Yeah, definitely. And the Graphcore case I think is interesting. And yeah, they spent like 700 million pounds and they made like 4 million pounds in revenue at their peak, which to be fair to them is just bad market timing and you just go down the wrong direction and these startups are hard. I think now lots of the team from Graphcore after it went bankrupt and then got bought out by Softbank is now this new AI chip startup, Fractile.

And it’s all very cool and I wish them the best, but yeah, they just don’t solve this strategic question we’re trying to solve. And I think that’s where something like middleware is maybe the better way to be thinking about it, which is basically here, I’m thinking through all of the future things we would expect to emerge that sit in between—if you’re going down a sort of value chain, if you imagine it and you’re moving down the semiconductor supply chain to making the AI chips, to putting them in data centers, to training the foundation models.

Right now, there’s not very much below that. It’s kind of like OpenAI trains a model, they put it into ChatGPT, you use ChatGPT. This is a very simple, vertically integrated setup. But I think you should expect this to just become much more complicated in the future. And for many more parts of the value chain to sit between the foundation model and the end use.

Where the end use here is almost certainly going to be an enterprise user, because this will be the much bigger market over time than the current consumer-dominated chatbot products. And just enterprises are going to have loads and loads of really different needs in terms of how they integrate these systems and build new workflows around them and do all these interesting things where they’re actually automating stuff and diffusing these deeply into their processes in the way that we actually get all the productivity gains that we want out of this technology.

So a few of the things I would put in this middleware bucket would be if you’re taking models and you’re altering them in various ways. So this could be fine-tuning models on specific datasets, you’re a downstream developer where you’re taking the OpenAI model, you’re then just using other data to change it to suit it to what specific task you’re trying to achieve.

And then you’re either vertically integrating that into your product or you’re selling that on to someone else who’s putting that into a product that they’re then selling to an enterprise. Or you have this new mode of reinforcement learning that’s become very powerful. Lots of the frontier model companies are pursuing building these reinforcement learning environments to make models much more capable at very specific tasks. The most successful one so far being coding tasks.

Where you’re able to just get lots and lots of gains by having models do this reinforcement learning process. If this process is really effective, which it seems to be, you would expect to want to do this for basically every other sector of the economy by the end of getting to 100% automation or whatever crazy future endpoint. That you’re going to want to find the right data and you’re going to want to find the task and you’re going to want to train the model to be just superhumanly good at doing that task.

OpenAI and Anthropic and Google simply can’t do this work for every sector of the economy. Eventually, they just become too big, they become too unwieldy, there’s regulatory barriers, there’s recruiting the actual people to do this thing. Apparently, there was some article, OpenAI has hired 100 old investment bankers to try to get data to do better financial modeling. But it’s just hard to do this for every industry on the planet.

Philip Bell: I mean I was really surprised—I’ve been quite surprised how many different industries OpenAI are trying to target. They’re like, I saw they’re trying to make an animated feature film which seems just really surprising. I have no real idea why they decided to do that but it’s intriguing. But no, I mean the general point I totally—I mean that just seems to make a lot of sense that there needs to be kind of specific parts of society that are specific verticals that integrate AI into the business process in kind of specific deep ways. And I guess, yeah, it tallies with a lot of kind of historical research about how technological change happens. It takes quite a long time to show up in productivity and involves changing business processes, et cetera, which yeah, so that’s really interesting.

And I guess specifically for the UK, because this kind of AI middleware idea—I’ve always thought the UK is in a great, when I say great position, I always thought the UK is in a central position in terms of AI, given that DeepMind was started in the UK and it’s arguably one of the most, well probably the most successful AI company in terms of the range of different breakthroughs they’ve kind of—I mean, in terms of AI research, it’s probably potentially the most successful in terms of all the different breakthroughs, like AlphaFold, AlphaGo, et cetera. And there’s also just a lot of important organizations like Stability and Eleven Labs, organizations like that and Anthropic and OpenAI have offices in London, for example.

So I’ve always kind of assumed that the UK was in a kind of quite pivotal position, but I think reading your article actually made me slightly rethink that, in quite a useful way. But I guess on this idea of AI middleware, do you think that the UK specifically is in kind of a better position to be able to come in and be part of that kind of AI middleware because of the fact that it does have essentially a lot of expertise, I suppose?

Hamish Low: Yeah, definitely. I think the UK’s strength is being really, really central on talent, right? We do really well on talent. We’re probably the best AI talent hub outside of the US and China. Our key weakness is that we don’t have very much compute. We don’t have lots of this infrastructure based here. We haven’t managed to have one of the leading frontier companies based here. DeepMind sort of being the weird exception where it used to be more UK weighted and then they merged with Google Brain and now it’s a bit harder to tell how you should understand them.

But I think this means that we have lots of talent here, but we’re not necessarily getting a lot of benefit out of that talent. And we’re not necessarily getting benefit in strategic terms from having that talent here. You want to turn this into some more uniquely UK capabilities. And this comes back to some very long-lasting issues with British tech in terms of startups struggling to scale, often just selling to US buyers, chasing the US market instead of the UK market because of just slower growth in the UK and regulations and et cetera.

But yeah, I think middleware is one way that you lean into this talent advantage of what the leading frontier companies of the US are doing is this big bet on huge infrastructure. They’re building out colossal amounts of this AI infrastructure just gigawatts and gigawatts. There’s just constantly more deals. The UK just clearly cannot keep up with this. And I think it’s not worth trying to keep up with this. But we can pursue this different approach where lots of these middleware capabilities, you don’t need that much compute ultimately. Often you’re just kind of building software systems on top of these models to make them more useful.

So to give one example, maybe—lots of models right now, a big evolution this year has been model context protocol and the ability for models to natively use tools as they accomplish a task for you. And right now this is just existing random software tools or just the ability to use a sort of coding application natively as it works through your answer. But presumably over time you’ll want to build tools that the models can use that are entirely separate from the models, but which are entirely AI native.

It might be that for the telecoms industry, you have a tool for monitoring the health of a network that is functionally just a software tool that does this job, but just in a way that is built for the model to be able to use it effectively. But you don’t actually want the model to probably be zooming through every aspect of BT’s network trying to find some fault. You want it to be able to access an actual system dedicated to do this and be like, okay, yes, check. This is all good. I can move on to the next stage, whatever I’m working through. And that’s a business right there.

You can build that tool and you can sell that tool. And I think that’s one of the things where I’m envisioning middleware here. And so I think this is one where if you just have lots of talent and you just get the right framework around being able to have them create startups, build these capabilities and sell them on, that you could create this just web of UK software focused companies, which means we can build products faster. They can scale them faster. We can actually create this interdependence much faster than if we were trying to build hardware capabilities where it just takes a decade to create something at the end of a long R&D process and get this to market and get it to scale. That actually you can move at the pace of AI development in this sense by being very software heavy and avoid this compute heavy models, but instead we’re trying to build on these and create useful things that just allow us to use these models in the real world. And I think, yeah, this leans into this talent heavy approach that I think is the right one for the UK.

Philip Bell: That’s really interesting. Is there kind of a government policy that would help with that? Or is that something... or there are other kind of conditions that would maybe stimulate that?

Hamish Low: Yeah, and I think the government has been doing some very useful things here. I think the Sovereign AI Unit is a very interesting project and has been doing some good work. They have 500 million pounds of budget to spend on achieving this aim. I think this is good. I think the AI Security Institute is a hugely helpful thing here in terms of this is one area where you’re leaning into this talent advantage. You’re just getting a really good cluster of talent within the government, building the government’s state capacity, but then also just hopefully building up an agglomeration of different things around this.

So one thing they’ve done is they’re looking into the AI assurance industry as one potential useful area where the UK has an advantage and DSIT has given some money to this and it’s very much on their radar. My view is basically you should just be doing all this but just doing it much more, that I think there’s all this work happening, but there’s insufficient urgency behind it for how quickly things could change and how powerful the capabilities could get. I really want the UK to be having this leverage, not in 10 years, but in two years or five years, that I think you could have very important decision points in the near future where I want the UK to have this leverage, which means I think we just need to be much more aggressive in doing industrial policy in a way that the UK government has not traditionally done.

And I think this is just—you basically have your key levers, which is capital and talent and compute, where compute is kind of a function of capital. And you just need to go super hard on these. So you have the Sovereign AI Unit. I think basically you should also create a national AI investment fund and you should create another fund and then another fund.

And you should be very okay with just duplicating all these different government investment vehicles and being like, taking this almost much more Chinese approach to industrial policy of just opening a floodgate of capital and just being very willing to just be—we’re going to change this market deliberately. This is what we’re trying to do. We’re just going to try crowd in much more competition to these sectors we think are strategically important. And we’re going to do this in a really big way, which ultimately has a lot of downsides. You know, it ends up being—you will create waste, you will make mistakes, some of these companies will fail, you have no guarantee of success. But I think if you really understand the urgency I think we should have here, that you’re willing to do this, you’re willing to put a lot of money after this goal.

Same thing on talent. I think that if the Sovereign AI Unit invests in you and has done their due diligence, they see you as strategically important. You should just be able to hire the talent you want. You could just have a visa where you, startup, decide you want to hire someone and as part of that process, they get a visa to come to the UK. You just skip all the Home Office fees, you skip all the Home Office vetting, pretty much, and you just delegate this power to the firms that you think are really important. And you could have this be just a super limited scheme, but I think this is one way of just speeding everything up and just doing everything we can to bring talent to the UK.

As well as just kind of other interesting ideas around the Center for British Progress has this idea for a kind of Center for Talent Excellence of deliberate headhunters in government who are trying to recruit the best AI engineers into the UK. I think these are great ideas as well. I think you basically just want to be maximally aggressive and just pulling these levers of capital and talent to just bring as many people to the UK and remove every barrier to them creating the companies that we want to exist.

Philip Bell: Yeah that’s a really interesting point because I suppose right now given some of the kind of restrictions on studying in the US and stuff like that I imagine there’s lots of PhD students who might be interested to come to the UK. It also reminds me of—because I mean I know Joel Mokyr just won the Nobel Prize for economics and as far as I understand, one of his main arguments about why Britain was central to Industrial Revolution was this idea of tinkerers and people who... a kind of dispersed network of different intellectual societies, like people from the Birmingham Lunar Society who went on to tinker with different types of technologies, which became hugely influential.

I sometimes wonder about that, like is there something around supporting a dispersed experimental culture? And maybe the best example I’ve seen of this is maybe Kaggle. I don’t know if you know Kaggle, but it’s an online data science competition thing. And like, I’d feel like that’s a kind of—I mean, yeah, that’s one example of kind of a tinkering sort of experimental culture. So yeah, so I think it’s a really interesting idea to lean into the UK’s kind of wealth of expertise and kind of capacity of the people. I think that’s really interesting.

And I guess because, but one thing I was going to ask about what you were saying about the kind of funding. I thought that was a really interesting point about basically just providing more of it and it’s fine if there’s overlapping kind of pools of funding. And I also thought your point about the AI assurance industry is really interesting because I mean it does feel like the UK has a lot of important kind of AI safety and AI governance organizations.

And but one thing I was going to say is with that kind of funding policy, some people might say, okay, we shouldn’t be picking winners. I have heard Tim Wu, I think he’s called Tim Wu, he’s an American kind of academic slash policy—I think he’s involved in the Biden administration, but he has the argument that you can pick sectors and ecosystems kind of thing rather than picking particular companies. What would you say to that? Is that potentially a challenge to funding, even funding the kind of expertise and stuff and also things like funding a more thriving safety and assurance industry kind of thing. Is that, do you think that’s a challenge? That thing of like maybe picking—it’s hard to pick the right ones.

Hamish Low: Yes, I think this is just always the challenge with industrial policy. I think maybe a counter example of who I think is doing a bad job on this is France or Canada, where both of them have kind of landed on national champion model development companies, where you have Mistral in France, where ASML invested in Mistral, and they’re getting lots of support from the French government. And it’s like the French government wants to preferentially use their models and this kind of thing.

You have the same in Canada with Cohere, where the government has given quite a lot of money, hundreds of millions of Canadian dollars, towards helping Cohere build out AI infrastructure in Canada. Clearly, viewing this as a national champion firm, I think this is a really bad idea. Both these companies have demonstrably fallen quite far behind the frontier in terms of the actual models they’re producing. They’ve fallen really far behind in the infrastructure they have access to, and they’re really far behind in how much actual money they’re making, I think like 100 to 150 million euros slash Canadian dollars when OpenAI and Anthropic are charging heads to tens of billions of dollars worth of revenue a year. And this is just a complete disparate relationship. And the frontier model developers are just going to be on this insane flywheel and just be way too far ahead of you. And they make models that are way better. They make their open source models an afterthought and the open source model is better than your closed source one. Like your business model is kaput.

So I think this is a really bad case of them picking winners. I think this sector approach is much more interesting. Picking another one from the recent Nobel winners, Philippe Aghion has this really interesting paper about Chinese industrial policy where he identifies one of the successful features of Chinese industrial policy is this kind of picking sectors where they pick a sector like electric vehicles, where they decide that this is productive and strategically important. And essentially by just funneling loads and loads of subsidies into the sector, they essentially just create much more competition because it encourages so many new entrants into this market that suddenly you just have this incredibly competitive market, which can go wrong. You sometimes end up where they leave the subsidies in too long and get this kind of involution—insane price competition where nobody makes any profit and everybody’s very sad.

But the core principle I think is good, that you can pick out sectors that are strategically important or that you think are economically important. And you can do this process of just making lots of capital clearly available. And this just crowds in lots of new entrants and makes this market more competitive. I think this is the vision for how this goes well in the UK, is that you identify some of these sectors you think there’s a lot of opportunity here. We want to put a lot of capital on the table that we’re willing to basically subsidize the market here in a sort of anti-market way. You’re trying to shape the market to not develop how it otherwise would. With the goal of crowding in just loads and loads of new players who are then going to be competing with each other very intensively, you’re going to get these kind of talent agglomeration effects, which I think is one way you get these kind of useful networks of talent, is that you just have a shared sector and you have a lot of capital available so that people are like, I’ve just done my PhD, I don’t want to move to the US, I’ll go to the UK, there’s loads of money around, I can get a job working on this cool AI sector.

So you get these networks of talent, you get these new firms, you get a very competitive market, and then the hope is that out of this very competitive market, you then just get very innovative, strong firms who are able to scale up and then become globally competitive players. And ultimately then, this means the US will start increasingly buying their stuff. And then hopefully, you have enough Fortune 500 companies that are buying their software solution for whatever this AI issue is they’re trying to solve. And this is what’s giving you this kind of interdependence.

Philip Bell: That’s really interesting. Yeah, I’d like to look at that article actually. I always think it’s—I mean, I kind of follow the kind of China Watchers community. And I think as far as I understand from listening to some of the podcasts like China Talk and other podcasts like that, I think one of the kind of insights that they have is that policymakers in the UK and the US and other places don’t really pay enough attention to what’s actually going on in China and don’t really understand China enough given its importance. And I do always think, because whenever we make policy—or at least whenever I hear discussions about policy, it seems all of the time we kind of take ideas from other European countries, which I guess makes sense because we’re a similar size, maybe. But yeah, I always think it’s interesting that people don’t seem to, or at least people that I hear talking don’t seem to take as many ideas from Southeast Asia or China.

One thing I was going to—yeah, one thing I was going to ask though is, are there any countries that you think are doing this well? Like they’re approaching AI sovereignty well and maybe creating some sort of interdependence. I mean, because I thought the France and Canada examples were really interesting about—I hadn’t really thought about that actually about how Canada is supporting—I didn’t actually even realize that Canada was supporting Cohere and then the France supporting Mistral, that was a really interesting point. But yeah, are there any countries that you think are doing well?

Hamish Low: Yeah, I think it’s basically just countries that have been able to figure out what their advantage is and be able to capitalize on this in some way. So I think Singapore has done very well here, which is partly a result of the fact that Singapore, I think, just has really good state capacity and had already done lots of work around getting data centers up and their industry, building up themselves as this key hub between the US and China. I think they’ve basically just been doubling down on this strategy that makes a lot of sense for Singapore.

That your entire gambit is that you’re sitting between the US and China. You’ll be a really good talent hub. They have their own national AI strategy 2.0. They’re trying to triple their AI practitioners in Singapore to 15,000 people. And I think this is just robustly a smart strategy. If you’re Singapore, this is the strategy you want to do. And they’ve just been very successful—Singapore and now Johor in Malaysia just across the border is the key AI data center hub in all of Southeast Asia. There’s huge investments by ByteDance working with Oracle in Malaysia. Singapore has become a really key hub here. They have the Singapore AI Safety Hub. This is just, I think, one example where you have a middle power that just has a really clear sense of what they’re trying to achieve and just has a state that’s very focused and willing to achieve this. And I think this is going well.

The UAE is another interesting one where I think, again, they sort of understand what they have to offer, which is loads of capital and loads of energy. Realized that these were two things that everybody else wanted and capitalized on this quite well. You know, they signed all these various deals with the US, all the kind of major US AI players coming and building out data centers in the UAE. I think once you sort of get below the surface of it, it’s slightly less clear how good this is for the UAE. They were very, very smart in that they timed their announcement really well, where when they announced all these different data centers, they were the largest ones in the world that have been announced. And subsequently, much bigger ones have been announced in the US that all come before the ones in the UAE. But still, everyone has the impression of, wow, the UAE’s at the frontier, which they probably aren’t. And the business model of how good it is to basically just be a landlord for a giant warehouse in the desert is slightly questionable.

But I think it’s still a good strategy and clearly gives them leverage. And they have some interesting ideas around exporting AI to the global south. And this is—it’s a strategy and they’re making lots of bets on it. They’re making the right bets, I think, if you were trying to carry this strategy off. And I think this is—they clearly have a sense of what they were trying to achieve and they’re working to achieve it. And I think otherwise it’s just—it’s hard to find countries that have a sense what their advantage is, partly because for most countries they don’t have much of an advantage. This is an incredibly sophisticated value chain that just doesn’t flow through that many countries.

You have a lot of challenges as well of a lot of what you really care about is talent, but talent is very mobile. So it’s a big issue for Latin American countries where you might have good AI researchers, but they’re all just going to move to the US or Europe. It’s hard to build up a sort of domestic base and work from there, which actually makes these kind of AI sovereignty discussions really, really difficult. You’re really struggling to build any advantage, let alone build a strategy to capitalize on that advantage.

Philip Bell: That’s really interesting. Yeah, I mean because to your point earlier about every country wants to have an ASML like the Dutch do, there are—I mean because Germany and France I believe do actually have quite important companies in the kind of AI supply chain. I believe that Germany has the Carl Zeiss SMT—Zeiss, the optics and laser which I think Germany is always quite strong in terms of making glass. They’ve got a long tradition of... I actually have a friend who goes to this, he’s an artist, but he goes to this glass making conference in the Black Forest every summer. So I think there is kind of—it’s probably like an ancient, not ancient but really old tradition of kind of German expertise in kind of glass which is kind of interesting that they do seem to make lenses for ASML and stuff like that. And I think they have chemicals which again is a really old German industry like the chemicals companies and stuff. But yeah, as you said, there are only a handful of companies. It’s basically France, Germany, the Netherlands, and then South Korea maybe, Japan. But other than that, is that right? They’re kind of the countries that do actually have companies in the supply chain.

Hamish Low: Yeah, because I think right now, the question of leverage is just very tied to the semiconductor supply chain. Most countries don’t have a lot of presence in the semiconductor supply chain. Yeah, South Korea, Japan, Netherlands, Germany, they really do. And that just sort of gives you a leg up.

Philip Bell: Yeah. And it’s interesting because I suppose going back to your point about the UAE had worked out that their advantage is, or part of their advantage is that they have loads of energy, loads of fuel based energy. I mean, do you think that the UK should accept that the UK relative to other countries doesn’t have an energy advantage, I suppose? Do you think the UK should kind of just accept that or do you think—I don’t know if this is very naive, but I have heard some people arguing that AI is an opportunity to build really comprehensive renewable energy infrastructure. Do you think that’s a kind of realistic possibility that a country like the UK should be considering?

Hamish Low: Yeah, it brings me great pain, the energy situation in the UK. I would wish it were not how it is. And I do think that AI is an opportunity to try to change this. But right now we’re just in a very bad position on energy. This is very bad for AI. It’s also just very bad for loads of other industries and lots of other reasons. And I think there is a good case that AI is—it helps you encourage lots of these more innovative energy solutions.

It kind of is a bit of a forcing mechanism to help you confront some of these just long running—the grid is super slow and we need to spend lots of money on expanding the grid, on building interconnections. We need to just build out lots more generation capacity of renewables. I’m very supportive of trialing new things like small modular nuclear reactors and just building out more of the kind of traditional nuclear power stations. I think all of this is really good. The issue is it’s just—even if you speed it up to my dreams of how fast it could be, it’s still quite slow. You’re trying to compress a real decades long project into maybe a decade if you’re being super ambitious. But we’re just coming from a really weak position here. UK industrial electricity prices are over 300 US dollars per megawatt hour compared to 82 in the US.

This is just really tough. You’re just dealing with way higher prices and you’re dealing with just this long accumulation of issues and challenges. And yeah, I think AI is a good forcing mechanism to help us try solve some of these, but I think the UK cannot rely on fixing these problems in the medium term. I think we keep working at them, but if we’re trying to build our strategy for trying to achieve things in the world, we can’t rely on solving these problems because I think they’re just so difficult.

Philip Bell: Yeah, is—I mean, yeah, because partly I am excited about the idea that it could be an opportunity to build renewable energy infrastructure. Because I think solar is, I think I’m right saying that solar energy is one of the kind of quickest forms of energy to build capacity in. And also wind energy is quite quick as well in terms of kind of getting everything improved and stuff. But yeah, it, as you said, I guess the UK is kind of coming from relatively a position of disadvantage. My girlfriend’s dad lives in Dubai and one of his favourite questions, well not one of his favourite questions, but something he quite often asks me is what’s the energy price per R in the UK? And it’s like 20 times more than it is in Dubai.

Yeah, I don’t know, I’d hope that it could be an opportunity, but yeah, it’s a good point about how long it takes to build the infrastructure and that’s one of the considerations. I actually read about your article on Jeffrey Ding’s substack, which is called ChinAI, which is a really interesting substack about articles from China about AI. And they seem to be a country who have leaned in most to thinking about their energy system in relation to AI. I guess, I mean, this is a relatively speculative question, but do you think there’s anything that the UK can kind of learn from that? Because I saw recently they had in their China AI plus policy, they have an AI plus policy for every different industry. Or maybe not every industry, but they definitely have an energy one. But yeah, do you think there’s anything that the UK could learn from what they’re trying to do?

Hamish Low: Yeah, I think in terms of just making clear plans, having them be ambitious and then just really ruthlessly executing on them is just really strong. It’s just back to these industrial policy questions. I think they do this well. I think this has clearly really paid off in energy.

China has this insane energy advantage on AI now relative to everyone else. I was looking at some of the stats recently. It’s like they built 278 gigawatts of solar in 2024, which is the entire UK capability 3.7 times. That’s a lot. It’s 3.7 UK grids they’ve built in a year of just solar. And they’re growing their grid at just such a rapid rate that it’s giving them a huge advantage here. Where right now the US is doing really well on AI infrastructure and energy, largely because there’s lots of just stranded energy from US deindustrialization.

So you can get a lot out of basically just all these random plants in the Midwest that no longer make cars but there’s still some legacy transmission infrastructure that you can capitalize on and just very quickly build up new data centers. The issue for the US is then in a couple years they start to run into actual bottlenecks of you’ve used up all this easier spare capacity lying around. You have to do much more difficult things to actually grow your grid and connect all these new energy sources to the data centers and there’s lots of interesting solutions people are talking about of building behind the meter energy, of doing some ways to reduce peak energy demand such that you just shut the data center down for a couple days a year when it’s super cold or super hot and when the grid is under peak strain. So there’s stuff there.

But then the advantage for China is that they’re scaling their grid by a huge amount every year. So when you’re like, oh, we need to add another 10% for AI, they’re like, ah, sure, why not? We can build more solar panels in the Gobi Desert. And I think they just have such a big advantage here. The issue for China, if we’re kind of diving into Chinese AI, is they don’t necessarily have the compute and some of the other pieces to bring this strategy together. Their AI plus strategy, I think is very interesting. And I think the kind of adoption focused mindset is very smart.

So yeah, Jeffrey Ding was my supervisor for doing lots of this work and his book is great on this in terms of general purpose technologies, the really important process for national power is you need to diffuse them very widely through your economy and do this in a very deep and meaningful way to unlock the productivity that just accelerates all your other aspects of national power. I think the challenge is that this is just hard to do. The targets in their AI plus strategy are really ambitious. It’s 70% penetration of AI by 2027, 90% by 2030. One, it’s unclear what this means. Two, it’s unclear if you can actually do this.

There was this whole DeepSeek craze earlier this year where suddenly the central government goes, everyone needs to adopt DeepSeek. And there was this very interesting article Jeffrey translated that was telling the story of how this actually went, which was people bought these DeepSeek in a box where it’s a big AI chip that just has DeepSeek already running on it. And all of these state-owned enterprises, they buy this thing because they’ve been told they need DeepSeek. And they use the thing for a week. And they realize that actually DeepSeek v3 is—it’s a fun model, but it’s not that useful. You know, there’s a reason that we haven’t seen all the crazy productivity gains yet. Just having this model does not unlock incredible AI gains. And I think basically the vibe of the article was that now all these just sit in a cupboard somewhere. So you’ve ticked the box of adoption. But the challenge is that adoption is really hard and you need to bring together having really capable models with having large scale compute infrastructure and having just a very sophisticated cloud business and digital economy, which is one area where China actually is still quite far behind.

In that their cloud market is just generally much lower quality than Western cloud offerings. The adoption of digital cloud services is just a lot lower. You have this situation of there’s a very advanced tech industry in China—Alibaba, Tencent, Huawei, these are leading incredible tech companies. But China is huge, and most Chinese companies are not like Alibaba, Tencent, and Huawei, where actually the median US Fortune 500 company is much more technologically sophisticated than maybe the median Shanghai Composite Index 500 companies. And this is a big issue for China.

So I think there’s definitely things to learn in terms of having this very adoption focused mindset. But then we can also kind of learn from, what do we think is actually gonna hold them back? Which is really having this just sophisticated firms, having the compute availability, having access to the best models, which is where actually we can do all these things.

Philip Bell: That’s really interesting. I did not know that about the DeepSeek in a box and it kind of makes me think about generally in this space. One of the differences that I’ve heard people talk about between building AI infrastructure compared to building other technological infrastructure in the past, like fiber optic cables for internet or train lines or electricity infrastructure is that the GPUs go out of date really quickly.

Kind of like DeepSeek in a box kind of goes out of date quite quickly potentially if there’s new models and it kind of—I feel like it does also kind of add weight to your argument about AI middleware can be kind of lighter weight and it’s more kind of adaptable to a relatively uncertain sort of situation where things are changing quickly and hardware can kind of go out of date quickly. So yeah, I think that’s a really interesting anecdote. I didn’t know about that.

I also think when you were just talking about the US, one of the reasons they’ve been using this spare infrastructure from deindustrialization. That’s another interesting one because it would be amazing if AI could support a reindustrialization kind of program in the US or elsewhere. But the problem I suppose is that data centers don’t really have any jobs associated with them. So sadly, I don’t think it will. Yeah, but I thought, but yeah, that’s a really interesting point. I also didn’t really know that about basically a lot of the data centers are using up that spare capacity.

So this is, I mean, it’s interesting because we’ve talked about in this conversation kind of historical sort of comparisons or historical analogies that might be needed and, or are used and needed. And I was interested to get your take on the argument of David Edgerton. So he wrote a book called The Shock of the Old where he basically argued that—well, he tried to think about use rather than invention as the most important process in technological change. And then also he argued that lots of new technologies are wrapped up in older ones. And I wondered whether you think is that way of thinking useful at all do you think in considering AI? And AI sovereignty, like the idea that maybe older technologies are going to be as important as kind of newer ones in potential AI advancement.

Hamish Low: Yeah, I think it is very helpful. I think you already kind of see this in practice, with some of the infrastructure build out in the US where you have OpenAI in Texas and the things that they desperately want more of are these natural gas turbines that no one has cared about their production for two decades and everyone thought that the production forecasts were basically going to completely die off because of renewables build outs and heating and ventilation technicians, which is two things that you would not immediately identify.

Philip Bell: Yeah. Yeah and SMRs because you were talking about small modular reactors as well. The nuclear energy small modular reactor because I mean I think they were invented by the British Navy in the 1950s or something like that but they yeah they’re kind of coming back in. But yeah and that’s really interesting.

Hamish Low: Yeah, and I think there are probably just lots of ways that AI interacts with older technology. I think it’s very interesting. And I think it’s very hard to reason through. I think you end up in this weird world where right now AI capabilities are just very jagged in the sense that there’s this jagged frontier of AIs are really good at coding, but they’re still really bad at lots of other things we might care about.

I think there’s good reasons to just expect this to continue being true. That probably you see just lots more progress in domains of traditional white collar work. They get way, way better at coding and making PowerPoint documents and doing incredible mathematical reasoning. But probably maybe struggle on things that you think wouldn’t trip them up and struggle with this sort of transitioning into functioning in the real world and in industries in the real world.

Which does lead you into this very interesting fusion of old and new technologies and how these might interact. You have cases like cyber security, where there’s interesting ways where the two of them interact, where you have really, really rapid progress potentially in the ability to do automated hacking attacks. You could just have, basically you can just spin up the most sophisticated team of hackers that you could have had in 2015 and just deploy them to attack any given target around the world. And this is really, really scary. Lots of companies around the world and governments rely on these incredibly old legacy outdated tech stacks. There’s a bunch of US government tech that runs on coding languages that were popular in the 1960s. And there’s the one guy who’s retired, they have to bring in every time it breaks. So it’s very concerning.

But then maybe you could use AI to just retranslate all of these coding languages to much more modern ones and like refactor all this security and this kind of thing. So it’s like you end up with these weird ways where the two of these things interact of very old technologies, very new technologies and probably you end up with lots of parts of the economy where there’s some really old process or technology. You know, there’s some literal fax machine or incredibly outdated piece of hardware. And then there’s some AI software system running on top of it that’s trying to use it and interact with this old system and we’re just gonna live in this really weird world where the two of these things are both true. You have the super intelligence in your coffee maker or whatever, where it’s some very established technology but that is getting some benefit out of having all of this just cognitive ability plugged into it.

Philip Bell: Yeah, that’s really interesting. Another, yeah. I also wonder, just as you were speaking there, I was thinking that’s a really interesting point about kind of short, you know, old programming languages, could AI help to kind of retranslate those in some way. And it made me think about the trade, the kind of the ratio between building versus maintaining.

Technology because I think that’s something else that David Edgerton argues. He argues that we think about kind of building new technologies much more than kind of maintaining repairing old ones. Whereas maintaining and repairing old ones is actually more important according to his argument. But it’s an interesting—I think that’s an interesting question around in an AI future, would—could AI actually mean that there’s more maintenance and repair needed if there’s, let’s say if there’s, you know, just more complicated and larger scale systems at play.

But, but yeah, I mean, the use of historical analogies, I think is really ubiquitous in talking about AI, maybe too ubiquitous. I heard Sam Altman pitch—he compared AI to fire. If you listen to Demis Hassabis or really any AI leader, they often talk—compare AI to I think the industrial revolution or other things. And I’ve heard, I mean, some people say that China according to its policy seems to consider AI more akin to electricity or the internet, whereas the US, according to its policy, and this might be a simplification, seems to compare, seems to be thinking about AI more akin to developing the atom bomb. And that kind of informs their policy in that US policymakers are very worried about takeoff scenarios, maybe more so than Chinese policymakers.

But yeah, in general, do you think that using historical analogies is more useful or more distracting when thinking about AI and generally AI sovereignty?

Hamish Low: I think generally they’re useful. I mean, I studied history, so I’m contractually obliged to give this answer. But I do think it’s very helpful. I think just from the kind of general—in forecasting, you have the outside view and the inside view, where the outside view is just ignoring the specific thing you’re trying to study and just figuring out, okay, what’s the right reference class? What’s the way we’ve understood previous incarnations of whatever this thing is?

I think that’s where historical analogies are just really, really helpful—you just want a reference class you can base off of. I think this is where these kind of previous general purpose technologies are super useful. Electricity and steam are just two really good examples of previous technologies that apply across every sector of the economy. And we can draw a lot of interesting conclusions from these, kind of what Jeffrey Ding does in his work of studying in detail quantitatively how these diffuse across an economy and when the productivity effects kick in.

So we can learn general purpose technologies, the productivity effects probably come two decades to three decades after their widespread diffusion. This is really good to know. But then you can do the kind of inside view, which is where you really are just interrogating the specific thing you’re interested in. And I think this is where you get to a bit more like AI is the atom bomb—there’s things about AI that are kind of crazy and actually very different.

You know, that if AI can function as just a substitute for human labor, this has lots of really crazy consequences that no other technology does. And I think it’s really useful to actually interrogate these differences because maybe previous general purpose technologies took two or three decades. But maybe one of the reasons why they took so long is that there just weren’t that many good metallurgists and iron engineers and people who were able to make steam engines and so the kind of skilled engineers who actually learned how to make a steam engine and then went and worked in this completely unrelated industry and was like you should plug in a steam engine. There just wasn’t that many people. But if AI can substitute for these people then maybe these processes can go much faster.

So then you can kind of more easily debate—okay if previous general purpose technologies took three decades, how fast do we think AI can speed up this diffusion process? Maybe it pushes it down to a decade, maybe it’s 15 years, maybe it’s five. And I think this is where they can be very helpful is because they just give you something to base these discussions off. Because I think otherwise you can end up in very—if you go too inside view and you go too, AI will change absolutely everything. Then you get lots of these economic models of explosive growth from AI where just you have a model of what growth is and you’re like, okay, AI is a perfect substitute now, what happens? And the answer to what happens is that you get stratospherically crazy economic growth.

And I think this is helpful in some ways, but also you just have lost some of this reference class and understanding the history that maybe grounds your answer a bit more and brings you a bit closer to what I think is a better forecast of the future.

Philip Bell: Do you think that it’s useful to, or do you think as policymakers, people should think about a number of different scenarios and kind of hedge their policy bets, I suppose? Or do you think it’s more useful to kind of basically decide what scenario is most likely, like let’s say 2028 takeoff or something, and kind of develop policy accordingly?

Hamish Low: Yeah, I think you basically just have to go in with different scenarios and just understand the differences between them. I think ultimately states are big and they have lots of money and they can do these multiple things. I think you should be worried about the 2028 takeoff. We just have lots of uncertainty about whether this is possible or whether it could happen. And probably you want the people in the AI Security Institute really thinking this through in detail and trying to figure out what the mitigations are. And we should put some work to preparing for that world.

But then equally, I think the AI as electricity, the AI as normal technology is also true that there’s just going to be lots of actually in practice very boring mundane ways that AI is diffused across the economy. You know, there’s going to be so many really dull pitches to venture capitalist investors about AI for X random thing. And lots of them will be very useful and will give us good economic growth. But they’re just not very interesting or flashy and you should just have people working on thinking about both of these worlds.

And yeah, from a policy perspective, it’s tricky because sometimes you’ll end up in trade-offs between the two of these. You know, there are some things you might do to prepare for this crazy takeoff world that are unhelpful for this other general slow diffusion normal technology world. I think probably those are worth taking the trade-off of. I’m quite scared of the takeoff world. But I guess we just need to go through the process of just trying to sketch out these possible futures and just having people debate between them and figure out what we think is important and what are the best, most robust ways we can navigate this.

Philip Bell: I agree. What—this is the last question that I was going to ask. What is your kind of best case scenario for kind of how AI plays out?

Hamish Low: You’re leaving me on the hardest possible question, which is slightly mean, but I’ll give it a go. I think we have the first easy box of it doesn’t kill us all, which is yay, this is excellent, and it’s a pretty necessary condition. We have the second tick box, which is that it doesn’t lead to some crazy concentration of power that destroys all our democratic institutions, which I think is also a real worry. So we want safe AI that’s developed in a kind of diffuse and democratic way.

But then I think if you’re able to start satisfying these conditions, it just can be really, really good. I think there’s big issues to navigate in terms of a lot of people in this space are not very honest about how far AI is just an automating technology. This is all it does is it automates people. Broadly, this is its main economic effect and any vision of the future where you’re like it will create new jobs—I mean, yes, but these will be a fraction of the jobs that it’s just taken away.

But I think even still we can envision very positive scenarios because we just have a much richer world here. We just have many more resources from all the things that AI can do for us. And I think we just get incredible medical goods. You just get to have your own personal AI advisor. There’s been lots of interesting work around ways that AI can augment democracy, of actually making some maybe really interesting direct democracy systems possible because I can tell my AI in much richer detail what my political preferences are. And then AIs themselves can aggregate these much more easily in a way that electoral systems are just not very good aggregating functions for people’s democratic preferences.

So I think ultimately the kind of flourishing positive world is one where everyone is richer, everyone has much better healthcare, everyone has better governance, and people can just live freer, happier, richer lives. And I think this is hopefully the goal that we can work towards.

What's really happening inside models when they generate text? Lee Sharkey, Principal Investigator at Goodfire AI and co-founder of Apollo Research, and I discuss mechanistic interpretability - the emerging science of reverse-engineering neural networks to understand how they actually work.

Lee works at Goodfire AI, an AI interpretability research lab focused on understanding and intentionally designing advanced AI systems ⁠Company⁠. In this conversation, we explore how researchers are using techniques like sparse autoencoders to decode the internal representations of large language models, discovering everything from "Golden Gate Bridge features" to Barack Obama neurons ⁠Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet⁠.

We discuss what we actually know about models, the challenges of working in high-dimensional spaces, and why understanding AI systems might be crucial for safety as they become more powerful. Lee also shares insights from his background in computational neuroscience and how similar methods are being applied to artificial neural networks.

Topics covered include induction heads, sparse dictionary learning, the "grown not made" nature of neural networks, and whether there might be universal structures in how both humans and AI systems organize knowledge.

Articles by Lee:

Open Problems in Mechanistic Interpretability

Sparse Autoencoders Find Highly Interpretable Features in Language Models