In his article ‘Time Machines’ Nicklas Berild Lundblad outlines the thesis that AI has the potential to be a ‘temporal mediator’, enabling a decoupling of computational and biological time. Nicklas is a writer, investor, and formerly Head of Global Policy and Public Affairs at DeepMind.

Philip Bell: You’re an investor, advisor, writer and researcher working across a number of different think tanks and writing a blog which I’m a big fan of—which is why I asked you to come on today. I previously worked at DeepMind on policy, so I loved your article “Time Machines” and it was really thought-provoking. It really made me consider some of the possibilities of how an AI-influenced world might sort of play out. So that’s why I was really keen to talk to you.

My first question would be: how will AI influence our experience of time?

NBL: The article has this fundamental idea that there are two kinds of time. There’s biological time and computational time. Everything that we’ve done as human beings—our relationships, our institutions, our society, our economies—they all evolved to run in biological time, in our time. Biological time is evolutionary time. It’s the kind of time of seasons. It has this particular pace.

But as we developed computers and machines, we were able to create this other pace layer—to use a term from the literature—this computational time that’s much, much faster. We can now do calculations at a speed that no human being could do. We could play a million chess games in a very short time.

So we have developed this computational layer of time, and there’s a tension between the two. Many of the institutions that we have developed are not necessarily built to run in computational time. Markets, for example, for a long time struggled with figuring out how to deal with high-frequency trading.

My idea in the article is essentially this: maybe we can use artificial intelligence as this mediator that understands biological time because it can communicate with us, but also can operate in computational time. So it can help us prioritize across the many different things that happen in computational time and give us an effective interface. Artificial intelligence becomes a temporal interface between these two different pace layers.

That’s the basic idea of the article. Not revolutionary in any sense, but I think it addresses this notion that we have of everything accelerating, everything speeding up, and at the same time, we have this upper limit beyond which we cannot speed up. We have 24 hours a day of attention and that’s it. That’s what we can spend.

Philip Bell: One of the really interesting ideas in the article was you put it that perhaps most significantly this bifurcation will enable individualized relationships with time itself. That made me think: if people are able to have individualized relationships with time, that might influence identities themselves.

Many scholars have talked about how the experience of time has helped form identities. For example, Benedict Anderson in his famous book about the origins of nationalism describes people reading a newspaper across different parts of—let’s say Germany—for the first time in the late 19th century, which created a kind of simultaneous feeling that you’re inhabiting the same time because everyone would read the newspaper at the same time in the morning. He described this as “national time.”

Could this new capability—if people are able to have their own individualized relationship with time—influence identities? Do you think that could allow small communities to have their own identities based around time? How do you see that influencing culture and identity?

NBL: That’s a great question. The way to think about this is to think about the way that we constitute our “now.” Essentially what Benedict Anderson is talking about is this notion of a national now, national time, national rhythm.

There’s plenty to be learned about how we constitute a now even just looking at our own nervous system, because whatever you’re experiencing is in the past. You’re not experiencing the now directly because your nervous system needs to collate all of the signals from your body, all of the impressions and perceptions that you have into something that can be a single, coherent now.

That’s why, paradoxically, if you—God forbid—were to be shot in the head, you wouldn’t experience it. Because what would actually happen is that before you can constitute the now of that moment, everything would go black. Just like—spoiler alert—in the last episode of The Sopranos.

One of the things that I think is interesting is to think about: how do we then constitute now across different groups? You said nations, communities can constitute their own nows. And yes, I do believe that artificial intelligence could be a core part of constituting that now. But we should also remember that there’s a really interesting question here about how artificial intelligence constitutes its own now.

Because now we’re talking about multi-agent systems. They need to coordinate with each other, they need to find a pulse, a sync, so that they can start to figure out what a now is. I think a little bit about this like the Empire of Rome. The Emperor of Rome had a now, but the pace of that was essentially what it took to ride from one part of the Roman Empire to the center. So that was the fastest possible now that Rome as an empire could experience.

I think this is true also for technology and human beings: the fastest possible now we can experience is the least common denominator when it comes to speed. So in hybrid communities, we will still be limited by biological time. But you’re right—the pace, the time, the temporal experience, the constitution of the now is essential to identity.

You can also probably say that you have different identities in which you constitute your now at different pace layers. You have one identity which is your personal experience of the world around you. And then you have a communal identity—you get together with friends and you update each other. That’s the first thing you do, right? “What’s up? What’s been happening lately?” That’s you building your now together. So there are all of these different nows that you operate through and in.

Philip Bell: That’s a really interesting point. Going back to the comparison with what does the now mean for AI—I think with current large language models, one difference between the way humans think and the way large language models process information is that they can’t really dwell on anything for a particular period of time. Information is processed between layers at a fixed time.

I actually read about a new paper in Europe by a company called Sakana—they created this idea called “continuous thought machines” where they basically introduce time into computations so that there is the ability for the model to dwell on different pieces of information according to different time speeds. That is an interesting difference between humans and AI currently—time, in a sense, is fixed in how AI processes information.

Historically some scholars have said reading was quite important for human culture in that it allowed for asynchronous thinking. Whereas dialogue—one has to think on the spot, in the moment—reading allowed for asynchronous thought. You could say chain-of-thought in large language models is sort of allowing for some element of that, buying time to some degree. But that is a sort of difference. How do you see that playing out?

NBL: You can turn the question on its head to some degree. What you can say is: how do we build things like dwelling machines, a machine that can dwell on something?

One of the things we often do is that we try to say, “This is how the machine thinks, this is how the human being thinks, and here’s the difference between them.” A much more interesting approach, I find, is constructive. To say, “Okay, how do we build a dwelling machine? How do we do it? How do we build a machine that can be filled with regret?”

All of these are temporal feelings. They all have to do with how we interact with time. So if we accept that it’s like an architectural construction problem, then we have to ask some really hard questions about: how would we model regret? How do we model dwelling on something?

Dwelling on something sounds a little bit like a loop, right? So I come back to this thing again and again. I’m not coming back to it necessarily in order to resolve it. It’s not like I’m doing a loop until I can finish the calculation. I’m coming back to it because I believe that the change in me when I come back to it will be meaningful for how I can approach the thing I’m working on.

For example, in art, I might be able to do something really quickly and just write it once and be done with it. But that’s not what poets do. Poets write a poem, they come back to it. They feel it because they change in between interacting with the thing. So there is this question: Okay, I’m building dwelling. I know it’s a loop, but it’s a loop where the object of my dwelling is not changing as much as I am in between the loops, in between the cycles. So what is that structure in me that then needs to change? How do I capture that?

What all of this teaches us is that temporal architectures are really complex, but they’re probably also a really core part of what it means to be intelligent. And of course, you also have—and I’ve written about this on the blog—you also have the ultimate temporal horizons. A lot of our intelligence is structured the way it’s structured because we die. We are finite beings. Being a finite being means you have to structure your experience in a certain way. You can’t spend infinite amounts evaluating two almost equal options. You just have to pick. So that means that death is the great tiebreaker. It constantly forces you from afar to make decisions.

That’s another temporal architecture and intelligence that’s deeply embedded in everything that we do. I think that the more we think about temporal architectures and intelligence, the more we realize that there is a lot here that actually is key to human intelligence. And that’s not replicated in what is sometimes a more atemporal model of artificial intelligence. Not saying that you can’t—I think you really can. But I think we will see—I could easily imagine that a specialization in AI research in the next 10 years or so will be the design of temporal architectures for intelligence, like thought machines or what we talked about before: context, chain-of-thought, all of those different things. Memory.

We speak about memory as if it was something we could add on. We need memory and planning, we say, when we’re building the next generation AI on the path to AGI. And I think that’s a simplification because memory comes in so many different forms. If you read the works of Paul Ricœur, for example, you realize that memory, history and forgetting are super complex—not even individual mechanisms, but they’re deeply embedded in our social interaction with each other. The way we produce history, forgetting, memory—it has to do with how we interact.

And this is another thing that I think is sometimes lost in the discussion: time exists between people. It’s a relational property of a system. It’s not an individual characteristic. An individual doesn’t have time. It experiences time vis-à-vis others. And modeling that relationship, I think, is a really interesting philosophical and probably also architectural question.

Philip Bell: That’s really interesting. Is that why—because you mentioned previously about multi-agent systems—is that why that introduces a new kind of temporal element, because there’s the temporal relation between the different agents?

NBL: And you can pace them in different ways. You can almost imagine orchestrating basically on temporal aspects. Here’s a slow agent that comes in once every 10 cycles and asks the same question. Here’s a very fast agent that collects a lot of information, synthesizes it, produces outcomes. And then as you design or orchestrate your multi-agent framework, you can use time as a key variable in how you design it. There are probably also safety and security measures that you can implement that have to do with the way you design temporal interaction in multi-agent frameworks.

Philip Bell: Yeah, that’s really interesting. Going back to the point you made previously about death—to bring it back to death...

NBL: Yes, let’s go back to death. More death.

Philip Bell: Isn’t that what Heidegger said? We should spend more time thinking about death. But no, I believe I watched a lecture by Daniel Dennett where he was saying that one of the differences between AI and humans is that humans have more skin in the game because we’re mortal, because we die. So I suppose that’s an interesting point about how that structures our temporal relationships with one another and because our agency is more or less, in a subconscious way, related to the fact that we’re going to die.

But one thing I wanted to ask is: I think the vision that you paint in the article is really positive. You say that, for example, AI systems adapt to us while we adapted to computers. I personally feel like sometimes social media kind of speeds up my life in a way that isn’t necessarily great for me. So I think that’s really interesting that you paint such a positive vision of AI systems adapting to us rather than us adapting to them.

How likely do you think is this scenario to play out—that AI systems adapt to us? Are there other possible scenarios? And is it partly our choice as to how these different scenarios play out?

NBL: It’s definitely our choice. It’s our choice also how we interact with social media. The fact is that we have agency and we can choose to deploy it or not. And I think sometimes it’s hard because architectures can capture us in ways that make it harder for us to exercise our agency. It’s an old problem.

Both Plato and Aristotle spoke about something called akrasia, which is weakness of will. Now Plato was quite rough about it. He was like, “There’s no weakness of will. If you realize what’s right, then you do what’s right. So it’s just a question of you not knowing what’s right. That’s why you spend too much time on Facebook.” Whereas Aristotle was like, “Well, hang on, let’s give people a little bit of benefit of the doubt. Some people may do something that they know is not actually good for them, because we have weakness of will. It really exists. It’s not just a lack of knowledge.”

And I think that one really interesting question that we will have to address is: how do we build architectures that augment our autonomy in different ways? And I think this is the natural evolution of the privacy debate. The privacy debates have been a lot about how do we make sure that information about me is not leaked and used against me in ways that reduce my ability to do things. And I think the natural extension of that is to say: how do we design not privacy-enhancing technology, but autonomy-enhancing technologies? Technologies that allow us to deal with and eliminate akrasia wherever it comes up.

So it’s partly an architectural problem, partly an agency problem. You have to choose it. You have to want to have stronger agency. And then the other thing that we have to do is to figure out: are there really good ways in which we can build artificial intelligence to be autonomy-augmenting rather than akrasia-augmenting, for lack of a better term, feeding on our will in different ways?

And this is philosophically a really difficult question because if I design a technology to strengthen my will, is my will then stronger or weaker because I needed a technology? And now I’m reliant on this will-amplifying technology. Does that mean that my will has actually weakened? And it’s like the question of: should I really ever use a car because then I will not be as fit as if I run everywhere?

And we have to decide how we think about ourselves and what we think is reasonable to include in our identities when it comes to different kinds of technologies. The car is reasonable because it’s such a great advantage over just running everywhere. And we will probably find equivalents of that with certain artificial intelligence technologies that are just so good that they help us actually do more of what we want, but they remain silent on the question of what we choose.

That’s going to be a really interesting design problem: how do you build a technology that amplifies your will without directing it in different ways? And in some cases we want our will to be directed too. I have a personal trainer. He’s great. He’s not necessary. I mean, my will is not necessarily to turn up at 6 a.m. in the morning, but I use him as a will-amplifying, augmenting technology in order to get into shape.

So this question of how you build autonomy into a system is really important. And I’m an optimist. The reason I have this bright perspective in the essay is I think we have the ability and will to do this and we need to choose it. And I think we can if we want to. I don’t think there’s any technological determinism that leads to an outcome in which we’re necessarily the slaves of the machine.

Philip Bell: I totally agree that a lot of the time it feels techno-deterministic the way people talk about technologies, and I think that’s also an interesting point in itself. But in terms of supporting agency and autonomy, it’s interesting because, as you said, it feels like that isn’t something that’s commonly talked about—at least in the debates I hear in the mainstream about utilizing new technologies such as AI.

Do you think that there is a need for a new kind of vocabulary and intellectual scaffold for thinking about and defining what agency might be? Because I was sort of thinking of Amartya Sen’s work on capabilities. Or do you think we already have it? Why is it that we talk a lot about privacy, but we don’t think about autonomy and agency in terms of introducing AI currently? Is it because there hasn’t been enough work on building an intellectual scaffold around how to define these things, or is there another reason?

NBL: No, I think you’re on to something. I think in many ways, artificial intelligence is this computational science giant on philosophical clay feet. It needs to figure out how to deal with a lot of the philosophical concepts that go into things like intelligence, agency, autonomy—all of these different things. The integration of time into all these perspectives. But I think it’s happening. There are some really good people out there thinking about it. Some strong philosophers really trying to understand this. There’s plenty of my former colleagues at DeepMind who are brilliant at this stuff.

And to some degree, I think what we need to do is to find a way to perhaps frame the discussion about artificial intelligence in these terms: how do we want artificial intelligence to work? We should also focus on risks. We do, there are lots of risks. But we should have more of a discussion around who do we want to be with a machine, rather than what will the future look like with the machine. So it’s a question of how you frame your future perspectives to some degree.

I’m worried that often we slip into techno-determinist thinking when we talk about whatever technology it might be. “This technology enables this, thus this will happen.” Well, it also enables all these other things that we can make happen if we want to. And the challenge of course is that we have to figure out what it is that we collectively want.

Philip Bell: Yeah, that’s a really interesting point. I wonder because in the article you described that technology can speed some things up, but then there are other areas of life where you can’t really just speed things up due to using technology, like biological processes, for example.

NBL: Or art. One example in the article is that you can imagine playing all of Bach’s work in a microsecond, just executing the notes, but then you won’t really have played Bach’s work. So I think there are some things that only exist if they’re performed temporally in a certain way.

Philip Bell: Exactly. I sometimes listen on 2x speed to audiobooks and sometimes my girlfriend looks at me like, “Why are you doing that? You shouldn’t listen on 2x speed.”

NBL: Yes. But it’s a bit slow, I do that too. I never listen at speed to Bach though.

Philip Bell: Yeah, true. I think listening to music is a whole different thing.

NBL: If you find somebody doing that, you would have found somebody who has a really different attitude to music, where the point of the music was the informational content, not necessarily... I can imagine that maybe there’s a professional musician that does that, listens to music sped up. But it’s a really interesting thought experiment: who would actually be listening to classical music at 2x, 3x pace because they just wanted to consume it faster? That’s interesting to me. I would love to know if there’s someone out there doing that.

Philip Bell: Yeah, you could listen to multiple albums in an evening. I guess if you’re trying to experiment and think about new types of music, maybe you might just warp music in lots of different ways including speeding it up and slowing it down. But it wouldn’t be, as you said, in that scenario it wouldn’t be just trying to get the information content of the music.

One thing I was going to ask about this is: Mancur Olson’s work on building coalitions is interesting in this regard because he argues that coalition building is necessarily a very long-term endeavor. And I’ve been really interested recently in the argument by Henry Farrell—he’s been arguing that AI should be considered a cultural technology that’s equivalent to the market or equivalent to democracy in that it’s basically a coordinating technology because it can classify at new speeds.

And I guess what I was wondering was: building institutions, how is that time-bottlenecked? Is that something that could be sped up using AI, do you think? Or is that one of the things where it’s got to find its time speed?

NBL: Yeah. No, I think it’s hard to speed up building institutions, partly because institutions are the result of repeated interactions. Now you could imagine artificial intelligences building institutions of their own much faster, because they can have repeated interactions much faster than we can as human beings. So there is this world in which you have a set of institutions that are solely, exclusively catering to different kinds of agents.

So these institutions would then be different contracts, say, between agents that they all agree to, standardized ways of thinking about delivery of goods, those kinds of things. And maybe you could get a Lex Agentia, which would be like the Lex Mercatoria between traders back in the day. And they would have their own law, they would have their own institutions, their own way of resolving disputes.

But if you think about any institution that contains a human being, that human being needs to repeatedly be able to interact with someone for the institution to arise. And then they need to invest their intentionality in something. John Searle famously said that institutions are the product of collective intentionality. And I think there is a lot of truth to that, which means that the pace at which you build institutions is the pace of intention. So that’s where you sort of end up.

And I think, again, when it comes to hybrid institutions where we have both agents—artificial agents and people—the lowest common denominator holds. We need to feel that we interact with the technology in such a way that we over time come to trust it.

Philip Bell: That’s really interesting. And I guess going back to your previous point about supporting agency, I wonder whether AI could help build collective agency as well as individual ones. I know that the Collective Intelligence Project have been doing work on getting AI to quickly find the commonalities between 30 people in a room, for example, which would be hard for a human to do in real time, but would facilitate a real-time conversation. I think that’s an interesting, also potentially positive vision of use of AI—supporting collective agency as well.

NBL: Yeah. And you can almost turn it around. You could say: I think that agency primarily is collective. The notion of individual agency flows only from this idea that we together create agency. If you had one single person in the universe, would they then have agency? Would it be possible to have agency then? Whereas agency is actually dependent on being a part of a group where you can direct your will in certain ways and others can either join with you or oppose you.

So I think that a lot of the agency we have is collective first and individual agency is a result of that. It’s just like individuality. We have this idea that we are born with an inner individuality and then that individuality expresses itself and we are who we are. But in reality, to go back to your quote from Heidegger, Heidegger notes I think somewhere that we’re “strewn in the eyes of others.” So identity is constructed by the eyes of others. And if that is the case, then I think agency might well be the same.

Identity and agency might both be collective phenomena before they become individual, because identity and agency are tightly coupled. So maybe the question then is: what happens to this collective formation of agency first when it also has artificial content? Can artificial agents be a part of forming our agency so that we actually want new things when there are artificial agents in the collective that shapes our agency in the first place?

And I do think that’s true. I think we’ll see certain very simple effects of that. Not wanting necessarily the same as the machine wants, but wanting in ways that are reminiscent of what the machine wants. Framing the world in ways that are reminiscent of the way that machine frames the world. And there’s this constant interplay between us and the machines that we use. Just like Heidegger—what are you? There’s the constant interplay between us and the tool or Zeug that we use in practical—say carpentry or something like that.

Philip Bell: That’s really interesting. That was just making me think, because I think one thing that’s important in my identity is memory. I know you were speaking about memory earlier, and I’m now interested to read Paul Ricœur’s work. One thing that I think is sort of an interesting possible use of AI is in supporting human memory.

We tend to think that as a society, relative to past societies, we’ve kind of forgotten the importance of memory to a large degree and our information systems are quite sparse and don’t support memory. Whereas in the past—this is slightly generalized—but in the past societies tended to have much more cyclical and thick knowledge environments which enabled people to retain information better. But in an AI world, I think it’s possible that AI could enable people to remember roughly whatever they want. Whereas today, most people kind of remember things based on chance. And I know...

NBL: But is that true though? I mean, I wonder if you remember based on chance because my pushback will be: I think you remember on the basis of who you are, the stories you tell yourself, the narrative structures you put into your life, and those have a selection mechanism built into them so that some memories surface. You come back to some memories because they’re constitutive of the story you tell yourself.

So there’s this distinction in some philosophy between history and memory where history is the things that happened and memory is the way you structure them and forget. And I think that there is a lot going on—consciously, or maybe not consciously, but going on in memory that is not random. I don’t think that what you remember is random.

It’s not as if you wake up one day and then you remember, “Oh, I had a glass of orange juice on the 4th of February in 1994.” If that were to happen, you would probably be quite surprised and you would worry. And your first question would actually be this: “What does that mean? Why do I now remember that glass of orange juice in February 1994?” That must mean something. And that sense of meaning is deeply tied with a sense of memory because it’s tied to the sense of story.

So we remember what we want to remember in the story that we tell about ourselves. And that’s not necessarily the historical events. And I do agree with you that AI could change that. It could make retrieval of historical facts easier, which might actually allow people to rewrite themselves. Because currently you strengthen the memory that’s in the story you tell about yourself. But the things that could tell a different story are forgotten. Your forgetting is also quite strategic. You forget who you do not want to be.

So to some degree, you could then imagine an AI that could retrieve what was recorded—not remembered, but recorded—and allow you to remember it, to make it a part of your memory and then rewrite yourself.

Philip Bell: That is really interesting. So it’s kind of a corrective almost—or well, corrective’s maybe a normative term—but yeah, it provides maybe a counterweight to the narrative you’ve built up. Now that’s a really good point actually. I mean, I think you’re right that maybe it’s not totally random. I guess that it doesn’t feel to me to be totally deliberate either. There are lots of things that I would like to remember that I can’t remember. I would say I forget most of the things that I read about and I don’t want to forget most of the things that I read about. And I guess that’s what I think is to me quite exciting about AI—it seems to provide the possibility that I will be able to remember more of the things that I want to be able to remember.

NBL: I think you want to make a distinction between retrieve and remember, because the AI might—you might talk to it about a book you read. This I recommend actually, I actually might do this when I read a paper or book: I speak to Claude and then I make sure that there’s a small note of it so I can come back to it and I can retrieve that conversation. The reason I want to retrieve the conversation rather than what was written in the book is I want to retrieve how I read the book because my reading of the book is how I want to remember it. The retrieval I want is the retrieval of the remembering, if that makes sense.

So to some degree, I think computational technology already allowed you to search, right? You had information retrieval technologies—they had nothing to do with AI. And information retrieval is different from remembering. But how we then retrieve remembrances is an interesting question.

And I think this goes to the point we made earlier that all of these mechanisms are much more layered, complex, interconnected and messy than we would like for them to be. The idea we think about memory in a von Neumann machine is clean. There’s this random-access memory, there’s this long-term memory and then you retrieve things from it and you put it into the computation. That’s not how we work. The narrative we have structures the remembering we do. Some things we want to be able to retain—like information, you retain information rather than remember it. And then you want to be able to retrieve that information later at some point. But if you do, is that really remembering or is that just retrieval?

There are all these nuances in language that we have evolved over time. And if we pay attention to the way we talk about it, we realize that these are different mechanisms, still reproducible I think. And I think it still would be really valuable to build an AI that can make a difference between remembering something and retrieving something. But it would be very different from just boiling it all down to the simple model where there’s memory and processing.

Philip Bell: That’s a really good point. I wonder also whether—because in some sense AI recalls older attempts to unify human experience through knowledge. I was really interested by an article by Patrick Hutton I read recently where he talks about different mnemonic systems of a time, including Renaissance mnemonic systems. They had memory palaces and Camillo built a memory theater and there was a memory theater in London built by Fludd. And he even talks about how Freud in some ways thought that you could get to some sort of unifying human experience through memory. But he argues that since roughly the ‘60s and post-structuralism we have actually lost that aim of trying to unify human experience through connected knowledge.

And I suppose that’s something that AI can, just by its affordances, support—connecting knowledge and enabling someone to reach into a common thread of human experience through that knowledge.

One thing I was going to ask you actually is: I know in one of your other articles you talk about how sensors might become more important in an AI age partly because of the ability to compute lots more information. And I wonder how that might interact with human memory and identity. I suppose if I’m able to actually get much more empirical data about the world... I don’t know if that’s how you were thinking about it, but can you foresee a world where I’m able to get much more empirical data about the world through other systems as well?

NBL: Yeah, no, you can definitely get the data, but then you have to translate it into your own sensorium. So you’re limited by the senses that evolution deigned to give you. You’re going to interpret it through your own sensorium anyway. But it’s such a good question because it gives us the opportunity to rephrase a paper by Thomas Nagel. Thomas Nagel once wrote a paper that was called “What Is It Like to Be a Bat?” He essentially asked this question very simply: because a bat has echolocation and we don’t, can we imagine bat consciousness with echolocation? Because the senses actually structure consciousness differently.

And going back to our discussion, then you can now actually ask the question: what is it to remember like a bat? What is it when a bat remembers and how is that different from how we remember? It’s a really interesting question because bats probably remember other things than we remember in a structure that we can’t replicate.

The reason I wrote the essay about sensors was I was fascinated by the fact that an AI essentially could collect any kind of physical measurement data. Anything you physically can measure can become a sensor. You can imagine that you track all of these different senses—and there could be millions of them—and then you could structure an understanding of the world from all of these different senses. And that in itself would give the AI an ability to understand our world in a very different way than we do. And you would have to ask the Nagel question: what is it like to be a multi-sensorial AI? And what is it like to remember like one?

And you can sort of go on and you would probably end up in worlds where the amount of senses... Say for the sake of argument that you had an AI that had a million senses—a million different physical measurements fed into a massive AI that in some way construes its understanding of the world from that, finds patterns, interactions, correlations—you can find different kinds of affordances in that world that we are not even aware exist between different kinds of physical quantities that we hardly notice. That kind of AI would have a very different consciousness, a very different intelligence from ours, and it may well end up being completely impossible to communicate with, because communication presupposes some overlap in sensoria, in the senses we have.

Evolution was extraordinarily parsimonious when it gave us our senses. We have, give or take, 6 to 11 depending on which biologist you talk to. And those senses are what we make do with. But if you look at the amount of possible senses that you could provide an AI with, that’s just a tiny, tiny fraction. The human world we inhabit is very small. AIs could essentially be sent out on missions to understand the world across sensoria that would be enormously different from ours. And I find that fascinating because I do think that if this were to happen, it would probably teach us a lot about the world if we could only translate it. And the translation problem interests me.

Philip Bell: That is so interesting actually. Yeah, and I guess it also would get us back into old philosophical questions about empirical information versus other sorts of information. And yeah, that is an interesting point about: even if the data exists, how do we translate it into information that we can understand ourselves?

NBL: Yeah. And we think about that with echolocation and bats too. When we think about how a bat thinks, we think that it thinks with this... we do our own voice, we’re like... I mean, we think it gets an echo and that’s roughly how it has to be. Now we know it’s not exactly like that because we know that their echolocation sense is much more advanced, but we have to translate it down to our own senses. So we translate it down to our own hearing. So we think it’s like hearing, sort of kind of, and then we translate it into our sensorium.

And I think that’s what we’d have to do with an AI that was able to sense different kinds of quantum fluctuations or different kinds of radiation that we can’t detect or multiple other things that we can’t understand. And we would have to figure out analogies or metaphors for what that is for us. Maybe quantum fluctuation is a little bit like taste—it tastes metallic. But I don’t know. So you would have to find these analogs in order to translate and understand. And in many cases you might accept that you cannot.

I mean, what if we get an AI—and this is the funny thing—what if you get an AI that has a million senses, it discovers a set of really interesting natural laws that we have no access to, but they can be translated into super cool machines that can be really good at stuff. And so we get these machines, we have no idea of how they work, because they work on the basis of data that’s been collected that’s way beyond what we can sense. We can get some kind of theoretical description of it, but we will essentially literally have built black boxes from these different sense datasets that we’ve been working with. And that’s not entirely out of the question that something like that could happen.

In fact, I mean, that’s sort of how evolution works. We’re just discovering now that evolution is a-theoretical. For the longest time, I stupidly thought that evolution would have to take into account how much we have understood by physics. So of course, it would just use Newtonian or Einsteinian physical phenomena. But now we’re finding out that evolution has been using quantum phenomena for a long time in photosynthesis, possibly in our ability to smell or in the ability of birds to navigate. We have this growing field of quantum biology.

And you have this a-theoretical force—evolution—that takes every small advantage that is physically detectable or sensible and turns it into fitness. And so we shouldn’t be surprised that there are quantum effects being utilized if those can lead to fitness. Now you can build an AI that has the same ability to become almost a-theoretical in the way it explores the world because it just adds senses and can measure all of the physical things that exist within its environment, and it can have that same a-theoretical exploration capacity.

Philip Bell: Wow. That is really interesting. And I guess that is quite post-human in having this a-theoretical... or revolutionary. But it makes me think: maybe that’s how we become more ecological in some sense. Or more... I don’t know, I mean... That’s a really interesting idea also, that through collecting information that our senses wouldn’t otherwise be able to pick up, we could actually understand what it means to be—well, maybe we can never understand, but we can maybe appreciate better what it means to be a bat or a tree or something.

NBL: We can be together with a bat, right? Our own “we” can slightly expand to accommodate the bat as well. And then some of the experience of the bat-likeness is in the bat and some of the experience of it is in us. So it’s this question of identity again that you raised in the beginning. What if this notion of individuality and identity is not necessarily as robust as we think it is?

Philip Bell: Yes, and I guess we probably, in most cases, we can’t be with them on a temporal level. I mean, I would love to be on the time scale of a tree, but I don’t think it’s going to happen anytime soon.

NBL: No, I think that’s right. And trees are interesting, temporally super interesting biological organisms. And as are mushrooms, for example. So there are tons of really interesting examples in nature of things that exist in different temporal dimensions, which is sort of what we’re saying that we’re building here. First wave of evolution was biological and then evolution could easily be machine-driven. A-theoretical, but at a different level with different capacities.

Philip Bell: Yes, and we didn’t get into religion. I suppose that is another interesting area that could be influenced by this. I just had one more question: you’ve already given me the Paul Ricœur suggestion. Did you have any other book recommendations?

NBL: Yeah, yeah, yeah. There’s actually a really good... if you like identity you should really read Oneself as Another, which is a great book by Paul Ricœur.

Plenty. So I mean, I think when we’re talking about these things, you mentioned Heidegger—Being and Time is a heavy read, but there is plenty of good commentary. There’s a lot of good stuff in there. I really like, in terms of how one approaches issues methodologically, I quite like Philosophical Investigations, Wittgenstein. I think it’s a really good book. I think it’s, I really think it’s readable. And a lot of people go like, “Wittgenstein, that’s really hard.” No, I think it’s actually just very interesting to dip in and out of books like that and sort of try to figure out how people are thinking. So that’s another one. Those are classics, so they’re obvious.

Ricœur I think is really good. There is a guy called—if I remember correctly, I don’t want to mess this up—it’s Pierre Hadot I think, who writes a lot about philosophy and identity. He was the inspiration for a lot of the work that Foucault did. Really good.

You’ve already mentioned Dennett. His From Bacteria to Bach and Back is an excellent book. Dennett generally is just very readable, and especially when you don’t agree with him, because then you really have to do hard work because he’s so robust in the way that he argues.

So I think those are some of the books, but there’s so much out there. Generally, I think one of the things we should do is to read more of the classics when it comes to artificial intelligence. And I think it’s good to go back to Turing’s original paper and just read it properly from the beginning. It’s good to go back to the early stuff from Minsky, especially now in the time of agents. Society of Mind is underestimated. It’s really good to look at things like what Margaret Boden did early on.

Or if you want to take the other tack and look at it from the human side, you should look at Hannah Arendt and The Human Condition or Simone Weil and The Need for Roots. Great books that discuss what it means to be human in the nature of machines. So I think that there is no shortage of reading. Don’t get me started.

Philip Bell: Awesome. Thank you so much. This has been so enjoyable and so... Yeah, it’s really helped me think about these things. So thanks so much and thanks for your article.

NBL: Thank you so much for having me and good luck with the podcast.

Philip Bell: To start off, could you outline the key arguments in your book, How Progress Ends?

Carl Benedikt Frey: The purpose of the book is to push back against the narrative that technological progress is inevitable. If progress was inevitable, it wouldn’t have taken humanity 200,000 years to have an industrial revolution. If progress was inevitable, most places around the world would be rich and prosperous today, and we wouldn’t see places that once prospered stagnating or declining or collapsing.

I think the reason that progress isn’t inevitable is that as technology moves on, institutions need to adjust as well. Different institutional settings are more conducive to different stages of the technology lifecycle. Early on, when you’re exploring, you don’t know if something is going to catch on or not, so you’re better off having a system where people take different bets and then you see what works out.

The Soviet Union was the most centralised economy the world had ever seen. If you were an aircraft engineer in the Soviet Union, you could develop a new engine and go to the Red Army to ask for funding. If they declined, maybe you had two or three other options. If they all declined, your idea would die with you. That’s quite different from the US system, where Bessemer Ventures famously declined to invest in Google back in 1999. They probably regret it today, but it also underlines that Google wasn’t a safe bet at the time—AltaVista and Yahoo were dominating search. To know if something will catch on, someone needs to take the risk and invest.

On the other hand, when technology is more mature, it’s more conducive to planning. Airplane technology was quite well established when Europe set up Airbus as a competitor to Boeing. The jet engine—the last significant innovation—had already been invented. You were catching up to a static target, not trying to catch a moving one. When that’s the case, it’s much easier to plan, coordinate, and scale.

The implication is that you need to move between these two phases. You explore, you get a prototype, you scale—and sometimes that runs into diminishing returns, so you need to move on to something new. That requires more decentralised structures. Some things are conducive to both exploration and scaling, though: having a large connected marketplace where people can move around and spread ideas, low transportation costs, good distribution networks. That helps both innovation and scaling.

Philip Bell: I really liked that argument—the idea that you need the push of exploitation and the pull of exploration. You illustrated it through the tinkerers and inventors of 18th and 19th century Britain, and then the centralised bureaucracy of Bismarck’s Germany and Meiji Japan. One term you used which I found evocative was the idea of adapting your “ecological niche.” Is it easy for contemporaries to understand what ecological niche they’re in? Is it obvious at the time, or is there also luck involved?

Carl Benedikt Frey: I think the big question is “make or buy.” Businesses do this all the time: do we invent something internally from scratch, or do we take existing technology, tweak it, and scale it?

When the Industrial Revolution took off in Britain, I think it was a fairly straightforward decision for many firms and states on the continent to buy rather than make—although there was a bit of both. States pursued various tactics to attract talent from Britain who could make things in Germany and France. It was more a question of catching up to the technological frontier than pushing it forward. Germany did come to push the frontiers, particularly during the Second Industrial Revolution. But the key question is: are you lagging behind or are you at the frontier? That determines which path you choose.

Philip Bell: That’s interesting for the current question about geopolitical sovereignty in the age of AI. Canada has championed Cohere, France has championed Mistral—even though those models don’t benchmark as well as Google, OpenAI, or Anthropic. Do you think the make-or-buy question is relevant today for AI specifically?

Carl Benedikt Frey: Most certainly. The question is how easy it is to access the technology you need from abroad. During the post-war period, American technology and know-how was readily available through Marshall Aid, largely because of concerns about Soviet influence on Europe. America wanted Europe as a buffer, and that paved the way for much of the technology transfer we saw. It’s less clear today that Europe will have access to some of the technology being developed in the United States.

One of the puzzles in my mind is that for the past couple of decades—this predates the Trump administration—Europe has been less dynamic, which is maybe not surprising. But it has also not caught up in digital. Europe managed to catch up in mass production in the post-war period but failed to do the same in digital. Why?

I think a big part of the answer is that the single market is much more harmonised for goods than for services. The IMF estimates that if you take all barriers to trade inside the European Union and add them up, they amount to something like a 110% tariff. Trump Liberation Day tariffs, self-imposed inside the European Union. That obviously caps the return to investment in digital in Europe. What both China and the United States have in common is large domestic markets that firms can scale into. For Europe, a key priority needs to be harmonising the single market for services.

Philip Bell: I’d never thought about it that way—that Europe hasn’t caught up in digital. There are European tech companies like Spotify and Klarna, but they’re few and far between. And it’s not a reflection of talent, because major tech companies open large offices in London to hire talented individuals at cheaper prices than in the US.

In the book, it seems like the nature of the specific technology has some bearing on what institutions are most fruitful. You argue that centralised Soviet bureaucracy was useful for building heavy industry, but not for digital. Is there a specific institutional arrangement that will particularly help in the age of AI?

Carl Benedikt Frey: I don’t know is the honest answer. With regard to the Soviet Union, they did well in heavy industry because they prioritised it—it aided the military-industrial complex, and defence was the key priority. As long as technology was static, Soviet elites could benchmark factory performance and hold managers accountable. The problem was that when mass production petered out, something new was needed for growth. That new thing was the computer revolution, to which Soviet contributions were essentially none.

Part of the reason is that when you introduce a new technology in production, you can’t really benchmark. It becomes much harder to monitor performance. And unlike China, which is much more regionally decentralised, there was very little space to experiment in the Soviet Union without pulling the rug on the entire system.

I don’t think AI is that different in that regard. AI development is quite concentrated in a few places. When technologies are implemented, they proliferate more rapidly than manufacturing industry did. And you see work beginning to migrate abroad as well—law firms reducing headcount in London, hiring more in Poland and India to save labour costs. AI aids that process because it reduces the productivity differential between a worker in London and a worker in India.

There are probably things that will be different with AI, but it’s too early to say what institutional arrangements will be needed. What we can say is that certain things matter regardless of technology: barriers to entry are important for innovation, and having a large homogeneous market helps scaling.

Philip Bell: Do you think AI will change the cost of exploration differently from how it changes the cost of exploitation? I’m building a startup, and AI has accelerated our implementation to an extreme degree. But the rate of exploration has increased much less—the people using our app still use it at a similar rate. We can build things quicker, but do you think AI will change the relationship between exploration and exploitation?

Carl Benedikt Frey: I think yes, but probably in similar ways to the internet and the personal computer. When I have an idea, I can check fairly quickly using my preferred AI tool whether somebody else had the same idea. But the internet did that too. AI will reduce the cost of exploration further.

The surprising thing is that despite the cost of setting up a company and exploring going down so much because of technology—the cloud has been enormously helpful to smaller firms—we’re seeing business dynamism in decline. Something is pushing in the other direction and has more than offset the advantages created by ICT and arguably AI.

What is that? I’m at the university, and we see new rules and regulations and forms almost every week. That might be extreme, but I do think regulation has something to do with increasing costs. If you take an extreme example: most people believe AI will have a material impact on medical discovery. Even if that’s true, you still have to go through clinical trials, which is tremendously expensive. You probably need to partner with a large pharmaceutical company. I’m not suggesting we get rid of clinical trials—they fulfil a purpose. But those safety measures do come at a cost, primarily prohibitive for solo inventors and smaller firms.

Another relevant aspect is what we incentivise. In academia, it’s publish or perish. When a new productivity tool arrives, we can do one of two things: use it to drill more holes, or use it to dig deeper and focus on one thing. It seems we’re incentivising people to do many things at any given point in time, which means attention is spread more thinly across multiple projects. We have research showing that the more projects you pursue at any given point in time, the less likely you are to make a real breakthrough. Those incentive structures might be another reason we haven’t seen a real upsurge in breakthrough innovation despite having tools that clearly aid exploration.

Philip Bell: That reminds me of Marconi, who developed the radio on his estate near Bologna. He probably didn’t have loads of distractions on his phone and could concentrate. It feels like today it would be harder to do that. Do you think AI could facilitate more of that tinkerer culture coming back?

Carl Benedikt Frey: Marconi had some help from the family butler too, so most people don’t have that. It’s entirely plausible that AI will democratise innovation. If everybody has access to their army of personal tutors and research assistants, that should tremendously aid people with good ideas. We could see an era of more decentralised exploration, unlike what we saw in the 19th century.

What makes me think that’s probably not going to happen is the experience of the past two decades. ICT has already provided many tools that should make exploration cheaper and easier, and we haven’t seen the re-emergence of the solo inventor to the same degree as in the 19th century.

It is true that the computer revolution in the United States was driven by new firms—Apple, Microsoft, Google, Amazon. But since the 2000s, we have a trend where new firms exit by acquisition rather than IPO. WhatsApp, Instagram, YouTube didn’t become firms in their own right. That’s not necessarily bad, but the trend towards killer acquisitions—where incumbents buy smaller firms just to shut them down and avoid competition—is concerning. The dynamism we saw during the early days of the computer revolution is no longer quite there, despite computers being much better today than in 1980.

Philip Bell: How useful is it to use historical analogies to understand AI? Sundar Pichai compared AI to fire, Demis Hassabis compared it to the Industrial Revolution, Alex Karp compared building AI to the Manhattan Project. China’s AI Plus plan compares it to the internet or electricity. Is it useful to think about AI in comparison to analogous technologies, or should we take it on its own terms?

Carl Benedikt Frey: It depends on the point you want to make. Going back to my previous book, The Technology Trap, I think the framework of replacing technologies that automate existing work versus enabling technologies that create new types of tasks—developed by Daron Acemoglu and Pascual Restrepo—sheds light on AI as well.

If AI is just used for automation, we’re more likely to see something similar to the First Industrial Revolution, where wages were stagnant and probably even falling at the lower end of the income distribution. But if AI is used to create new products and industries, we’ll see something more similar to the Second Industrial Revolution, where the automobile industry emerged as the largest industrial undertaking the world had ever seen, along with supplier industries, electrical industries, and essentially every household appliance you have in your kitchen.

But to say AI is inevitably going to reproduce the 1990s or 20th century pattern—I think that’s probably not right.

Philip Bell: In The Technology Trap, you describe how technologies have sometimes caused short-term disruption but enabled long-term progress, partly determined by political impact. Do you think society has begun to perceive technology as more deterministic? When Dario Amodei says 20% of white-collar workers will be automated, it feels quite techno-determinist. Does that change how institutions, governments, and populations react to technology?

Carl Benedikt Frey: I’m not sure if people have become more techno-determinist. Generally speaking, the further back in time you go, the smaller the cross-section of society you get on record. If you go back to the 18th and 19th century, Marx, Malthus, Ricardo—they all believed in some form of iron law of wages. They didn’t think technology could improve standards of living over the long run.

In the mid-20th century, there was enormous optimism about electricity and personalised travel. Today there’s a lot of anxiety over AI, and a lot of people saying, “It’s the way it is, let’s hope for the best and maybe plan for the worst.” I think there have always been some of these impulses, but I may be wrong.

Philip Bell: I’ve seen a debate between Arvind Narayanan, who argues AI is a normal technology, and AI 2027’s Daniel Kokotajlo, who argues AI is completely different because it will be self-replicating. Henry Farrell recently argued we should think of AI as a cultural technology, similar to the market or bureaucracy or democracy, because it allows classification on a large scale, which has never been possible before. Do you think it’s too early to make that classification?

Carl Benedikt Frey: I haven’t read those articles, but I don’t think AI is there yet. AI is not going to replace an organisation or a firm anytime soon. Anthropic did an experiment recently where they had a vending machine being run by Claude—stocking it and selling products, a relatively straightforward operation. But Anthropic employees are a bit more experimental, and they managed to trick it into selling everything at 25% discounts and stocking the machine with metal cubes.

This is a relatively straightforward operation, and I wouldn’t want AI running anything too complex right now. When you apply AI for fairly well-defined tasks with a lot of precedent, it does pretty well. But the messier something gets, the more moving parts you have that affect each other, the more unpredictable it becomes. AI generally doesn’t do very well at generalising to situations it hasn’t seen before in training.

Advances will likely emerge in the next few years. Time will tell. Right now, I think it’s best to think about the AI we have as a tool, similar to the internet or the personal computer. In the future, maybe—but I’m not sure.

Philip Bell: Do you think historical research will change in the age of AI? I saw an economist recently write a paper about how to use AI agents for economics. Are there going to be huge teams of AI agents going through archives?

Carl Benedikt Frey: I’m not a historian, I’m an economist, so I’m not sure I’m best placed to answer. But speaking to historian colleagues, the challenge is that for AI to be of much use, things need to be digitised. AI might help that process, but that’s the first step. I suspect historians who spend much of their time in the archives will be some of the last researchers to be replaced—but I may be wrong.

Philip Bell: That’s a good point. A lot of things aren’t digitised, and that information will become increasingly precious because AI companies want very specific data involving lots of reasoning. Do you personally use AI much in your work?

Carl Benedikt Frey: I use it a bit. Partly because I write about it, I feel like I have to. I find it quite useful for some tasks, less useful for others. But the only way of knowing is trying it. So yes, I use it on a daily basis now.

Philip Bell: One last question: do you have any book or article recommendations? It could be about the political economy of technology or anything else.

Carl Benedikt Frey: The book that made me want to become an academic was Joel Mokyr’s The Lever of Riches, so I have to plug that one. I really like Doug Irwin’s Clashing Over Commerce—it’s a wonderful economic history of the United States in general, and a wonderful history of trade policy in particular. And Adam Tooze’s The Wages of Destruction: The Making and Breaking of the Nazi Economy is a book that’s beautifully executed and a pleasure to read.

Philip Bell: Thank you so much, Carl. It’s been such a pleasure to speak to you.

Carl Benedikt Frey: Thank you. It’s been a pleasure, enjoyed it myself.

I have found his argument that politicians (and the public) feel we are in a situation of ‘temporal claustrephobia’ really useful for understanding our current landscape. Our actions feel both urgent, uncertain and irreversible.Are you feeling a sense of temporal claustrephobia or do you think there are ways to avoid this? Comment below!

Phil: Jonathan, you’re a professor of politics at LSE, and I was really keen to speak to you about your book In the Long Run, because your argument that it feels like in politics the future is being foreclosed has really structured the way I think about a lot of issues, including climate breakdown. But also, on an emotional level, I feel this temporal claustrophobia that you talk about in my own personal life. So welcome to the Tech Futures Project.

Jonathan: Thanks very much for having me, Phil. Glad you found something interesting in the book.

Phil: Could you summarize the argument in In the Long Run and maybe say a little bit about whether you’ve updated any of the ideas since you wrote it a year ago?

Jonathan: Sure. The book came out of a course I was teaching at LSE on how the future is used and abused in politics—the ways in which different outlooks on the future get promoted by different types of ideologies and interest groups, and the political implications that come with these.

It’s a book with a tortoise on the cover, which indicates something to do with questions of how much time one has—whether there are certain virtues of perseverance and deliberation that presuppose a certain way of approaching political time, and that get extinguished when one approaches it in the register of high-speed emergency.

A running theme is how the question of how much future is felt to be available—whether it’s open and abundant, or whether it’s felt to be closing in on the present—varies historically. The book starts with an earlier phase of modernity 200 years ago when revolutionary politics was very much focused on immediate transformation. Then over the course of the 19th into the early 20th century, you get conversely a lot of politics with a sense of abundant time. The slow timescales of evolution made their way into political thinking—incrementalism, gradualism. The Fabians in this country used the tortoise as their emblem.

This is quite in contrast to the present moment where, in a range of contexts—you mentioned climate change, ideas of climate deadlines, five years, ten years, twelve years in which to make certain key political decisions—this is clearly how we’ve thought about climate change for some years now. But it’s not just that. There’s also economic inequality and the sense that we have a critical window to arrest runaway concentrations of economic power, or somehow it’ll be too late. That sense of living in a critical moment where what can’t be done now risks going permanently undone.

You asked how my thoughts have changed since I wrote the book. Clearly one thing that’s happened is Trump coming to power in the US. I’m perhaps more aware now of the ways in which certain kinds of far-right, potentially fascist forms of politics prosper under these conditions of temporal constraint and the sense of the closed future—that things can’t be otherwise, that we’re living with a limited array of possibilities. So many mainstream political parties seem willing to push this, including Labour in this country at the moment. I think this really creates an opening that Trump made use of in the US and potentially that Farage and Reform will make use of in this country—an opening for that desire to occupy the territory of the future from a far-right perspective with ideas of disruption, of breakdown, of imminent civil wars.

We live in a time of temporal claustrophobia, and I think too many parties of the center and the left are too quick to adapt to this and become very reactive and emergency-focused in their politics. This then is part of what the appeal is of those types of far-right politics, which are all about saying the future is going to be different. It may be bad, it may be a breakdown of some sort, but you don’t have to simply reckon with the stasis of the present. Change is coming. Perhaps we’re going to be the agents of change as well.

Phil: That’s really interesting. The word “abundance”—I know there’s a whole abundance movement now. It’s interesting to consider whether we could have an abundant concept of time as well. What are the conditions that create this sense of emergency that politicians are operating within? Do you think it’s partly to do with our information environment—the short-term social media dopamine-fueled news cycle that we’re in?

Jonathan: I think it’s certainly partly that—the sense that everything is in such permanent flux, including the news cycle and attention cycle, that really the only way to get heard or politicize anything is to turn it into some kind of emergency that cannot wait. Anything that can be postponed is going to fall out of reckoning with this type of media environment.

But I think what that touches on, which is perhaps slightly deeper, is a sense of weak power. Things become an emergency when you feel powerless to address them. If you feel you’ve got plenty of control over a situation, or you feel you’ve got time on your side—even if maybe you haven’t got immediate control, at least you think you can play the long game—then you have the capacity to set out your agenda. Conversely, things become this reactive logic of emergency when you’re just dealing with problems as they arise, precisely when you doubt your capacity to really control them. The best you feel you can do is respond to them in real time and cope in some fashion.

This applies to different fields of politics. Politicians are working in that media environment you mentioned, but they’re also working with institutions that have been ravaged by austerity, that lack strong state institutions on which to rely. They may also doubt the capacity to command the allegiance of people that maybe in another age would have been long-term supporters. Think of party leaders who realize that electoral volatility is very pronounced and you can’t really rely on people sticking around with you. Again, you have to accelerate your actions because you can’t depend on long-term support.

In wider society, it’s not just how politicians see the world. If you think of emergency discourse that comes from below, from social movements—something like Extinction Rebellion, where this was pretty central to the framing of the climate change predicament as one of climate emergency—again, I think it’s got something to do with the fact that when you doubt your capacity to influence people who have political and economic power, and perhaps you doubt the public’s engagement with the question, then this logic of emergency is one of the few resources that you still have.

These general patterns play out in different ways across the political spectrum. The left has its emergencies to do with climate change and economic inequality. Right-wing politics has its own emergencies—things it constructs as such to do with migration, racial replacement. The language of emergency tends to be quite central to how the right and far right approach things they consider problematic. And there are centrist emergencies to do with AI or global geopolitics, breakdown of the liberal international order. From all these different political orientations, each has different motivations and content, but there’s a common theme of political actors responding to a world in flux and doubting their capacity to control it, therefore reliant on a register of urgency and quick decision-making.

Phil: That’s really interesting. It made me think of Antonia Matuschek’s book Hyperpolitics, where she talks about exit costs being much lower from organizations like political parties. I think she mentioned Five Star in Italy purposefully didn’t have any physical spaces they organized in as a party, which made it easy to get into the party and leave it.

I wonder whether that logic might extend to other aspects of trying to control your life in the medium term. In the UK, young people are buying houses much less than previously—there are loads of reasons for that—they’re also having fewer children, going to university less, and there’s a big attendance crisis in schools. To me, these are all medium-term investments. Peter Mandler wrote a book where he argued that in the 20th century there was only one decade where university enrollment went down—the 1970s—and he argued that people in that decade were losing faith in the future. Do you think there’s any link between those trends and the idea of a future being foreclosed?

Jonathan: I think there is. All these things you mentioned are long-term projects, often individual personal or family projects, but they require you to have a certain confidence in the possibility of planning, of thinking ahead by a decade or two. These are things that may have short-term costs—you only really engage them if you can see the light at the end of a tunnel you have to pass through to reap the benefits.

When you see a world that seems to be very fast-changing, that makes planning seem impossible or somewhat naive, that clearly makes all these activities that require a longer-term perspective less appealing. All the more so for those who are especially precarious in their economic conditions—this has clear class differentiations. Those in the most precarious situations are least able to count on things like even basic job security, and that’s a foundation on which to construct everything else you’ve mentioned.

One way of thinking about it: the role of political organizations like parties and movements has perhaps always been to counterbalance that, to give people collective perspectives on the future that allow them to have a future orientation despite conditions of economic insecurity they may be stuck in. If parties themselves stop making available more collective visions of the future, people are left reliant only on the futures they can muster from their own life course. Then they’re all the more challenged by the loss of confidence that comes with being unable to plan their personal economic or social security.

These all have multiple origins, multiple circular logics. To the extent that people struggle to buy a house or find accommodation to plan their personal lives, if you feel yourself locked out of these pursuits, then you start to lack long-term reference points. This in turn undermines the confidence in the future that you might otherwise have. There’s plenty of potential for vicious circles here, but the links are important. One of the stakes of the political abdication of the future is that it accentuates these problems as they play out at the individual level.

Phil: One of the things I found really fascinating about your book was your exploration of fascism. I hadn’t really thought about fascism in terms of impulsiveness as a tendency within it and their general approach to the future and history. Do you think there’s a risk that fascism becomes more appealing in this moment?

Jonathan: I think yes, partly as I said, it’s got everything to do with what other parties are doing. Whenever you think about the appeal of a certain political orientation, it’s maybe at most 50% about the intrinsic appeal of that orientation and at least as much about the appeal—or lacking appeal—of everything around it. Precisely as other parties of center and center-left become less willing to develop visions of the future, visions of alternatives, that does leave a vacuum.

But also, fascism is undemanding in what it asks people to think about the future. This is not like socialism, where maybe you have to have confidence in progress, confidence in the measurability of progress, the capacity to find evidence for it. Various forms of liberalism likewise require credibility that comes with accurate prediction. This is not really the game that fascism is in.

If we can speak of it as an ideology, it’s one where perhaps a lower weight is placed on internal coherence than many other ideologies like liberalism and socialism have tended to place. Fascism, going back to the ‘30s and even ‘20s, often emphasizes this as a doctrine of action and not of theory. People around Mussolini and Mussolini himself said exactly this: don’t judge us by some notion of putative logic—we are about action first and decision. We make our truths rather than simply formulating them in advance and then acting retrospectively in adhesion to them.

It’s a politics where you can make it up as you go along, where you don’t have to apologize too much for incoherence but also for uncertainty, for surprises, for not doing what you say you’ll do, because this is somehow all priced in with fascism. It’s a politics of unpredictability and wrong-footing opponents, sometimes even deliberately contradicting your own line so that it’s much harder for critics to hold a normative standard against you.

In crisis times where everything is changing fast anyway, where it’s very hard to plausibly make predictions or develop coherent programs of action, fascism just says: fine, we don’t do that. All we do is disrupt. We throw things in the air. We’re going to have maybe a bit of fun as we go about it. There’s an emotional appeal to the exhilaration of disruption and chaos, but also to the violence that comes with that, to the pain that comes with that. It embraces the chaos and unpredictability of crisis times and makes that somehow its calling card.

What I call the impulsiveness of fascism—the act-first-and-think-later, if at all, aspect to it—is always going to be peculiarly well suited to times in which that seems to be what is happening anyway, that things are just changing with rapidity that makes it very hard to be coherent and considered in your actions.

Phil: That indeterminacy is hard to fight against as well, or hard to argue with, because it’s a moving target. It reminds me of seeing Donald Trump being really nice to Zelenskyy recently, which wrong-footed a lot of people. That seems characteristic of his relatively indeterminate approach.

Jonathan: Exactly. It’s very hard to know what you’re against when you’re against someone of that disposition. In addition to the wrong-footing effect, it can allow such an individual to present themselves as free-thinking, charismatic. Precisely because they’re unpredictable they can say they’re authentic—”I say how I see it, I’m not trying to stick to a line just because it’s correct or something.”

This impulsiveness is not just bemusing to those around it, but acts as an indicator of unfiltered self-expression, which for certain types of audience can be read as a sign of authenticity, of strong will. Deliberate self-contradiction, or at least indifference to norms of consistency and coherence, is part of the appeal. It allows opportunism—you can take advantage of whatever seems like the opportunity at hand without thinking too much about how it coheres with other situations past and to come.

Phil: I read an article about lots of TV shows that are popular right now being game shows where you win lots of money—like Squid Game. The journalist was arguing that this displays some form of idea that our society is zero-sum and get-rich-quick schemes are one of the only ways out of a difficult situation. That felt like it spoke to what we were talking about earlier—lack of control over one’s life, and the only way out is potentially a game show.

Jonathan: Yeah, and the lottery would be the obvious example as well. In other words, rolling the dice is the best hope of a different future.

Phil: Do you see any signs of either politicians or institutions trying to regain a sense of the medium term?

Jonathan: Well, in some ways, sure, at the political margins on the left. There are plenty of experiments of efforts to organize in a way that’s not just about, or not even primarily about, the next election. In this country, clearly the Greens—and not just under Zack Polanski, but I think for some time this has been their position—obviously you’re not going to win the next election, but you have to think of how to develop some kind of cause that is nonetheless consequential regardless of whether you can understand that in terms of electoral success.

Under Polanski, that’s taken a much sharper and more influential form insofar as this is now speaking a left-wing language that seems to be pretty much absent in the rest of the parliamentary parties in Westminster. I wouldn’t want to say it’s long-termism at the expense of immediacy or urgency. Clearly the Greens, more than any political party, are aware of the climate science and the need to act quickly. But what you see here is a sense of beginning with urgency but continuing indefinitely—thinking the longer term, not just about how to win the next election but how to influence public discourse, how to influence other political parties that maybe will win the next election, how to build a cause that will outlive the particular electoral cycle.

Jeremy Corbyn and Sultana’s new party—we’ll see where that goes—is clearly some effort again to try and conceive a party that’s more centered on a political vision rather than an electoral machine. And possibly Zohran Mamdani in New York is something different. To some degree you have to realize that success is not the first priority. You have to find a way of building some kind of movement that’s going to maybe be successful in a few cycles further down the track in order to have any real political profile and substance in the nearer term.

These are experiments perhaps speaking to frustrations with the reactive logic of emergency politics, trying to pursue something that’s much more like a normative vision of politics rather than simply firefighting problems as they arise. What’s also interesting about a number of these experiments is it’s not just at the level of rhetoric, it’s about the organizational structure that underpins this.

There’s a fake response to the charge of short-termism which you see articulated almost everywhere, including in the last Labour manifesto, which is at the level of rhetoric saying, “Well, we know that everyone is frustrated with short-termism and crisis management, so we’re going to think a little bit more long-term, we’re going to think about goals for the next ten years”—we’ll give you however many thousand nurses by 2030 or whatever. These very specific, almost accountancy-like ways of sketching out the further future. I think that’s a fake response to short-termism because it doesn’t really go much beyond goals set by a leadership which itself is basically entirely in control of. It doesn’t really make itself accountable to anyone that’s going to hold it to those goals.

If you really want long-termism in politics—however you define it, simply something longer than the next electoral cycle—you have to create the political structures that will mean that leaders are held to those goals. You need to strengthen the position of activists in parties, for example, who are the ones that tend to join parties because they care about the longer-term project, the cause. If you don’t empower those people, you’re probably going to find that even the most well-intentioned leaders are going to make pretty severe compromises pretty quickly.

Insofar as you see any interesting experiments in the present, they’re interesting not just about talking about how we need to be a bit more long-term or visionary, but they’re also looking at that question of how do you empower those people for whom that is second nature—the people that join collectives without expecting any real immediate power but want to sign up to something they believe in. Empower those people and then you’ve got a more authentic kind of long-term politics.

Phil: It’s interesting you said some of those projects had a normative dimension to them, because I’ve often wondered—given the ecological crisis and climate breakdown and all the different competing candidates for what the most important emergencies are—what would be a coherent moral framework to use? I was even reading about how David Runciman reviewed a book arguing that having children could be good for climate breakdown because it changes the population structure so you can have more progressive politics. That stunned me, because I’d assumed having children was one of the worst things you could do for the climate.

Is one of the reasons why it’s difficult to approach these issues from a normative or moral point of view because it’s not clear what morals work in this current situation? I always think about Keir Starmer—he appears to know the answers, or portrays himself as knowing them. That’s a big contrast to how Trump presents himself, because he doesn’t present himself as knowing the answers since he changes them day to day. Do you think that’s part of the problem—that we don’t really have a coherent framework?

Jonathan: It’s certainly the case that any type of moral position comes under a certain strain in conditions that are plausibly seen as unprecedented. Morality presupposes some type of continuity with the past. Morals are those things that are understood to be inherited in some way—of course refined, something you can put under critical scrutiny, but not something you simply make up on the spot, because that’s not quite how we normally see morality.

Yet when you’re in a situation where the major issues of the day are so often said to be unprecedented—that they are unlike anything in the past—and clearly climate change is often presented in these terms, but artificial intelligence likewise—these all seem to put you on the cusp of a quite different type of historical situation. The susceptibility that society has to emergency claims in general is that if you can present a situation as unique, then you can say it needs a unique response that has nothing to do with existing normative precepts—that you’re in a new world, therefore let’s not worry about the morality we bring to it from the old world.

That is one of the genuinely confusing and disorienting aspects of the present. I think at some level one has to almost refuse a little bit the idea that the contemporary situation is quite so unique and unprecedented, because I think the cost of accepting that point is basically to embrace this disorientation. Then you can quickly find yourself acquiescing in the mere exercise of power and not really having any type of normative standpoint from which to take distance from realpolitik and the exercise of power by the strongest.

That is part of our difficulty, but I don’t think it can be allowed to be insurmountable. I think we have to assert the continuities. The world has always been a changing environment, and morality is always somehow requiring a leap of faith whereby we can say that ideas that developed before the present nonetheless have purchase in the present. That’s a leap of faith that is hardly new to the present.

The revolutionary moments we talked about earlier in the 18th century are first instances of that. Is the revolutionary moment one in which there’s no normativity, no morality? Well, maybe sometimes it looks like that when you get a lot of executions in quick succession, but I think at the same time one wants to say that even in these exceptional moments there has to be some kind of appeal to limits on what can be done, or at least some kind of moral compass has to be applied even if we disagree where that’s exactly going to point us to.

It’s part of the challenge, but it’s a challenge that has to be refused. Therefore it’s going to be about how to take existing normative frameworks, ideologies, and find ways of reconceiving them for exactly this kind of context. Ideas of 21st-century socialism, ways in which we might say there is something that is enduring in claims of equality—precisely what you want to see as living and dead in 19th-century socialism is clearly the key issue here—but there are certain enduring features of ideas of equality, even in ideas of progress and of the economic conditions that one has to pursue in order to make these things more than simply formal goods.

These are enduring questions. The things the left is talking about today—wealth taxes, land taxes, how to find forms of justice in production as well as in distribution—these are all questions that are just as important in a climate emergency or an escalating emergency of economic concentrations of wealth. The situations may look new, but I think there’s still a lodestar to be found in some of the more familiar ideological perspectives if we really think what they might mean in a new context.

For me that’s the position. Yes, this is what’s disorienting about unique circumstances or ones that seem to be such, but nonetheless the challenge must indeed be to find those continuities and distill from them ways of thinking normatively in the political moment.

Phil: That’s a really good point. I had one final question about the long-termism movement. There are people who seem to be thinking about the long term in a lot of detail, including effective altruists. People like Toby Ord have written books about the importance of considering the long-term view. In some ways, unlike the Fabians, they’re probably not gradualist—maybe they’re more accelerationist long-term thinkers. Because you talk about thinking about an abundance of time, these people seem to be thinking about an abundance of time to some degree. Would you say that’s true for this long-termism movement?

Jonathan: Yes, in a sense. They’re clearly stepping back to take the longest timescales into perspective—the timescales of transhumanism, of interstellar travel. Something I try to suggest in the book is that there are twin challenges to democratic politics that come both from what you might call short-termism and also from perhaps a kind of excessive long-termism.

On one hand, the first is more intuitive. Short-termism is a limit on political imagination because there are long-term goals that we might want to pursue—of equality, liberty—things that take time, that are not to be equated with transformations you can pursue within a year or two. These are projects. Clearly a politics that is deaf to the longer timescales required for deep structural transformations of society is going to miss something.

At the same time, this fascination with the furthest futures is something that can also be quite paralyzing, at least for democratic politics, insofar as it has a sort of flattening effect on the significance of the present. It’s not just these contemporary philosophical movements that have that effect. There are forms of utopian thought going back a number of centuries where you see the same interest in the far future, several centuries ahead, where the effect is somehow to diminish the present and to diminish the choices of the present.

If you’re thinking in terms of many centuries, does it really matter what you do in the near term? Anything less than an existential threat to the human race in the present becomes almost no problem at all, because the meaningful timescales of change are well beyond that. And conversely, anything that is an existential threat, or can be presented as such, trumps all competing considerations.

The long-termism that one sees in these effective altruist movements—philosophically it’s got some substance, but in terms of what its democratic implications may be, it’s really something that I think is almost as pernicious as severe short-termism, because it really doesn’t accommodate democratic disagreement very easily. It has a tendency to treat certain types of policy as trumps for the sake of the long-term survival of the human race, and likewise to see certain types of inequality as essentially irrelevant or perhaps even functional to longer-term gains in utility.

If you’re thinking about these questions of the future from a democratic perspective rather than a perspective in a certain kind of moral philosophy, then I think one has to—and here there’s no magic number and it’s somewhat arbitrary—but there is a kind of democratic future that I would put something on the scale of decades. Long enough that one can meaningfully think about structural transformations of society, and yet not so long as to make the contribution of living individuals and their commitment somehow epiphenomenal to the future that plays out. That still makes agency somehow meaningful in the present.

To that extent, I wouldn’t want to be an advocate of long-termism on the scale of millennia. While it may have some philosophical coherence—or at least the philosophical coherence is a separate question—in terms of its political implications, it’s clearly quite either paralyzing or quite deterministic.

Phil: Thank you so much. That was brilliant. I really enjoyed it, and thank you for the book. I struggle to stop thinking about it. I think about a lot of things through the lens of the book now.

Jonathan: Thank you. Thanks for reading it. It’s been a pleasure, and good luck with the podcast.

This is a really important question and Sean is a leading thinker and practitioner in this field. Do you believe China is racing for AGI? Why or why not? (Comment below!)

Philip Bell: Hamish, you wrote this article as part of the Center for Governance of AI summer fellowship. You’re now working as an AI policy fellow, which is really cool. And I guess it’d be really useful if you could just lay out the main argument of your article, which is called “Compute is Not the Answer to AI Sovereignty.”

Hamish Low: Yeah, definitely. Thank you very much for having me on. It’s great to be here. Yeah, so this is work I did at the Center for Governance of AI for three months, really trying to explore these ideas of AI sovereignty, basically trying to get to the bottom of what people actually mean when they’re talking in this space about AI sovereignty. I think there are not that many good answers once you start trying to scratch the surface.

So one of the first things I tried to do is just get a better definition of what AI sovereignty actually is, because it’s a bit of a wishy-washy term. I think it’s trending towards becoming a bit of an “everything is in” term, where you just sort of add “sovereign” in front of literally any AI thing you want to do, so it just sounds more serious and strategic. But actually it just isn’t the case—lots of things are not sovereign AI and are still good, but we shouldn’t be putting everything into this bucket.

But I think really the way you should think about sovereign AI is that it’s this big picture strategic dilemma. In the AI value chain, the two dominant players are just by far the US and China. Both of these two just have the strongest positions, they have the frontier models, they’re producing the AI accelerators, they’re way ahead of everyone else. So for every other country in the world, you have to figure out how do we fit into this world where there’s two very dominant players. And I think ultimately this is just a generic strategic dilemma. And the answer will be different for any individual country.

So each country has to figure out for itself what sovereign AI means, basically based on what they want to accomplish. For lots of developing countries who are trying to pursue export of digital services, this AI situation could be an existential risk to their development model, in which case it’s all primarily economic. It’s just like we need to salvage, make sure that the US doesn’t just re-onshore all of what we’re trying to do through AI.

I think for the UK, it’s slightly different. I think we are in a position where we can sit sort of in between the US, China and the EU. I think we’ve built up really impressive state capacity within the UK government around AI. And I think ultimately we have a much more ambitious goal, which is being able to freely regulate AI to help shape AI’s development on a much more fundamental level. And ultimately it’s about exercising freedom of action in terms of how AI actually affects UK society and economy.

And this is a much more ambitious aim and it’s much harder to achieve than if you’re just looking for economic benefits. And importantly it’s quite separate from economic benefits. You could have a lot of influence on the development of AI but not really get any of the benefits and you would have still achieved your AI sovereignty goal. So I think that’s how I see it for the UK. You have this AI sovereignty goal that’s about achieving freedom of action.

And I think ultimately that boils down to you want to influence the US. We’re going to be closer to the US than we are going to be to China. The US is just the clear leader and the key hub here. And so for the UK, the way I define AI sovereignty is it’s about creating interdependence with the US as opposed to where we are right now, which is this kind of one-way dependence relationship. We’re essentially getting all of these AI products from the US and we have very little stake in how these are being made and how these are being deployed.

And what we need to do is develop capabilities in the UK that the US comes to be dependent on, such that the US understands that we’re in this interdependent relationship, that there are important things that they rely on us for in AI. That means that when we reach some kind of crunch point, when we’re debating regulation or how this technology should be developed or safety issues, that they will have us in the room, that we clearly have an important stake on this, and that we have some leverage over them. That if they want to cut us out, this would be bad for them because they’re relying on us for important aspects of how they’re able to deploy AI and use AI within their own economy and military and government.

So I think this is ultimately the most powerful frame—trying to get towards this interdependence situation. The challenge is that this is incredibly difficult. The thing is that everybody wants their own ASML, the Dutch chip semiconductor manufacturing equipment maker, who has this insane monopoly over the most advanced lithography machines that are essential to making advanced AI accelerators. The issue is that there’s not lots of these lying around.

ASML was the result of this incredible process of decades-long consolidation in the uniquely neoliberal 1990s. In 2025, that’s not the case. Everybody is trying to find their niche in the AI value chain. Everybody wants their ASML. And so it’s very hard to find one. So I think a lot of AI sovereignty discussions in the UK are a bit just hand-wavy in terms of what’s actually valuable for us to build.

A couple years ago it was like, “BritGPT, we should be training models.” I think people now understand probably that this is not as good of a model and this doesn’t actually really get as much leverage. And then there’s questions around like, okay, so Nvidia chips are the big deal. We want our own AI chips, which is fine, you can make a case for this. There are cool startups in the UK working on this. And I think this is good.

But I think it’s just not clear that this is useful from an AI sovereignty perspective. I think you can make the case that this is economically good for the UK. If we have big successful chip makers, this is good for the UK in general. But I think it just isn’t very good for the UK’s AI sovereignty. Because if you’re trying to create this interdependence, you just don’t really get this if you’re trying to sell the US AI chips. Because ultimately, the US has other AI chips to buy. Most of the other key players here, whether it’s Nvidia, AMD, Google, Amazon, they’re all doing this themselves.

And even if you carve out the specific bit of the value chain, if you say we’re going to make our own AI chips, this does not give you in any way independence in this value chain. You’re using all of the exact same supply chain aspects that everyone else is using. You’re getting them manufactured at TSMC, you’re getting South Korean high bandwidth memory, you’re having to buy networking switches from this random Taiwanese company.

You’re incredibly dependent on the rest of this value chain, which functionally is very dominated by US regulatory regimes and the foreign direct product rule and their ability to put export controls on these capabilities. So even if you’ve built an AI chip, this in itself doesn’t give you any sovereignty. The US could cut you off from this wider supply chain and you’re screwed. There’s nowhere to go. You can’t make any more. You’re still very dependent on the US.

And you’re not generating something where the US is dependent on you. So I think this just doesn’t have a very clear AI sovereignty advantage. And I think this is true for just a lot of aspects of the AI value chain, whether you’re talking about building data centers in the UK, which I think is another useful initiative. And there’s certainly lots of merits of having data centers and AI compute in the UK. It lets you build out your public compute resources. It is investment, which is always nice from an economic perspective. You can build up some UK companies like EnScale, who’s building out with OpenAI in the UK. This is good, but again, it doesn’t solve your AI sovereignty question. It’s nice for other reasons, but putting a bunch of Nvidia chips in a very fancy warehouse in the middle of the UK is not making the US dependent on you. You’re just strictly a customer here. And it has other benefits downstream in how you use that compute. You can use that to build new products and do new things with it. But it doesn’t solve this core question.

So then I think what you need to think through is, okay, where actually do we get leverage? And where can we build this dependence? I think ultimately the view I’ve come to is you have to look at what are the future nodes of the AI value chain that are going to be important. Because for almost all of the existing areas, there’s just very powerful US incumbents that it’s really hard to compete with. Even if you deployed the most incredible industrial policy the UK was capable of, I think you would struggle to really generate US dependence. So yeah, this is where I then get into this idea of AI middleware, which I can walk you through.

Philip Bell: That’s super interesting. I think that’s such a useful phrase—interdependence rather than independence. I think that is a really kind of both clear but powerful encapsulation of your argument. When you were talking about the chip sovereignty idea, it made me think of Graphcore, because I think they were kind of an early competitor with Nvidia and they created kind of IPUs. And it kind of reminds me of the nuclear energy competition, because I think that Britain had their own kind of rival form of nuclear energy to the US in the form of gas-cooled nuclear reactors.

The idea’s got kind of interesting historical parallels, but yeah, it’s a really interesting point that actually that isn’t really what we should be focusing on at all because it doesn’t even create interdependence if we were to create a Nvidia competitor. And it also makes me think of the Jeffrey Ding book, the recent Jeffrey Ding book about fusion rather than kind of coming leading sector.

But yeah, I think it would be really useful if you could lay out that idea of AI middleware or where you think the UK could, or any country I suppose, could kind of become interdependent.

Hamish Low: Yeah, definitely. And the Graphcore case I think is interesting. And yeah, they spent like 700 million pounds and they made like 4 million pounds in revenue at their peak, which to be fair to them is just bad market timing and you just go down the wrong direction and these startups are hard. I think now lots of the team from Graphcore after it went bankrupt and then got bought out by Softbank is now this new AI chip startup, Fractile.

And it’s all very cool and I wish them the best, but yeah, they just don’t solve this strategic question we’re trying to solve. And I think that’s where something like middleware is maybe the better way to be thinking about it, which is basically here, I’m thinking through all of the future things we would expect to emerge that sit in between—if you’re going down a sort of value chain, if you imagine it and you’re moving down the semiconductor supply chain to making the AI chips, to putting them in data centers, to training the foundation models.

Right now, there’s not very much below that. It’s kind of like OpenAI trains a model, they put it into ChatGPT, you use ChatGPT. This is a very simple, vertically integrated setup. But I think you should expect this to just become much more complicated in the future. And for many more parts of the value chain to sit between the foundation model and the end use.

Where the end use here is almost certainly going to be an enterprise user, because this will be the much bigger market over time than the current consumer-dominated chatbot products. And just enterprises are going to have loads and loads of really different needs in terms of how they integrate these systems and build new workflows around them and do all these interesting things where they’re actually automating stuff and diffusing these deeply into their processes in the way that we actually get all the productivity gains that we want out of this technology.

So a few of the things I would put in this middleware bucket would be if you’re taking models and you’re altering them in various ways. So this could be fine-tuning models on specific datasets, you’re a downstream developer where you’re taking the OpenAI model, you’re then just using other data to change it to suit it to what specific task you’re trying to achieve.

And then you’re either vertically integrating that into your product or you’re selling that on to someone else who’s putting that into a product that they’re then selling to an enterprise. Or you have this new mode of reinforcement learning that’s become very powerful. Lots of the frontier model companies are pursuing building these reinforcement learning environments to make models much more capable at very specific tasks. The most successful one so far being coding tasks.

Where you’re able to just get lots and lots of gains by having models do this reinforcement learning process. If this process is really effective, which it seems to be, you would expect to want to do this for basically every other sector of the economy by the end of getting to 100% automation or whatever crazy future endpoint. That you’re going to want to find the right data and you’re going to want to find the task and you’re going to want to train the model to be just superhumanly good at doing that task.

OpenAI and Anthropic and Google simply can’t do this work for every sector of the economy. Eventually, they just become too big, they become too unwieldy, there’s regulatory barriers, there’s recruiting the actual people to do this thing. Apparently, there was some article, OpenAI has hired 100 old investment bankers to try to get data to do better financial modeling. But it’s just hard to do this for every industry on the planet.

Philip Bell: I mean I was really surprised—I’ve been quite surprised how many different industries OpenAI are trying to target. They’re like, I saw they’re trying to make an animated feature film which seems just really surprising. I have no real idea why they decided to do that but it’s intriguing. But no, I mean the general point I totally—I mean that just seems to make a lot of sense that there needs to be kind of specific parts of society that are specific verticals that integrate AI into the business process in kind of specific deep ways. And I guess, yeah, it tallies with a lot of kind of historical research about how technological change happens. It takes quite a long time to show up in productivity and involves changing business processes, et cetera, which yeah, so that’s really interesting.

And I guess specifically for the UK, because this kind of AI middleware idea—I’ve always thought the UK is in a great, when I say great position, I always thought the UK is in a central position in terms of AI, given that DeepMind was started in the UK and it’s arguably one of the most, well probably the most successful AI company in terms of the range of different breakthroughs they’ve kind of—I mean, in terms of AI research, it’s probably potentially the most successful in terms of all the different breakthroughs, like AlphaFold, AlphaGo, et cetera. And there’s also just a lot of important organizations like Stability and Eleven Labs, organizations like that and Anthropic and OpenAI have offices in London, for example.

So I’ve always kind of assumed that the UK was in a kind of quite pivotal position, but I think reading your article actually made me slightly rethink that, in quite a useful way. But I guess on this idea of AI middleware, do you think that the UK specifically is in kind of a better position to be able to come in and be part of that kind of AI middleware because of the fact that it does have essentially a lot of expertise, I suppose?

Hamish Low: Yeah, definitely. I think the UK’s strength is being really, really central on talent, right? We do really well on talent. We’re probably the best AI talent hub outside of the US and China. Our key weakness is that we don’t have very much compute. We don’t have lots of this infrastructure based here. We haven’t managed to have one of the leading frontier companies based here. DeepMind sort of being the weird exception where it used to be more UK weighted and then they merged with Google Brain and now it’s a bit harder to tell how you should understand them.

But I think this means that we have lots of talent here, but we’re not necessarily getting a lot of benefit out of that talent. And we’re not necessarily getting benefit in strategic terms from having that talent here. You want to turn this into some more uniquely UK capabilities. And this comes back to some very long-lasting issues with British tech in terms of startups struggling to scale, often just selling to US buyers, chasing the US market instead of the UK market because of just slower growth in the UK and regulations and et cetera.

But yeah, I think middleware is one way that you lean into this talent advantage of what the leading frontier companies of the US are doing is this big bet on huge infrastructure. They’re building out colossal amounts of this AI infrastructure just gigawatts and gigawatts. There’s just constantly more deals. The UK just clearly cannot keep up with this. And I think it’s not worth trying to keep up with this. But we can pursue this different approach where lots of these middleware capabilities, you don’t need that much compute ultimately. Often you’re just kind of building software systems on top of these models to make them more useful.

So to give one example, maybe—lots of models right now, a big evolution this year has been model context protocol and the ability for models to natively use tools as they accomplish a task for you. And right now this is just existing random software tools or just the ability to use a sort of coding application natively as it works through your answer. But presumably over time you’ll want to build tools that the models can use that are entirely separate from the models, but which are entirely AI native.

It might be that for the telecoms industry, you have a tool for monitoring the health of a network that is functionally just a software tool that does this job, but just in a way that is built for the model to be able to use it effectively. But you don’t actually want the model to probably be zooming through every aspect of BT’s network trying to find some fault. You want it to be able to access an actual system dedicated to do this and be like, okay, yes, check. This is all good. I can move on to the next stage, whatever I’m working through. And that’s a business right there.

You can build that tool and you can sell that tool. And I think that’s one of the things where I’m envisioning middleware here. And so I think this is one where if you just have lots of talent and you just get the right framework around being able to have them create startups, build these capabilities and sell them on, that you could create this just web of UK software focused companies, which means we can build products faster. They can scale them faster. We can actually create this interdependence much faster than if we were trying to build hardware capabilities where it just takes a decade to create something at the end of a long R&D process and get this to market and get it to scale. That actually you can move at the pace of AI development in this sense by being very software heavy and avoid this compute heavy models, but instead we’re trying to build on these and create useful things that just allow us to use these models in the real world. And I think, yeah, this leans into this talent heavy approach that I think is the right one for the UK.

Philip Bell: That’s really interesting. Is there kind of a government policy that would help with that? Or is that something... or there are other kind of conditions that would maybe stimulate that?

Hamish Low: Yeah, and I think the government has been doing some very useful things here. I think the Sovereign AI Unit is a very interesting project and has been doing some good work. They have 500 million pounds of budget to spend on achieving this aim. I think this is good. I think the AI Security Institute is a hugely helpful thing here in terms of this is one area where you’re leaning into this talent advantage. You’re just getting a really good cluster of talent within the government, building the government’s state capacity, but then also just hopefully building up an agglomeration of different things around this.

So one thing they’ve done is they’re looking into the AI assurance industry as one potential useful area where the UK has an advantage and DSIT has given some money to this and it’s very much on their radar. My view is basically you should just be doing all this but just doing it much more, that I think there’s all this work happening, but there’s insufficient urgency behind it for how quickly things could change and how powerful the capabilities could get. I really want the UK to be having this leverage, not in 10 years, but in two years or five years, that I think you could have very important decision points in the near future where I want the UK to have this leverage, which means I think we just need to be much more aggressive in doing industrial policy in a way that the UK government has not traditionally done.

And I think this is just—you basically have your key levers, which is capital and talent and compute, where compute is kind of a function of capital. And you just need to go super hard on these. So you have the Sovereign AI Unit. I think basically you should also create a national AI investment fund and you should create another fund and then another fund.

And you should be very okay with just duplicating all these different government investment vehicles and being like, taking this almost much more Chinese approach to industrial policy of just opening a floodgate of capital and just being very willing to just be—we’re going to change this market deliberately. This is what we’re trying to do. We’re just going to try crowd in much more competition to these sectors we think are strategically important. And we’re going to do this in a really big way, which ultimately has a lot of downsides. You know, it ends up being—you will create waste, you will make mistakes, some of these companies will fail, you have no guarantee of success. But I think if you really understand the urgency I think we should have here, that you’re willing to do this, you’re willing to put a lot of money after this goal.

Same thing on talent. I think that if the Sovereign AI Unit invests in you and has done their due diligence, they see you as strategically important. You should just be able to hire the talent you want. You could just have a visa where you, startup, decide you want to hire someone and as part of that process, they get a visa to come to the UK. You just skip all the Home Office fees, you skip all the Home Office vetting, pretty much, and you just delegate this power to the firms that you think are really important. And you could have this be just a super limited scheme, but I think this is one way of just speeding everything up and just doing everything we can to bring talent to the UK.

As well as just kind of other interesting ideas around the Center for British Progress has this idea for a kind of Center for Talent Excellence of deliberate headhunters in government who are trying to recruit the best AI engineers into the UK. I think these are great ideas as well. I think you basically just want to be maximally aggressive and just pulling these levers of capital and talent to just bring as many people to the UK and remove every barrier to them creating the companies that we want to exist.

Philip Bell: Yeah that’s a really interesting point because I suppose right now given some of the kind of restrictions on studying in the US and stuff like that I imagine there’s lots of PhD students who might be interested to come to the UK. It also reminds me of—because I mean I know Joel Mokyr just won the Nobel Prize for economics and as far as I understand, one of his main arguments about why Britain was central to Industrial Revolution was this idea of tinkerers and people who... a kind of dispersed network of different intellectual societies, like people from the Birmingham Lunar Society who went on to tinker with different types of technologies, which became hugely influential.

I sometimes wonder about that, like is there something around supporting a dispersed experimental culture? And maybe the best example I’ve seen of this is maybe Kaggle. I don’t know if you know Kaggle, but it’s an online data science competition thing. And like, I’d feel like that’s a kind of—I mean, yeah, that’s one example of kind of a tinkering sort of experimental culture. So yeah, so I think it’s a really interesting idea to lean into the UK’s kind of wealth of expertise and kind of capacity of the people. I think that’s really interesting.

And I guess because, but one thing I was going to ask about what you were saying about the kind of funding. I thought that was a really interesting point about basically just providing more of it and it’s fine if there’s overlapping kind of pools of funding. And I also thought your point about the AI assurance industry is really interesting because I mean it does feel like the UK has a lot of important kind of AI safety and AI governance organizations.

And but one thing I was going to say is with that kind of funding policy, some people might say, okay, we shouldn’t be picking winners. I have heard Tim Wu, I think he’s called Tim Wu, he’s an American kind of academic slash policy—I think he’s involved in the Biden administration, but he has the argument that you can pick sectors and ecosystems kind of thing rather than picking particular companies. What would you say to that? Is that potentially a challenge to funding, even funding the kind of expertise and stuff and also things like funding a more thriving safety and assurance industry kind of thing. Is that, do you think that’s a challenge? That thing of like maybe picking—it’s hard to pick the right ones.

Hamish Low: Yes, I think this is just always the challenge with industrial policy. I think maybe a counter example of who I think is doing a bad job on this is France or Canada, where both of them have kind of landed on national champion model development companies, where you have Mistral in France, where ASML invested in Mistral, and they’re getting lots of support from the French government. And it’s like the French government wants to preferentially use their models and this kind of thing.

You have the same in Canada with Cohere, where the government has given quite a lot of money, hundreds of millions of Canadian dollars, towards helping Cohere build out AI infrastructure in Canada. Clearly, viewing this as a national champion firm, I think this is a really bad idea. Both these companies have demonstrably fallen quite far behind the frontier in terms of the actual models they’re producing. They’ve fallen really far behind in the infrastructure they have access to, and they’re really far behind in how much actual money they’re making, I think like 100 to 150 million euros slash Canadian dollars when OpenAI and Anthropic are charging heads to tens of billions of dollars worth of revenue a year. And this is just a complete disparate relationship. And the frontier model developers are just going to be on this insane flywheel and just be way too far ahead of you. And they make models that are way better. They make their open source models an afterthought and the open source model is better than your closed source one. Like your business model is kaput.

So I think this is a really bad case of them picking winners. I think this sector approach is much more interesting. Picking another one from the recent Nobel winners, Philippe Aghion has this really interesting paper about Chinese industrial policy where he identifies one of the successful features of Chinese industrial policy is this kind of picking sectors where they pick a sector like electric vehicles, where they decide that this is productive and strategically important. And essentially by just funneling loads and loads of subsidies into the sector, they essentially just create much more competition because it encourages so many new entrants into this market that suddenly you just have this incredibly competitive market, which can go wrong. You sometimes end up where they leave the subsidies in too long and get this kind of involution—insane price competition where nobody makes any profit and everybody’s very sad.

But the core principle I think is good, that you can pick out sectors that are strategically important or that you think are economically important. And you can do this process of just making lots of capital clearly available. And this just crowds in lots of new entrants and makes this market more competitive. I think this is the vision for how this goes well in the UK, is that you identify some of these sectors you think there’s a lot of opportunity here. We want to put a lot of capital on the table that we’re willing to basically subsidize the market here in a sort of anti-market way. You’re trying to shape the market to not develop how it otherwise would. With the goal of crowding in just loads and loads of new players who are then going to be competing with each other very intensively, you’re going to get these kind of talent agglomeration effects, which I think is one way you get these kind of useful networks of talent, is that you just have a shared sector and you have a lot of capital available so that people are like, I’ve just done my PhD, I don’t want to move to the US, I’ll go to the UK, there’s loads of money around, I can get a job working on this cool AI sector.

So you get these networks of talent, you get these new firms, you get a very competitive market, and then the hope is that out of this very competitive market, you then just get very innovative, strong firms who are able to scale up and then become globally competitive players. And ultimately then, this means the US will start increasingly buying their stuff. And then hopefully, you have enough Fortune 500 companies that are buying their software solution for whatever this AI issue is they’re trying to solve. And this is what’s giving you this kind of interdependence.

Philip Bell: That’s really interesting. Yeah, I’d like to look at that article actually. I always think it’s—I mean, I kind of follow the kind of China Watchers community. And I think as far as I understand from listening to some of the podcasts like China Talk and other podcasts like that, I think one of the kind of insights that they have is that policymakers in the UK and the US and other places don’t really pay enough attention to what’s actually going on in China and don’t really understand China enough given its importance. And I do always think, because whenever we make policy—or at least whenever I hear discussions about policy, it seems all of the time we kind of take ideas from other European countries, which I guess makes sense because we’re a similar size, maybe. But yeah, I always think it’s interesting that people don’t seem to, or at least people that I hear talking don’t seem to take as many ideas from Southeast Asia or China.

One thing I was going to—yeah, one thing I was going to ask though is, are there any countries that you think are doing this well? Like they’re approaching AI sovereignty well and maybe creating some sort of interdependence. I mean, because I thought the France and Canada examples were really interesting about—I hadn’t really thought about that actually about how Canada is supporting—I didn’t actually even realize that Canada was supporting Cohere and then the France supporting Mistral, that was a really interesting point. But yeah, are there any countries that you think are doing well?

Hamish Low: Yeah, I think it’s basically just countries that have been able to figure out what their advantage is and be able to capitalize on this in some way. So I think Singapore has done very well here, which is partly a result of the fact that Singapore, I think, just has really good state capacity and had already done lots of work around getting data centers up and their industry, building up themselves as this key hub between the US and China. I think they’ve basically just been doubling down on this strategy that makes a lot of sense for Singapore.

That your entire gambit is that you’re sitting between the US and China. You’ll be a really good talent hub. They have their own national AI strategy 2.0. They’re trying to triple their AI practitioners in Singapore to 15,000 people. And I think this is just robustly a smart strategy. If you’re Singapore, this is the strategy you want to do. And they’ve just been very successful—Singapore and now Johor in Malaysia just across the border is the key AI data center hub in all of Southeast Asia. There’s huge investments by ByteDance working with Oracle in Malaysia. Singapore has become a really key hub here. They have the Singapore AI Safety Hub. This is just, I think, one example where you have a middle power that just has a really clear sense of what they’re trying to achieve and just has a state that’s very focused and willing to achieve this. And I think this is going well.

The UAE is another interesting one where I think, again, they sort of understand what they have to offer, which is loads of capital and loads of energy. Realized that these were two things that everybody else wanted and capitalized on this quite well. You know, they signed all these various deals with the US, all the kind of major US AI players coming and building out data centers in the UAE. I think once you sort of get below the surface of it, it’s slightly less clear how good this is for the UAE. They were very, very smart in that they timed their announcement really well, where when they announced all these different data centers, they were the largest ones in the world that have been announced. And subsequently, much bigger ones have been announced in the US that all come before the ones in the UAE. But still, everyone has the impression of, wow, the UAE’s at the frontier, which they probably aren’t. And the business model of how good it is to basically just be a landlord for a giant warehouse in the desert is slightly questionable.

But I think it’s still a good strategy and clearly gives them leverage. And they have some interesting ideas around exporting AI to the global south. And this is—it’s a strategy and they’re making lots of bets on it. They’re making the right bets, I think, if you were trying to carry this strategy off. And I think this is—they clearly have a sense of what they were trying to achieve and they’re working to achieve it. And I think otherwise it’s just—it’s hard to find countries that have a sense what their advantage is, partly because for most countries they don’t have much of an advantage. This is an incredibly sophisticated value chain that just doesn’t flow through that many countries.

You have a lot of challenges as well of a lot of what you really care about is talent, but talent is very mobile. So it’s a big issue for Latin American countries where you might have good AI researchers, but they’re all just going to move to the US or Europe. It’s hard to build up a sort of domestic base and work from there, which actually makes these kind of AI sovereignty discussions really, really difficult. You’re really struggling to build any advantage, let alone build a strategy to capitalize on that advantage.

Philip Bell: That’s really interesting. Yeah, I mean because to your point earlier about every country wants to have an ASML like the Dutch do, there are—I mean because Germany and France I believe do actually have quite important companies in the kind of AI supply chain. I believe that Germany has the Carl Zeiss SMT—Zeiss, the optics and laser which I think Germany is always quite strong in terms of making glass. They’ve got a long tradition of... I actually have a friend who goes to this, he’s an artist, but he goes to this glass making conference in the Black Forest every summer. So I think there is kind of—it’s probably like an ancient, not ancient but really old tradition of kind of German expertise in kind of glass which is kind of interesting that they do seem to make lenses for ASML and stuff like that. And I think they have chemicals which again is a really old German industry like the chemicals companies and stuff. But yeah, as you said, there are only a handful of companies. It’s basically France, Germany, the Netherlands, and then South Korea maybe, Japan. But other than that, is that right? They’re kind of the countries that do actually have companies in the supply chain.

Hamish Low: Yeah, because I think right now, the question of leverage is just very tied to the semiconductor supply chain. Most countries don’t have a lot of presence in the semiconductor supply chain. Yeah, South Korea, Japan, Netherlands, Germany, they really do. And that just sort of gives you a leg up.

Philip Bell: Yeah. And it’s interesting because I suppose going back to your point about the UAE had worked out that their advantage is, or part of their advantage is that they have loads of energy, loads of fuel based energy. I mean, do you think that the UK should accept that the UK relative to other countries doesn’t have an energy advantage, I suppose? Do you think the UK should kind of just accept that or do you think—I don’t know if this is very naive, but I have heard some people arguing that AI is an opportunity to build really comprehensive renewable energy infrastructure. Do you think that’s a kind of realistic possibility that a country like the UK should be considering?

Hamish Low: Yeah, it brings me great pain, the energy situation in the UK. I would wish it were not how it is. And I do think that AI is an opportunity to try to change this. But right now we’re just in a very bad position on energy. This is very bad for AI. It’s also just very bad for loads of other industries and lots of other reasons. And I think there is a good case that AI is—it helps you encourage lots of these more innovative energy solutions.

It kind of is a bit of a forcing mechanism to help you confront some of these just long running—the grid is super slow and we need to spend lots of money on expanding the grid, on building interconnections. We need to just build out lots more generation capacity of renewables. I’m very supportive of trialing new things like small modular nuclear reactors and just building out more of the kind of traditional nuclear power stations. I think all of this is really good. The issue is it’s just—even if you speed it up to my dreams of how fast it could be, it’s still quite slow. You’re trying to compress a real decades long project into maybe a decade if you’re being super ambitious. But we’re just coming from a really weak position here. UK industrial electricity prices are over 300 US dollars per megawatt hour compared to 82 in the US.

This is just really tough. You’re just dealing with way higher prices and you’re dealing with just this long accumulation of issues and challenges. And yeah, I think AI is a good forcing mechanism to help us try solve some of these, but I think the UK cannot rely on fixing these problems in the medium term. I think we keep working at them, but if we’re trying to build our strategy for trying to achieve things in the world, we can’t rely on solving these problems because I think they’re just so difficult.

Philip Bell: Yeah, is—I mean, yeah, because partly I am excited about the idea that it could be an opportunity to build renewable energy infrastructure. Because I think solar is, I think I’m right saying that solar energy is one of the kind of quickest forms of energy to build capacity in. And also wind energy is quite quick as well in terms of kind of getting everything improved and stuff. But yeah, it, as you said, I guess the UK is kind of coming from relatively a position of disadvantage. My girlfriend’s dad lives in Dubai and one of his favourite questions, well not one of his favourite questions, but something he quite often asks me is what’s the energy price per R in the UK? And it’s like 20 times more than it is in Dubai.

Yeah, I don’t know, I’d hope that it could be an opportunity, but yeah, it’s a good point about how long it takes to build the infrastructure and that’s one of the considerations. I actually read about your article on Jeffrey Ding’s substack, which is called ChinAI, which is a really interesting substack about articles from China about AI. And they seem to be a country who have leaned in most to thinking about their energy system in relation to AI. I guess, I mean, this is a relatively speculative question, but do you think there’s anything that the UK can kind of learn from that? Because I saw recently they had in their China AI plus policy, they have an AI plus policy for every different industry. Or maybe not every industry, but they definitely have an energy one. But yeah, do you think there’s anything that the UK could learn from what they’re trying to do?

Hamish Low: Yeah, I think in terms of just making clear plans, having them be ambitious and then just really ruthlessly executing on them is just really strong. It’s just back to these industrial policy questions. I think they do this well. I think this has clearly really paid off in energy.

China has this insane energy advantage on AI now relative to everyone else. I was looking at some of the stats recently. It’s like they built 278 gigawatts of solar in 2024, which is the entire UK capability 3.7 times. That’s a lot. It’s 3.7 UK grids they’ve built in a year of just solar. And they’re growing their grid at just such a rapid rate that it’s giving them a huge advantage here. Where right now the US is doing really well on AI infrastructure and energy, largely because there’s lots of just stranded energy from US deindustrialization.

So you can get a lot out of basically just all these random plants in the Midwest that no longer make cars but there’s still some legacy transmission infrastructure that you can capitalize on and just very quickly build up new data centers. The issue for the US is then in a couple years they start to run into actual bottlenecks of you’ve used up all this easier spare capacity lying around. You have to do much more difficult things to actually grow your grid and connect all these new energy sources to the data centers and there’s lots of interesting solutions people are talking about of building behind the meter energy, of doing some ways to reduce peak energy demand such that you just shut the data center down for a couple days a year when it’s super cold or super hot and when the grid is under peak strain. So there’s stuff there.

But then the advantage for China is that they’re scaling their grid by a huge amount every year. So when you’re like, oh, we need to add another 10% for AI, they’re like, ah, sure, why not? We can build more solar panels in the Gobi Desert. And I think they just have such a big advantage here. The issue for China, if we’re kind of diving into Chinese AI, is they don’t necessarily have the compute and some of the other pieces to bring this strategy together. Their AI plus strategy, I think is very interesting. And I think the kind of adoption focused mindset is very smart.

So yeah, Jeffrey Ding was my supervisor for doing lots of this work and his book is great on this in terms of general purpose technologies, the really important process for national power is you need to diffuse them very widely through your economy and do this in a very deep and meaningful way to unlock the productivity that just accelerates all your other aspects of national power. I think the challenge is that this is just hard to do. The targets in their AI plus strategy are really ambitious. It’s 70% penetration of AI by 2027, 90% by 2030. One, it’s unclear what this means. Two, it’s unclear if you can actually do this.

There was this whole DeepSeek craze earlier this year where suddenly the central government goes, everyone needs to adopt DeepSeek. And there was this very interesting article Jeffrey translated that was telling the story of how this actually went, which was people bought these DeepSeek in a box where it’s a big AI chip that just has DeepSeek already running on it. And all of these state-owned enterprises, they buy this thing because they’ve been told they need DeepSeek. And they use the thing for a week. And they realize that actually DeepSeek v3 is—it’s a fun model, but it’s not that useful. You know, there’s a reason that we haven’t seen all the crazy productivity gains yet. Just having this model does not unlock incredible AI gains. And I think basically the vibe of the article was that now all these just sit in a cupboard somewhere. So you’ve ticked the box of adoption. But the challenge is that adoption is really hard and you need to bring together having really capable models with having large scale compute infrastructure and having just a very sophisticated cloud business and digital economy, which is one area where China actually is still quite far behind.

In that their cloud market is just generally much lower quality than Western cloud offerings. The adoption of digital cloud services is just a lot lower. You have this situation of there’s a very advanced tech industry in China—Alibaba, Tencent, Huawei, these are leading incredible tech companies. But China is huge, and most Chinese companies are not like Alibaba, Tencent, and Huawei, where actually the median US Fortune 500 company is much more technologically sophisticated than maybe the median Shanghai Composite Index 500 companies. And this is a big issue for China.

So I think there’s definitely things to learn in terms of having this very adoption focused mindset. But then we can also kind of learn from, what do we think is actually gonna hold them back? Which is really having this just sophisticated firms, having the compute availability, having access to the best models, which is where actually we can do all these things.

Philip Bell: That’s really interesting. I did not know that about the DeepSeek in a box and it kind of makes me think about generally in this space. One of the differences that I’ve heard people talk about between building AI infrastructure compared to building other technological infrastructure in the past, like fiber optic cables for internet or train lines or electricity infrastructure is that the GPUs go out of date really quickly.

Kind of like DeepSeek in a box kind of goes out of date quite quickly potentially if there’s new models and it kind of—I feel like it does also kind of add weight to your argument about AI middleware can be kind of lighter weight and it’s more kind of adaptable to a relatively uncertain sort of situation where things are changing quickly and hardware can kind of go out of date quickly. So yeah, I think that’s a really interesting anecdote. I didn’t know about that.

I also think when you were just talking about the US, one of the reasons they’ve been using this spare infrastructure from deindustrialization. That’s another interesting one because it would be amazing if AI could support a reindustrialization kind of program in the US or elsewhere. But the problem I suppose is that data centers don’t really have any jobs associated with them. So sadly, I don’t think it will. Yeah, but I thought, but yeah, that’s a really interesting point. I also didn’t really know that about basically a lot of the data centers are using up that spare capacity.

So this is, I mean, it’s interesting because we’ve talked about in this conversation kind of historical sort of comparisons or historical analogies that might be needed and, or are used and needed. And I was interested to get your take on the argument of David Edgerton. So he wrote a book called The Shock of the Old where he basically argued that—well, he tried to think about use rather than invention as the most important process in technological change. And then also he argued that lots of new technologies are wrapped up in older ones. And I wondered whether you think is that way of thinking useful at all do you think in considering AI? And AI sovereignty, like the idea that maybe older technologies are going to be as important as kind of newer ones in potential AI advancement.

Hamish Low: Yeah, I think it is very helpful. I think you already kind of see this in practice, with some of the infrastructure build out in the US where you have OpenAI in Texas and the things that they desperately want more of are these natural gas turbines that no one has cared about their production for two decades and everyone thought that the production forecasts were basically going to completely die off because of renewables build outs and heating and ventilation technicians, which is two things that you would not immediately identify.

Philip Bell: Yeah. Yeah and SMRs because you were talking about small modular reactors as well. The nuclear energy small modular reactor because I mean I think they were invented by the British Navy in the 1950s or something like that but they yeah they’re kind of coming back in. But yeah and that’s really interesting.

Hamish Low: Yeah, and I think there are probably just lots of ways that AI interacts with older technology. I think it’s very interesting. And I think it’s very hard to reason through. I think you end up in this weird world where right now AI capabilities are just very jagged in the sense that there’s this jagged frontier of AIs are really good at coding, but they’re still really bad at lots of other things we might care about.

I think there’s good reasons to just expect this to continue being true. That probably you see just lots more progress in domains of traditional white collar work. They get way, way better at coding and making PowerPoint documents and doing incredible mathematical reasoning. But probably maybe struggle on things that you think wouldn’t trip them up and struggle with this sort of transitioning into functioning in the real world and in industries in the real world.

Which does lead you into this very interesting fusion of old and new technologies and how these might interact. You have cases like cyber security, where there’s interesting ways where the two of them interact, where you have really, really rapid progress potentially in the ability to do automated hacking attacks. You could just have, basically you can just spin up the most sophisticated team of hackers that you could have had in 2015 and just deploy them to attack any given target around the world. And this is really, really scary. Lots of companies around the world and governments rely on these incredibly old legacy outdated tech stacks. There’s a bunch of US government tech that runs on coding languages that were popular in the 1960s. And there’s the one guy who’s retired, they have to bring in every time it breaks. So it’s very concerning.

But then maybe you could use AI to just retranslate all of these coding languages to much more modern ones and like refactor all this security and this kind of thing. So it’s like you end up with these weird ways where the two of these things interact of very old technologies, very new technologies and probably you end up with lots of parts of the economy where there’s some really old process or technology. You know, there’s some literal fax machine or incredibly outdated piece of hardware. And then there’s some AI software system running on top of it that’s trying to use it and interact with this old system and we’re just gonna live in this really weird world where the two of these things are both true. You have the super intelligence in your coffee maker or whatever, where it’s some very established technology but that is getting some benefit out of having all of this just cognitive ability plugged into it.

Philip Bell: Yeah, that’s really interesting. Another, yeah. I also wonder, just as you were speaking there, I was thinking that’s a really interesting point about kind of short, you know, old programming languages, could AI help to kind of retranslate those in some way. And it made me think about the trade, the kind of the ratio between building versus maintaining.

Technology because I think that’s something else that David Edgerton argues. He argues that we think about kind of building new technologies much more than kind of maintaining repairing old ones. Whereas maintaining and repairing old ones is actually more important according to his argument. But it’s an interesting—I think that’s an interesting question around in an AI future, would—could AI actually mean that there’s more maintenance and repair needed if there’s, let’s say if there’s, you know, just more complicated and larger scale systems at play.

But, but yeah, I mean, the use of historical analogies, I think is really ubiquitous in talking about AI, maybe too ubiquitous. I heard Sam Altman pitch—he compared AI to fire. If you listen to Demis Hassabis or really any AI leader, they often talk—compare AI to I think the industrial revolution or other things. And I’ve heard, I mean, some people say that China according to its policy seems to consider AI more akin to electricity or the internet, whereas the US, according to its policy, and this might be a simplification, seems to compare, seems to be thinking about AI more akin to developing the atom bomb. And that kind of informs their policy in that US policymakers are very worried about takeoff scenarios, maybe more so than Chinese policymakers.

But yeah, in general, do you think that using historical analogies is more useful or more distracting when thinking about AI and generally AI sovereignty?

Hamish Low: I think generally they’re useful. I mean, I studied history, so I’m contractually obliged to give this answer. But I do think it’s very helpful. I think just from the kind of general—in forecasting, you have the outside view and the inside view, where the outside view is just ignoring the specific thing you’re trying to study and just figuring out, okay, what’s the right reference class? What’s the way we’ve understood previous incarnations of whatever this thing is?

I think that’s where historical analogies are just really, really helpful—you just want a reference class you can base off of. I think this is where these kind of previous general purpose technologies are super useful. Electricity and steam are just two really good examples of previous technologies that apply across every sector of the economy. And we can draw a lot of interesting conclusions from these, kind of what Jeffrey Ding does in his work of studying in detail quantitatively how these diffuse across an economy and when the productivity effects kick in.

So we can learn general purpose technologies, the productivity effects probably come two decades to three decades after their widespread diffusion. This is really good to know. But then you can do the kind of inside view, which is where you really are just interrogating the specific thing you’re interested in. And I think this is where you get to a bit more like AI is the atom bomb—there’s things about AI that are kind of crazy and actually very different.

You know, that if AI can function as just a substitute for human labor, this has lots of really crazy consequences that no other technology does. And I think it’s really useful to actually interrogate these differences because maybe previous general purpose technologies took two or three decades. But maybe one of the reasons why they took so long is that there just weren’t that many good metallurgists and iron engineers and people who were able to make steam engines and so the kind of skilled engineers who actually learned how to make a steam engine and then went and worked in this completely unrelated industry and was like you should plug in a steam engine. There just wasn’t that many people. But if AI can substitute for these people then maybe these processes can go much faster.

So then you can kind of more easily debate—okay if previous general purpose technologies took three decades, how fast do we think AI can speed up this diffusion process? Maybe it pushes it down to a decade, maybe it’s 15 years, maybe it’s five. And I think this is where they can be very helpful is because they just give you something to base these discussions off. Because I think otherwise you can end up in very—if you go too inside view and you go too, AI will change absolutely everything. Then you get lots of these economic models of explosive growth from AI where just you have a model of what growth is and you’re like, okay, AI is a perfect substitute now, what happens? And the answer to what happens is that you get stratospherically crazy economic growth.

And I think this is helpful in some ways, but also you just have lost some of this reference class and understanding the history that maybe grounds your answer a bit more and brings you a bit closer to what I think is a better forecast of the future.

Philip Bell: Do you think that it’s useful to, or do you think as policymakers, people should think about a number of different scenarios and kind of hedge their policy bets, I suppose? Or do you think it’s more useful to kind of basically decide what scenario is most likely, like let’s say 2028 takeoff or something, and kind of develop policy accordingly?

Hamish Low: Yeah, I think you basically just have to go in with different scenarios and just understand the differences between them. I think ultimately states are big and they have lots of money and they can do these multiple things. I think you should be worried about the 2028 takeoff. We just have lots of uncertainty about whether this is possible or whether it could happen. And probably you want the people in the AI Security Institute really thinking this through in detail and trying to figure out what the mitigations are. And we should put some work to preparing for that world.

But then equally, I think the AI as electricity, the AI as normal technology is also true that there’s just going to be lots of actually in practice very boring mundane ways that AI is diffused across the economy. You know, there’s going to be so many really dull pitches to venture capitalist investors about AI for X random thing. And lots of them will be very useful and will give us good economic growth. But they’re just not very interesting or flashy and you should just have people working on thinking about both of these worlds.

And yeah, from a policy perspective, it’s tricky because sometimes you’ll end up in trade-offs between the two of these. You know, there are some things you might do to prepare for this crazy takeoff world that are unhelpful for this other general slow diffusion normal technology world. I think probably those are worth taking the trade-off of. I’m quite scared of the takeoff world. But I guess we just need to go through the process of just trying to sketch out these possible futures and just having people debate between them and figure out what we think is important and what are the best, most robust ways we can navigate this.

Philip Bell: I agree. What—this is the last question that I was going to ask. What is your kind of best case scenario for kind of how AI plays out?

Hamish Low: You’re leaving me on the hardest possible question, which is slightly mean, but I’ll give it a go. I think we have the first easy box of it doesn’t kill us all, which is yay, this is excellent, and it’s a pretty necessary condition. We have the second tick box, which is that it doesn’t lead to some crazy concentration of power that destroys all our democratic institutions, which I think is also a real worry. So we want safe AI that’s developed in a kind of diffuse and democratic way.

But then I think if you’re able to start satisfying these conditions, it just can be really, really good. I think there’s big issues to navigate in terms of a lot of people in this space are not very honest about how far AI is just an automating technology. This is all it does is it automates people. Broadly, this is its main economic effect and any vision of the future where you’re like it will create new jobs—I mean, yes, but these will be a fraction of the jobs that it’s just taken away.

But I think even still we can envision very positive scenarios because we just have a much richer world here. We just have many more resources from all the things that AI can do for us. And I think we just get incredible medical goods. You just get to have your own personal AI advisor. There’s been lots of interesting work around ways that AI can augment democracy, of actually making some maybe really interesting direct democracy systems possible because I can tell my AI in much richer detail what my political preferences are. And then AIs themselves can aggregate these much more easily in a way that electoral systems are just not very good aggregating functions for people’s democratic preferences.

So I think ultimately the kind of flourishing positive world is one where everyone is richer, everyone has much better healthcare, everyone has better governance, and people can just live freer, happier, richer lives. And I think this is hopefully the goal that we can work towards.

What's really happening inside models when they generate text? Lee Sharkey, Principal Investigator at Goodfire AI and co-founder of Apollo Research, and I discuss mechanistic interpretability - the emerging science of reverse-engineering neural networks to understand how they actually work.

Lee works at Goodfire AI, an AI interpretability research lab focused on understanding and intentionally designing advanced AI systems ⁠Company⁠. In this conversation, we explore how researchers are using techniques like sparse autoencoders to decode the internal representations of large language models, discovering everything from "Golden Gate Bridge features" to Barack Obama neurons ⁠Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet⁠.

We discuss what we actually know about models, the challenges of working in high-dimensional spaces, and why understanding AI systems might be crucial for safety as they become more powerful. Lee also shares insights from his background in computational neuroscience and how similar methods are being applied to artificial neural networks.

Topics covered include induction heads, sparse dictionary learning, the "grown not made" nature of neural networks, and whether there might be universal structures in how both humans and AI systems organize knowledge.

Articles by Lee:

Open Problems in Mechanistic Interpretability

Sparse Autoencoders Find Highly Interpretable Features in Language Models