Buckle Up: Is the U.S.S. Enterprise About to Take Off?

I recently came across a post by the AI Futures Project with a rather unsettling title: Q3.5 Timelines Update: Looks Like the Intelligence Explosion Will Happen in 2027 After All.

The basic idea is that we could be much closer than previously expected to AI systems capable of doing AI research better than the best human researchers. And, if that happens, those systems could help build even more capable AI, potentially accelerating the whole process.

I honestly don't know how to feel about this.

I've always been a fan of science fiction. I've loved imagining different versions of the future, whether it's the terrifying world of Terminator 2, the mind-bending reality of The Matrix, or the optimistic, technology-driven universe of Star Trek.

But one particular science-fiction story has stayed with me more than any other: Isaac Asimov's short story The Last Question.

It's a fascinating story that follows humanity and increasingly advanced computers across an almost unimaginable stretch of time. Throughout the story, humans repeatedly ask a computer whether the universe's entropy can ever be reversed. Each time, the computer becomes more sophisticated, but its answer remains the same: there isn't enough information to answer the question yet. Eventually, the story reaches a conclusion that is difficult to forget.

What struck me about the story wasn't simply the idea of an incredibly powerful computer. It was the scale of what intelligence might eventually become, and how far beyond our current understanding it could take us. It made me think about a future in which technological progress reaches a point where we can no longer meaningfully predict what comes next.

That's closely related to an idea known as the technological singularity.

The term is borrowed from physics, where a singularity describes a point at which our existing theories can no longer adequately explain what is happening. In technology, it's used as a metaphor for a hypothetical point when progress becomes so rapid, and its consequences so profound, that our ability to predict what comes next breaks down.

But how could we get there?

One proposed path is something known as an intelligence explosion.

Imagine AI becoming capable enough to meaningfully contribute to developing the next generation of AI. That next generation would be even better at research and engineering, allowing it to build an even more capable generation after that. Each improvement could make the next improvement happen faster.

In theory, this creates a feedback loop: AI helps improve AI, which makes AI even better at improving itself. If this process becomes sufficiently powerful, the pace of technological progress could accelerate dramatically.

This is the intelligence explosion: a hypothetical period of extremely rapid growth in machine intelligence.

And if that growth becomes powerful enough to fundamentally transform the pace and direction of technological development, it could lead to what we call the technological singularity.

In other words, the intelligence explosion is a possible route to the singularity. The singularity is the hypothetical point beyond which our ability to predict technological progress breaks down.

It's an extraordinary possibility. And it's also difficult to get our heads around.

Why is exponential growth so difficult to understand?

Part of the problem is that exponential growth is not something humans are naturally good at understanding.

We're pretty good at imagining one thing getting a little bigger, then a little bigger again. We're much worse at imagining something doubling over and over. The numbers look small at first, then suddenly become ridiculous.

Perhaps that's because most of the things we encounter in everyday life don't behave this way. A tree grows gradually. A house gets built one brick at a time. Our salaries, savings and everyday experiences tend to change in ways that feel relatively linear. We rarely encounter something that appears almost insignificant at first and then grows to an unimaginable size in a surprisingly short period.

But exponential growth can do exactly that.

There's a famous legend that makes this easy to picture.

A king offers a reward to the person who invents chess. The inventor asks for one grain of rice on the first square of the chessboard, two on the second, four on the third, and so on—doubling the amount on every square.

The king thinks this sounds like a modest request. It isn't.

A chessboard has 64 squares, so the total is:

1 + 2 + 4 + 8 + ... + 2^63 = 2^64 - 1

That's 18,446,744,073,709,551,615 grains of rice. Around 18 quintillion.

Suppose, very roughly, that each grain weighs 25 milligrams. The total would weigh about 461 billion metric tonnes. That's roughly 1.3 million Empire State Buildings' worth of rice, using a rough building-weight estimate of 365,000 metric tonnes.

The first few squares barely matter. Almost all the rice is on the last few.

That's the part of exponential growth our intuition misses: the ending can be so much bigger than the beginning that it's hard to believe they belong to the same process.

And this is why discussions about an AI intelligence explosion are so difficult to evaluate intuitively. If AI development starts feeding into its own improvement, we might be looking at a process whose consequences are much harder to imagine than the technologies we're used to seeing develop around us.

A different way to think about the future

There's another way to appreciate just how extraordinary this could become.

In its recent assessment of Anthropic's AI research and development, METR identified a striking change in the rate of progress. Its analysis of benchmark trends found that the more recent trend had approximately 1.9 times the slope of the earlier one.

Now, let's take that 1.9 figure and consider a hypothetical scenario.

Imagine that the rate of technological progress increases by a factor of 1.9 every year, and that this acceleration compounds.

After ten years, we get:

1.9^10 ≈ 613

In other words, the rate of progress in year ten would be approximately 613 times what it was at the beginning.

Think about what that means. One year of progress at that future rate could potentially deliver as much advancement as 613 years at today's starting rate.

Of course, this is a mathematical thought experiment, not a prediction. METR's 1.9 figure describes the relative slope of a particular benchmark trend, not a claim that all technological progress will compound by 1.9 every year. Progress doesn't necessarily accelerate at a constant rate, and technological breakthroughs don't translate neatly into units of time.

But the exercise gives us a sense of the scale we're talking about.

Now, consider how much the world has already changed.

Much of what we consider modern technology today has emerged during the roughly 250 years since the Industrial Revolution. Electricity, telecommunications, aviation, computers, the internet and now artificial intelligence have transformed almost every aspect of human life.

My own grandmother grew up in a remote area of Spain where electricity wasn't even available.

Think about that for a moment.

Within a single human lifetime, we've gone from a world where some people lived without electric lighting to one where we can interact with increasingly capable AI agents that can reason, write code, conduct research and perform tasks on our behalf.

My grandmother has witnessed a transformation that would have seemed like science fiction to the world she was born into.

And if humanity achieved so much in roughly a century, what could the next few decades bring if the pace of progress continues to accelerate?

What happens if the next ten years deliver hundreds of times the technological progress we currently experience in a year?

Perhaps the most difficult thing isn't imagining what AI might become. It's recognising that our own experience of technological change may be a very poor guide to what's coming next.

Buckle up indeed.

So, is AI about to do the same thing?

Some people think AI could help accelerate the development of even better AI—and that this feedback loop might produce an intelligence explosion as soon as 2027.

That sounds like a lot. But it helps to be precise about what the claim actually is.

The recent AI Futures Project timelines update forecasts that Superhuman AI Researcher (SAR) could arrive sometime in 2027, assuming there isn't a substantial slowdown in AI development.

What exactly does that mean?

The authors describe a Superhuman AI Researcher as roughly the point at which an AI system could perform the research work of the best human researchers at an advanced AI company, while also being able to automate its coding. In other words, a company could potentially prefer an entirely AI-based research team to even its best human-only research team.

That's a pretty extraordinary milestone.

There is some evidence behind the forecast. METR's recent assessment found that progress on certain well-specified AI tasks had accelerated substantially compared with an earlier period.

More specifically, its benchmark-trend analysis suggested that the recent rate of progress was approximately 1.9 times the earlier rate, although there is considerable uncertainty around the estimate.

This is particularly interesting because it suggests that AI isn't simply becoming more capable at helping us with everyday tasks. It may also be helping researchers build the next generation of AI systems more quickly.

However, there are important caveats.

METR's findings concern one company and particular kinds of tasks. Benchmark improvements don't necessarily translate directly into equivalent improvements in real-world research capabilities. The report also acknowledges significant uncertainty, and some of the evidence most relevant to its conclusions was redacted by Anthropic.

Turning those findings into a forecast for when AI could outperform top human researchers requires additional assumptions and modelling.

In fact, the AI Futures Project's different forecasting approaches don't all land on exactly the same date. Some point to late 2027 or even 2028; others give earlier estimates.

The authors themselves acknowledge that their modelling is preliminary and that their forecasts depend on AI development not slowing down substantially.

So I'd describe 2027 as a plausible, evidence-informed scenario—not a date we know is coming.

And what could this mean for our future?

Something I've increasingly noticed is how normal it is becoming to hear incredibly wild predictions about AI.

We're talking about machines that could outperform the best human researchers, technological progress accelerating beyond anything we've experienced before, robots potentially transforming the economy, and a future where traditional ideas about work, money and retirement might no longer make sense.

Elon Musk, for example, has made some extraordinary predictions about AI and robotics creating a world of abundance, where goods and services become so cheap that everyone could enjoy a very high standard of living. He's even suggested that retirement savings might become largely irrelevant.

The underlying argument is that if AI and robotics make production extraordinarily cheap and efficient, we could eventually reach a world where many of the economic constraints we take for granted today no longer apply.

Imagine robots building houses, growing food, producing goods and providing services, while increasingly capable AI systems handle much of the intellectual work.

In theory, the cost of producing many things could fall dramatically.

But there's a rather large assumption hidden in this vision: that the benefits of all this productivity would be distributed widely across society.

Technology alone doesn't guarantee economic equality. Even if we develop machines capable of producing extraordinary amounts of wealth, we still have to decide who owns those machines, who benefits from their productivity and how society adapts.

For an example of the kind of claims Musk has made, see this report on his AI-abundance predictions.

But this brings me back to something I've been wondering about.

Are we actually approaching something extraordinary, or are we just getting caught up in the hype?

Are these people genuinely convinced that we're approaching the technological singularity? Or is some of this simply hype, driven by companies, investors and people trying to capture our attention?

I sometimes wonder whether we're becoming so accustomed to dramatic headlines that we've developed a kind of mental filter. We read that AI could outperform the best human researchers by 2027, or that robots could fundamentally change the economy, and our first instinct is to dismiss it as another exaggerated prediction.

Perhaps that's because the implications are so enormous that we don't really know how to process them. It's easier to assume it's nonsense than to seriously consider that some of these things might actually happen.

And that's what I find particularly fascinating, but also a little unsettling.

There is genuine evidence that AI is accelerating parts of AI research and development. There are also serious researchers trying to understand what happens if that acceleration continues. But there is a huge difference between observing faster progress today and knowing exactly where it will lead tomorrow.

We don't know whether the intelligence explosion will happen in 2027, in 2030, or at all. We don't know whether the technology will hit significant bottlenecks, whether progress will slow down, or whether some of the more extraordinary predictions will turn out to be wildly optimistic.

The chessboard story teaches us why exponential growth can surprise us. The compounding thought experiment gives us a glimpse of how extraordinary the numbers could become. But neither tells us that AI progress will keep accelerating indefinitely.

Personally, I find worrying endlessly about these possibilities rather stressful. There are enough things to worry about in the present without spending every day wondering whether we're approaching an event that could fundamentally change civilisation.

So I'd rather remain optimistic.

When I think about the future of AI, I'd much rather imagine a Star Trek-like future than a Terminator one. A world where technology helps us solve problems, explore the universe, improve medicine, eliminate scarcity and spend more time doing the things that make life worthwhile.

Maybe that's wishful thinking. There are plenty of reasons to be cautious, and we shouldn't ignore the risks or assume that technological progress automatically makes the world a better place.

But perhaps that's also the exciting part. We don't know exactly what comes next.

The U.S.S. Enterprise isn't about to lift off, and science fiction isn't about to become reality all at once. But maybe, just maybe, we're getting closer to a future where some of the things that once seemed impossible start becoming ordinary.

And I'd quite like to be around to see that.

So, buckle up. It could be a wild ride.