Jaan Tallinn’s Case for Certifying Frontier AI

Jaan is the founding engineer of Skype, an early board member and Series A backer of DeepMind (alongside Mr. Musk and Peter Thiel), and the contrarian who led Anthropic’s $124M Series A at a $623M post-money valuation in 2021, an investment that, excluding dilution, has returned ~250x as Anthropic raised $13B at a $183B valuation, Jaan Tallinn has spent the last many years warning that humanity’s biggest risk is not climate change or pandemics, but losing control over the very intelligence we are building.

Why was it contrarian at the time?

(i) Timing vs. proof: GPT-3 had been released a year earlier and showed the promise of scaling laws, but it was still unproven whether transformer-based LLMs would sustain progress. Betting big at that stage was against consensus. (ii) Valuation risk: Tallinn led a $124M Series A at a $623M post-money valuation for a company that had just been founded. That was viewed as extremely high and risky for such an early AI lab.

This week, we’re including a bit of a long read, but it’s worth it. We promise.

However the TLDR is:

• Jaan Tallinn’s top risk: losing control of frontier AI: a meta-problem unlike pandemics or climate.

• Calls for global safety certifications for high-risk data centers with real, enforceable kill-switches.

• Backs capability-triggered moratoria (pause if AI crosses thresholds like novel bio-weapon design).

• Expects continued rapid capability gains (beyond current regime); control could be lost without safeguards.

• Near-term: society must adapt to current GPT-level disruption; long-term superhuman AI could be existential.

• COVID showed policy can move (e.g., gain-of-function limits) and highlighted a tangible risk: AI-designed pathogens.

Talk conducted during GPT-4 era.

Q&A - Jaan Tallinn, founding engineer @ skype

BlackWood:

How worried should we be? You often focus on existential risks for humanity, and the Doomsday Clock, a symbolic countdown to human extinction, is currently at 90 seconds to midnight. That is the closest it has been since it was established in 1947. What are the main threats that concern you most, and how much time do you think we have left to address them?

Jaan:

My main focus is on the control problem: creating AIs that we will not be able to control, much like chimpanzees cannot control humans. I am happy if others focus on different problems. It is not that other risks do not matter, but my primary concern and area of expertise is rogue AI

BlackWood:

You’ve also spoken about data centers. Could you break down your idea of requiring safety certifications for high-risk data centers, and what you see as the biggest obstacles to enforcing these certifications globally?

Jaan:

The idea behind data center certification is that we need constraints on what happens in these increasingly large facilities. Right now, frontier AIs are developed with very little oversight. The current model is essentially a black box: you take vast amounts of data and compute, run it through a relatively simple program, let the process run for months, and then see what kind of system emerges. We rely on the fact that the resulting AI is not very capable yet to maintain control. But building ever smarter systems while counting on their current weakness as our safety mechanism is not sustainable. Certification would mean that these digital “alien-summoning experiments” only take place in environments with oversight.

The oversight I have in mind includes mechanisms like off switches. A common response I hear is: why be afraid of AI if we can always pull the plug? The problem is that you cannot simply unplug a data center. They are built to resist power cuts. To remain in control of these experiments, we need different standards than those applied to today’s data centers.

BlackWood:

And on that point, you have described the race to build ever more powerful AI as an AI suicide race and compared developing advanced AI to launching a rocket.

When we hosted Sam Altman, he suggested that advanced AI might require something like an international atomic energy agency model for superintelligence, where regulation would apply above a certain capacity threshold. What would you like policymakers and technology companies to do in terms of specific mechanisms or regulations to address this AI suicide race and pull us back from the cliff edge?

Jaan:

So I think one important observation is that people have very different ideas about where AI will be in the next year or ten years. Our preparations should therefore be sensitive to those expectations. If we believe smarter-than-human AIs could emerge later this decade, we should prepare very differently than if we are confident nothing close to human-level competence will appear. One approach is to design conditional policies, such as conditional moratoria.

The idea is that both groups, those who do not expect highly capable AI this decade and those who do,  could agree that if AI crosses certain capability thresholds, specific policies would take effect.

For example, if an AI became more capable than humans at creating biological weapons, we could agree to halt the development of frontier AIs at that point. Those who expect such capabilities soon would obviously support the moratorium, while those who do not would also be fine with it, since they do not expect the threshold to be triggered.

BlackWood:

On that point, Jaan, what do you expect? We’ve spoken before, and I recall you mentioning something a bit gloomy, that we may have entered the last phase of human extinction, or words to that effect. So what do you think will happen, and what timeline are we talking about?

Jaan:

My simple answer is: I don’t know. But it’s important to note that not knowing does not mean we are safe. I would much rather know that we will not develop very capable AIs. Still, when I look at the trajectory, I’m puzzled by people who expect progress to stall. It’s strange to see GPT-2, GPT-3, and GPT-4, and then assume GPT-5 will be no more capable than GPT-4. If I extrapolate, speculative as that always is, I find it more likely than not that we won’t hit a glass ceiling anytime soon. And if the jump from GPT-4 to GPT-5 resembles the leap from GPT-3 to GPT-4, then there is a significant risk we may no longer be able to control it.

BlackWood:

And what about the societal changes that would come from that? Other than putting in risk mitigation techniques, how do we ensure those changes are more positive than negative? Or is it just a matter of putting barriers in place to maintain safety?

Jaan:

Yeah, again, I think it’s important to distinguish between a subhuman competence regime and a superhuman competence regime. Clearly, we are currently in a subhuman competence regime when it comes to AI. And even now, there is already significant disruption happening in the world. Just ask high school teachers who can no longer tell the difference between essays written by their students and GPT, or artists and translators facing similar challenges. So even if we stopped pushing the AI frontier, I think society would spend the next decade simply adjusting to GPT-4–level systems.

But if we push beyond the human regime, I just don’t see how we can remain in control. If we lose control over what is happening on this planet, we will likely go extinct shortly after, because humans are very finely tuned to this particular environment. The entire global problem of climate change is about the temperature shifting only slightly. And I say slightly because, from an astronomical perspective, a five-degree change is nothing. But AI is unlikely to care about the environment at the precision humans require for survival.

BlackWood:

Speaking of warning shots, you once described COVID-19 as a minimum viable catastrophe. What are the key lessons we should take from it to prepare for future global risks, particularly in relation to AI?

Jaan:

In some ways, it’s safer to live in a post-COVID world than a pre-COVID one. I read about new restrictions in the U.S. on gain-of-function research, which I see as a direct outcome of the pandemic, and a positive one.

I think that’s very much a direct result of COVID. Another lesson the pandemic gave us, from an AI perspective, is clarity on how AI could become lethal. The easiest example for people to grasp is AI developing novel pathogens. Given AI’s growing competence in synthetic biology, and our lived experience with COVID, it’s now much easier to talk about this risk. It feels far more tangible.