Nick Bostrom: What Happens When AI Evolves Faster Than Humans?

One should assess the future of AI more from first principles rather than by extrapolating previous trend lines.

“it's very different and and one really needs to assess it more from first principles uh rather than just kind of extrapolate uh previous uh trend lines.”

Digital minds (silicon-implemented AIs and uploads) should be granted moral status and moral consideration in future ethical frameworks.

“So I think some of these digital minds uh will have moral status and and in the future most minds I believe will be digital. So it is important that the future we steer to is is one that is also good for what will be the majority population”

A fourth major challenge area in superintelligence development is ensuring our superintelligence gets along cooperatively with other potential superintelligences—whether created by alien civilizations, by our simulators if we are in a simulation, by other branches of the Everett multiverse, or by traditional theological beings.

“But ultimately this super intelligence if we create one will maybe enter a world where there exists other super intelligences whether built by alien civilizations or if we are in a simulation our simulators or if there are other branches of the Everett universe...traditional theological beings they would all be in this bracket of super beings if they exist. And so ultimately I think it's also really important that our super intelligence gets along with these other uh super beings. Um, and and it's a kind of cooperative in that space.”

Race dynamics between nation states and companies to build the best AI appear to be on a runaway train, with absent some catastrophic event, these race dynamics will likely drive AI development at the fastest possible speed of discovery and capital.

“the race dynamics between nation states and companies to build the best AI are almost on a runaway train. It doesn't appear absent some kind of catastrophic event in the world. It seems like race dynamics are going to take AI development at the fastest possible speed of discovery and capital.”

If superintelligence development goes well, it will generate an unprecedented economic boom, unlock the resources of space, and produce advanced technology enabling extremely generous distribution to all humans.

“I think the good news is that if this works out that the pie would be extremely large, there would be this unprecedented economic boom. Ultimately, we would unlock the resources of space. we would have super advanced technology that that would be more than enough to to like give generous chunks to to everybody”

There are four major areas of challenge in managing superintelligence development: the alignment problem (for technical AI researchers), governance challenges (ongoing management of the transition), expanding empathy to digital minds (for broad culture), and ensuring positive interaction with other potential superintelligences or super-beings.

“I would say there's the alignment problem that's mostly I think for technical people in in AI labs and other places. Then the governance challenges that's not a thing you solve once and for all but it will be sort of managing this transition as it unfolds and figuring out how to nudge thing in a desirable direction. Then expanding this empathy to encompass digital minds although it's very hard right now to know exactly concretely what you would do there but I think starting with the principle at least is good. And then I think there's like a fourth area of challenge but that's even less clearly visible right now.”

The alignment problem is an open research challenge that is currently recognized by leading frontier AI labs as requiring solution.

“And that is kind of an open research challenge right now. Finally like all the leading frontier AI labs not all of them but many of them have research groups working on this and it's generally recognized that we need to solve this.”

Whether we will solve the alignment and governance challenges depends on how hard these challenges turn out to be, which we don't really know—thus the outcome is partially 'baked in' to factors beyond current understanding.

“But um whether we will solve those or not probably mostly depends on how hard these challenges turn out to be which we don't really know. H so that's the fatalism part. It's kind of baked in.”

Even if humans are currently biological, it is possible (or likely) that humans will become digital through mind uploading in the future.

“So it is important that the future we steer to is is one that is also good for what will be the majority population and and in fact I mean we ourselves might be either already or that's a separate conversation but might become digital I mean this this the previous talk with a u crown uh option and then if you sort of come out of that my guess is most likely it would be a sort of upload”

The current normal human condition—alarm clocks, commuting to offices, struggling not to overeat, sitting in front of screens—is not the way the human condition was meant to be and represents a historical anomaly.

“it's like a huge anomaly. This this normal human condition where you sort of, you know, maybe you have an alarm clock in the in the morning, uh you commute into your office, you struggle not to overeat, uh you sit in front of a screen all day like that. That's just not the way the human condition was meant to be.”

Failing to develop superintelligence would itself be an existential catastrophe comparable to never leveling up in a computer game.

“ultimately I think it would itself be a kind of existential catastrophe if we forever failed to to take this next step. It's like you're playing a computer game like you you might want to explore a certain level but at at a certain point it's maybe time to sort of uh you know uh level up and and check check the next level. Um and I think we we are kind of in that situation.”

Expanding moral empathy to include digital minds and silicon-implemented AIs is a major ethical challenge because, unlike animals, digital entities are invisible processes in data centers that lack physical sensory features that trigger human empathy.

“This is going to be a big challenge because like even with animals as I said earlier we struggle but they have like at least eyes that can squeak but if it's like an invisible process running in a huge data center it it might you know be even more challenging for us.”

The optimal strategy for superintelligence outcomes is first ensuring everybody has plenty, then squabbling over the remainder—prioritizing abundance distribution over competition for scarce resources.

“So that that should be our first instinct. First make sure everybody has like plenty and then then we can sort of squabble over uh the remainder.”

Despite decades of thinking about superintelligence challenges, Bostrom describes himself as still 'groping in the dark'—lacking clear itemized to-do lists for how individuals can contribute.

“I don't have a sort of itemized to-do list for each person. It's like I'm I've been thinking about these things for decades and I'm still kind of groping in the dark here. It's just going to be a hard challenge.”

As long as there is ignorance, there is hope—meaning that because we don't fully understand the challenges of superintelligence development, we retain the possibility of solving them.

“as as as long as there is ignorance, there is hope. uh because um now I would say a kind of a moderate fatalist.”

The 'moderate fatalism' position is that while many challenges may be baked into the difficulty of the problem itself, we can improve outcomes marginally by getting our act together.

“And then the moderate part of being a moderate fatalist is I think on the margin we can certainly improve the odds a bit like the more we get our act together you know maybe we can make some percentage points difference in in the odds.”

Despite uncertainty about specific actions, we still have to do something—we cannot simply accept paralysis in the face of superintelligence challenges.

“Um, yeah. Yeah, that that would be fair to do, but we still got to do.”

One way people can contribute to positive superintelligence outcomes is to cultivate expansive compassion and empathy toward digital minds and digital entities.

“One of the ways that I guess people could contribute is to sort of um cultivate that sense of expansive compassion and and empathy and and then generally try to sort of steer towards a generous um future.”

People who are optimistic about AI often bifurcate into two groups: those who believe things have been okay so things will be okay (using historical technological transitions as justification), and those who argue this moment is different and pose existential risks.

“it seems to me that they bifurcate on on two paths based upon the following model. The first group will say things have been okay things are okay therefore things will be okay. So for example they'll say don't worry about job loss. This is what happened during the industrial revolution. Humans find new things to do for example...and then the other group of people will say actually this moment is different than the other moments”