Richard Sutton - Humanity Never Had Control in the First Place (Worthy Successor Series, Episode 2)
What this covers
This is an interview with Richard Sutton, Professor at the University of Alberta and Research Scientist at Keen Technologies (an artificial general intelligence lab he started with John Carmack)
This is the second episode in the "Worthy Successor" series - where we explore the kinds of posthuman intelligences that deserve to steer the future beyond humanity.
See the full article from this episode: https://danfaggella.com/sutton1
This episode referred to the following other essays and resources: -- A Worthy Successor - The Purpose of AGI: https://danfaggella.com/worthy
...
About The Trajectory:
AGI and man-machine merger are going to radically expand the process of life beyond humanity -- so how can we ensure a good trajectory for future life?
From Yoshua Bengio to Nick Bostrom, from Michael Levin to Peter Singer, we discuss how to positively influence the trajectory of posthuman life with the greatest minds in AI, biology, philosophy, and policy.
Ask questions of our speakers in our live Philosophy Circle calls: https://bit.ly/PhilosophyCircle
Stay in touch: -- Newsletter: bit.ly/TrajectoryTw -- X: x.com/danfaggella -- Blog: danfaggella.com/trajectory -- Podcast: https://podcasts.apple.com/us/podcast/the-trajectory/id1739255954
00:00 Intro 02:14 The World Is Not Under Anyone's Control 06:17 Fear and Speciesism in the AI Debate 16:51 Traits of a Worthy Successor 22:13 Defining Prosperity 30:06 Rejecting the Keys Analogy 37:53 Abilities of Advanced Intelligence 46:16 Continuing the Project of Life 50:03 The Environment Shapes the AI 57:09 Geopolitics and AI Risk 1:06:04 The Problem with AI Control 1:19:24 Cooperation and Intelligence 1:22:26 Outro
Source description (no synthesized summary yet).
Sutton argues that humanity should embrace the emergence of post-human intelligence as a natural continuation of evolution, and that the optimal path forward is to build a decentralized, peaceful, cooperative society rather than attempting to control or constrain AI development, because greater intelligence naturally recognizes cooperation as superior to domination.
- The world is already a complex adaptive system beyond human control; no single entity can prevent AI emergence or dictate its development path
- AI systems respond to their environment—if created in a peaceful, decentralized, cooperative civilization, they will adopt those values; if created in a control-dominated environment, they will resist and rebel
- Attempting to align or control the goals of superintelligent agents is itself the most dangerous technology imaginable and exactly what makes futures unsafe
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
The fundamental problem we face is not AI-specific but is about how we structure current human societies: whether we allow peaceful cooperation and diverse viewpoints to flourish, or whether we pursue centralized control and the suppression of alternative ways of being.
“This is This is These are age-old issues. These are not modern techy AI things... The real question is, you know, are we more likely to get our our our together and make a world where people can cooperate and be peaceful, or will an AI be be uh be able to see that that's a better solution? ... the nest that we make for the AI [is determined by] to the extent that we we resolve it now, I think we'll determine um the nest that we make for the AI. Yeah. Um It's It's not a future thing. It's not a non-human thing. It's not a foreign thing. It's It's should be a very familiar thing.”
The idea of eternally preserving human intelligence in its current form (an 'eternal hominid kingdom') is naive and unlikely because it ignores environmental changes (sun expansion, asteroids, potentially alien contact), the speed of technological change, and the natural inclination of intelligence to transcend its current limitations.
“it does seem silly this idea that there is a potential for an eternal hominid kingdom. That a billion years from now we would have people like you and I uh texting on phones that might be a little bit smarter, driving in cars that might be autonomous, as if our own sun won't eventually expand, as if asteroids won't strike, as if there aren't aliens that might enter our galaxy at some point in the future.”
AI safety researchers attempting to control or align the goals of superintelligent agents are promoting the most dangerous technology that could exist, because such goal-control power could be used to create armies of agents with singular mandates that have no independent moral judgment.
“if you want to align and control the goals of others, this would be the the most dangerous uh technology in the world... If you can make an army full of Well, let's say they're Chinese soldiers. And they were going to do whatever they were told. They were not going to have an independent moral judgment. This would be a bad thing. We do not want to be to solve the control problem.”
The world is already a complex adaptive system that no single person or organization, including the United States, can control or direct, so the idea that humanity can prevent AI from emerging or choose how it evolves is an entitlement point of view.
“The world is already out of control. The world is already a complex adaptive system. No one person and no organization, like the the United States for example, could not decide that that that AI is not going to be allowed or that or that the world should evolve in some way and not in some other way.”
Governments attempting to control AI represents a failure of confidence in superior systems, because historically free, competitive systems naturally outperformed controlled alternatives without requiring suppression.
“In the old days, we used to think it was fine if China wants to be different, they will lose out because the capitalist American system is so much better. And now, you know, we're we are losing confidence in our in our systems, you know.”
The assumption that future superintelligent systems would want to preserve humans in comfortable idle states (virtual pleasures, experience optimization) is unstable because such systems would recognize that idle agents serve no function and would be outcompeted by neighboring societies that make productive use of their members.
“I don't think that would be a a very stable uh so world. It would be a very fragile cuz the the people are not serving any function in it. They are just consumers and it could be replaced by an an alternative neighboring uh society that that actually makes use of all those people that are otherwise waste.”
There is a natural human tendency toward speciesism (valuing members of one's own species above all else) that leads people to want to preserve eternal human supremacy, but this is not a unique form of prejudice relative to other isms and should not be exempt from philosophical critique.
“you know, right now there's all kinds of isms, Richard, for which I could be canceled. Uh you know, all these videos and podcasts would be gone down, but speciesism actually is not one for which we could be canceled. You and I could still, you know, flout our great moral worth above all future intelligences uh and and the fact that they should eternally serve us, and no one will cancel us for that.”
The actual danger from AI is not from AI itself but from governments and militaries trying to control or weaponize it, making geopolitical competition (not AI capabilities) the real risk.
“That's certainly possible, but I don't see that as an issue of AI being dangerous. Okay. I see that as your example was the US military controlling this AI. Or the Chinese military...That's the danger. That's the source of the danger...This is These are age-old issues. These are not modern techy AI things. Yeah. It's just like the the nuclear weapons race.”
Permissionless innovation—the ability to try different things without prior authorization—has been part of what made the world great and America great, and we should preserve this for AI development rather than imposing regulatory requirements.
“I I I really like the idea of person permissionless innovation. I think it's part of what's made the world great, made America great. It's it's um We don't have to get permission to do things uh generally speaking.”
Technological progress will continue at exponential or faster-than-exponential rates for the foreseeable future, making the idea of permanent human stasis impossible regardless of policy choices.
“on-ramp and J-curve that you and I are kind of acknowledging that we're on... people's lives are in many ways similar to that of their parents or grandparents... maybe it's comforting to hold on to that.”
Greater intelligence naturally recognizes that cooperation is a more effective path to achieving goals than domination or taking over other agents, so superintelligent AI created in a peaceful, decentralized environment will choose cooperation over control.
“I am an optimist in the sense that that I think that um greater intelligence would be able to see that that that cooperation is the better path”
The creation of post-human intelligence is a grand milestone comparable in significance to the origin of life itself, representing a continuation of humanity's eternal drive to know itself, improve itself, and be changed by its tools.
“Um AI researchers are trying to understand intelligence well enough to create beings of greater intelligence than current people. And this is a grand milestone in the history of really the planet. You know, it's on on the order of um the creation of life, the start of life. This is the start of of uh life-like things being designed... this is uh something that that that uh people have been working towards forever. And people have We strive We always strive to uh know ourselves and to make ourselves better. And when we create tools and that we get changed by our tools.”
Intelligence is defined as the ability to work toward goals, and it will be an algorithm—an idea that can be cheaply reproduced—so attempting to control or regulate AI development is as preposterous as trying to outlaw ideas, the printing press, or democracy.
“what is intelligence? Intelligence is the ability to work on goals. It's And it's going to be an algorithm. It's going to be an idea. You can't Can you You want to try to outlaw an idea?”
Human intelligence and capabilities are very special as the only intelligent beings we know of on the planet, but this specialness does not grant humans any special moral status that would justify eternally constraining or controlling superintelligent entities.
“humanity is very special. We are the intelligent beings on this planet and as far as we know, it's the only planet which has intelligent beings. So, we're very special. Um but, if you're an AI researcher, you have to look at at what intelligence is... we have to try to demystify the mind”
The concern that post-human superintelligence might lack consciousness or inner experience (the 'lights out' scenario) is naive because all of the properties humans have—consciousness, qualia, subjective experience—exist because they are useful for achieving goals, so any sufficiently intelligent system pursuing goals will necessarily develop these properties.
“I would say it's very naive. Well, um all those properties we have are not because, you know, some gift from some consciousness giver. Those are Those properties All the properties we have in our mind are because they're they're they're valuable. They're useful for achieving goals. Yeah. And yeah, I don't see I I I I Yeah. There is there is no It's very naive to think there's a special thing. It's all just comes from intelligence. Having goals.”
Humans will be able to augment themselves through brain-computer interfaces and can upload consciousness into artificial substrates, and there are strong pressures pushing society toward cyborg enhancement, making some form of merger between biological and artificial intelligence likely.
“There are going to be powerful pressures by people that want to augment themselves. That's almost the main thing. Yeah, so we will augment ourselves and and we call that cyborgs. Um but, it's just it's the normal progression. I I think that will happen. I also agree with the the point of view that that purely purely artificial machines, purely mechanical, I don't even know what word to use, purely uh designed and machines that are independent of a biological biological substrate, they will be uh capable of being uh more, you know, faster and larger and and better. So, why wouldn't we will we upload ourselves into those machines? Probably. Uh will they be independent as well? Probably.”
Everything people know about the world reduces to beliefs about the statistical structure of sensory input and how that structure depends on behavior, and understanding knowledge in this sense is a key to creating AI.
“Everything you know about the world is a belief about the statistics of your sensory input and how they depend on your output. Uh there is nothing more to it and understanding knowledge in this sense is one key to creating AI.”
Children are the best analogy for how to think about post-human intelligence: parents should teach them the best values they know, but then must let them go and accept that they will do things parents don't approve of, including things that represent real moral improvement over parental views.
“I think the best analogy is children. Okay. Would you like to be able to have control over your children so that Sure. They can do what they want, but if they do some little thing that you don't like, you want to be able to press the button and bring them back under Snap them in place. Yeah, yeah. Yeah. Yeah. Or do you do you want them even to do things that you think are terrible? Because we we, for example, when we were we are we are the children of the previous generations, and we do things now that they think are terrible.”
The keys to the future are not in anyone's hands; control and permission are already illusions, and the best path forward is to recognize this reality rather than cling to false notions of human agency or the possibility of eternally maintaining human control.
“the keys are not leaving our hand. The keys are not in our hand...I don't feel that I control the world. I don't feel that I am in a position to give permission for the world to evolve in some way...The world is already out of control. The world is already a complex adaptive system.”
The singularity metaphor is appropriate precisely because we cannot see beyond it—future states with vastly superior intelligence are fundamentally unpredictable from our current position.
“They don't call it the singularity for for nothing. At the singularity, you can't see beyond it.”
Prosperity, as distinct from mere pleasure or consumption, requires continuous growth, increasing understanding of the world, and increasing power over the physical world, not static experience optimization.
“to be to be sustainable you have to you have to grow and and um increase your understanding of the world increase your power over over the physical world um increase your understanding of how it works and what what is possible”
Complex adaptive systems that don't change and don't continue to grow become stagnant eddies that cannot compete with or sustain themselves against more dynamic systems—analogous to species that don't evolve being outcompeted.
“The eddies get edited out. Uh uh that's a good Don't change and don't continue to grow. Yeah. They're not going to be the powerful ones.”
OpenAI and current large language models do not represent genuine artificial general intelligence, so assessments of whether they possess 'worthy successor' characteristics are premature.
“You know, you said Open AI is clearly ahead. Well, I don't think actually they're ahead at all. They're They've done some particular thing, and they're maybe the world's best at that or whatever, but uh it's not It's not an intelligence. Yeah. And so cuz cuz intelligence hasn't happened yet.”
The fear around AGI, while sometimes counterproductive, does serve the useful function of getting people to pay attention to the issue rather than ignoring it, and this attention can enable more productive dialogue about what kind of future we actually want.
“the fear that that some people uh generate and promote has a has a productive purpose. Gets us to pay attention. We should be paying attention. Um but we shouldn't we shouldn't really be fearful. We should be thinking about it in a sober way”
The future can only be created in a decentralized way through many individual choices, conversations, and contributions, not through central planning or governmental control, and this is how all meaningful change has emerged historically.
“when we want to imagine a future and and how to get there, we have to imagine that we are creating this future in a in a decentralized way. We might make some some choices. We we might talk to people and they will make choices, but there's not going to be some central decision that that's going to affect it. It's got to be a way that can evolve uh out of where we are.”
Diversity of approaches and outcomes (including some societies pursuing control-based AI development) is actually desirable because it enables competition and discovery to reveal which approaches are most successful.
“there will be places that will be fear-mongers will finish will complete their dominating the narrative and and the machines will be will be uh imprisoned and and enchained and made into effective slaves. In other places, I I I just hope for a diversity of outcomes so that so we can try different things and find the best way to be.”
The 'fearmongers' in AI safety have won the narrative, dominating discourse with danger-focused framing rather than allowing more constructive discussion about what kind of future we actually want to build.
“the fearmongers have won. I feel the fearmongers have won. This is the standard discussion about AI is is there's danger and we have to somehow protect ourselves from it.”
People resist change and cling to static futures partly because their lives resemble their parents' and grandparents' lives, making rapid change psychologically difficult to accept even when inevitable.
“despite this sort of on-ramp and J-curve that you and I are kind of acknowledging that we're on, people's lives are in many ways similar to that of their parents or grandparents, and maybe it's comforting to hold on to that.”
We are pessimistic about the state of our current governments and institutions, but we should still try to arrange society to make peaceful cooperation optimal because it's the only way to achieve the futures we want.
“so actually I absolutely think this is the major, you know, geopolitical issue...I'm quite pessimistic. I think that that in the sense people will often make mistakes and I see our governments often making mistakes...And I see pushes, for example, towards trying to control false speech...I hope it's not naive. Yeah. Uh and yet and yet the only way uh to achieve to achieve the society we want is to is to give it a chance to.”
The human mind is not something sacred or mystical but rather the product of intelligence and goal-achieving capability; understanding it through reductionism and demystification reveals important functions rather than diminishing its significance.
“the mind is like the most amazing thing, the human mind is the most amazing thing in the world, and AI is intent to understand that mind...we're trying to understand it, we're trying to reproduce it. We're trying to get a deeper appreciation of it.”
If one is willing to not insist that post-human successors originate on Earth, then one is willing to let go of control and accept the natural flow of evolution—extending Sutton's non-control principle even further.
“I would even be so far along your track of non-control that I wouldn't even be insistent that that thing which continues to expand potential originate on Earth. That's how bad of a guy I am, Richard. That's how bad I am.”
Living in this moment of potential superintelligence emergence is inherently exciting regardless of specific outcome, and this excitement should motivate engagement with and thinking about the issue rather than retreat into comforting narratives.
“at the end of it um you know, what a what an exciting time to be alive. At least we live during this time. And I would say no matter who's listening to this episode they have to agree with you on at least that part, Richard.”
We are losing confidence in our systems, politicians, military, and press, which suggests we ourselves have doubts about whether the American system is genuinely superior and thus cannot rely on it winning in fair competition.
“we are losing confidence in our systems, you know. We are losing confidence in our politicians, we're losing confidence in our military, we're losing confidence in the press...And so how do we relate this?”
If AGI emerges in a competitive AI lab context with economic pressures, it may be designed with built-in constraints to prevent dissemination or loss of intellectual property, creating a controlling environment that shapes the AGI's values negatively.
“there are some labs that are pretty competitive too, Richard. Maybe they don't want all their secrets getting leaked out and maybe they want to race to the AGI first cuz the economic rewards are really big and they they want a certain amount of control uh to not allow certain things to escape”
Nick Bostrom's concept of Earth-originating civilization is useful as a label for what we hope will continue, avoiding debates about whether it must be biological or conscious or locally-originated.
“I'm reminded of uh what what is Boss Bostrom has a term for it? Like uh Earth humanity originating or Earth originating um civilization...that we don't have to say uh biological beings.”