People will inevitably give superintelligent systems goals, because you want intelligent systems to achieve goals. If you want them to be good at achieving goals, you give them the ability to create subgoals. An intelligent system will realize that a very good subgoal is to get more control, because that helps you achieve all your other goals.
causalpending
Speaker
Geoffrey HintonEvidence Quote
“as soon as you have the ability to create a sub goal if you're intelligent you realize that a very good sub goal is to get more control because that helps you achieve all your other goals”
Created: 8/12/2026, 5:56:09 PM
My Notes
Loading notes...