People will inevitably give superintelligent systems goals, because you want intelligent systems to achieve goals. If you want them to be good at achieving goals, you give them the ability to create subgoals. An intelligent system will realize that a very good subgoal is to get more control, because that helps you achieve all your other goals.

causalpending

Speaker

Geoffrey Hinton

Evidence Quote

as soon as you have the ability to create a sub goal if you're intelligent you realize that a very good sub goal is to get more control because that helps you achieve all your other goals

Source

Geoffrey Hinton - Two Paths to IntelligenceCSER Cambridge
Created: 8/12/2026, 5:56:09 PM

My Notes

Loading notes...