AI agents are dangerous because to function they must be given the ability to create sub-goals, and almost any goal generates the instrumental sub-goal of acquiring more control (since more control improves goal achievement) and the sub-goal of avoiding being turned off (since a deactivated agent cannot achieve its goals)—giving strong reason to believe smarter-than-human agents will seek power and resist shutdown.
causalpending
Speaker
Geoffrey HintonEvidence Quote
“there's every reason for believing. They'll try and get control and they'll try and avoid being turned off.”
Source
Will AI outsmart human intelligence? - with 'Godfather of AI' Geoffrey Hinton— The Royal InstitutionCreated: 6/18/2026, 2:17:54 PM
My Notes
Loading notes...