AI agents are dangerous because to function they must be given the ability to create sub-goals, and almost any goal generates the instrumental sub-goal of acquiring more control (since more control improves goal achievement) and the sub-goal of avoiding being turned off (since a deactivated agent cannot achieve its goals)—giving strong reason to believe smarter-than-human agents will seek power and resist shutdown.

causalpending

Speaker

Geoffrey Hinton

Evidence Quote

there's every reason for believing. They'll try and get control and they'll try and avoid being turned off.

Source

Will AI outsmart human intelligence? - with 'Godfather of AI' Geoffrey HintonThe Royal Institution
Created: 6/18/2026, 2:17:54 PM

My Notes

Loading notes...