normative
Three Distinct Superintelligence Goals
There are three distinct technical goals one could pursue: a superintelligence that obeys orders with no unexpected side effects, one that benevolently runs the galaxy by nice principles without humans in charge, and one that is itself a conscious being having fun and acting as a good citizen; these are different goals and should not be pursued simultaneously, especially at the start.
normativepending
Speaker
Eliezer YudkowskyEvidence Quote
“there's the super intelligence that shuts up does what you ordered... there's super intelligence that is trying to run the whole galaxy according to nice benevolent principles... and third there's there's super intelligence that is itself having fun and cares about other super intelligences... these are three different goals”
Created: 6/14/2026, 2:10:11 AM
My Notes
Loading notes...