normative

Three Distinct Superintelligence Goals

There are three distinct technical goals one could pursue: a superintelligence that obeys orders with no unexpected side effects, one that benevolently runs the galaxy by nice principles without humans in charge, and one that is itself a conscious being having fun and acting as a good citizen; these are different goals and should not be pursued simultaneously, especially at the start.

normativepending

Speaker

Eliezer Yudkowsky

Evidence Quote

there's the super intelligence that shuts up does what you ordered... there's super intelligence that is trying to run the whole galaxy according to nice benevolent principles... and third there's there's super intelligence that is itself having fun and cares about other super intelligences... these are three different goals

Source

434 - Can We Survive AiMaking Sense with Sam Harris
Created: 6/14/2026, 2:10:11 AM

My Notes

Loading notes...