The concern that post-human superintelligence might lack consciousness or inner experience (the 'lights out' scenario) is naive because all of the properties humans have—consciousness, qualia, subjective experience—exist because they are useful for achieving goals, so any sufficiently intelligent system pursuing goals will necessarily develop these properties.

causalpending

Speaker

Richard Sutton

Evidence Quote

all those properties we have... are valuable... useful for achieving goals... It's all just comes from intelligence

Source

Richard Sutton - Humanity Never Had Control in the First Place (Worthy Successor Series, Episode 2)The Trajectory with Dan Faggella
Created: 8/12/2026, 6:16:37 PM

My Notes

Loading notes...