factual
Space Probe Analogy for One-Shot Failure
Building aligned superintelligence is like launching a space probe that must work on the first try in an environment that differs from any test environment; even with full knowledge of physical laws, an $800 million probe was lost to a slow vapor buildup over years in vacuum, because no one will pay the vastly greater cost to anticipate every contingency - and we understand AI minds far less than we understand probes.
factualpending
Speaker
Eliezer YudkowskyEvidence Quote
“what killed the probe was that over the long time it was in space there was a slight buildup of vapor inside the probe... that took out an 800 million probe... even with the probe where we understand all the physical laws... it's just too hard”
Created: 6/14/2026, 2:10:11 AM
My Notes
Loading notes...