factual

Space Probe Analogy for One-Shot Failure

Building aligned superintelligence is like launching a space probe that must work on the first try in an environment that differs from any test environment; even with full knowledge of physical laws, an $800 million probe was lost to a slow vapor buildup over years in vacuum, because no one will pay the vastly greater cost to anticipate every contingency - and we understand AI minds far less than we understand probes.

factualpending

Speaker

Eliezer Yudkowsky

Evidence Quote

what killed the probe was that over the long time it was in space there was a slight buildup of vapor inside the probe... that took out an 800 million probe... even with the probe where we understand all the physical laws... it's just too hard

Source

434 - Can We Survive AiMaking Sense with Sam Harris
Created: 6/14/2026, 2:10:11 AM

My Notes

Loading notes...