Building autonomous weapons systems is a catastrophic policy error because it escalates to a race-to-the-bottom where every nation must develop weapons it doesn't want, guaranteeing eventual use, and the incentive structure mirrors nuclear proliferation: once one player builds them, all others must follow, locking in mutual vulnerability.