The attempt to guard-rail ChatGPT by having it refuse to utter racial slurs has failed to generalize, and broader attempts at value alignment through reinforcement learning have been very sloppy and don't reliably do what is intended.