RLHF works via subtle aggregated human preferences

RLHF works so well because human preference data contains a huge amount of subtle information — different people pick up on small things (like correct semicolon usage) that an observer wouldn't even notice — and the model learns across all domains and contexts what humans want, similar to how deep learning beats hand-coded edge detection.