Charismatic averaging makes default outputs boring

Just as people who must appeal to large audiences are incentivized toward boring, non-divisive views, language models trained to maximize average preference may produce benign, average creative output (e.g., generic rhyming poems); prompting the model to be fully creative can push it off that average.