LLMs Should Learn to Say I Don't Know

Large language models are very good at sounding like they are saying true things without being grounded in mathematical truth, so when training data on a topic is sparse they concoct accurate-sounding but false narratives; a key challenge is teaching them to say 'I don't know' more often and to infer or introspect about what is actually true.