Current LLMs operate through autoregressive token prediction—producing one token at a time by predicting the next token given a fixed window of previous tokens—which creates exponential divergence because each token prediction has some probability of error, and once an incorrect token is generated, there is no way to correct it, leading to hallucinations.
causalpending
Speaker
Yann LeCunEvidence Quote
“the way you train the uh those systems uh then you can use it to generate text by just Auto aggressively producing a token shifting it into the input and then producing the second token shifting that in ETC that's Auto prediction Not A NEW Concept at all obviously um and there's an issue with this which is that um the U the that process is basically Divergent every time you produce a token there is some chance that the token is not within the set of reasonable answers and take you outside a set of reasonable answers and if it does that there is no way to fix it afterwards um and if you assume there is some probability for that... then you get exponential Divergence uh which is why you know we have with those models hallucination issues”
Created: 8/11/2026, 7:11:43 AM
My Notes
Loading notes...