Direct pixel or token reconstruction objectives contain full entropy from noise sources like text variations without right answers and lighting changes in images, forcing models to accommodate unpredictable details even though they are fundamentally irrelevant to meaning.
causalpending
Evidence Quote
“As the problem of directly working with pixels or tokens is that the objective is full of entropy. For example, text variations that does not have a right answer while all meaning roughly the same thing, and lighting changes of background within an image that doesn't even matter, but the reconstruction error will always be full of noise.”
Created: 8/12/2026, 6:16:15 PM
My Notes
Loading notes...