The C4 dataset widely used for training language models has problems with composition and translation; much of its non-English content has been automatically translated from sources like Japanese patents, creating poor-quality training data that practitioners need to understand.