The downside of contrastive methods is they rely heavily on negative samples, requiring a large batch of other images to push away from, meaning you need very large batch sizes or memory banks, which makes training computationally expensive and harder to scale.