FineWeb-Edu and the case for educational data in pre-training: what changed in MMLU and ARC
FineWeb-Edu is a 1.3T-token educational subset of FineWeb whose paper reports large gains on knowledge- and reasoning-heavy evaluations, including higher MMLU and ARC scores than the base FineWeb subset — but the lift comes from a carefully filtered educational slice, not from adding more generic web text.