Should you extend context or retrain for long-context workloads? Lessons from RULER and LongBench v2
RULER shows that many models look near-perfect on vanilla needle-in-a-haystack yet suffer large drops as context length and task complexity rise, while LongBench v2 shows the best direct-answer model still reaches only 50.1% accuracy and o1-preview reaches 57.7% — but that gap does not automatically justify retraining, because the right choice depends on whether your workload needs deeper reasoning, not just longer windows.