If solving a problem requires composing too many distinct strategies, mode-collapsed models will struggle, especially with outcome rewards. Possible fix: identify base skills, decompose environments to these mini-environments, and then train sequentially.
I suspect a model trained through environment n in the curriculum might even beat the base model on environment n at pass@k, even at high k and low n.
Loosely related promising empirical results: Cognition’s swe-check using a 2-step post-training process for reasoning and latency, and Li et al.’s “Reasoning and Tool-use Compete in Agentic RL.”