[AkshitaB]

Summary

Everything the scaling-ladders modularity evals (ladders/olmoe3/workloads/evals/modularity_full) need from olmo-eval, on top of current main. It replaces the pins to ryanyxw/ladder-evals and akshitab/ladder-eval-runtime-fixes; Jackson's long-context task commits on those branches are not needed here and are left out.

• Namespaced dataset ids for GSM8K, CSQA, PIQA, SocialIQA, WinoGrande and HumanEval (openai/gsm8k, tau/commonsense_qa, …). With current huggingface_hub, load_dataset("gsm8k", "main") fails from an empty cache ("Repository id must be 'namespace/name'"); it only works where an older cache already holds the dataset. The namespaced ids load with the same row counts. The data source is part of a task's identity, so the recorded hashes of the tasks reading these datasets change (task_hashes.json regenerated with --update-task-hashes; 60 entries).
• Pipeline parallelism: an instance spans tensor_parallel_size × pipeline_parallel_size GPUs, in the GPU planner and in the Beaker GPU request, and each inference worker gives its engine tensor_parallel_size = GPUs / pipeline_parallel_size (it used the worker's GPU count, so a 4-GPU pipeline-parallel worker asked for TP 4 too). Needed to run a model larger than one GPU with provider.kwargs.pipeline_parallel_size (the OLMoE3 vLLM plugin supports PP, not TP).
• Beaker launcher runtime hooks (Ryan's): OLMO_EVAL_RUNTIME_TORCH_EXTRA_PACKAGES installs torchvision/torchaudio builds with a runtime Torch override (cu130 on B300), and OLMO_EVAL_RUNTIME_POST_INSTALL_CMD runs a snippet after all installs.
• Provider package index: with OLMO_EVAL_RUNTIME_TORCH_INDEX_URL set, provider packages resolve build dependencies from PyPI with the Torch index as an extra index, not as their only one.

Test plan

• pytest tests --ignore=tests/integration: 5289 passed, 8 failed; the same 8 fail on main (test_sandbox_manager_capacity, test_check_file_sizes).
• Each renamed dataset id loads from an empty HF cache with the expected row count.
• End to end through scaling-ladders' modularity evals on Beaker: a 4-GPU pipeline-parallel worker gets GPUs [0, 1, 2, 3] with TP 1 / PP 4 (small hero on H100s, 24/24 items).
• scaling-ladders' modularity tasks (5 GSM8K variants, 34 MMLU-by-category tasks) format byte-identical requests and sampling parameters on this branch and on the previous pin.

🤖 Generated with Claude Code

[gregorybchris]

Approved, with the two inline findings below to address and resolve before merging. Review validation: 191 focused tests passed; auxiliary GPU allocation was checked with mocked server startup. Lint reports three added long lines, and formatting needs correction. No real-model run was performed.