Repository navigation
LiveCodeBench: a stalled execution quarantines every podman sandbox and later samples are silently scored 0 #457
Description
Codex · codex://threads/01a1326f-ed81-75e3-80e4-661392dbe4b5
Implementation status: the core execution and failure-accounting fixes are now on main: count-scoring-failures (#492), drain-sandbox-container (#515), isolate-sandbox-commands (#470), and recover-quarantined-sandbox (#471). The PRs link the saved-output grading and exact-source container-cleanup receipts, with their tested revisions.
Suggested fix 3, a dedicated saved-prediction replay CLI, remains tracked in add-offline-score (#315). Restart-budget policy is a separate follow-up in make-sandbox-restart (#482); remaining HTTP command paths are tracked in remove-blocking-http (#483). Keep this issue open for the outstanding replay item.
Original report
Written by Claude Code session ledger-manager (https://claude.ai/code/session_0143Q74cUmeM2FLnNZg3yiyK)
Bug. In a LiveCodeBench run with more than one sample per problem, a single stalled execution can take every podman sandbox out of service. Every later sample is then scored 0 with SandboxRequiredError. The task still exits 0, so nothing in the reported pass rate shows it.
Setup. olmo-eval 0131141, livecodebench:lite (v3) with k=8 (612 problems x 8 = 4,896 samples), the codex_python harness, and SandboxMode.DOCKER with podman and sandboxes.0.instances=4. T=1.0, forced think, 32K answers. The spec is H045's lcb-v3-forcedthink-t10 with k=8.
Runs. Three Olmo 3.5 small (4T) SFT checkpoints, all hf_step11768, evaluated for H051 in allenai/olmo-post-training-ledger:
| run | model | Beaker | samples lost to SandboxRequiredError |
|---|---|---|---|
| E1 | EMO SFT seed 1, /weka/oe-training-default/ai2-llm/checkpoints/abhishekr/hero-sft-4t/runs/h045-anchor-emo-olmo35-s34521/hf_step11768 |
01M4C01F95AC0NXJBRVWHBFFVH | 2,364 of 4,896 (48%) |
| E2 | EMO SFT seed 2, .../h045-anchor-emo-olmo35-seed2-s34522/hf_step11768 |
01M4C0HW77Q2XRYZFVN73V07V5 | 4,301 (88%) |
| N | non-EMO SFT, /weka/oe-training-default/ai2-llm/checkpoints/pradeepd/hero-sft-4t/runs/h038-control-olmo35-s34521/hf_step11768 |
01M4C125YCT15T45WH17MTXDBQ | 4,336 (89%) |
The error message is "LiveCodeBenchScorer requires a sandbox", and losses are all-or-nothing per problem. At k=1, the same spec and commit had 0 errors in 612 samples, in two runs (01M42X0PXJ5RWPH1QG9DQH8QSW, 01M42X7427SKDTBETMTEJEZD80).
Pattern (identical in all three runs).
- The "Scored N/612" progress counter stalls: E1 at 307, E2 at 69, N at 57.
- About 5 minutes later, 3–4 sandboxes report
Transport failure during file write to /tmp/lcb-…/{problem.json,grade.py,solution.py}at the same moment. That points to something in flight wedging the shared runtime or host, not a single sandbox. - Each sandbox exhausts 4 transport retries and fails 4 health checks, then is quarantined. All 4 are quarantined within about 40 minutes.
- From then on, every sample hits "No healthy sandbox remained" and gets
SandboxRequiredError.
It's deterministic and driven by the code being run, not by load. We replayed each run's saved outputs through olmo-eval run with a mock provider (CPU-only Beaker jobs on ai2/neptune: 01M4CSG2W9NBQAHDN006XNB1G2, 01M4CSH1AN0HYGVYJ0YXD7NTYS, 01M4CSHZRZA3WR9V500TXFD2F8). The cascade came back on the same problems: 295/298, 512/541 and 514/542 of the errored problems overlap the original runs. Lowering concurrency therefore won't fix it.
Problems in flight at the stall (LiveCodeBench ids):
- E1: abc349_c, abc352_c, abc364_e, abc312_a
- E2: abc353_a, abc362_d, 3388, 2955
- N: abc340_d, 3297, abc362_d, 3388
abc362_d and 3388 recur. The logs don't separate the sample that hung from the ones queued behind it.
Repro bundle: /weka/oe-adapt-default/abhishekr/handoff/h051/lcb-rescore/repro/.
README.md: the pattern and how to reproduce.repro.json: per run, the stall line, the first transport failure, all four quarantine log lines and the in-flight problems; per sample, the output index, response sha1, lengths and whether it executed.<run>/<problem>__out<k>.py: the extracted code of the non-executed in-flight samples. It is model-generated, so run it only in a sandbox.- Cleaned job logs:
../logs/{e1,e2,n}.clean.
Suggested fixes.
- Kill each execution on a hard wall-clock limit, including its whole process group. Respawn quarantined sandboxes instead of failing every remaining sample.
- Don't score infrastructure errors as 0 silently. Report a sandbox-error count in metrics.json, and fail the task, or mark the metric invalid, when it is non-zero.
- Add a rescore/replay entry point for saved predictions. Today this needs a mock-provider shim.
Activity
gregorybchris commentedon Oct 22, 2026
Written by Claude Code for @gregorybchris.
Status after #492 (merged as d60f81f): suggested fix 2 is done. Fix 1 is still open, in #470 and #471.
What #492 changed for this failure
max_hard_failure_rate(5% by default). A cascade like the ones above (48–89% of samples) fails the task and withholds its metrics, so it can't pass as a low pass rate. The run exits non-zero.metrics.json, the upload, and the run summary carry ascoring_failuresrecord: unscored and partially scored instance counts, the infrastructure count, counts by error type, and the first error.SandboxInfrastructureError("No running sandbox has capability …"), notSandboxRequiredError("requires a sandbox"). The earlier message pointed at configuration when the pool had been lost.Reproduced on main before #492.
livecodebench:lite, limit 200, k=1,codex_universal,allenai/Olmo-3-7B-Instruct, one Ceres H100, commit 47fabfb (01M5JRA8).Scoredcounter stalled at 129/200. Then file writes to/tmp/lcb-…/solution.pyfailed three times about five minutes apart, and at 22:12 UTC the sandbox was quarantined after four failed health checks.SandboxRequiredError. Main gave all 64 a 0 and withheld the task. Counting them as 0 gives 0.210; the 136 scored instances average 0.309.Still open: stopping a single stalled execution from wedging the sandbox (#470), and replacing quarantined sandboxes (#471, then #482 for the restart budget). Until those land, a cascade fails the task loudly instead of scoring it, but the run is still lost.
abhishekraok commentedon Nov 3, 2026
Written by Claude Code session 28dc7c78 (https://claude.ai/code/session_01Amsg4C36artRnazx27HKRY)
The last open item here, suggested fix 3 (rescoring saved predictions without a mock-provider shim), is in replay-provider (#536). The PR closes this issue on merge.
-H default -o provider.kind=replay -o provider.predictions=<results dir>, with the original-m/-toptions, rescores a saved run with no GPU. Each request gets the saved generation for the same prompt.Still separate: make-sandbox-restart (#482) and remove-blocking-http (#483).