Repository navigation
Add offline score-only / replay path: rescore a predictions JSONL without a model #315
Description
Ask
A way to run a task's scorers and metrics over an existing predictions file, with no model and no GPU. Something like:
olmo-eval score -t popqa:chat --predictions path/to/predictions.jsonl
or equivalently a replay provider (provider.kind=replay, provider.predictions=<path>) so the whole runner/metric path is exercised without inference.
oe-eval already has this as --recompute-metrics ("Recompute metrics for cached predictions"). olmo-eval has no equivalent: olmo-eval metrics only queries stored inference telemetry, and MockProvider returns fixed canned responses for unit tests rather than replaying a file.
Why it's worth it
1. Port certification becomes nearly free, and much stronger. Certifying #307/#308/#309 against oe-eval consumed a lot of GPU time comparing aggregate scores — a weak test, since T=0.6 sampling at n=500 gives SE ≈ 0.018 and only catches gross errors. The strong test is paired: take oe-eval's own stored generations and score them with olmo-eval's scorer, then diff per-instance verdicts. Sampling noise disappears entirely.
Done retroactively for popqa (Qwen3-8B, oe-eval's stored generations, olmo-eval's PopQAContainsScorer):
instances : 14267
per-instance agreement : 14267/14267 = 100.0000%
oe-eval aggregate : 0.31289
olmo-eval scorer : 0.31289
paired diff : 0.00000
That is a far stronger statement than the z=0.06 aggregate cell on #307, and it cost zero GPU-seconds. It only required hand-rolling a script because there's no supported entry point.
2. Scorer changes currently require re-running the model to see their effect. #296 (IFEval determinism) and #314 (IFEval robustness) were both pure scorer fixes. Measuring their impact today means regenerating; with a replay path you'd rescore stored predictions in seconds. Same for anyone tuning an answer-extraction cascade.
3. It's the natural regression test for scorers. Check in a small predictions fixture per task, assert scores don't move.
Notes for implementation
- Both harnesses' prediction formats carry what's needed: raw generation, gold/label (oe-eval's popqa
labelis the full alias list), and per-instance metrics for comparison. - A cross-harness mode (read oe-eval's
task-*-predictions.jsonltoo) would make it a one-command parity gate for the remaining ports — MMLU:cot and LiveCodeBench are still to come. - Incidental perf note found while doing this by hand: the regex cascade in
common/answer_extraction.pyis slow on long reasoning traces (500 omega items with ~35k-token generations took minutes of pure CPU). Not a correctness issue, but a replay path would make it easy to profile.
🤖 Generated with Claude Code
Activity
abhishekraok commentedon Aug 25, 2026
Concrete pending use case, now that #320 exists: it lists two bugs in oe-eval's
extract_answerthat olmo-eval reproduces deliberately (a bare42extracts to''because the unanchored(.*)\.?fallback matches at end-of-string and[-1]takes the empty match;The answer is: 42extracts': 42'because the colon-bearing prefixes sit after their shorter forms and are unreachable).Fixing either means diverging from the reference, which breaks comparability with published Olmo 3 numbers — so the decision hinges on how many real responses are actually affected. Today that question costs a GPU re-run per candidate fix. With a replay path it's a rescore of predictions already sitting in the result datasets: run both extractors over the same stored generations and count the instances where the verdict moves.
That generalizes to any change in
common/answer_extraction.py— the cascade is shared across omega_500 and the GPQA CoT tasks, so "what would this cost us" is a question that will keep coming up.abhishekraok commentedon Sep 17, 2026
Claude Code · https://claude.ai/code/session_01LMFjgUqHjQA3aCeB4QmEHZ
Re-checked against main
73770ec, still valid; assigned to @gregorybchris as eval owner. Noscore/replay subcommand or replay provider exists (cli/has beaker, metrics, results, run, suite, task). Every paired parity check since (mmlu:cot, ifeval, popqa, omega_500 for reasoning-regime-ports (#348) / strip-thinking (#375)) was a hand-rolled script.abhishekraok commentedon Nov 3, 2026
Written by Claude Code session 28dc7c78 (https://claude.ai/code/session_01Amsg4C36artRnazx27HKRY)
replay-provider (#536) adds the replay-provider form of this ask:
provider.kind=replay provider.predictions=<olmo-eval results dir>. It runs the full runner, scorer and metric path on saved generations. It does not read oe-evaltask-*-predictions.jsonlfiles yet. That cross-harness parity mode is still open here, so the PR doesn't close this issue.