Repository navigation
Add a replay provider that rescores saved generations - #536
Conversation
provider.kind=replay with provider.predictions=<file or results dir> answers each request with the generation an earlier run saved for the same prompt, matched through the requests file saved beside each predictions file. Unmatched prompts and sample-count mismatches fail as generation errors, so the failure budget catches a replay against the wrong run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Amsg4C36artRnazx27HKRY
Fail image and tool requests, which the saved files cannot pair with the right image or native tool calls, and reject directories that mix models. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Amsg4C36artRnazx27HKRY
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 919f967ed3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Two configurations of a task can save the same prompt with different sample counts; serving them from one queue could fail every request or hand one configuration's generations to the other. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Amsg4C36artRnazx27HKRY
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f9f3f751d4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 80fe4018d7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…text Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cb6a9202eb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rations A request does not say which instance it is and the runner shuffles requests, so serving repeated prompts in arrival order could swap instances' generations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Written by Claude Code session 28dc7c78 (https://claude.ai/code/session_01Amsg4C36artRnazx27HKRY); updated by Claude Code session 27c852a2-737f-40d7-a147-784d52aa3a9b
Closes #457.
Adds a
replayprovider: rescores a saved run's generations with no model or GPU. It covers the last open item of lcb-stalled-execution (#457) (suggested fix 3) and the replay-provider form asked for in add-offline-score (#315).contextplussystem_prompt) is identical. A request does not say which instance it is and the runner shuffles requests, so a prompt saved for several instances replays only when every save serves the same generation; otherwise its requests fail.original_textwhen reasoning was stripped), so the task strips and extracts the answer again.finish_reason, the token counts, the raw Harmony decode, its parser error and the context-cut marker carry over.num_samples, the request is marked as a generation error. A replay against the wrong run or the wrong task options then fails through the failure budget instead of scoring 0.Checks
tests/providers/test_replay_provider.py.gsm8kwith limit 4 onmock, rewrote 2 of the 4 saved answers to the gold answers, and replayed them. The replay scored 0.5, both with-m mockand with-m allenai/OLMo-2-1124-7B(no model load). A chat task,ifeval, also replays. Replaying limit 8 against 4 saved answers fails withFailureBudgetExceeded: 4 of 8 instances get "replay: no saved generation matches this prompt".ruff,tyand the unit suite pass, apart from 4tests/scripts/test_check_file_sizes.pytests. Those 4 fail the same way on clean main in this checkout.🤖 Generated with Claude Code
https://claude.ai/code/session_01Amsg4C36artRnazx27HKRY