Repository navigation
Added Browsecomp-Plus - #373
Open
wu-ming233 wants to merge 13 commits into
Open
wu-ming233 wants to merge 13 commits into
wu-ming233 wants to merge 13 commits into
Conversation
…omp web task - Raise the BrowseComp presets to 150 turns and enable model thinking; the tasks strip thinking traces before scoring. - Retry transient judge errors and record trajectory measurements before grading, so a judge failure no longer zeroes them. - Treat runs the scaffold abandoned (turn limit, context limit, tool error, refusal) as incomplete instead of grading placeholder text. The scaffold now reports a context-length overflow as a scored, unanswered instance with the trajectory gathered so far. - Add a `nojudge` variant that records trajectories and judge-free metrics without calling GPT-4.1, plus `completed` and `tool_errors` metrics. - Let the evaluation process start the retrieval server itself when BROWSECOMP_PLUS_RETRIEVER is set, so one cluster job serves both the model and the retriever. - Add a `browsecomp` task and harness preset for the original benchmark on the live web, sharing the grader and metrics with BrowseComp-Plus. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…B preset - Move the scorer, metrics, judge call and run-completion checks shared by BrowseComp and BrowseComp-Plus into browsecomp_common, so neither task imports from the other. - Ship the upstream MIT license next to the copied reference code. - Commit the BrowseComp-Plus task, data, server, setup and tool tests. - Add a qwen3.5-4b model preset with text-only serving and the model card's thinking-mode sampling. - Document the live-web BrowseComp task in the task README. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- The Serper search and fetch tools and the crawl4ai browse tool no longer return or open pages on huggingface.co and hf.co, where benchmark data is widely mirrored. OLMO_SEARCH_BLOCKED_DOMAINS replaces the default list. - Raise the BrowseComp presets' model-call timeout from 120s to 30 minutes so long thinking turns no longer hard-fail a question. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- Add a jina_fetch_webpage_content tool that reads a page through Jina Reader in its default format, honors the blocked-domain list (including after redirects), and returns up to 20,000 characters. - Switch the browsecomp preset and task to Serper search plus the Jina reader. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The tasks rewrote their data source to the local cache path when constructed, so the task hash depended on the machine's cache directory. They now keep the canonical source in the config and switch to the prepared local file only when loading instances. Pin the task hashes for the new BrowseComp and BrowseComp-Plus specs.
The judge fixture patched openai.AsyncOpenAI, which fails where the optional SDK is not installed. Stand in for the module instead.
Ask Jina Reader for its JSON response, which carries the title, final URL and content as fields, and drop the regex that split the plain-text header.
The reference evaluation sends only the user prompt, so the preset no longer adds a system prompt of its own.
Restore the OpenAI Agents scaffold to its upstream behaviour: a context-length error is an ordinary failed instance again, not a scored placeholder answer. Drop the matching check from the BrowseComp completion test.
- Drop the code that started the retrieval server from inside a tool call; the server is started separately, as the README describes. - Fold the shared scoring back into the BrowseComp-Plus task module and remove the separate common module with its function-level imports. - Remove the `completed` and `tool_errors` metrics and the scaffold-specific completion heuristics; a run is complete when it gives an answer and is not cut off at the turn limit. - Write each response's scores once, after grading.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds BrowseComp-Plus evaluation
get_documentsettings.Official BrowseComp-Plus use Qwen3-32B to judge, but GPT-4.1 is close in judging numbers and also used by official evals. Continuing with GPT-4.1.