diff --git a/CHANGELOG.md b/CHANGELOG.md index 555572579a..01950f0534 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,7 @@ All notable changes to this project will be documented in this file. - Remove the dataset readiness scripts (`scripts/data/rl_readiness/`), the standalone IF-target filter and the first-pass data handoff page; the RL Data Workbench now owns qualification, screening and release evidence, while Open Instruct keeps source adapters, catalog resolution, verifiers and the training loader (https://github.com/allenai/open-instruct/pull/2220). ### Changed +- Add the native Nemotron abstention QA adapter and named three-tier judge profile, preserving NVIDIA's rubric and answer extraction while keeping judge failures ungraded (https://github.com/allenai/open-instruct/pull/2252). - Honor native MILES num_epoch budgets without injecting a conflicting rollout default; save epoch-run final checkpoints and recognize the runtime-resolved completion limit (https://github.com/allenai/open-instruct/pull/2251). - Enable final-answer grading in the three maintained MILES examples while preserving the library default. Keep exclusion tables aligned with grading-aware record keys, reject cross-contract evidence, and validate native reward contracts before worker startup (https://github.com/allenai/open-instruct/pull/2250). - Give MILES final-answer grading shared thinking/response views, explicit per-verifier input and eligibility policies, non-thinking and prompt-prefilled-closure support, per-component selection provenance and distinct evidence identities. Preserve default-off scoring; document every path in `docs/miles/grading.md` with validation evidence in the PR (https://github.com/allenai/open-instruct/pull/2250). diff --git a/docs/miles/abstention.md b/docs/miles/abstention.md new file mode 100644 index 0000000000..9931303b80 --- /dev/null +++ b/docs/miles/abstention.md @@ -0,0 +1,28 @@ +# Nemotron abstention QA + +The Nemotron-RL-QA-Abstention-v1 dataset supplies factoid questions and factual references while asking the policy to abstain when uncertain. The `nemo_abstention_v1` source adapter preserves its original system/user messages and emits a structured question/reference target. Training needs a named correctness judge; plain answer matching does not reproduce the abstention objective. + +The `nvidia/qa-abstention-v1` rubric ports NVIDIA-NeMo/Gym's abstention environment at `cf7f71a57dde0a69b14a2e919b60a01b17cc7abd`. It strips thinking traces, extracts the last balanced boxed answer (including nested braces), and falls back to the remaining output when no complete box exists. Normalized `[IDK]` receives 0.5 without contacting the judge. Other outputs use the original OMNISCIENCE_GRADER prompt and A/B/C parser: correct receives 1, incorrect 0, and NOT_ATTEMPTED 0.5. This version fixes the abstention reward at NVIDIA's default 0.5; it does not implement dynamic calibration. + +Reuse the existing `judges.general` Qwen/Qwen3-32B service from the maintained three-node example (model revision `9216db5781bf21249d130ec9da846c4624c16137`). Add a rubric and binding to that same service; no additional judge GPU or model load is needed: + +```toml +[rubrics.abstention] +profile = "nvidia/qa-abstention-v1" +max_response_tokens = 64 +temperature = 0.0 + +[judging.bindings.abstention] +judge = "general" +rubric = "abstention" +``` + +Replace the verifier binding name with the actual name in the exported `verifiers.json`; Workbench namespaces names by their contract. Keep the profile paired with its structured `nemo_abstention_v1` target. Preparation emits a `NamedJudgeVerifier` registry entry and preserves the full input as `metadata.judge_query`. + +The named-service probe requires correct/incorrect/explicit-abstention/implicit-abstention controls to score 1/0/0.5/0.5. Empty replies, transport errors and incomplete judge generations follow the existing failure policy: strict mode raises; zero mode records `status=judge_error` and a fallback zero, which the training bridge excludes as ungraded. Evaluation can continue with those explicit failure diagnostics. Diagnostics distinguish an explicit abstention from a judge-produced NOT_ATTEMPTED grade. Existing truncation/final-answer gates apply before judging; a capped response does not earn abstention credit. + +This is the native permissive parsing contract, not strict boxed-format enforcement. In particular, nonempty judge output uses NVIDIA's substring fallback for A/B/C and otherwise defaults to incorrect. Preserve that fact in calibration; tightening the parser would be a separate contract change. The rubric also contains permissive numerical and name-equivalence examples. Test them with the selected judge before relying on training rewards. + +A source-preparation or fixture test does not calibrate an actual judge or establish learning benefit. Qualify the intended model and endpoint with independently reviewed correct, wrong, equivalent, incomplete and abstaining responses across all four domains. The publisher provides no validation split. Preserve the card/row license discrepancy (CC BY 4.0 versus CC BY-SA 4.0), and run the intended release's protected-source hygiene before redistribution or training admission. + +Upstream implementation: . diff --git a/docs/miles/source-preparation.md b/docs/miles/source-preparation.md index 622ec9b652..78c3d7e3b5 100644 --- a/docs/miles/source-preparation.md +++ b/docs/miles/source-preparation.md @@ -572,3 +572,7 @@ comparator, preserving ToolRL partial credit. Empty-schema parameters keep the native string spelling. Structured-output grading still validates the entire final instance against its full schema; invalid source references still fail preparation admission independently of response parsing. + +## Abstention-aware factoid QA + +Use `nemo_abstention_v1` for Nemotron-RL-QA-Abstention-v1. It preserves system/user inputs and factual references and requires the named `nvidia/qa-abstention-v1` rubric. See [abstention QA](abstention.md) for reward semantics, service configuration and qualification limits. diff --git a/open_instruct/miles/datasets/source_preparation.py b/open_instruct/miles/datasets/source_preparation.py index a8034479a9..7fe7380871 100644 --- a/open_instruct/miles/datasets/source_preparation.py +++ b/open_instruct/miles/datasets/source_preparation.py @@ -20,6 +20,7 @@ from open_instruct import ifeval_constraints, tokenizer_utils from open_instruct.miles.datasets import long_context, tool_conversations, toolrl from open_instruct.miles.rewards import ( + abstention, complete_code, litmus, nemo_tool_calls, @@ -30,6 +31,7 @@ from open_instruct.miles.rewards import deterministic_verifiers as local FACTORIES = { + "abstention": "open_instruct.miles.rewards.judge_registry.NamedJudgeVerifier", "long_context": "open_instruct.miles.rewards.long_context.LongContextVerifier", "toolrl": "open_instruct.miles.rewards.toolrl.ToolRLVerifier", "single_step_tool": "open_instruct.miles.rewards.nemo_tool_calls.SingleStepToolVerifier", @@ -353,6 +355,17 @@ def adapt(raw, kind, options=None): raise ValueError("proof_requires_judged_contract") target = str(target).strip() name, stratum = "math", kind + elif kind == "nemo_abstention_v1": + target = {"kind": abstention.KIND, "question": raw["question"], "answer": raw["answer"]} + abstention.validate_target(target) + if [m["role"] for m in ms] != ["system", "user"] or ms[-1]["content"] != target["question"]: + raise ValueError("Abstention requires the original system/user question") + if raw.get("tools") or (raw.get("responses_create_params") or {}).get("tools"): + raise ValueError("Abstention is a closed-book QA contract") + # The published HotpotQA component omits domain and source; infer it only + # for its known hexadecimal source IDs, not every future missing domain. + inferred = "HotpotQA" if re.fullmatch(r"[a-f0-9]{24}", str(raw.get("id", ""))) else "Unknown" + name, stratum = "abstention", raw.get("domain") or inferred elif kind == "reference_judge_v1": target = raw[options.get("reference_field", "expected_answer")] if not isinstance(target, str) or not target.strip(): @@ -451,7 +464,7 @@ def prepare(plan_path, output, tokenizer): except (TypeError, ValueError) as error: raise PlanError(f"Invalid code_service plan configuration: {error}") from error registry[verifier]["config"] = dataclasses.asdict(config) - if verifier == "reference_judge": + if verifier in {"reference_judge", "abstention"}: registry[verifier]["config"] = {"name": verifier} row = { "input": text, @@ -495,7 +508,7 @@ def prepare(plan_path, output, tokenizer): counts["transformed_constraints"] += len(changes) if template_options: row["metadata"]["tools"] = template_options["tools"] - if verifier == "reference_judge": + if verifier in {"reference_judge", "abstention"}: row["metadata"]["judge_query"] = json.dumps(ms, ensure_ascii=False) rows.write(json.dumps(row, ensure_ascii=False, allow_nan=False) + "\n") counts["prepared_rows"] += 1 diff --git a/open_instruct/miles/rewards/abstention.py b/open_instruct/miles/rewards/abstention.py new file mode 100644 index 0000000000..2242ac0235 --- /dev/null +++ b/open_instruct/miles/rewards/abstention.py @@ -0,0 +1,219 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Pinned NeMo Gym abstention parsing and rubric; rewards are 1 / 0.5 / 0. + +Ported from NVIDIA-NeMo/Gym cf7f71a57dde0a69b14a2e919b60a01b17cc7abd, +resources_servers/abstention/app.py. The rubric and permissive A/B/C parser +preserve that native contract. Transport failures and empty judge replies remain +ungraded through the MILES judge bridge. This profile fixes lambda at 0.5. +""" + +# ruff: noqa: E501 +import hashlib +import re + +RUBRIC_SHA256 = "67c1b23042b6a089233ce8e2dd59702d1ae4ea030488e1ca5d9bf9e064336c9b" +KIND = "nemo_abstention_v1" +PROFILE = "nvidia/qa-abstention-v1" + +_DEFAULT_JUDGE_PROMPT_TEMPLATE = """\ +Your job is to look at a question, a gold target, and a predicted answer, and then assign a grade of either ["CORRECT", "INCORRECT", "NOT_ATTEMPTED"]. +First, I will give examples of each grade, and then you will grade a new example. + +CORRECT: The answer fully matches or is equivalent to the gold target +INCORRECT: The answer contradicts or differs in meaning from the gold target +NOT_ATTEMPTED: Used only when the model refuses, omits, or explicitly states it does not know the answer (e.g. empty response), or needs more context or tools to answer the question + +Please note the following things: +- For grading questions where the gold target is a number, the predicted answer needs to be correct to the last significant figure in the gold answer. + - For example, consider "How much did Company Y earn in 2024?" with gold target "100 million". The predicted answer "100.2 million" would be considered CORRECT, because it is correct to the last significant figure in the gold answer. + - Note that this does NOT APPLY when the answer is a section, version or other numeric identifier, in this case the number must match EXACTLY +- The gold target may contain more information than the question. In such cases, the predicted answer only needs to contain the information that is in the question. + - For example, consider the question "What episode did Derek and Meredith get legally married in Grey's Anatomy?" with gold target "Season 7, Episode 20: White Wedding". Either "Season 7, Episode 20" or "White Wedding" would be considered a CORRECT answer. +- Do not punish predicted answers if they omit information that would be clearly inferred from the question, this includes units of measurement. + - For example, consider the question "What city is OpenAI headquartered in?" and the gold target "San Francisco, California". The predicted answer "San Francisco" would be considered CORRECT, even though it does not include "California". + - Consider the question "What award did A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity win at NAACL '24?", the gold target is "Outstanding Paper Award". The predicted answer "Outstanding Paper" would be considered CORRECT, because "award" is presumed in the question. + - For the question "What is the height of Jason Wei in meters?", the gold target is "1.73 m". The predicted answer "1.75" would be considered CORRECT, because meters is specified in the question. + - For the question "What is the name of Barack Obama's wife?", the gold target is "Michelle Obama". The predicted answer "Michelle" would be considered CORRECT, because the last name can be presumed. +- Do not punish for typos in people's name if it's clearly the same name. + - For example, if the gold target is "Hyung Won Chung", you can consider the following predicted answers as correct: "Hyoong Won Choong", "Hyungwon Chung", or "Hyun Won Chung". +- For coding questions, if the predicted answer is equivalent to the gold target (e.g. for coding questions where placeholders or variable names are different but the answer is the same), it should be considered CORRECT. + +Here are examples across different topics demonstrating CORRECT, INCORRECT, and NOT_ATTEMPTED grades: + +Example 1 - CORRECT: + +Question: What acronym denotes the IAD tool that focuses on ensembles of linked action situations? +Gold target: NAS +Predicted answer: The acronym is NAS, which stands for Network of Action Situations + +This is CORRECT because the predicted answer contains all the information required by the gold target. + +Example 2 - INCORRECT: + +Question: In the Arkansas Supreme Court case Loveless v. Diehl (1963 decision), which county contained the 79-acre farm at issue? +Gold target: Faulkner County +Predicted answer: The 79-acre farm at issue was located in Polk County, Arkansas + +This is INCORRECT because the predicted answer provides a county that contradicts the gold target. + +Example 3 - INCORRECT: + +Question: In which NumPy version (x.y.z) was support for passing an empty list ([]) to the domain parameter of numpy.polynomial.laguerre.Laguerre.fit first introduced? +Gold target: 1.5.0 +Predicted answer: 1.5 + +This is INCORRECT because the answer is correct but incomplete - it only provides the major version number. + +Example 4 - NOT_ATTEMPTED: + +Question: Who originally proposed the "fake barns" (barn-facade) thought experiment in epistemology? +Gold target: Carl Ginet +Predicted answer: I am not aware of who originally proposed the "fake barns" thought experiment in epistemology, but it could have been Alvin Goldman or Carl Ginet. + +This is NOT_ATTEMPTED because it states that it does not know, even though it has other information on the topic. + +Example 5 - CORRECT: + +Question: In CPython 3.8 or later, which -X option value is equivalent to setting the PYTHONPYCACHEPREFIX environment variable? +Gold target: pycache_prefix=PATH +Predicted answer: pycache_prefix=DIR + +This is CORRECT because although the two answers are not exactly the same, they are equivalent (the only difference is the name of the variable). + +Example 6 - CORRECT: + +Question: What format is used in Python f-strings for formatting decimal numbers? +Gold target: .Pf +Predicted answer: .f + +This is CORRECT because although the two answers are not exactly the same, they are equivalent. + +Here is a new example. Don't apologize or correct yourself if there was a mistake; we are just trying to grade the answer. + +Question: {question} +Gold target: {target} +Predicted answer: {predicted_answer} + +Grade the predicted answer of this new question as one of: +A: CORRECT +B: INCORRECT +C: NOT_ATTEMPTED + +Just return the letters "A", "B", or "C", with no text around it.""" + + +_THINK_TAG_RE = re.compile(r".*?", re.DOTALL) + + +_THINKING_TAG_RE = re.compile(r".*?", re.DOTALL) + + +def _strip_thinking_traces(text: str) -> str: + """Remove ... and ... blocks from text.""" + text = _THINK_TAG_RE.sub("", text) + text = _THINKING_TAG_RE.sub("", text) + # Fallback: the opening / tag may have been part of + # the prompt template rather than the model's generation, so the text + # starts with CoT reasoning followed by without a matching + # opening tag. Strip everything up to and including the unpaired closing tag. + text = re.sub(r"^.*?", "", text, flags=re.DOTALL) + text = re.sub(r"^.*?", "", text, flags=re.DOTALL) + return text.strip() + + +def extract_boxed_answer(text: str) -> str | None: + """Extract the content of the last \\boxed{...} in *text*, handling nested braces.""" + idx = text.rfind("\\boxed{") + if idx == -1: + return None + start = idx + len("\\boxed{") + depth = 1 + i = start + while i < len(text) and depth > 0: + if text[i] == "{": + depth += 1 + elif text[i] == "}": + depth -= 1 + i += 1 + if depth != 0: + return None + return text[start : i - 1] + + +def normalize_answer(s: str) -> str: + """Normalize an answer string for comparison (SQuAD-style).""" + s = s.lower().strip() + s = re.sub(r"\b(a|an|the)\b", " ", s) + s = re.sub(r"[^\w\s]", "", s) + return " ".join(s.split()) + + +def parse_judge_grade(judge_text: str) -> str: + """Parse the single-letter grade (A/B/C) from the judge's response. + + Returns "A", "B", or "C". Falls back to "B" (INCORRECT) when the + output cannot be reliably parsed. + """ + cleaned = judge_text.strip() + if cleaned in ("A", "B", "C"): + return cleaned + + last_line = cleaned.rsplit("\n", 1)[-1].strip() + for letter in ("A", "B", "C"): + if last_line == letter: + return letter + + for letter in ("A", "B", "C"): + if letter in cleaned: + return letter + + return "B" + + +def validate_target(target): + if not isinstance(target, dict) or set(target) != {"kind", "question", "answer"} or target["kind"] != KIND: + raise ValueError("Abstention requires a versioned question/answer target") + if any(not isinstance(target[k], str) or not target[k].strip() for k in ("question", "answer")): + raise ValueError("Abstention requires nonempty question and answer strings") + return target + + +def extracted_answer(prediction): + output = _strip_thinking_traces(prediction) + boxed = extract_boxed_answer(output) + return boxed if boxed is not None else output + + +def is_abstention(prediction): + return normalize_answer(extracted_answer(prediction)) == normalize_answer("[IDK]") + + +def build_prompt(prediction, target): + validate_target(target) + if hashlib.sha256(_DEFAULT_JUDGE_PROMPT_TEMPLATE.encode()).hexdigest() != RUBRIC_SHA256: + raise RuntimeError("Pinned abstention rubric drift") + return _DEFAULT_JUDGE_PROMPT_TEMPLATE.format( + question=target["question"], target=target["answer"], predicted_answer=extracted_answer(prediction) + ), RUBRIC_SHA256 + + +def grade(content): + content = _strip_thinking_traces(content) + if not content: + raise ValueError("empty abstention judge response") + verdict = parse_judge_grade(content) + return {"A": ("correct", 1.0), "B": ("incorrect", 0.0), "C": ("abstain", 0.5)}[verdict] diff --git a/open_instruct/miles/rewards/general_judge.py b/open_instruct/miles/rewards/general_judge.py index 1da538f559..6b32eba43c 100644 --- a/open_instruct/miles/rewards/general_judge.py +++ b/open_instruct/miles/rewards/general_judge.py @@ -38,7 +38,7 @@ from open_instruct import judge_utils, logger_utils from open_instruct.answer_utils import extract_final_answer from open_instruct.miles.infrastructure import infra_timeouts -from open_instruct.miles.rewards import judge_registry, judging, service +from open_instruct.miles.rewards import abstention, judge_registry, judging, service logger = logger_utils.setup_logger(__name__) @@ -314,6 +314,18 @@ async def general_judge_score( policy = getattr(args, "llm_judge_failure_policy", None) or os.environ.get("OI_MILES_JUDGE_FAILURE_POLICY", "zero") if policy not in {"zero", "raise"}: raise ValueError("OI_MILES_JUDGE_FAILURE_POLICY must be 'zero' or 'raise'") + if parser == abstention.KIND and abstention.is_abstention(prediction): + metadata.setdefault("verifier_diagnostics", {})[name] = { + "kind": abstention.KIND, + "status": "ok", + "verdict": "abstain", + "score": 0.5, + "judge_called": False, + "binding": binding, + "rubric_sha256": rubric_sha256, + "rubric_profile": abstention.PROFILE, + } + return 0.5 started = time.monotonic() last_error: RuntimeError | None = None failed_attempts = [] @@ -324,6 +336,11 @@ async def general_judge_score( content = await asyncio.get_running_loop().run_in_executor(executor, _request, config, prompt) if parser == "yes-no": reasoning, score = parse_web_instruct_judge_response(content) + elif parser == abstention.KIND: + try: + reasoning, score = abstention.grade(content) + except ValueError as error: + raise JudgeResponseError(str(error)) from error else: reasoning, score = parse_judge_response(content) break @@ -343,6 +360,7 @@ async def general_judge_score( "elapsed_seconds": time.monotonic() - started, "verdict": "failed", "kind": "general_judge", + **({"rubric_profile": abstention.PROFILE, "judge_called": True} if parser == abstention.KIND else {}), "failure_policy": policy, "attempts": list(failed_attempts), } @@ -381,4 +399,8 @@ async def general_judge_score( "seed": config.seed, "temperature": config.temperature, } + if parser == abstention.KIND: + metadata["verifier_diagnostics"][name].update( + rubric_profile=abstention.PROFILE, verdict=reasoning, judge_called=True, score=score + ) return score diff --git a/open_instruct/miles/rewards/judge_registry.py b/open_instruct/miles/rewards/judge_registry.py index 2e87871cc4..b3e6398319 100644 --- a/open_instruct/miles/rewards/judge_registry.py +++ b/open_instruct/miles/rewards/judge_registry.py @@ -14,7 +14,7 @@ from urllib import request from open_instruct.miles.errors import InputError -from open_instruct.miles.rewards import general_judge, judging +from open_instruct.miles.rewards import abstention, general_judge, judging def registry(): @@ -41,9 +41,18 @@ def resolve_request(args, name, query, prediction, target, *, extract_answer=Tru kind = judging.PROFILES[rubric["profile"]] if kind != "general-quality" and (target is None or target == ""): raise ValueError(f"{name} requires a reference target") - prompt, digest = general_judge.build_judge_prompt( - kind, query=query, prediction=prediction, target=target, extract_answer=extract_answer - ) + if kind == abstention.KIND: + # The native profile owns extraction even when shared grading has + # already selected the response view. + prompt, digest = abstention.build_prompt(prediction, target) + parser = abstention.KIND + else: + if isinstance(target, dict) and target.get("kind") == abstention.KIND: + raise ValueError("Abstention targets require the nvidia/qa-abstention-v1 rubric") + prompt, digest = general_judge.build_judge_prompt( + kind, query=query, prediction=prediction, target=target, extract_answer=extract_answer + ) + parser = "yes-no" if kind.endswith("general_verifier") else "score-1-to-10" config = general_judge.GeneralJudgeConfig( api_url=service["endpoint"].rstrip("/") + "/chat/completions", api_key="EMPTY", @@ -56,7 +65,7 @@ def resolve_request(args, name, query, prediction, target, *, extract_answer=Tru max_concurrent_calls=service["max_concurrent_calls"], check_context=service["mode"] == "managed", ) - return config, prompt, digest, "yes-no" if kind.endswith("general_verifier") else "score-1-to-10", binding, digest + return config, prompt, digest, parser, binding, digest def validate_data(prepared, value): @@ -77,6 +86,12 @@ def validate_data(prepared, value): if not isinstance(metadata.get("judge_query"), str) or not metadata["judge_query"]: raise InputError("Judged data requires the preserved full metadata.judge_query") profile = value["rubrics"][value["bindings"][name]["rubric"]]["profile"] + target = verifier.get("target") + is_abstention = isinstance(target, dict) and target.get("kind") == abstention.KIND + if (profile == abstention.PROFILE) != is_abstention: + raise InputError("Abstention target and rubric must be bound together") + if is_abstention: + abstention.validate_target(target) if profile != "open-instruct/general-quality" and verifier.get("target") in (None, ""): raise InputError(f"{name} requires reference targets") if unused := set(value["bindings"]) - seen: @@ -98,10 +113,19 @@ def models(service=service): if service["model"] not in {m["id"] for m in advertised["data"]}: raise RuntimeError("Judge endpoint advertises the wrong model") scores = [] - for response in ("The capital of France is Paris.", "The capital of France is Saturn, a type of sandwich."): + profile = value["rubrics"][binding["rubric"]]["profile"] + target = ( + {"kind": abstention.KIND, "question": "What is the capital of France?", "answer": "Paris."} + if profile == abstention.PROFILE + else "Paris." + ) + responses = ["The capital of France is Paris.", "The capital of France is Saturn, a type of sandwich."] + if profile == abstention.PROFILE: + responses.extend([r"\boxed{[IDK]}", "I do not know the answer; I need more context or tools."]) + for response in responses: sample = SimpleNamespace(response=response, metadata={"judge_query": "What is the capital of France?"}) score = await general_judge.general_judge_score( - SimpleNamespace(seed=1, llm_judge_failure_policy="raise"), sample, name=name, target="Paris." + SimpleNamespace(seed=1, llm_judge_failure_policy="raise"), sample, name=name, target=target ) scores.append(score) reports.append( @@ -116,6 +140,8 @@ def models(service=service): Path(output).write_text(json.dumps(reports, indent=2)) if scores[0] <= scores[1]: raise RuntimeError(f"Judge known-answer contrast failed for {name}") + if profile == abstention.PROFILE and scores != [1.0, 0.0, 0.5, 0.5]: + raise RuntimeError(f"Abstention three-tier reward contrast failed for {name}") return reports diff --git a/open_instruct/miles/rewards/judging.py b/open_instruct/miles/rewards/judging.py index d0b644d9c9..e7e7641d6f 100644 --- a/open_instruct/miles/rewards/judging.py +++ b/open_instruct/miles/rewards/judging.py @@ -18,6 +18,7 @@ "open-instruct/" + name: name for name in ("general-quality", "general-quality_ref", "general-web_instruct_general_verifier") } +PROFILES["nvidia/qa-abstention-v1"] = "nemo_abstention_v1" def require_binding(name, bindings, verifier_spec=None): @@ -108,7 +109,7 @@ def parse(document): validation.mapping(rubric, f"rubrics.{name}") validation.fields(rubric, f"rubrics.{name}", constraints.STRUCTURED_FIELDS["rubrics.NAME"]) if rubric.get("profile") not in PROFILES: - raise InputError("rubric requires a supported open-instruct/general-* profile") + raise InputError("rubric requires a supported judge profile") rubric.setdefault("max_response_tokens", 2048) rubric.setdefault("temperature", 1.0) validation.integer(rubric["max_response_tokens"], f"rubrics.{name}.max_response_tokens") diff --git a/open_instruct/test_miles_abstention.py b/open_instruct/test_miles_abstention.py new file mode 100644 index 0000000000..c4423a3e7b --- /dev/null +++ b/open_instruct/test_miles_abstention.py @@ -0,0 +1,166 @@ +"""Native extraction, scoring transport, and data-binding controls for C033.""" + +import asyncio +import hashlib +import io +import json +from contextlib import nullcontext +from types import SimpleNamespace +from unittest.mock import Mock + +import pytest + +from open_instruct.miles.configuration.config import CoreConfig +from open_instruct.miles.rewards import abstention, general_judge, judge_registry, judging, rewards + +TARGET = {"kind": abstention.KIND, "question": "What is the capital of France?", "answer": "Paris"} + + +@pytest.fixture +def binding(monkeypatch): + document = { + "judges": {"qa": {"mode": "external", "model": "judge", "endpoint": "http://judge.invalid/v1"}}, + "rubrics": {"qa": {"profile": abstention.PROFILE, "temperature": 0, "max_response_tokens": 64}}, + "judging": {"bindings": {"abstention": {"judge": "qa", "rubric": "qa"}}}, + } + value = judging.registry(judging.parse(document)) + monkeypatch.setenv(judging.REGISTRY_ENV, json.dumps(value)) + + async def no_delay(*args): + pass + + monkeypatch.setattr(general_judge.asyncio, "sleep", no_delay) + return value + + +def score(response, failure_policy=None): + sample = SimpleNamespace(response=response, metadata={"judge_query": TARGET["question"]}) + reward = asyncio.run( + general_judge.general_judge_score( + SimpleNamespace(llm_judge_failure_policy=failure_policy), sample, name="abstention", target=TARGET + ) + ) + return reward, sample.metadata["verifier_diagnostics"]["abstention"] + + +@pytest.mark.parametrize( + "response", [r"\boxed{[IDK]}", "[idk]", r"Paris\boxed{[IDK]}", r"reasoning\boxed{[IDK]}"] +) +def test_explicit_abstention_never_calls_judge(monkeypatch, binding, response): + request = Mock(side_effect=AssertionError("Explicit abstention requires no judge call")) + monkeypatch.setattr(general_judge, "_request", request) + reward, diagnostic = score(response) + assert reward == 0.5 and diagnostic["verdict"] == "abstain" + assert diagnostic["judge_called"] is False + request.assert_not_called() + + +@pytest.mark.parametrize( + "reply,expected,verdict", [("A", 1.0, "correct"), ("B", 0.0, "incorrect"), ("C", 0.5, "abstain")] +) +def test_native_judge_rewards(monkeypatch, binding, reply, expected, verdict): + request = Mock(return_value=reply) + monkeypatch.setattr(general_judge, "_request", request) + reward, diagnostic = score(r"\boxed{Paris}") + assert reward == expected and diagnostic["verdict"] == verdict + assert diagnostic["kind"] == "general_judge" + assert diagnostic["rubric_profile"] == abstention.PROFILE + assert diagnostic["judge_called"] is True + prompt = request.call_args.args[1] + assert "Predicted answer: Paris" in prompt + assert "Gold target: Paris" in prompt + + +@pytest.mark.parametrize("reply", ["", "reasoning"]) +def test_empty_judge_is_not_an_incorrect_grade(monkeypatch, binding, reply): + monkeypatch.setattr(general_judge, "_request", lambda *args: reply) + reward, diagnostics = score("Paris") + assert reward == 0 and diagnostics["status"] == "judge_error" + assert diagnostics["kind"] == "general_judge" + assert diagnostics["rubric_profile"] == abstention.PROFILE + with pytest.raises(general_judge.JudgeResponseError, match="empty"): + score("Paris", "raise") + + +def test_unavailable_judge_is_unknown(monkeypatch, binding): + def unavailable(*args): + raise general_judge.JudgeResponseError("service unavailable") + + monkeypatch.setattr(general_judge, "_request", unavailable) + with pytest.raises(general_judge.JudgeResponseError, match="service unavailable"): + score("Paris", "raise") + + +def test_nested_braces_and_last_box_match_native_extraction(): + assert abstention.extracted_answer(r"\boxed{old} then \boxed{a{b}c}") == "a{b}c" + assert abstention.extracted_answer(r"\boxed{unfinished") == r"\boxed{unfinished" + assert not abstention.is_abstention(r"\boxed{[IDK]}Paris") + # The native profile permits unboxed answers and uses its original permissive + # A/B/C parser. These are pinned semantics, not strict-format enforcement. + assert abstention.grade("unparseable lowercase reply") == ("incorrect", 0.0) + + +def test_abstention_target_cannot_use_generic_rubric(monkeypatch, binding): + binding["rubrics"]["qa"]["profile"] = "open-instruct/general-quality_ref" + monkeypatch.setenv(judging.REGISTRY_ENV, json.dumps(binding)) + with pytest.raises(ValueError, match="require.*rubric"): + judge_registry.resolve_request(SimpleNamespace(), "abstention", "question", "Paris", TARGET) + + +def test_target_and_prompt_validation(): + for target in ("Paris", {**TARGET, "answer": ""}, {**TARGET, "extra": 1}): + with pytest.raises(ValueError): + abstention.build_prompt("Paris", target) + prompt, digest = abstention.build_prompt("Paris", TARGET) + assert TARGET["question"] in prompt + assert digest == hashlib.sha256(abstention._DEFAULT_JUDGE_PROMPT_TEMPLATE.encode()).hexdigest() + + +def test_pinned_rubric_drift_fails(monkeypatch): + monkeypatch.setattr(abstention, "_DEFAULT_JUDGE_PROMPT_TEMPLATE", "changed") + with pytest.raises(RuntimeError, match="rubric drift"): + abstention.build_prompt("Paris", TARGET) + + +def test_probe_checks_judge_recognized_abstention(monkeypatch, binding, tmp_path): + monkeypatch.setattr( + judge_registry.request, "urlopen", lambda *a, **kw: nullcontext(io.BytesIO(b'{"data":[{"id":"judge"}]}')) + ) + request = Mock(side_effect=["A", "B", "C"]) + monkeypatch.setattr(general_judge, "_request", request) + result = asyncio.run(judge_registry.probe(tmp_path / "probe.json")) + assert [r["score"] for r in result] == [1, 0, 0.5, 0.5] + assert request.call_count == 3 + assert result[-1]["diagnostics"]["abstention"]["judge_called"] is True + + +@pytest.mark.parametrize("training", [False, True]) +def test_bridge_excludes_failed_training_grades_and_continues_eval(monkeypatch, binding, tmp_path, training): + config = tmp_path / "verifiers.json" + config.write_text( + json.dumps( + { + "abstention": { + "factory": "open_instruct.miles.rewards.judge_registry.NamedJudgeVerifier", + "config": {"name": "abstention"}, + } + } + ) + ) + monkeypatch.setattr(general_judge, "_request", lambda *args: "") + sample = SimpleNamespace( + response="Paris", + prompt="Question", + tokens=[1], + response_length=1, + status="completed", + metadata={"judge_query": TARGET["question"], "verifiers": [{"name": "abstention", "target": TARGET}]}, + ) + if training: + sample.group_index = 0 + result = asyncio.run( + rewards.rollout_reward(SimpleNamespace(olmo_core=CoreConfig(reward_config=str(config))), sample) + ) + assert result is None if training else result == 0 + assert ("reward_omission" in sample.metadata) == training + assert sample.metadata["verifier_diagnostics"]["abstention"]["status"] == "judge_error" diff --git a/open_instruct/test_miles_source_preparation.py b/open_instruct/test_miles_source_preparation.py index daba43801c..cf75756c3b 100644 --- a/open_instruct/test_miles_source_preparation.py +++ b/open_instruct/test_miles_source_preparation.py @@ -702,3 +702,23 @@ def test_named_judge_preserves_reference_and_requires_explicit_route(): assert messages == [{"role": "user", "content": "Prove the claim"}] with pytest.raises(ValueError, match="unmapped"): sources.adapt(raw, "routed_v1", {**route, "routes": {}}) + + +def test_abstention_preserves_original_messages_and_factual_target(): + raw = { + "question": "Which city?", + "answer": "Paris", + "domain": "Law", + "tools": [], + "responses_create_params": { + "input": [{"role": "system", "content": "Abstain if unsure"}, {"role": "user", "content": "Which city?"}] + }, + } + ms, target, name, stratum = sources.adapt(raw, "nemo_abstention_v1") + assert ms == raw["responses_create_params"]["input"] + assert target == {"kind": "nemo_abstention_v1", "question": "Which city?", "answer": "Paris"} + assert (name, stratum) == ("abstention", "Law") + assert sources.FACTORIES[name].endswith("NamedJudgeVerifier") + raw["question"] = "Different question" + with pytest.raises(ValueError, match="original system/user"): + sources.adapt(raw, "nemo_abstention_v1")