Support native Nemotron abstention QA rewards in MILES

CHANGELOG.md

@@ -7,6 +7,7 @@ All notable changes to this project will be documented in this file.
 - Remove the dataset readiness scripts (`scripts/data/rl_readiness/`), the standalone IF-target filter and the first-pass data handoff page; the RL Data Workbench now owns qualification, screening and release evidence, while Open Instruct keeps source adapters, catalog resolution, verifiers and the training loader (https://github.com/allenai/open-instruct/pull/2220).
 
 ### Changed
+- Add the native Nemotron abstention QA adapter and named three-tier judge profile, preserving NVIDIA's rubric and answer extraction while keeping judge failures ungraded (https://github.com/allenai/open-instruct/pull/2252).
 - Honor native MILES num_epoch budgets without injecting a conflicting rollout default; save epoch-run final checkpoints and recognize the runtime-resolved completion limit (https://github.com/allenai/open-instruct/pull/2251).
 - Enable final-answer grading in the three maintained MILES examples while preserving the library default. Keep exclusion tables aligned with grading-aware record keys, reject cross-contract evidence, and validate native reward contracts before worker startup (https://github.com/allenai/open-instruct/pull/2250).
 - Give MILES final-answer grading shared thinking/response views, explicit per-verifier input and eligibility policies, non-thinking and prompt-prefilled-closure support, per-component selection provenance and distinct evidence identities. Preserve default-off scoring; document every path in `docs/miles/grading.md` with validation evidence in the PR (https://github.com/allenai/open-instruct/pull/2250).

docs/miles/abstention.md

@@ -0,0 +1,28 @@
+# Nemotron abstention QA
+
+The Nemotron-RL-QA-Abstention-v1 dataset supplies factoid questions and factual references while asking the policy to abstain when uncertain. The `nemo_abstention_v1` source adapter preserves its original system/user messages and emits a structured question/reference target. Training needs a named correctness judge; plain answer matching does not reproduce the abstention objective.
+
+The `nvidia/qa-abstention-v1` rubric ports NVIDIA-NeMo/Gym's abstention environment at `cf7f71a57dde0a69b14a2e919b60a01b17cc7abd`. It strips thinking traces, extracts the last balanced boxed answer (including nested braces), and falls back to the remaining output when no complete box exists. Normalized `[IDK]` receives 0.5 without contacting the judge. Other outputs use the original OMNISCIENCE_GRADER prompt and A/B/C parser: correct receives 1, incorrect 0, and NOT_ATTEMPTED 0.5. This version fixes the abstention reward at NVIDIA's default 0.5; it does not implement dynamic calibration.
+
+Reuse the existing `judges.general` Qwen/Qwen3-32B service from the maintained three-node example (model revision `9216db5781bf21249d130ec9da846c4624c16137`). Add a rubric and binding to that same service; no additional judge GPU or model load is needed:
+
+```toml
+[rubrics.abstention]
+profile = "nvidia/qa-abstention-v1"
+max_response_tokens = 64
+temperature = 0.0
+
+[judging.bindings.abstention]
+judge = "general"
+rubric = "abstention"
+```
+
+Replace the verifier binding name with the actual name in the exported `verifiers.json`; Workbench namespaces names by their contract. Keep the profile paired with its structured `nemo_abstention_v1` target. Preparation emits a `NamedJudgeVerifier` registry entry and preserves the full input as `metadata.judge_query`.
+
+The named-service probe requires correct/incorrect/explicit-abstention/implicit-abstention controls to score 1/0/0.5/0.5. Empty replies, transport errors and incomplete judge generations follow the existing failure policy: strict mode raises; zero mode records `status=judge_error` and a fallback zero, which the training bridge excludes as ungraded. Evaluation can continue with those explicit failure diagnostics. Diagnostics distinguish an explicit abstention from a judge-produced NOT_ATTEMPTED grade. Existing truncation/final-answer gates apply before judging; a capped response does not earn abstention credit.
+
+This is the native permissive parsing contract, not strict boxed-format enforcement. In particular, nonempty judge output uses NVIDIA's substring fallback for A/B/C and otherwise defaults to incorrect. Preserve that fact in calibration; tightening the parser would be a separate contract change. The rubric also contains permissive numerical and name-equivalence examples. Test them with the selected judge before relying on training rewards.
+
+A source-preparation or fixture test does not calibrate an actual judge or establish learning benefit. Qualify the intended model and endpoint with independently reviewed correct, wrong, equivalent, incomplete and abstaining responses across all four domains. The publisher provides no validation split. Preserve the card/row license discrepancy (CC BY 4.0 versus CC BY-SA 4.0), and run the intended release's protected-source hygiene before redistribution or training admission.
+
+Upstream implementation: <https://github.com/NVIDIA-NeMo/Gym/blob/cf7f71a57dde0a69b14a2e919b60a01b17cc7abd/resources_servers/abstention/app.py>.

docs/miles/source-preparation.md

@@ -572,3 +572,7 @@ comparator, preserving ToolRL partial credit. Empty-schema parameters keep the
 native string spelling. Structured-output grading still validates the entire
 final instance against its full schema; invalid source references still fail
 preparation admission independently of response parsing.
+
+## Abstention-aware factoid QA
+
+Use `nemo_abstention_v1` for Nemotron-RL-QA-Abstention-v1. It preserves system/user inputs and factual references and requires the named `nvidia/qa-abstention-v1` rubric. See [abstention QA](abstention.md) for reward semantics, service configuration and qualification limits.

open_instruct/miles/datasets/source_preparation.py

@@ -20,6 +20,7 @@
 from open_instruct import ifeval_constraints, tokenizer_utils
 from open_instruct.miles.datasets import long_context, tool_conversations, toolrl
 from open_instruct.miles.rewards import (
+    abstention,
     complete_code,
     litmus,
     nemo_tool_calls,
@@ -30,6 +31,7 @@
 from open_instruct.miles.rewards import deterministic_verifiers as local
 
 FACTORIES = {
+    "abstention": "open_instruct.miles.rewards.judge_registry.NamedJudgeVerifier",
     "long_context": "open_instruct.miles.rewards.long_context.LongContextVerifier",
     "toolrl": "open_instruct.miles.rewards.toolrl.ToolRLVerifier",
     "single_step_tool": "open_instruct.miles.rewards.nemo_tool_calls.SingleStepToolVerifier",
@@ -353,6 +355,17 @@ def adapt(raw, kind, options=None):
             raise ValueError("proof_requires_judged_contract")
         target = str(target).strip()
         name, stratum = "math", kind
+    elif kind == "nemo_abstention_v1":
+        target = {"kind": abstention.KIND, "question": raw["question"], "answer": raw["answer"]}
+        abstention.validate_target(target)
+        if [m["role"] for m in ms] != ["system", "user"] or ms[-1]["content"] != target["question"]:
+            raise ValueError("Abstention requires the original system/user question")
+        if raw.get("tools") or (raw.get("responses_create_params") or {}).get("tools"):
+            raise ValueError("Abstention is a closed-book QA contract")
+        # The published HotpotQA component omits domain and source; infer it only
+        # for its known hexadecimal source IDs, not every future missing domain.
+        inferred = "HotpotQA" if re.fullmatch(r"[a-f0-9]{24}", str(raw.get("id", ""))) else "Unknown"
+        name, stratum = "abstention", raw.get("domain") or inferred
     elif kind == "reference_judge_v1":
         target = raw[options.get("reference_field", "expected_answer")]
         if not isinstance(target, str) or not target.strip():
@@ -451,7 +464,7 @@ def prepare(plan_path, output, tokenizer):
                         except (TypeError, ValueError) as error:
                             raise PlanError(f"Invalid code_service plan configuration: {error}") from error
                         registry[verifier]["config"] = dataclasses.asdict(config)
-                    if verifier == "reference_judge":
+                    if verifier in {"reference_judge", "abstention"}:
                         registry[verifier]["config"] = {"name": verifier}
                     row = {
                         "input": text,
@@ -495,7 +508,7 @@ def prepare(plan_path, output, tokenizer):
                         counts["transformed_constraints"] += len(changes)
                     if template_options:
                         row["metadata"]["tools"] = template_options["tools"]
-                    if verifier == "reference_judge":
+                    if verifier in {"reference_judge", "abstention"}:
                         row["metadata"]["judge_query"] = json.dumps(ms, ensure_ascii=False)
                     rows.write(json.dumps(row, ensure_ascii=False, allow_nan=False) + "\n")
                     counts["prepared_rows"] += 1

open_instruct/miles/rewards/abstention.py

@@ -0,0 +1,219 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Pinned NeMo Gym abstention parsing and rubric; rewards are 1 / 0.5 / 0.
+
+Ported from NVIDIA-NeMo/Gym cf7f71a57dde0a69b14a2e919b60a01b17cc7abd,
+resources_servers/abstention/app.py. The rubric and permissive A/B/C parser
+preserve that native contract. Transport failures and empty judge replies remain
+ungraded through the MILES judge bridge. This profile fixes lambda at 0.5.
+"""
+
+# ruff: noqa: E501
+import hashlib
+import re
+
+RUBRIC_SHA256 = "67c1b23042b6a089233ce8e2dd59702d1ae4ea030488e1ca5d9bf9e064336c9b"
+KIND = "nemo_abstention_v1"
+PROFILE = "nvidia/qa-abstention-v1"
+
+_DEFAULT_JUDGE_PROMPT_TEMPLATE = """\
+Your job is to look at a question, a gold target, and a predicted answer, and then assign a grade of either ["CORRECT", "INCORRECT", "NOT_ATTEMPTED"].
+First, I will give examples of each grade, and then you will grade a new example.
+
+CORRECT: The answer fully matches or is equivalent to the gold target
+INCORRECT: The answer contradicts or differs in meaning from the gold target
+NOT_ATTEMPTED: Used only when the model refuses, omits, or explicitly states it does not know the answer (e.g. empty response), or needs more context or tools to answer the question
+
+Please note the following things:
+- For grading questions where the gold target is a number, the predicted answer needs to be correct to the last significant figure in the gold answer.
+    - For example, consider "How much did Company Y earn in 2024?" with gold target "100 million". The predicted answer "100.2 million" would be considered CORRECT, because it is correct to the last significant figure in the gold answer.
+    - Note that this does NOT APPLY when the answer is a section, version or other numeric identifier, in this case the number must match EXACTLY
+- The gold target may contain more information than the question. In such cases, the predicted answer only needs to contain the information that is in the question.
+    - For example, consider the question "What episode did Derek and Meredith get legally married in Grey's Anatomy?" with gold target "Season 7, Episode 20: White Wedding". Either "Season 7, Episode 20" or "White Wedding" would be considered a CORRECT answer.
+- Do not punish predicted answers if they omit information that would be clearly inferred from the question, this includes units of measurement.
+    - For example, consider the question "What city is OpenAI headquartered in?" and the gold target "San Francisco, California". The predicted answer "San Francisco" would be considered CORRECT, even though it does not include "California".
+    - Consider the question "What award did A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity win at NAACL '24?", the gold target is "Outstanding Paper Award". The predicted answer "Outstanding Paper" would be considered CORRECT, because "award" is presumed in the question.
+    - For the question "What is the height of Jason Wei in meters?", the gold target is "1.73 m". The predicted answer "1.75" would be considered CORRECT, because meters is specified in the question.
+    - For the question "What is the name of Barack Obama's wife?", the gold target is "Michelle Obama". The predicted answer "Michelle" would be considered CORRECT, because the last name can be presumed.
+- Do not punish for typos in people's name if it's clearly the same name.
+    - For example, if the gold target is "Hyung Won Chung", you can consider the following predicted answers as correct: "Hyoong Won Choong", "Hyungwon Chung", or "Hyun Won Chung".
+- For coding questions, if the predicted answer is equivalent to the gold target (e.g. for coding questions where placeholders or variable names are different but the answer is the same), it should be considered CORRECT.
+
+Here are examples across different topics demonstrating CORRECT, INCORRECT, and NOT_ATTEMPTED grades:
+
+Example 1 - CORRECT:
+
+Question: What acronym denotes the IAD tool that focuses on ensembles of linked action situations?
+Gold target: NAS
+Predicted answer: The acronym is NAS, which stands for Network of Action Situations
+
+This is CORRECT because the predicted answer contains all the information required by the gold target.
+
+Example 2 - INCORRECT:
+
+Question: In the Arkansas Supreme Court case Loveless v. Diehl (1963 decision), which county contained the 79-acre farm at issue?
+Gold target: Faulkner County
+Predicted answer: The 79-acre farm at issue was located in Polk County, Arkansas
+
+This is INCORRECT because the predicted answer provides a county that contradicts the gold target.
+
+Example 3 - INCORRECT:
+
+Question: In which NumPy version (x.y.z) was support for passing an empty list ([]) to the domain parameter of numpy.polynomial.laguerre.Laguerre.fit first introduced?
+Gold target: 1.5.0
+Predicted answer: 1.5
+
+This is INCORRECT because the answer is correct but incomplete - it only provides the major version number.
+
+Example 4 - NOT_ATTEMPTED:
+
+Question: Who originally proposed the "fake barns" (barn-facade) thought experiment in epistemology?
+Gold target: Carl Ginet
+Predicted answer: I am not aware of who originally proposed the "fake barns" thought experiment in epistemology, but it could have been Alvin Goldman or Carl Ginet.
+
+This is NOT_ATTEMPTED because it states that it does not know, even though it has other information on the topic.
+
+Example 5 - CORRECT:
+
+Question: In CPython 3.8 or later, which -X option value is equivalent to setting the PYTHONPYCACHEPREFIX environment variable?
+Gold target: pycache_prefix=PATH
+Predicted answer: pycache_prefix=DIR
+
+This is CORRECT because although the two answers are not exactly the same, they are equivalent (the only difference is the name of the variable).
+
+Example 6 - CORRECT:
+
+Question: What format is used in Python f-strings for formatting decimal numbers?
+Gold target: .Pf
+Predicted answer: .<precision>f
+
+This is CORRECT because although the two answers are not exactly the same, they are equivalent.
+
+Here is a new example. Don't apologize or correct yourself if there was a mistake; we are just trying to grade the answer.
+
+Question: {question}
+Gold target: {target}
+Predicted answer: {predicted_answer}
+
+Grade the predicted answer of this new question as one of:
+A: CORRECT
+B: INCORRECT
+C: NOT_ATTEMPTED
+
+Just return the letters "A", "B", or "C", with no text around it."""
+
+
+_THINK_TAG_RE = re.compile(r"<think>.*?</think>", re.DOTALL)
+
+
+_THINKING_TAG_RE = re.compile(r"<thinking>.*?</thinking>", re.DOTALL)
+
+
+def _strip_thinking_traces(text: str) -> str:
+    """Remove <think>...</think> and <thinking>...</thinking> blocks from text."""
+    text = _THINK_TAG_RE.sub("", text)
+    text = _THINKING_TAG_RE.sub("", text)
+    # Fallback: the opening <think>/<thinking> tag may have been part of
+    # the prompt template rather than the model's generation, so the text
+    # starts with CoT reasoning followed by </think> without a matching
+    # opening tag. Strip everything up to and including the unpaired closing tag.
+    text = re.sub(r"^.*?</think>", "", text, flags=re.DOTALL)
+    text = re.sub(r"^.*?</thinking>", "", text, flags=re.DOTALL)
+    return text.strip()
+
+
+def extract_boxed_answer(text: str) -> str | None:
+    """Extract the content of the last \\boxed{...} in *text*, handling nested braces."""
+    idx = text.rfind("\\boxed{")
+    if idx == -1:
+        return None
+    start = idx + len("\\boxed{")
+    depth = 1
+    i = start
+    while i < len(text) and depth > 0:
+        if text[i] == "{":
+            depth += 1
+        elif text[i] == "}":
+            depth -= 1
+        i += 1
+    if depth != 0:
+        return None
+    return text[start : i - 1]
+
+
+def normalize_answer(s: str) -> str:
+    """Normalize an answer string for comparison (SQuAD-style)."""
+    s = s.lower().strip()
+    s = re.sub(r"\b(a|an|the)\b", " ", s)
+    s = re.sub(r"[^\w\s]", "", s)
+    return " ".join(s.split())
+
+
+def parse_judge_grade(judge_text: str) -> str:
+    """Parse the single-letter grade (A/B/C) from the judge's response.
+
+    Returns "A", "B", or "C". Falls back to "B" (INCORRECT) when the
+    output cannot be reliably parsed.
+    """
+    cleaned = judge_text.strip()
+    if cleaned in ("A", "B", "C"):
+        return cleaned
+
+    last_line = cleaned.rsplit("\n", 1)[-1].strip()
+    for letter in ("A", "B", "C"):
+        if last_line == letter:
+            return letter
+
+    for letter in ("A", "B", "C"):
+        if letter in cleaned:
+            return letter
+
+    return "B"
+
+
+def validate_target(target):
+    if not isinstance(target, dict) or set(target) != {"kind", "question", "answer"} or target["kind"] != KIND:
+        raise ValueError("Abstention requires a versioned question/answer target")
+    if any(not isinstance(target[k], str) or not target[k].strip() for k in ("question", "answer")):
+        raise ValueError("Abstention requires nonempty question and answer strings")
+    return target
+
+
+def extracted_answer(prediction):
+    output = _strip_thinking_traces(prediction)
+    boxed = extract_boxed_answer(output)
+    return boxed if boxed is not None else output
+
+
+def is_abstention(prediction):
+    return normalize_answer(extracted_answer(prediction)) == normalize_answer("[IDK]")
+
+
+def build_prompt(prediction, target):
+    validate_target(target)
+    if hashlib.sha256(_DEFAULT_JUDGE_PROMPT_TEMPLATE.encode()).hexdigest() != RUBRIC_SHA256:
+        raise RuntimeError("Pinned abstention rubric drift")
+    return _DEFAULT_JUDGE_PROMPT_TEMPLATE.format(
+        question=target["question"], target=target["answer"], predicted_answer=extracted_answer(prediction)
+    ), RUBRIC_SHA256
+
+
+def grade(content):
+    content = _strip_thinking_traces(content)
+    if not content:
+        raise ValueError("empty abstention judge response")
+    verdict = parse_judge_grade(content)
+    return {"A": ("correct", 1.0), "B": ("incorrect", 0.0), "C": ("abstain", 0.5)}[verdict]