Skip to content

fix(eval): reject ambiguous task IDs before comparing results - #3609

Open
linyuchao123 wants to merge 1 commit into
ag2ai:mainfrom
linyuchao123:fix/eval-unique-task-ids
Open

linyuchao123 wants to merge 1 commit into
ag2ai:mainfrom
linyuchao123:fix/eval-unique-task-ids

Conversation

@linyuchao123

Copy link
Copy Markdown

Why are these changes needed?

Two tasks with the same ID can silently hide a regression. With baseline scores [True, True] and current scores [False, True], the overall pass rate drops to 50%, but RunResult.diff() maps tasks by ID and keeps only the last task, reporting 100% to 100% with no regressions.

Validate non-empty string IDs and uniqueness in Suite and RunResult. This also rejects collisions between explicit IDs and generated IDs, and ambiguous historical JSON runs loaded through load_run. Missing/null IDs still receive the existing sequential defaults; explicit IDs are never silently renamed. Errors identify the source, duplicated ID, and task indices.

Validation:

  • New regression tests against the original implementation: 12 failed, 48 passed.
  • With the fix: python -m pytest test/eval -q --disable-warnings: 332 passed, 6 skipped (optional dependencies).
  • Ruff check/format and git diff --check passed for the changed files.
  • No real model calls were made.

Related issue number

No directly corresponding open issue found. This is separate from #3401, which addresses unpaired scorer denominators rather than duplicate task identity.

Checks

  • I've included any doc changes needed for https://docs.ag2.ai/.
  • I've added tests (if relevant) corresponding to the changes introduced in this PR.
  • I've made sure all auto checks have passed.

AI assistance

AI assistance was used for investigation, implementation, regression tests, and this description. Local validation and comparison with the original implementation are recorded above. The human-ownership attestations below are left for the contributor to complete after personal review.

  • I understand the changes in this PR and can explain them in my own words.
  • I have verified that the PR description accurately reflects the actual diff.
  • If AI assistance was used, I reviewed, tested, and validated the generated code/text before submitting.

@CLAassistant

CLAassistant commented Nov 7, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@github-actions github-actions Bot added the area:eval Offline evaluation framework (ag2/eval) label Nov 7, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:eval Offline evaluation framework (ag2/eval)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants