Evidence recall: a 12 s deadline on the packs, and say when one was left out

[sanil-23]

Summary

Evidence-mode recall (#276) builds one pack per kind scope and waited for all of them. A host bounds a memory tool call: OpenHuman kills it at 15 s. One scope's pack can take that long on its own, so the slowest scope decided whether the agent got any evidence at all. Two changes:

1. A 12 s deadline on the packs. Packs still building 12 s after the recall starts are left out with a warning. The evidence is rendered from the packs that arrived, still most specific first. A pack that fails still fails the recall.
2. A recall that left a pack out says so. If the packs that arrived hold nothing, the recall is Unavailable (so is one where no pack arrived). It is never an empty answer, which a host reads as "nothing stored": OpenHuman tells its agent "Nothing relevant is stored in memory". If something arrived, the evidence ends with a note that memory was only partly searched.

Measured in openhuman-benchmarks #38 (OpenHuman agent on CortexDB, LoCoMo conversation 0):

• Without the deadline: 2 of 13 answers were lost to the 15 s cut-off. With it, none were, at OpenHuman's own limits.
• Without the second change: a retrieval test over 199 questions showed 2–4 recalls per run coming back empty. The conversation scope missed the deadline while the empty document and learning scopes answered.

Related issue

None in this repo. Context: tinyhumansai/openhuman-benchmarks#38.

API or behavior changes

Behavior only, not breaking. Only RecallMode::Evidence on the CortexDB engine (direct and hosted) changes:

• a recall now returns within about 12 s, leaving out any scope still building;
• a recall that left a scope out and found nothing returns Unavailable instead of empty evidence;
• one that found something ends its evidence with a "(Memory was only partly searched: …)" line.

RecallMode::Answer is unchanged. The deadline is Timing::evidence (crate-internal, default 12 s), not public configuration.

Validation

Commands actually run, with their outcome:

• cargo fmt --all -- --check: clean
• cargo clippy --all-targets --all-features -- -D warnings: clean
• cargo build --all-targets --all-features: builds
• cargo test --all-features: all pass

Tests

The CortexDB test double gained slow_recall (hold recalls over a matching scope). New tests in cortex/engine/evidence_tests.rs:

• a_scope_still_building_at_the_deadline_is_left_out: the slow scope is absent, the recall returns well before the slow pack, and the evidence carries the partly-searched note.
• no_pack_by_the_deadline_is_unavailable_not_empty
• a_late_scope_beside_empty_ones_is_unavailable_not_nothing_stored
• a_recall_with_every_pack_in_time_carries_no_note

Not covered: the exact 12 s value. Tests shorten the deadline to 300 ms through the engine's timing.

Documentation

docs/architecture/cortex-flows.md, "Recall in evidence mode", step 0. Module docs of cortex/engine/evidence.rs ("Deadline").

Checklist

• The change is focused on one logical change
• No new #[allow(...)], #[ignore], or relaxed lints
• No secrets, tokens, or .env contents in the diff or the description

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Evidence-based recall now returns results from memory sources that finish within a 12-second window. If other sources are still building, the response indicates that the search was partial.
  • If no source is ready in time—or the available sources return no evidence—recall reports that results are unavailable rather than implying that nothing is stored. Failures from individual sources continue to fail the recall.
• Documentation
  • Updated the recall flow documentation to describe the deadline and partial-search behavior.

[tinysweeper]

Tiny Sweeper review

Tiny Sweeper reviewed this change across 5 lane(s) and found 0 active actionable finding(s). Detailed lane evidence and any incomplete work are listed below.

State: Incomplete
Priority: none
Reviewed head: 2db85f68f2d6
Updated: 2026-10-16T05:04:19Z

Review snapshot

Change surface	Files	Review signal	Count
Production	2	Active findings	0
Tests	4	Noted findings	0
Documentation	1	Resolved findings	0
Configuration	0	Pending checks/questions	4

Completeness: Incomplete
Test assessment: No supported feature-to-test mapping was available; this does not mean tests are absent or passed.

Features

None identified with supported citations.

Tests

No supported feature-to-test mapping was produced. Test execution is not inferred.

Findings

No active actionable findings.

Could not review: crates/tinymemory-integrations/src/cortex/engine/engine_test_support.rs, crates/tinymemory-integrations/src/cortex/log/mod.rs, crates/tinymemory-integrations/src/cortex/testing/mod.rs, crates/tinymemory-integrations/src/cortex/testing/routes.rs

Before merge

• Complete the security review for crates/tinymemory-integrations/src/cortex/engine/engine_test_support.rs, crates/tinymemory-integrations/src/cortex/log/mod.rs, crates/tinymemory-integrations/src/cortex/testing/mod.rs, crates/tinymemory-integrations/src/cortex/testing/routes.rs.

How this fits together

flowchart LR
  n0["render<br/>changed"]:::changed
  n1["..._an_excluded_thread_stays_in_the_evidence<br/>changed"]:::changed
  n2["Timing<br/>changed"]:::changed
  n3["recall<br/>changed"]:::changed
  n4["map"]:::impacted
  n5["clone"]:::impacted
  n6["Log"]:::impacted
  n7["Reply"]:::impacted
  n8["gate"]:::impacted
  n9["scopes"]:::impacted
  n0 -->|calls| n4
  n1 -->|calls| n4
  n1 -->|tests| n4
  n3 -->|calls| n5
  n3 -->|uses| n7
  n3 -->|calls| n8
  n6 -->|uses| n2
  n8 -->|calls| n5
  n8 -->|uses| n7
  n9 -->|calls| n4
  n9 -->|calls| n5
  n9 -->|uses| n7
  n9 -->|calls| n8
  classDef changed fill:#0d4429,stroke:#238636,color:#e6edf3
  classDef impacted fill:#161b22,stroke:#6e7681,color:#c9d1d9
  classDef flagged fill:#5a1e02,stroke:#d93f0b,color:#ffffff
  classDef blocking fill:#67060c,stroke:#f85149,color:#ffffff

Loading

Agent review details

critique

• Conclusion: Success
• Scope reviewed: all assigned evidence
• Lane summary: Reviewed 7 files; 0 findings. _The code index is behind this pull request (indexed at `fc656a7848f8`), so retrieved context may be out of date._

security

• Conclusion: Success
• Scope reviewed: incomplete; unanswered: crates/tinymemory-integrations/src/cortex/engine/engine_test_support.rs, crates/tinymemory-integrations/src/cortex/log/mod.rs, crates/tinymemory-integrations/src/cortex/testing/mod.rs, crates/tinymemory-integrations/src/cortex/testing/routes.rs
• Lane summary: Provisionally: the deadline refactor looks sound — packs arriving within the deadline are ordered by original index, the late/empty handling matches the tests — but I need to read `Timing::evidence` and `Log::recall` to confirm the deadline field's meaning and that `pack`'s network call is what `timeout_at` bounds, and to check the test double's `slow_recall` and `sample_items` to be sure the new tests actually drive the new code paths as claimed before giving a verdict. 4 files could not be reviewed: crates/tinymemory-integrations/src/cortex/engine/engine_test_support.rs, crates/tinymemory-integrations/src/cortex/log/mod.rs, crates/tinymemory-integrations/src/cortex/testing/mod.rs, crates/tinymemory-integrations/src/cortex/testing/routes.rs. 1 file was not security-reviewed: docs/architecture/cortex-flows.md (prose or tabular data). _The code index is behind this pull request (indexed at `fc656a7848f8`), so retrieved context may be out of date._

tests

• Conclusion: Success
• Scope reviewed: all assigned evidence
• Lane summary: _The code index is behind this pull request (indexed at `fc656a7848f8`), so retrieved context may be out of date._

commits

• Conclusion: Neutral
• Scope reviewed: all assigned evidence
• Lane summary: Nothing sensitive found in what this pull request commits.

description

• Conclusion: Success
• Scope reviewed: all assigned evidence
• Lane summary: I need to verify several things before a verdict: how `Timing::evidence` interacts with the test double (`visibility` vs a concrete duration — the diff sets `evidence: visibility`, which I cannot parse), how `recall_evidence` is reached in answer mode vs evidence mode (to check the claim that only `RecallMode::Evidence` changes), whether `buffer_unordered` on `packs_by_deadline` interacts with `PACKS_AT_ONCE` the way the docs describe, whether `Error::Unavailable` exists and how `render` decides emptiness, and whether the `if let Some((scope, ms)) = slow` line in routes.rs uses let-chains correctly for the edition in use. Also checking that `engine_test_support.rs` and `log/mod.rs` keep the timing struct coherent across constructors. Provisonal summary: the description seems largely accurate; the main risk is a mismatch between `evidence: visibility` in the test double and the real `Timing::evidence` default of 12s, and whether test `engine_with_slow` correctly shortens the deadline. I will resolve this by reading the surrounding definitions of `Timing`, `visibility`, `render`, `Error::Unavailable`, and `PACKS_AT_ONCE`. Leaving findings empty for now while I look these up; the next turn will carry the verdict based on what comes back. I have not yet determined whether the `while let Ok(next) = ...` loop correctly propagates pack errors (`pack?`) given the deadline timeout wraps `building.next()` — a pack error that arrives after the deadline would be silently dropped rather than failing the recall, which may contradict the module docs' claim that 'a pack that fails still fails the recall'. I will check `Error` and how `pack` errors currently propagate in the original `buffered().try_collect()` flow to decide whether this is a regression the description's Summary claims against. Also need to confirm `arrived.sort_by_key(|(index, _)| *index)` ordering matches the docstring 'in `paths` order' (it does), and that `late.len()` is correct given `arrived` may contain duplicate indices from `buffer_unordered` (it cannot; each scope is built once). These need the lookups below to settle before reporting anything concrete; if `visibility` resolves to a concrete short duration, the test helper and the `evidence` field default in `log/mod.rs` are coherent and nothing is reportable on that line. I also want to confirm `RecallMode::Answer` path does not call `recall_evidence`, to verify the description's API-behaviour claim about which mode changes. Lastly, checking whether `tokio::time::timeout_at` with a future deadline computed at call time from `self.log.timing.evidence` is correct when `evidence` is mutated by tests after engine construction (the deadline is read at call time, so the mutation is picked up; no defect). These are all pending the lookups. No findings yet; the next turn will carry the verdict based on what comes back. In particular the 'a pack that fails still fails the recall' claim in the module docs and the PR body depends on how `pack?` interacts with `timeout_at` in the new loop, versus the old `buffered().try_collect().await?` which propagated errors unconditionally. If a slow-and-failing pack is now dropped rather than failing the recall, the docs and the description both say otherwise and that is a real defect worth reporting. The lookups below settle it: reading the old and new `recall_evidence` control flow plus `Error::Unavailable` and `render` definitions, and the `Timing` struct/`visibility` definition in the test support file. I will also look at `PACKS_AT_ONCE` and the `recall` route to confirm the `slow_recall` double works as the tests assume. With those in hand I can either report a concrete regression with a fix or return an empty findings list with a clear summary. I am keeping the provisional summary honest: nothing is confirmed yet. The most likely candidate is the interaction of `pack?` with `timeout_at` — but `timeout_at` wraps `building.next()`, not the whole future, so a completed-but-errored pack within the deadline still propagates through `pack?` (the `?` is inside the loop, on `arrived.push((index, pack?))`), so errors within the deadline fail the recall as before; only a pack that would have failed after the deadline is dropped, which is consistent with leaving it out. So that may be fine. I will confirm with the reads. The `evidence: visibility` line in the test-support builder remains the most suspicious: I do not know what `visibility` is there, and it is not defined in this diff, so I must read it. If it is a `Duration` constant that is too short (or a bool), the code will not compile and the tests are broken. That is a concrete, reachable defect: the test-support builder is used by tests and, if it fails to compile, `cargo test --all-features` fails. The description claims `cargo test --all-features` passes, so if it does not compile that claim is false too. I need to read the file. Reading the whole `Timing` struct and its constructor(s) plus the `visibility` helper is the goal. Four lookups below cover: (1) `engine_test_support.rs` around the `Timing` build, (2) `log/mod.rs` around the `Timing` struct and `Default`, (3) `evidence.rs` around `render` and `recall_evidence` to check emptiness/unavailable logic and the mode gating, (4) a search for `PACKS_AT_ONCE` and `Error::Unavailable` to confirm their definitions. After that I will either finalise findings or return an empty list. If `visibility` is a `Duration` like 10 seconds, then `evidence: visibility` is a valid (if odd) choice for tests and the code compiles; the 12s default in `log/mod.rs` is what production uses. If it is a bool or something else, the change does not compile and that is the finding. Also worth confirming the description's claim that the deadline is not public configuration — `Timing` is `pub(crate)` per the diff (`pub(crate) erasure`), so that checks out from what is shown. The description says _The code index is behind this pull request (indexed at `fc656a7848f8`), so retrieved context may be out of date._

Evidence and run details

• Models: ladder/vectors-oai3, glm-5.3-flash
• Spend: $0.001938
• Tokens: 183508 input · 11552 output · 8960 cached · 531 embedding

Head	State	Pass summary
2db85f68f2d6	incomplete	0 active finding(s), 0 resolved finding(s) (at 2026-10-16T05:04:19Z)

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: d28403e7-c46a-46be-863f-18e80b69b229

📥 Commits

Reviewing files that changed from the base of the PR and between bb781ca and 2db85f6.

📒 Files selected for processing: 7

• crates/tinymemory-integrations/src/cortex/engine/engine_test_support.rs
• crates/tinymemory-integrations/src/cortex/engine/evidence.rs
• crates/tinymemory-integrations/src/cortex/engine/evidence_tests.rs
• crates/tinymemory-integrations/src/cortex/log/mod.rs
• crates/tinymemory-integrations/src/cortex/testing/mod.rs
• crates/tinymemory-integrations/src/cortex/testing/routes.rs
• docs/architecture/cortex-flows.md

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

📝 Walkthrough

Evidence-mode recall now gathers packs concurrently under a configurable deadline. It omits packs that are still building, reports when a search is partial, and returns Unavailable when the available results cannot support a complete search.

Changes: Evidence Recall

Layer / File(s)	Summary
Deadline and delayed-recall setup crates/tinymemory-integrations/src/cortex/log/mod.rs, crates/tinymemory-integrations/src/cortex/testing/*, crates/tinymemory-integrations/src/cortex/engine/engine_test_support.rs, crates/tinymemory-integrations/src/cortex/engine/evidence_tests.rs	Timing adds a 12-second evidence deadline. Test support can delay selected recall scopes, and test timing uses the visibility duration for evidence.
Pack collection and incomplete-search results crates/tinymemory-integrations/src/cortex/engine/evidence.rs, crates/tinymemory-integrations/src/cortex/engine/evidence_tests.rs, docs/architecture/cortex-flows.md	Recall gathers packs concurrently and retains those ready before the deadline. Late scopes are omitted with a warning. Empty results from an incomplete search and cases with no timely packs return Unavailable; nonempty partial results include a partial-search notice. Pack failures still fail recall.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant CortexEngine
  participant RecallPacks
  participant EvidenceAnswer
  CortexEngine->>RecallPacks: Start scope pack requests concurrently
  RecallPacks-->>CortexEngine: Return packs completed before the deadline
  CortexEngine->>EvidenceAnswer: Render available evidence and partial-search status

Loading

Suggested reviewers: senamakel

Merge Risk: ⚪ Minimal · up to 2db85

Evidence recall can return partial results at the deadline without masking a completed pack failure. No merge-blocking issue was identified.

Security architecture risk: 🔵 Low · up to 2db85

The change preserves existing access checks and explicitly distinguishes incomplete searches from empty results. Remaining uncertainty concerns how server-side work is handled after a request times out.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — The demonstrated exposure remains the scopes selected through the existing credential-bound recall path, rather than a newly exposed endpoint or privileged operation. Repeated timed-out calls may leave accepted backend work outstanding, but available client evidence cannot establish its duration or aggregate resource impact.

Trust Boundaries and Controls

• observed — Recall continues through the existing reach-bound call path. The unchanged transport resolves a credential before sending, attaches authorization, and requires a tenant for the proxied wire. The deadline collector does not replace these controls or introduce caller-selectable credentials or timing.

Resilience and Maintainability Implications

• inferred — Local collection state is per recall, with no persistent memory mutation or rollback step in the traced path. Expiry, an observed error, or caller interruption drops unfinished futures without an explicit server cancellation operation. Existing documentation describes server-held packs with a 60-second lifetime, but production disconnect handling and cleanup are not available; this is a coverage limitation, not an established leak.

Hardening Proposals

• proposed — Confirm the production recall service's disconnect, expiry and resource-accounting guarantees under repeated deadline expiry. Add explicit cancellation or bounded outstanding-work controls only if those guarantees do not already contain abandoned requests.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 71.43% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 14 functions across 6 files. (1 skipped: …	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly identifies the main changes: a 12-second deadline for evidence recall packs and notification when a pack is omitted.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 71.43% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 14 functions across 6 files. (1 skipped: 1 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches: 💡 1

🛠️ Fix failing CI checks: 💡

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{A rabbit counts the seconds down
While waiting packs arrive
The ready clues are gathered close
The late ones miss the call
A note says when the search is partial
Then hops back through the logs}

_{Comment @coderabbitai help to get the list of available commands.}

[tinysweeper]

tinysweeper found nothing blocking, but could not review everything, so this is not an approval: crates/tinymemory-integrations/src/cortex/engine/engine_test_support.rs, crates/tinymemory-integrations/src/cortex/log/mod.rs, crates/tinymemory-integrations/src/cortex/testing/mod.rs, crates/tinymemory-integrations/src/cortex/testing/routes.rs.

             $0.0019 · 183,508 in / 11,552 out · 8,960 cached (5%)  · ladder/vectors-oai3, glm-5.3-flash · 531 embedded
critique:    $0.0010 · 92,025 in  / 7,813 out  · 8,960 cached (10%) · glm-5.3-flash
security:    $0.0006 · 58,446 in  / 1,722 out  · 0 cached (0%)      · glm-5.3-flash
tests:       $0.0002 · 16,806 in  / 11 out     · 0 cached (0%)      · glm-5.3-flash
description: $0.0001 · 9,027 in   / 1,360 out  · 0 cached (0%)      · glm-5.3-flash