Preserve ordered worker contracts inside concurrent proof bundles #620

What's the problem

Independent bundles may overlap, but splitting RigForge’s connected worker sequence or relying on a partial summary can remove cross-step state proof.

What it impacts

Live worker/coordinator orchestration and summary diagnostics.

Related repositories: bench-ci (p2pool-starter-stack/bench-ci#2359); pithead-developer (p2pool-starter-stack/pithead-developer#141); pithead (p2pool-starter-stack/pithead#3887).

How I found it

The operator’s testing-and-CI review of 2026-11-02. RigForge CI median/p90 was 8.1/18.7 minutes, with 23 failures in 85 runs; Security was 0.3/8.5 minutes, with 14 failures in 90 runs. Shared E2E queue median/p90 was 195/895 minutes; execution was 45.5/64 minutes, with 101 failures in 270 submissions (43 cancellations, 22 errors). KVM queue was 0/170 minutes and execution 45/138 minutes, with 94 failures in 208 submissions. The October 23–November 2 fleet sample has no usable repository/PR attribution; these counts are not flake rates. The plan uses 11 inclusive days and excludes null durations.

How important is it

Contributes to the shared rank-5 8–20 gate elapsed hours/week scenario, with no RigForge-only saving measured. High risk: missing cross-step effects or racing a shared rig/stack; extra setup can consume more bench hours. Estimates are conditional, may fall to zero, overlap other ranks and are not engineering hours; do not sum related issues.

Acceptance criteria

• Inventory phase dependencies in tests/e2e-pithead.sh and tests/e2e-summary-diagnostics.sh; preserve connect → worker API → API impact → network ordering, fault/recovery and restoration within each stateful group.
• Use separate mutable rigs/stacks for genuinely independent groups after rank-1 capacity acceptance. Retain physical hashing/accepted-share, systemd/feed/donation, performance baselines and all currently required live/release witnesses.
• Emit exact head/ref, input, group and row identities plus cleanup for the required final AND. Regression controls for missing/duplicate/substituted/stale groups, unexplained skips and cleanup failure must block the same current-head product/release gate.
• Run affected local/hosted tests before merge and paired live serial/concurrent acceptance before rollout. Compare complete assertion/platform manifests, gate median/p90, occupied setup hours, failures and restoration; coordinate rank-7 immutable inputs.
• Before rollout, compare the complete old/new behavior, assertion, input and platform manifests at every affected gate; retain all existing required checks there. Missing, truncated, unknown or unjustifiably skipped evidence blocks acceptance; retain existing explicit skip and row-ownership rules.
• Record a baseline on normal authorized runs, then compare one week after rollout: paired PR/head latency, stage median/p90 minutes, occupied hours, failures and cleanup outcomes. Report sample sizes and load; roll back on lost evidence, cleanup failures or increased KVM queueing.

Possible solutions

One option is to retain the connected sequence as a complete bundle while extending its summary for bench-ci’s phase-aware join and pithead-developer’s evidence preflight. Parallelize only separately proven independent groups. Prerequisites: p2pool-starter-stack/bench-ci#2362, p2pool-starter-stack/pithead-developer#145, p2pool-starter-stack/pithead#3891, #625, p2pool-starter-stack/bench-ci#2355, p2pool-starter-stack/pithead-developer#137, p2pool-starter-stack/bench-ci#2359.

Owner rulings for this plan (2026-11-02): every proof that runs per PR today stays per PR; both independent bench-ci make check executions stay; capacity is tuned on existing benches and the owner is asked before any new lane.