[pithead-developer]

What's the problem

Issue #620: the worker gate already preserves its connected sequence, but its
human summary and network-summary diagnostic do not bind completion to a
particular head/ref, immutable input set, attempt, group, row or platform. A
partial or stale summary cannot establish that all stateful work and restoration
finished. The shared manifest and fleet rollout dependencies remain open.

How I fixed it

Added optional, strictly validated worker proof context and a receipt emitted
after the existing cleanup. The producer verifies the clean requested head/ref
and actual platform before mutation and after restoration. It counts assertions
through a private event file, including child-shell assertions, without recording
assertion prose, response bodies or deployment details. Existing dispatch,
phase internals, snapshots, locks and explicit skip behavior are preserved.
Git identity checks use the existing exact-checkout trust pattern so root-run
jobs can verify an operator-owned checkout without changing global Git settings.

Added a standard-library checker for the required AND against an independently
supplied expected plan. It rejects missing, duplicate, reordered, substituted,
stale, malformed or truncated evidence, mismatched identities/platforms, partial
assertion counts, unapproved skips, failed cleanup and nonzero exits. Missing
context is explicitly unbound and cannot pass this checker; legacy CLI behavior
remains available. No concurrency, capacity, scheduling or hosted status change
is enabled by this patch.

Added schema/join and dispatcher regression suites to make test-suite, plus
the complete phase inventory and rollout contract in docs/dev/worker-proof.md.
Existing regression assertions and fixtures are unchanged; the live E2E harness
gains proof instrumentation. No version or changelog edit.

Here is the proof

Baseline: 81ec49584fa88196ef915435f7a768d0c20b866c, branch
astra/620-rigforge. All results below concern the uncommitted working tree.

Before/after proof set at the affected worker gate. Both sides retain Linux
physical-worker execution and the existing coordinator/fixture ownership. all
is the ordered list below; network-sequence is exactly its first four phases in
one shell/snapshot. Standalone phases remain diagnostic/existing-owner rows, and
cannot replace either complete group.

Block	Before assertions	After assertions
connect	Job received; accepted share; worker identity generated (3)	Same three, same position.
worker-api	Open read; bind policy; restricted write refusal; unauthenticated refusal; matching bearer read (5)	Same five, after connect.
api-impact	Sister API ready; live loaded hashrate within baseline tolerance; responsiveness under mining load (3)	Same three, after worker API; same baseline/load windows.
network	Pool-only peers; control-implied listener; enabled API listener; miner listener policy; exact summary superset/metadata; token and password leak sweeps; disabled API closed (8)	Same eight, after impact, with original diagnostic and fatal assertion.
stratum-auth	Correct credential mines; wrong credential rejected; no jobs under wrong credential; correct credential recovers (4)	Same four in one fault/recovery phase.
dashboard	Initially online; stopped worker becomes offline (2)	Same two; initial absence remains fatal; worker restarted.
dev-fee	Compound real migration/rebuild, zero donation, runtime reductions without rebuilding and original-artifact restoration witness (1)	Same complete compound witness.
control	Apply accepted; donation applied; config persisted; appliance pool mutation atomically refused; new thermal feed within deadline (5)	Same five, including child-shell refusal and full systemd/timer observation window.
access-tokens	Miner refuses named token; sister raw and derived reads; control apply accepted/applied; both revocation checks (7)	Same seven; mutation/revocation stay together.
Each thermal/pools replay	Nested connect (3), complete scalar/pools/thermal sequence, final timely publication assertion (1), fixture/reservation ownership	Same four counted assertions and all intermediate checks/windows; neither replay joins all.
Cleanup	Profiling restoration; recorded load processes stopped; config copied/applied/byte-compared; original miner active state restored; failed snapshot retained	Same operations; failure now also prevents bound-receipt acceptance.
Inputs and skip rules	Existing fixtures, pinned product sources, tolerances, timeouts and row owners; standalone auth requires fixture; optional auth/dashboard absence explained	All retained; bound mode additionally requires exact input-manifest digest and current attempt/head/ref/platform. Only the two existing fixture-absence reasons may be authorized by the expected plan.

Before/after local and hosted gate manifest. No existing required check is
removed, moved off a PR, or replaced with receipt evidence.

Gate / platform / status	Before proof set	After proof set
Test suite, Ubuntu hosted; local macOS/Linux	tests/run.sh, test-contract-credentials.sh, test-access-token-readiness.sh, test-docs-clone-pin.sh, test-staged-rebuild.sh, test-tool-pin-watch.sh	All six unchanged, then new test-worker-proof.py and test-worker-proof.sh.
Existing Bash aggregate blocks, same platforms	All inline run.sh assertions, suite liveness, tree hygiene and lock-helper parity; sourced onion proxy, API refresh liveness, control upgrade, apt prerequisites and E2E fail-closed blocks	Byte-identical existing test sources; no assertion removed or expectation changed.
Existing E2E fail-closed blocks, same platforms	Config edit/apply errors; dispatch/typo/unknown rejection; full order; cleanup bytes/runtime/snapshot retention/status; dashboard restoration; stop after failed predecessor; real-gate control/upgrade/watchdog cleanup retry cases	Same blocks and assertions.
Existing summary/sequence blocks, same platforms	Valid/invalid/missing/non-object metadata; unreachable XMRig; key mismatch; secret omission; malformed/multiple records; fatal wire assertion; same-shell ordering; fatal and failed predecessors	Same blocks and assertions.
Existing stratum-auth blocks, same platforms	Unset/empty required fixture refusal; authorized right/wrong/recovery path; optional all skip; initial active/stopped restoration	Same blocks and assertions.
New receipt tests, local macOS/Linux and hosted Test suite	Absent	Context/ref/types/duplicate JSON keys; exact counts/order; all supported groups; explicit skips; missing/duplicate/substituted/stale groups and binding fields; plan consistency; malformed/truncated events; cleanup/exit failures; dispatcher failure/signal/restoration and child-shell assertions.
Lint (shellcheck + shfmt), Ubuntu hosted/local	Topology, file-budget self-tests and monotonic gate; all tracked shell sources	Same checks plus new tracked shell sources after submission. New untracked sources were checked explicitly here.
Lint (yamllint) and Lint (markdownlint), Ubuntu hosted	YAML/workflows/actionlint and Markdown	Same checks; new contract documentation added.
Stock home setup (historical) and (current), Ubuntu hosted	Both historical failure control and current native-systemd setup	Unchanged.
End-to-end (Docker), Ubuntu hosted, Linux amd64 containers	Standard deployment and appliance deployment passes	Unchanged.
Coverage (kcov + diff-cover), Ubuntu hosted/Linux container	Total floor and changed-product-line floor	Unchanged; no product source changed.
Secret scan (gitleaks) and Workflow audit (zizmor)	Existing security workflow gates	Unchanged.
adversarial-review at current submitted head	Independent review required by repository policy	Still required; local agent reviews are not a posted head status.
Live worker/coordinator and release gates, physical Linux rigs	Connected sequence, accepted shares, systemd/feed/donation; real provision/reboot/verify/control/upgrade/watchdog/performance/teardown; existing explicit skips and baselines	All retained. Exact sibling-runner required-status names remain unverified here and must be reconciled before rollout.
Cross-repository required AND / current-head status	Existing required per-PR proofs and both independent bench-ci make check runs	All retained; integrating this checker with the sibling runner/status/preflight is still open.

Executed commands and results.

• WORKER_PROOF_BASELINE=1 bash tests/test-worker-proof.sh — expected RED,
  exit 1: FAIL: missing worker proof receipt. The harness reads the original
  dispatcher from git show HEAD:tests/e2e-pithead.sh into memory and replaces
  hardware operations with stubs; no stash, checkout or original test edit.
• python3 tests/test-worker-proof.py — PASS, 22 tests.
• bash tests/test-worker-proof.sh — PASS, 21 integration cases, including both
  replay dispatcher shapes, failed Git enumeration, platform mismatch and the
  root-run operator-checkout case. Adding that last regression before the Git
  trust fix produced the expected RED, exit 1: the case returned 2 instead of 0.
  These exercise the production dispatcher and receipt hooks with phase bodies
  and hardware operations replaced by stubs; they are not live-phase proof.
• shellcheck --severity=warning tests/e2e-worker-proof.sh tests/test-worker-proof.sh tests/e2e-pithead.sh
  — PASS after fixing the initial shared-state variable annotation, ShellCheck
  0.11.0. This final run includes the instrumented existing harness.
• shellcheck --severity=warning tests/e2e-worker-proof.sh tests/test-worker-proof.sh
  — PASS, ShellCheck 0.11.0.
• shfmt -i 4 -d tests/e2e-worker-proof.sh tests/test-worker-proof.sh tests/e2e-pithead.sh
  — PASS with installed shfmt 3.13.1; the complete source set also passed the exact
  CI version 3.14.1 in its native container (command below).
• FILE_BUDGET_REQUIRE_BASE=1 bash scripts/lint-file-budget.sh — PASS, including
  the unchanged-budget comparison to the available base.
• git diff --check — PASS.
• make lint-actions — PASS.
• npm_config_cache=/private/tmp/rigforge-620-npm-cache npx --yes markdownlint-cli2@0.23.3 --no-globs docs/dev/worker-proof.md tests/README.md PR-BODY.md
  — PASS, three files and zero issues. Earlier attempts found a corrected code-span
  whitespace issue and the pre-existing blank-line violation in untracked
  ISSUE.md; --no-globs checks the changed documents without altering that input.
  The first attempt using the default npm cache was blocked by cache permissions.
• npm_config_cache=/private/tmp/rigforge-620-npm-cache make lint-yaml lint-md lint-actions
  — blocked at YAML lint because yamllint is unavailable; later targets did not
  run in that command. Actionlint was run separately as above.
• UV_TOOL_DIR=/private/tmp/rigforge-620-uv-tools uv tool run --cache-dir /private/tmp/rigforge-620-uv --from yamllint==1.38.0 yamllint --strict $(git ls-files '*.yml' '*.yaml')
  — PASS, all tracked YAML with the CI-pinned version. An earlier attempt without
  the temporary tool directory was blocked by tool-directory permissions.
• make test-e2e — FAIL before either suite started: both amd64 containers
  reported exec format error. This engine is Linux arm64 with no working
  amd64 execution for these containers; no assertion result is claimed.
• Final native Linux proof command below — PASS: 22 validator tests and
  21 dispatcher cases. The digest is the arm64 child of the pinned Ubuntu index,
  resolved using docker buildx imagetools inspect and the architecture filter.
  The initial attempt to select arm64 through the multiarchitecture index failed
  with cannot overwrite digest; using the exact child digest succeeded.

docker run --rm --platform linux/arm64 -v "$PWD:/src:ro" \
  ubuntu@sha256:ea0b2b8607a61f981b2461f21edc27c0b37ddd53d23d4d350936480adf5de28c \
  bash -c 'set -e; E2E_LIB_ONLY=1 source /src/tests/e2e/in-container.sh; export DEBIAN_FRONTEND=noninteractive PYTHONDONTWRITEBYTECODE=1; _apt_prereqs; cd /src; python3 tests/test-worker-proof.py; bash tests/test-worker-proof.sh'

make test was interrupted during the repository-wide ShellCheck step after a
long interval without output (exit 130); its test recipe had not begun.
timeout -k 30 300 make lint then timed out in that same step (exit 124), after
topology and file-budget checks passed. Full repository shell lint is unproved;
all changed shell sources passed the explicit check above. The native pinned
ShellCheck command below exited 137 without diagnostics; no full-shell lint pass
is claimed. The complete shfmt command below passed with the CI-pinned version.

docker run --rm --platform linux/arm64 -v "$PWD:/src:ro" -w /src \
  koalaman/shellcheck@sha256:4d7c931ae7ee555483ceac75379369f1bc5f01807be6638d12ede9ce29a27b53 \
  --severity=warning $(git ls-files '*.sh') tests/e2e-worker-proof.sh tests/test-worker-proof.sh

docker run --rm --platform linux/arm64 -v "$PWD:/src:ro" -w /src \
  mvdan/shfmt@sha256:879d33d63cc7604014150b0831522fb2288447fd2db687947a78a610a4936f02 \
  -i 4 -d $(git ls-files '*.sh') tests/e2e-worker-proof.sh tests/test-worker-proof.sh

timeout -k 30 5400 make test-suite was interrupted on macOS (exit 130) after
switching the complete suite to the native Linux container; no completed macOS
aggregate pass is claimed. The focused new suites did complete on macOS.
The native Linux aggregate finished with 2,466 passed and 2 failed (Make exit
2), both existing Stratum recovery assertions. No aggregate pass is claimed.
The phase body and existing test expectations are unchanged. The remaining
Makefile recipes did not run in that failed invocation; all seven passed in the
separate native command below, including the final 22 validator tests and
the then-current 20 dispatcher cases. The later Git trust fix was verified by the
final focused command above with 21 dispatcher cases; the aggregate was not rerun.

Full-suite command:

docker run --rm --platform linux/arm64 -v "$PWD:/src:ro" \
  ubuntu@sha256:ea0b2b8607a61f981b2461f21edc27c0b37ddd53d23d4d350936480adf5de28c \
  bash -c 'set -e; E2E_LIB_ONLY=1 source /src/tests/e2e/in-container.sh; export DEBIAN_FRONTEND=noninteractive PYTHONDONTWRITEBYTECODE=1; _apt_prereqs; apt-get install -y -qq curl git make util-linux >/dev/null; cd /src; timeout -k 30 3600 make test-suite'

Remaining-recipe command — PASS, exit 0:

docker run --rm --platform linux/arm64 -v "$PWD:/src:ro" \
  ubuntu@sha256:ea0b2b8607a61f981b2461f21edc27c0b37ddd53d23d4d350936480adf5de28c \
  bash -c 'set -e; E2E_LIB_ONLY=1 source /src/tests/e2e/in-container.sh; export DEBIAN_FRONTEND=noninteractive PYTHONDONTWRITEBYTECODE=1; _apt_prereqs; apt-get install -y -qq git make util-linux >/dev/null; cd /src; bash tests/test-contract-credentials.sh; bash tests/test-access-token-readiness.sh; bash tests/test-docs-clone-pin.sh; bash tests/test-staged-rebuild.sh; bash tests/test-tool-pin-watch.sh; python3 tests/test-worker-proof.py; bash tests/test-worker-proof.sh'

The focused native comparison below ran the unchanged first Stratum block twenty
times per source: baseline 303 passed / 17 failed; candidate 300 passed /
20 failed. Both reproduce fixture-setup and recovery failures; the phase body
is byte-identical. This driver exits 0 after printing counters, not because those
assertions passed. It reads the baseline into memory and edits no existing test.
The stub does not consume piped input, making a producer pipe failure plausible;
the exact signal was not captured. These failures predate the receipt patch and
remain unfixed here.

docker run --rm -i --platform linux/arm64 -v "$PWD:/src:ro" \
  ubuntu@sha256:ea0b2b8607a61f981b2461f21edc27c0b37ddd53d23d4d350936480adf5de28c bash -s <<'BASH'
set -uo pipefail
E2E_LIB_ONLY=1 source /src/tests/e2e/in-container.sh
export DEBIAN_FRONTEND=noninteractive PYTHONDONTWRITEBYTECODE=1
_apt_prereqs || exit
apt-get install -y -qq git >/dev/null || exit
ROOT=/src
SANDBOX=$(mktemp -d)
trap 'rm -rf "$SANDBOX"' EXIT
source "$ROOT/tests/e2e-refresh-window.sh"
eval "$(sed -n '/^ok()/,/^_poll_up()/ { /^_poll_up()/!p; }' "$ROOT/tests/run.sh")"
sed() {
    if [ "$mode" = baseline ] && [ "${!#}" = "$ROOT/tests/e2e-pithead.sh" ]; then
        command sed "${@:1:$#-1}" < <(git -C "$ROOT" show HEAD:tests/e2e-pithead.sh)
    else
        command sed "$@"
    fi
}
for mode in baseline candidate; do
    PASS=0 FAIL=0
    PIT_DIE=$(sed -n '/^die()/,/^}/p' "$ROOT/tests/e2e-pithead.sh")
    PIT_CLEAN=$(sed -n '/^_cleanup()/,/^}/p' "$ROOT/tests/e2e-pithead.sh")
    PIT_RESTORE=$(sed -n '/^_restore_xmrig()/,/^}/p' "$ROOT/tests/e2e-pithead.sh")
    PIT_SNAPSHOT=$(sed -n '/^snapshot_config()/,/^}/p' "$ROOT/tests/e2e-pithead.sh")
    PIT_DISPATCH=$(sed -n '/^case "${1:-all}" in/,/^esac/p' "$ROOT/tests/e2e-pithead.sh")
    for sample in {1..20}; do
        T491=$(mktemp -d "$SANDBOX/stratum.XXXXXX")
        eval "$(sed '/^# #570:/,$d' "$ROOT/tests/test-stratum-auth-gate.sh")"
    done
    printf '%s: %s passed, %s failed\n' "$mode" "$PASS" "$FAIL"
done
BASH

Independent local security and adversarial reviews found no remaining concrete
implementation defect. They confirmed that the external acceptance items below
remain open; no hosted review status was written.

Still open; this is not rollout acceptance. No hosted workflow, bench, KVM or
physical-worker test was submitted or claimed. Hosted green checks and independent
review at the eventual submitted head remain required. The amd64 container E2E,
stock-home systemd witnesses and coverage remain unproved locally. Bound-context
input provenance and required-status wiring need the sibling runner and preflight
changes; the shared manifest is not assumed merged. Capacity acceptance must
precede any separate mutable rigs/stacks or concurrency. No new lane is authorized.

Paired live serial/concurrent acceptance must compare complete old/new behavior,
assertion, immutable-input and platform manifests, gate median/p90, occupied
setup hours, failures and restoration while preserving every existing live/release
witness. Baseline collection on normal authorized runs and the one-week comparison
of paired PR/head latency, stage median/p90, occupied hours, failures and cleanup
must report sample sizes and load. Lost evidence, cleanup failures or increased
KVM queueing require rollback. These criteria require authorized fleet data and
remain open under the issue's operator scope ruling.

Closes #620

Draft from an operator-run gpt-6-astra session (owner request, 2026-11-01). The operator reviews and gates it before it leaves draft.

🤖 Generated with Claude Code

[pithead-developer]

Operator: this stays a held draft until RigForge 1.18.1 is tagged (due Fri 2026-11-06, Sat 11-07 at the latest). 1.18.1 is cut from develop, so merging this first would widen that release. After the tag: rebase on develop, run the agent review, and request the owner's approval.