[pithead-developer]

What's the problem

Closes #626

When a named bearer was revoked and the control receiver could not be replaced, the old receiver process kept the token it had loaded at startup. It went on accepting payout writes that the root consumer applied. install_control suppressed stop, enable and restart failures, and apply discarded control reconciliation failures.

How I fixed it

• _stop_control_receiver stops the receiver before any unit or credential change. If systemd rejects the stop, it kills the exact unit and stops it again. A missing unit is benign only when it is also inactive. If both stops fail, it warns that revocation is not confirmed and returns failure.
• install_control runs in a subshell with an EXIT trap that stops the receiver whenever the function fails. Every render, reload, enable and restart step returns failure.
  • Because it is a subshell, error() (for example "needs python3") now ends only install_control. The caller records _control_rc=1, setup goes on, and the trap stops the receiver.
• Setup, upgrade and apply keep the control failure, still run the firewall reconcile, and exit non-zero. Through apply, this also means control-apply and remote upgrade roll back when the receiver cannot be reconciled.
• The receiver unit is Type=notify with NotifyAccess=main and TimeoutStartSec=15. util/control-server.py sends READY=1 only after it has loaded credentials and bound its listener.
• With control disabled, teardown disables only the units whose files exist. Before, it named the remote-upgrade path even when control_upgrade was off, so real systemd failed the disable and the unit files survived. A leftover upgrade service without its path unit is also removed without naming the absent unit.
• docs/pithead-integration.md states where the revocation boundary sits:
  • A change the receiver accepted earlier is still applied.
  • A run that aborts before the control step leaves the old receiver serving.
• Shortcut: if systemd refuses both stops even after SIGKILL, the receiver can keep running. The run warns and fails, and nothing more is attempted. Upgrade when a real rig shows systemd ignoring a stop.
• Linux hosts without a reachable systemd still fail closed. Setup already aborts there at install_service's unguarded systemctl daemon-reload.
• Pre-existing comment lines in _apply_runtime, apply and the control server's __main__ stay removed. rigforge.sh (6303) and util/control-server.py (400) are exactly at their file-budget ceilings.

Here is the proof

Red, then green. Each check below was run on this Mac with the named old code swapped in, then restored:

• python3 tests/control-revocation.py "$PWD" with develop's rigforge.sh exits 1 at assert stale is None: the stale receiver accepted the revoked bearer's write. The stub rejects the first systemctl stop (the issue's trigger) and makes kill SIGKILL the receiver. The control is a fresh receiver that returns 401 for the revoked bearer and 202 for the master. At head it exits 0.
• bash tests/test-control-revocation.sh with cef9b66's elif line fails with "a leftover upgrade service without its path unit failed the control reconcile".
• With 83fdd91's disabled teardown it fails with "control-disabled teardown failed when the upgrade unit was absent".
• At head it exits 0. The suite also covers a failing disable (non-zero, and the receiver stopped on exit) and every caller's firewall continuation.

Local, at head: bash tests/test-control-revocation.sh, python3 tests/test-control-readiness.py, bash tests/test-sister-reconcile.sh, FILE_BUDGET_REQUIRE_BASE=1 bash scripts/lint-file-budget.sh (file budget OK), and shellcheck --severity=warning on every shell file passed. Formatting is proved by the pinned CI job: "Lint (shellcheck + shfmt)" is green on each head (the local shfmt here is older than CI's 3.14.1 pin and only disagrees on lines byte-identical to develop). Not run locally: make test, the full suite (GitHub's Test suite job runs it), and the container e2e (this Mac runs no Docker).

Real systemd: tests/test-stock-home.sh (GitHub "Stock home setup (current)", ubuntu-24.04) now runs apply with control enabled and remote upgrade off. It asserts that the receiver is active as Type=notify and answers its token on 8082. It then disables control, asserts the "Writable control path disabled" teardown, and checks that the receiver is inactive and its unit files are gone. On head 164a1b5 the "Stock home setup (current)" job passed with the enable-then-disable control stage. It asserts the teardown by state: receiver inactive, unit files removed. An earlier head asserted apply's log line, which apply hides, and that failed; the assertion is removed.

Base merge at 6cb1915: this branch has merged develop twice to resolve the base conflict, at 20990de and again at 6cb1915 to take in #641 (the worker-proof harness). Both times the only conflict was the Makefile test-suite list, where both sides had added test entries; each resolution keeps both. The resolved entries python3 tests/test-control-readiness.py, python3 tests/test-worker-proof.py and bash tests/test-worker-proof.sh were each run on the merged head and pass, and FILE_BUDGET_REQUIRE_BASE=1 bash scripts/lint-file-budget.sh is OK. The control-teardown code this issue changes is byte-identical to 609c568 below; the merges add only develop-side code (upgrade throttle state, thermal watchdog retries, HugePages ceiling, worker-proof harness) that develop already proved. All ten GitHub checks pass on 6cb1915: Lint (shellcheck + shfmt), Lint (markdownlint), Lint (yamllint), Test suite, End-to-end (Docker), Coverage (kcov + diff-cover), Stock home setup (current and historical), Secret scan (gitleaks), Workflow audit (zizmor).

Bench, tier4-e2e connect,control,access-tokens, job 3519 on 609c568. The head moved to 164a1b5 afterwards, and the only change since was tests/test-stock-home.sh, which tier4 does not run; the base merges add no change to the code this job proves:

bench-ci job 3519@609c568fbd6f2e38ffee5d5ed25d61bd734e7518 · rigforge · tier4-e2e

• options: {"phases": ["connect", "control", "access-tokens"]}
• status: completed · conclusion: success · exit 0 · 10m 54s
• baseline verified: yes · rig restored: yes
• node guard: not recorded · no comparison recorded
• steps: checkout success · reserve success · preflight success · run success · collect success · restore success · cleanup success
• rows: 15 pass · 0 fail · 0 skip
  Generated by bench-ci evidence 3519@609c568fbd6f2e38ffee5d5ed25d61bd734e7518; re-run it to check.

Ponytail pass: I cut the no-systemd shortcut I had added; it bought no supported host and could hide an orphaned receiver. The stale-spool and abort-before-control windows are documented, not closed, because they are outside the replacement-failure path this issue names.

🤖 Generated with Claude Code

[pithead-developer]

Operator: hold lifted so the runner's reviewer takes this draft. It fixes a confirmed finding from the operator's RigForge audit; bugs go into the next patch release (v1.18.1, cut from develop). It merges only after the agent review passes and the owner approves at the exact head.

[VijitSingh97]

Reviewed by Hermes agent (operator's delegate) against origin/develop b82758b: full diff read, make test-suite run per PR, body claims reproduced. Not a substitute for the owner's approval; merge stays with the operator.

Verdict: RETURN (mandatory rebase onto b82758b; design otherwise sound)

Security design verified: receiver stopped before units/credentials change, errors propagate, EXIT trap re-stops the receiver, firewall reconciliation completes before surfacing failure, Python receiver signals readiness only after loading credentials and binding. All three body-claimed pre-fix failures reproduced against develop; full suite SUITE_EXIT=0 here (the author's 10 Linux startup-timing failures did not reproduce).

Rebase required — b82758b rewrote the same apply() tail. Two content conflicts, both in rigforge.sh (tests/run.sh auto-merges):

1. ~:4207 — HEAD local _wd_rc=0 _api_rc=0 _control_rc=0 ordering vs develop's local-before-restart reorder; keep develop's order, add _control_rc to the declaration.
2. ~:4232 — keep both: the restart-reason := first-failure-wins idiom, the control reason block, and a final gate over _wd_rc/_api_rc/_control_rc/RIGFORGE_RESTART_FAILED.
   After rebase re-run the suite and b82758b's new restart-rejection case in tests/test-sister-reconcile.sh; add one assertion that restart+control failure keeps the first RIGFORGE_APPLY_FAIL_REASON.

Notes: error inside the install_control subshell (rigforge.sh:1451,1467) is now non-fatal to the outer command (fail-closed but silently softened; no test covers it); READY=1 send unguarded against stale NOTIFY_SOCKET (util/control-server.py:392-397); make test-suite now needs python3 unguarded (Makefile:24) — fine for the pinned CI image.

[VijitSingh97]

Rebased onto develop b82758b and resolved both apply() conflicts per the review recipe: develop's local-before-restart order kept with _control_rc added to the declaration; the final gate now covers _wd_rc/_api_rc/_control_rc AND RIGFORGE_RESTART_FAILED, preserving develop's first-failure-wins `:=" fallback for the restart reason. Added an assertion (tests/test-sister-reconcile.sh) that when both restart and control fail the first RIGFORGE_APPLY_FAIL_REASON wins. Evidence: make test-suite rc=0 on the rebased head; tests/test-sister-reconcile.sh PASS; python3 tests/test-control-readiness.py PASS; python3 tests/control-revocation.py . rc=0. Ready for the operator's approval.

[pithead-developer]

Operator review of the rebased head (not an approval): Lint fails on the file budget (rigforge.sh is 6308 lines against its recorded ceiling of 6307). Drop or fold one line, run FILE_BUDGET_REQUIRE_BASE=1 bash scripts/lint-file-budget.sh, then it needs a fresh agent review at the new head, since the earlier review was on the head the force-push replaced.

[pithead-reviewer]

FAIL

What I investigated

FAIL. The design fixes the reported hole: the stale receiver is stopped before replacement, failures propagate, the firewall step still runs, and readiness uses Type=notify. The new tests are real: they fail with the non-test hunks reverted and pass on the head. But the PR changes the receiver's systemd unit and rewrites install_control's stop, teardown and enable paths. It quotes no bench-ci job and says real-systemd and tier4 proof are still needed, and I confirmed bench-ci has no job on the head. The disabled-branch changes also have no test.

• Head: gh pr view 640 headRefOid is 83fdd91, which matches the worktree HEAD. Base is develop (the PR is a draft, not a finding). The merge-base with origin/develop b1c25d6 is current.

• gh pr checks 640: all 10 checks SUCCESS (Test suite, Coverage, End-to-end (Docker), both Stock home setups, shellcheck+shfmt, markdownlint, yamllint, zizmor, gitleaks).

• FILE_BUDGET_REQUIRE_BASE=1 bash scripts/lint-file-budget.sh printed 'file budget OK'. rigforge.sh is 6303 lines.

• Fix reverted in a scratch worktree (rigforge.sh, util/ and systemd/ taken from origin/develop, tests kept): bash tests/test-control-revocation.sh exit 1 ('FAIL: install_control hid apply-enable failure'); python3 tests/test-control-readiness.py exit 1 (Type=notify assertion); bash tests/test-sister-reconcile.sh exit 1; python3 tests/control-revocation.py . exit 1 (assert install_rc != 0). The scratch worktree was removed afterwards.

• On the head: bash tests/test-control-revocation.sh exit 0 (PASS); python3 tests/test-control-readiness.py exit 0; bash tests/test-sister-reconcile.sh exit 0; python3 tests/control-revocation.py . exit 0.

• I started a full local make test and killed it (exit 143) before it finished. No full-suite result of mine exists. The GitHub 'Test suite' check at the head is the tier 1-3 evidence.

• bench-ci: after source ~/.config/bench-ci/env, bench-ci list --all --fleet --commit 83fdd913e73ada4f0cb2fef89bc4a224018c0da7 printed 'queued 0, in progress 0, completed 0'. The same query for 2b2a016 also returned no jobs. The PR body quotes no bench-ci job.

• The PR body says: 'Still needed: real-systemd replacement/failure proof and the worker-to-coordinator tier4 gate on a reserved bench rig.' The Docker e2e stubs systemctl. The real-systemd path is the tests/e2e-real.sh control phase (usage line at tests/e2e-real.sh:29), which has not run on this head.

• Topology: detect-topology.py over the PR body, comments and reviews printed 'clean' (exit 0). Over the issue Stop stale receiver access when token revocation cannot restart the receiver #626 comments it also printed 'clean' (exit 0).

• Comment discipline: no added explanatory comments in the diff. The only added # lines are shebang strings and test labels.

• Issue Stop stale receiver access when token revocation cannot restart the receiver #626 acceptance criteria: (1) a revoked bearer cannot apply a write after failed replacement; (2) failure reaches every reconciliation caller and readiness confirms the replacement; (3) the stale receiver stays stopped or blocked and the firewall step still completes; (4) a real-write regression with a successful-replacement control. The tests cover these at stub level only.

• rigforge.sh:1452 (install_control converted from { } to ( )): error() (rigforge.sh:34-36) does exit 1. Inside the new subshell, the 'needs python3' error at line ~1471 now ends only the subshell and becomes _control_rc=1, so setup continues instead of aborting. The receiver is still stopped by the EXIT trap, so this is probably acceptable. State it under 'How I fixed it'.

• rigforge.sh:4139-4141, util/control-server.py (blank lines removed before if __name__, inside class _V6), tests/run.sh:9025: Comment deletion in functions the PR does not edit: the first three header lines of _apply_runtime were removed, leaving a comment that now starts mid-thought. There is also unrelated blank-line churn. The change is harmless, but the diff is wider than the issue.

Reviewed head: 83fdd913e73ada4f0cb2fef89bc4a224018c0da7.

What are the gaps

• systemd/rigforge-control.service.template:6-8; rigforge.sh:1452-1496 (install_control); PR body 'Still needed' paragraph: No real-systemd or bench-ci proof on the head. The diff adds Type=notify, NotifyAccess=main and TimeoutStartSec=15 to the receiver unit and rewrites install_control's stop, teardown, enable and restart sequence for the control plane and deploy path. Tiers 1-3 only stub systemctl (test-control-revocation.sh, tests/run.sh STUBS). bench-ci list --commit shows zero jobs for 83fdd91 and 2b2a016. The PR itself says the real-systemd and tier4 proof are still needed. Run a bench-ci tier4-e2e job on the head covering the control phase (tests/e2e-real.sh control) and the worker-to-coordinator contract gate. Quote its bench-ci evidence <id>@<sha> block in the body.
• rigforge.sh:1455-1464: Changed behaviour with no test: in the control-disabled branch systemctl disable --now rigforge-control.service rigforge-control-apply.path rigforge-control-upgrade.path ... || return 1 (formerly || true) names the upgrade path unit, which does not exist when control_upgrade is off. If systemctl exits non-zero for a missing unit (my inference; I could not run systemd on this Mac), the function returns before sudo rm -f "$svc" .... Unit files then survive and every later setup, apply and upgrade reports control failure. Separately, _stop_control_receiver || return 1 at line 1455 now fails a control-disabled host on a Linux box without a usable systemctl. test-control-revocation.sh only runs CONTROL_MODE=enabled, and tests/run.sh:7611,7636 use an always-succeeding systemctl stub. Add tests for the disabled branch with an absent upgrade unit and with a failing disable, and show the disabled-with-upgrade-off case passing on real systemd.

[pithead-reviewer]

PASS

What I investigated

Round-2 review of PR #640 (resolves #626) at head 164a1b5, covering the delta from 83fdd91 (cef9b66, bebd497, 609c568, 164a1b5). Every round-1 finding is fixed or answered with evidence that holds. The new tests fail without the fix and pass at the head. CI at the head is all green, including a real-systemd stage that enables control and then tears it down. Bench tier4-e2e job 3519 on 609c568 passed the control phase, and the only later change is one line of a test the bench does not run.

• git cat-file -e 83fdd913 printed present. git log --no-merges 83fdd913..164a1b54 lists 4 worker commits. The delta touches rigforge.sh (+7/-7), docs/pithead-integration.md (+3), tests/test-control-revocation.sh (+74), tests/test-stock-home.sh (+20), tests/control-revocation.py and tests/run.sh (+1 blank line).

• Round-1 finding (no real-systemd or bench proof): FIXED. gh pr checks 640 shows all 10 checks SUCCESS at 164a1b5, none skipped. In the CI run 42375651223 log (headSha 164a1b5), the 'Stock home setup (current)' job prints stock-home control GREEN: real systemd started the notify receiver, then tore control down with remote upgrade off. That stage asserts Type=notify, the receiver answering on 8082, then the receiver inactive and its unit files removed after disable.

• bench-ci evidence 3519@609c568fbd6f2e38ffee5d5ed25d61bd734e7518 printed exactly the block pasted in the PR body: tier4-e2e, phases connect/control/access-tokens, conclusion success, exit 0, rig restored, 15 pass, 0 fail, 0 skip. In the job log, the checkout line is rigforge@609c568…, the run fetches and checks out 609c568 in the rig's install tree, and the phases report phase connect: passed (3), phase control: passed (5) and phase access-tokens: passed (7).

• git diff --stat 609c568..164a1b54 shows only tests/test-stock-home.sh | 1 -. tier4-e2e does not run that file, so job 3519 covers the head.

• Round-1 finding (disabled branch named the absent upgrade unit, no test): FIXED. rigforge.sh:1458-1463 builds the disable list from unit files that exist. In a scratch worktree at the head, bash tests/test-control-revocation.sh exits 0. With rigforge.sh from 83fdd91 it exits 1 with FAIL: control-disabled teardown failed when the upgrade unit was absent.

• With only the new elif line (rigforge.sh:1484) reverted to its 83fdd91 form, the same test exits 1 with FAIL: a leftover upgrade service without its path unit failed the control reconcile.

• Round-1 fresh-host concern (_stop_control_receiver || return 1): answered. At rigforge.sh:1447 a not-found, inactive unit counts as stopped. The real-systemd control-enable stage on a host with no control unit passed at the head.

• Live-write regression: python3 tests/control-revocation.py with develop's rigforge.sh, util and systemd exits 1 at assert stale is None. At the head it exits 0.

• The CI Test suite log at the head shows control revocation regression: PASS and ✓ receiver revocation fails closed (#626). The test is registered at tests/run.sh:8874, and test-control-readiness.py is registered in the Makefile test-suite target.

• Round-1 note (subshell error()): closed. The PR body now states it under 'How I fixed it'.

• Round-1 note (comment and blank-line churn): answered. The tests/run.sh blank line is restored. The body explains that the removed _apply_runtime comment lines stay out because rigforge.sh and control-server.py are at their file-budget ceilings. No added comments in this delta.

• FILE_BUDGET_REQUIRE_BASE=1 bash scripts/lint-file-budget.sh prints file budget OK. make lint-topology prints topology classes OK. shellcheck --severity=warning and shfmt -i 4 -d on rigforge.sh, test-control-revocation.sh and test-stock-home.sh exit 0.

• The topology detector reported clean on the PR body, comments and reviews, and clean on the issue Stop stale receiver access when token revocation cannot restart the receiver #626 comments.

• Issue Stop stale receiver access when token revocation cannot restart the receiver #626 acceptance criteria and where each is met: the stale receiver is stopped before replacement and on any failure (EXIT trap); failure reaches setup, upgrade and apply (_control_rc); readiness uses Type=notify, and READY=1 is sent only after credentials load and the listener binds; the firewall reconcile still runs; the regression test attempts an actual write through the stale receiver after revocation, with successful replacement as its control. All are met in the code and tests above.

• rigforge.sh:1459: With control disabled, teardown runs only if rigforge-control.service, rigforge-control-apply.path or rigforge-control-upgrade.path exists. A leftover rigforge-control-apply.service or rigforge-control-upgrade.service on its own is never removed. The pre-PR code had the same gate, so this is not a regression, and both units are inert without their path or receiver.

Reviewed head: 164a1b541a7c48cc71bb6bbd8c8797a38ee6c637.

[pithead-reviewer]

Owner brief for 164a1b541a7c

What it does. When a named access token is revoked and the control receiver can't be replaced, setup, upgrade and apply now stop the old receiver and exit non-zero. Before, the old process kept running with the revoked token and could still accept payout writes. The receiver is also now a systemd Type=notify unit, so it only counts as started after it has loaded its credentials and bound its port. Turning control off now removes only the unit files that exist.

Why. Closes #626, a security blocker in milestone v1.18.1. Receiver restart failures were suppressed and apply threw away control failures. A revoked bearer therefore kept payout-write authority until the receiver was replaced.

Risk. Moderate blast radius. It rewrites install_control and changes the receiver's systemd unit, so a control-step failure now fails setup, upgrade and apply. This includes rollback for control-apply and remote upgrade. If systemd refuses both stops, even after SIGKILL, the receiver can keep running. In that case the run warns and fails. The docs state that an earlier-accepted change is still applied, and that a run aborting before the control step leaves the old receiver serving. Nothing here is irreversible.

Proof. CI: 11 success. Reviewer: PASS at 164a1b5 by pithead-reviewer, round 2 (recorded 2026-11-07T13:22:47Z). Bench: job 3519: success on 609c568, not the head.

What to check.

• rigforge.sh:1444-1494: _stop_control_receiver and the install_control subshell with its EXIT trap. Confirm the fail-closed ordering: stop first, then change units, and stop again on any failure.
• rigforge.sh:1470: error() inside the subshell now ends only install_control, so a missing python3 no longer aborts setup. Confirm that is acceptable.
• systemd/rigforge-control.service.template:6-8 and util/control-server.py end of file: Type=notify with a 15s start timeout. A slow start would now count as failure and stop the receiver.
• tests/test-stock-home.sh:113-131: the real-systemd enable-then-disable stage. It passed in CI at head. Bench job 3519 ran on the previous commit; the only change since is this test file, which that bench doesn't run.

Verdict. merge: round 2 reviewer PASS covers the head. The only open note is that a lone leftover apply or upgrade service file isn't removed when control is off. That gate existed before this PR

[pithead-reviewer]

PASS

What I investigated

Re-review of #640 (closes #626, stale control receiver kept accepting a revoked bearer) at head 20990de. Reviewed the worker's commits since 83fdd91 (cef9b66, bebd497, 609c568, 164a1b5) and the develop merge. Both round-1 return findings are fixed. The disabled-teardown and leftover-upgrade-service fixes each go red with the old hunk and green at the head. Real systemd covers the enable-then-disable control stage at the head, and the tier4-e2e control job 3519 covers 609c568, whose code outside the tests is the same as the head's.

• git cat-file -e 83fdd91: present. git log --no-merges 83fdd91..20990de: worker commits cef9b66, bebd497, 609c568, 164a1b5; the rest are develop (Preserve thermal watchdog retries after failed service commands #638 Fix rigforge#633 #642 Verify cached jq before use and preserve fresh coverage proofs #643 Protect root upgrade throttle state #644 fix(tests): wait for the sister snapshot to recover before the network wire check (#563) #654)

• PR delta vs merge-base at 609c568 and at 20990de, excluding tests and Makefile: the same apart from hunk line offsets. The develop side (b1c25d6..74d9a4b) touches _grub_proposed, _control_upgrade_throttle_ok and watchdog, not install_control

• gh pr checks 640: 10 checks pass (Lint shellcheck+shfmt, Test suite, End-to-end Docker, Coverage, Stock home current and historical, gitleaks, zizmor, markdownlint, yamllint); none skipped

• bench-ci evidence 3519@609c568f: tier4-e2e, phases connect/control/access-tokens, completed, success, exit 0, 15 pass 0 fail 0 skip, baseline verified, rig restored. Matches the PR's pasted block

• bench-ci log 3519: 'phase connect: passed (3 assertions)', 'phase control: passed (5 assertions)', 'phase access-tokens: passed (7 assertions)', 'conclusion: success'

• CI job 128459700535 (Stock home setup current) at 20990de prints 'stock-home control GREEN: real systemd started the notify receiver, then tore control down with remote upgrade off'. The script runs under set -Eeuo pipefail. Round-1 finding 1 is fixed

• CI job 128459700384 (Test suite) at 20990de: '✓ receiver revocation fails closed (Stop stale receiver access when token revocation cannot restart the receiver #626)', 'rigforge tests: 2477 passed, 0 failed'

• Scratch worktree at 20990de with 83fdd91's disabled block restored: bash tests/test-control-revocation.sh exits 1, 'FAIL: control-disabled teardown failed when the upgrade unit was absent'. With the old upgrade elif restored: exits 1, 'FAIL: a leftover upgrade service without its path unit failed the control reconcile'. At the head: exits 0, 'control revocation regression: PASS'. Round-1 finding 2 is fixed

• Round-1 finding 2, second half (receiver stop on a host without systemd): the answer holds. rigforge.sh:15 is set -Eeuo pipefail, and setup calls install_service (unguarded sudo systemctl daemon-reload) at line 1999, before install_control

• Round-1 note 3 (subshell error()): now stated under 'How I fixed it'. Closed

• Round-1 note 4 (comment deletion and blank-line churn): the _apply_runtime comment removal is disclosed and tied to the file-budget ceilings. The tests/run.sh blank line is restored in 164a1b5. Already ruled as a note

• git diff 74d9a4b..20990de -- rigforge.sh util systemd, grep for added '#' lines: none

• detect-topology.py on the PR body, PR comments and Stop stale receiver access when token revocation cannot restart the receiver #626 comments: rc=0 on each; a control probe with an IP gives rc=1

• FILE_BUDGET_REQUIRE_BASE=1 bash scripts/lint-file-budget.sh: 'file budget OK'

• apply now runs install_control with stdout hidden, as develop did, but keeps stderr warnings and the failure rc (rigforge.sh:4212)

• PR body, bench section: Job 3519 tested 609c568, not the head 20990de. The PR's code outside the tests is the same at both, the later commits are tests/test-stock-home.sh and a develop merge that does not touch install_control, so it is accepted. A tier4 run on the merged head would make the body match the head exactly.

• rigforge.sh:1460-1463: The disabled teardown still keys on control.service, apply.path and upgrade.path. A lone leftover rigforge-control-apply.service, with no other unit file present, is not removed. This is unchanged from develop's condition, not a regression.

Reviewed head: 20990de3dd874df2369b9b8db92b12a3b8c49932.

[pithead-reviewer]

Owner brief for 20990de3dd87

What it does. When a revoked access token can't be cleared because the control receiver fails to restart, setup, upgrade and apply now stop the old receiver, still run the firewall step, and exit with an error. The receiver's systemd unit now waits for a ready signal, which the receiver sends only after it has loaded its credentials and bound its port. With control turned off, teardown now disables only the unit files that exist.

Why. Closes #626, a v1.18.1 blocker with the bug and security labels. Before, restart, enable and stop failures were suppressed and apply discarded control failures. A revoked named bearer could keep submitting payout writes through the old receiver, and the root consumer applied them.

Risk. Setup, upgrade and apply on Linux now exit non-zero when the control receiver can't be reconciled. Control-apply and remote upgrade roll back in that case, so a flaky systemd could newly fail runs that used to pass. If systemd refuses both stops even after SIGKILL, the receiver keeps running, and the run only warns and fails. The new notify unit has a 15-second start timeout, which a slow-starting rig could hit. The change is reversible, with no data migration.

Proof. CI: 11 success. Reviewer: PASS at 20990de by pithead-reviewer, round 2 (recorded 2026-11-07T16:43:28Z). Bench: job 3519: success on 609c568, not the head.

What to check.

• rigforge.sh:1444-1494: _stop_control_receiver and the install_control subshell with its EXIT trap. An error() call now ends only the subshell, so setup continues instead of aborting.
• rigforge.sh:1460-1466: the control-disabled teardown now returns failure instead of ignoring it. It keys on which unit files exist, so a lone leftover rigforge-control-apply.service is still not removed. That matches develop's behaviour.
• systemd/rigforge-control.service.template:6-8 and util/control-server.py (end of file): Type=notify with TimeoutStartSec=15, and READY=1 sent only after the listener is bound.
• rigforge.sh apply() near line 4218: a control failure now sets the apply failure reason and the exit status.

Verdict. merge: bench job 3519 ran on the earlier head 609c568, not 20990de. The later changes are a test file and a develop merge that doesn't touch the control code

[pithead-reviewer]

PASS

What I investigated

Round 4 re-review of PR #640 (closes #626) at head 6cb1915. 83fdd91 is in the clone. The worker's own commits since that review are cef9b66, bebd497, 609c568 and 164a1b5; the other commits in the range came in through two develop merges (20990de, 6cb1915). Both round-1 returns are fixed. The disabled-control teardown now names only units that exist, and each new teardown hunk has a test that fails with that hunk reverted and passes at head. Real-systemd proof is in place: the Stock home (current) check exercises the control stage at head, and tier4-e2e job 3519 covers connect, control and access-tokens. The PR's own product hunks are byte-identical between 609c568 and head. All 10 GitHub checks are green at head, and the topology scan is clean.

• git cat-file -e 83fdd913… succeeded. git log --graph shows the worker's commits as cef9b66, bebd497, 609c568 and 164a1b5. 6d41d94, 8f21693, 1156f1c, dab40f6, 74d9a4b and 780727e are all ancestors of origin/develop, so they arrived through the 20990de and 6cb1915 merges.

• git diff 83fdd913..164a1b5 --stat: 6 files changed, +110/-10 (rigforge.sh, tests/control-revocation.py, tests/test-control-revocation.sh, tests/test-stock-home.sh, tests/run.sh, docs/pithead-integration.md).

• gh pr checks 640 --json: all 10 checks pass (both Lint jobs, yamllint, Test suite, End-to-end (Docker), Coverage, Stock home setup (current) and (historical), gitleaks, zizmor). The Stock home (current) job 128477502355 has head_sha 6cb1915 and conclusion success. Its log contains 'stock-home control GREEN: real systemd started the notify receiver, then tore control down with remote upgrade off'.

• bench-ci evidence 3519@609c568f… printed a block identical to the one in the PR: tier4-e2e with phases connect, control and access-tokens; completed, conclusion success, exit 0; rows 15 pass, 0 fail, 0 skip; rig restored: yes. The job log shows the checkout of 609c568 and that the connect (3 assertions), control (5) and access-tokens (7) phases passed. bench-ci list --commit 6cb1915… returned 0 jobs.

• Coverage of the head by job 3519: for rigforge.sh, util/control-server.py, systemd/rigforge-control.service.template, tests/control-revocation.py, tests/test-control-revocation.sh and docs/pithead-integration.md, the +/- lines of git diff <merge-base 609c568,develop> 609c568 match the +/- lines of git diff origin/develop HEAD exactly, so all six report the same. 609c568..164a1b5 changes only tests/test-stock-home.sh, which tier4 does not run. The rigforge.sh hunks the merges brought in (_grub_proposed, control_apply, control_upgrade, watchdog) are develop's own code.

• Red/green in a scratch worktree at head. Reverting the worker's rigforge.sh diff (83fdd91..164a1b5) made bash tests/test-control-revocation.sh exit 1 with 'FAIL: control-disabled teardown failed when the upgrade unit was absent'. Reverting only the upgrade-off elif hunk failed with 'FAIL: a leftover upgrade service without its path unit failed the control reconcile'. At head it exits 0 and prints 'control revocation regression: PASS'.

• Round 1 [return], no real-systemd or bench-ci proof: FIXED. Job 3519 covers the control phase on 609c568, and the PR's hunks are unchanged since that commit. The Stock home (current) check at head runs apply with control enabled and confirms the receiver is active as Type=notify (tests/test-stock-home.sh:113-131). It then disables control and confirms the receiver is inactive and its unit files are gone.

• Round 1 [return], disabled branch named the absent upgrade path unit with no test: FIXED. rigforge.sh:1459-1463 builds the units list from the files that exist. tests/test-control-revocation.sh disabled_case absent-upgrade and disabled_case fail cover both cases: an absent unit is not named, and a failing disable returns non-zero and stops the receiver again on exit. The red run is quoted above. The fail-closed stop on hosts without systemd (609c568) is in rigforge.sh:1445-1451.

• Round 1 [note], subshell error(): ANSWERED. The PR body states it under 'How I fixed it'.

• Round 1 [note], comment and blank-line churn: ANSWERED. The tests/run.sh:9026 blank line is restored in this range. The removed comments sit in hunks the PR edits, which the comment policy requires, and the body states they stay removed.

• apply's install_control >/dev/null || _control_rc=1 (rigforge.sh:4204): develop called install_control >/dev/null 2>&1 || true. stdout stays quiet as before, while stderr (warn/error) and the failure rc are now kept. This is consistent with the fix.

• Comment scan of git diff origin/develop...HEAD: the only added # line is a # shellcheck disable= pragma, which is functional.

• FILE_BUDGET_REQUIRE_BASE=1 bash scripts/lint-file-budget.sh at head: rc=0, 'File budget OK'.

• Issue Stop stale receiver access when token revocation cannot restart the receiver #626 acceptance criteria are covered by the current PR diff and its tests: control-revocation.py makes an authorized write through the stale receiver with a fresh receiver as the control, and the callers' firewall continuation is tested in test-control-revocation.sh caller_case. The only non-App commenter on the PR is the owner (2026-11-04 rebase note). Both issue comments are from the pithead-developer App.

• detect-topology.py on the PR body, the PR comments and the issue comments: rc 0 for each (no TOPOLOGY hit). A canary line with an IP address and a home path returned rc 1, which confirms the detector catches such text.

• Ponytail pass over the worker's diff: the units loop and the brace-grouped elif are minimal, and I found no extra machinery.

• PR body, 'Real systemd' paragraph: One sentence says the stock-home stage 'asserts the "Writable control path disabled" teardown'. The same paragraph, and tests/test-stock-home.sh:126-131 at head, say it asserts by state only: receiver inactive, unit files removed. Delete the log-line claim so the body matches the test.

Reviewed head: 6cb1915bf863a04147ba6ded35eacb1d7477124d.

[pithead-reviewer]

Owner brief for 6cb1915bf863

What it does. When a named access token is removed and the writable control receiver can't be replaced, setup, upgrade and apply now stop the old receiver. They finish the firewall step, then exit with an error. Before, the old process kept the removed token and went on accepting payout changes. The receiver's systemd unit now reports ready only after it has loaded credentials and bound its port. With control turned off, teardown now disables only the unit files that exist.

Why. Closes #626, a security blocker in milestone v1.18.1. Restart failures were suppressed with || true and apply discarded the control result. A revoked bearer therefore kept payout-write authority whenever replacement failed.

Risk. Setup, upgrade and apply now fail when control reconciliation fails, where they used to succeed quietly. Through apply, control-apply and remote upgrade will roll back on that failure. install_control is now a subshell, so a "needs python3" error no longer aborts setup. It only fails the control step. If systemd refuses both stop attempts, the receiver can keep running. The run warns and fails, and nothing more is attempted. The PR accepts this as a known gap. No data loss or irreversible step found. The only receiver-unit change is the notify settings.

Proof. CI: 11 success. Reviewer: PASS at 6cb1915 by pithead-reviewer, round 2 (recorded 2026-11-07T17:58:37Z). Bench: job 3519: success on 609c568, not the head.

What to check.

• rigforge.sh:1444-1496: _stop_control_receiver and the EXIT trap in install_control. Check that every failure path returns non-zero and the stop-then-kill-then-stop order is sound.
• rigforge.sh:1452-1464: the control-disabled teardown. It now builds the unit list from files that exist and returns 1 if disable or rm fails. The reviewer notes a lone leftover apply.service or upgrade.service still isn't cleaned. That matches develop and isn't a regression.
• systemd/rigforge-control.service.template:6-8 with util/control-server.py:395-400: Type=notify and TimeoutStartSec=15. Check that READY=1 is sent only after the bind succeeds, and that 15s is long enough on slow rigs.
• docs/pithead-integration.md:242-249: the stated revocation boundary. A change accepted earlier is still applied, and a run that aborts before the control step leaves the old receiver serving.

Verdict. merge: the tier4 bench job ran on an earlier commit, not the head. The reviewer confirmed the product code is identical between that commit and the head