Skip to content

fix: don't restart a cluster worker twice when ready arrives during stop - #6179

Open
cestercian wants to merge 2 commits into
Unitech:developmentfrom
cestercian:fix/late-ready-during-stop
Open

cestercian wants to merge 2 commits into
Unitech:developmentfrom
cestercian:fix/late-ready-during-stop

Conversation

@cestercian

Copy link
Copy Markdown
Contributor

When a cluster worker sends ready (or hits the exit-before-ready path) while it's already STOPPING, pm2 flipped it back to online. The stop's exit then got treated like a crash, so restart_task kicked in on top of the reload/restart path starting the worker, and you ended up with an orphan worker in the round robin.

This keeps a late ready from promoting a stopping worker, so the stop exit is handled as a normal stop.

Tests: added test/programmatic/issue_6178_late_ready_during_stop.mocha.js with a fixture that sends ready late during stop, and checks there's only one start and no stray worker.

Fixes #6178

A ready message (or the exit-before-ready path) that lands while a worker
is STOPPING was promoted back to online, so the stop's exit was handled as
a crash. That scheduled restart_task and the reload/restart path started
the worker too, leaving an orphan in the round robin.

Fixes Unitech#6178
@Unitech

Unitech commented Nov 3, 2026

Copy link
Copy Markdown
Owner

Thanks, the fix is right. I reproduced #6178 with the issue's CLI steps: on development the daemon ends up with 3 workers for 2 tracked, with this PR it has 2. The full Docker suite passes on Node 20 (184/184).

The PR fixes two more things:

  • pm2 stop and pm2 delete on a wait_ready app that hasn't sent ready yet. Since 7.0.4 the app gets started again, in fork mode too.
  • With a restart delay set, the handleExit change keeps autorestart: false, stop_exit_codes and too-many-restarts apps from showing waiting restart forever.

A few changes before merging:

  1. Please retarget the PR to development. lib/God.js is the same on both branches, so it applies as is.

  2. The vizion path still runs during a stop. With PM2_ENABLE_GIT_PARSING=true, finalizeProcedure sees STOPPING and cancels, but leaves vizion_running set to true. The next start then takes the shortcut, sends online while the status is still stopping, and never computes versioning again. Please guard the whole block:

    if (!stopping) {
      if (cst.ENABLE_GIT_PARSING === true &&
          proc.pm2_env.vizion !== false && proc.pm2_env.vizion !== "false")
        God.finalizeProcedure(proc);
      else
        God.notify('online', proc);
    }
  3. Fixture order. process.send('booted') runs before process.on('SIGINT') is installed. Under CI load, the test can send SIGINT in that gap. The worker then dies right away and its ready never arrives during STOPPING. Test 1 still passes, but only through the exit-before-ready path. Please install the SIGINT and message handlers first, then send booted.

  4. Missing assertions:

    • Callback count. Check that the first restartProcessId callback is called exactly once. Otherwise a readyCb that returns early during a stop without calling cb would pass all 4 tests and leave pm2 restart hanging.
    • Fork mode. readyCb is shared, and on development fork mode had the same orphan, both for ready during stop and for exit before ready. One fork-mode case would cover it.
    • The handleExit change. No test fails when it is reverted. Two possible checks:
      • prev_restart_delay stays unset after the reload;
      • a short test where an app with exp_backoff_restart_delay crashes past max_restarts and must end errored, not waiting restart.
  5. CI wiring.

    • Add runUnitTest $D/issue_6178_late_ready_during_stop.mocha.js to test/unit.sh, after cluster_log_lifecycle.
    • Leaving it out of test/windows.sh is correct, because a SIGINT handler can't delay the exit there. A # Excluded: comment, like the one for signals.js, would say why.
  6. Optional. The test file is 363 lines. You could cut it down:

    • one shared waitFor(pred, ms, cb) helper instead of the six poll loops;
    • one driver for tests 1 and 2;
    • drop the unused go-exit handler.

We'll add the CHANGELOG lines on our side when merging.

@cestercian
cestercian changed the base branch from master to development November 3, 2026 15:15
@cestercian

Copy link
Copy Markdown
Contributor Author

Thanks a lot for digging into this and running the full Docker suite. Retargeted to development. I've got the rest done locally (vizion guard, fixture order, fork mode, callback count, the handleExit checks, unit.sh/windows.sh wiring and a slimmer test) and will push it shortly.

…llback and backoff checks

- skip the whole finalizeProcedure/notify block while stopping so
  vizion_running is never left set
- install the SIGINT and message handlers before sending 'booted'
- assert the restart callback runs once, cover fork mode, and check that
  an intentional stop doesn't advance the backoff and that an overlimit
  app with a backoff ends errored
- run the test in unit.sh, note why it's excluded from windows.sh
- shared waitFor/holdFor helpers, one driver for the stop cases
@cestercian

Copy link
Copy Markdown
Contributor Author

pushed the review fixes: vizion is guarded while stopping, handlers go in before booted, callback once plus the fork mode and backoff checks, and the unit.sh entry. also cleaned up the test helpers. lmk if anything else needs a tweak

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reloading a cluster worker that is still booting starts two workers for one pm_id and orphans one (7.1.1, 7.0.4)

2 participants