Windows: `pm2 restart` starts fork-mode application twice with `shutdown_with_message`

[GIDustin]

Disclaimer: I am a hobbyist programmer. I had an issue with pm2 starting multiple instances of my app. After workarounds that didn't solve the issue, I used controversial "tools" to find the problem, and the solution generated did fix my issue. I am not the caliber of programmer to contribute to major public repos in the form of pull requests, but I can at least post an issue with what was found, and let the powers to be do with it as they please.

Windows: pm2 restart starts fork-mode application twice with shutdown_with_message

Description

On Windows, restarting a fork-mode application configured with shutdown_with_message can start two replacement processes.

The original process shuts down cleanly and exits with code 0. PM2 detects that its process tree has stopped and starts the intended replacement. The old child’s exit event is then delivered after the replacement has occupied the same PM2 ID.

God.handleExit() looks up the process using only the PM2 ID. It therefore retrieves the new replacement process instead of recognizing that the event belongs to the old child. Since the replacement is already launching, PM2 interprets the stale exit event as an unexpected exit and schedules another start.

The two replacements then compete for the same listening port, causing one to fail with EADDRINUSE.

Environment

• OS: Windows
• PM2: 7.0.3
• Node.js: 24.16.0
• Execution mode: fork
• shutdown_with_message: enabled
• Application listens for the shutdown message, performs graceful cleanup, and calls process.exit(0)

Application shutdown handler

The application follows PM2's documented Windows shutdown pattern:

process.on('message', function(msg) {
  if (msg == 'shutdown') {
    gracefulShutdown().then(function() {
      process.exit(0);
    });
  }
});

Steps to reproduce

1. Start a fork-mode Node.js application on Windows with shutdown_with_message: true.
2. Have the application listen for the shutdown message and exit cleanly after graceful cleanup.
3. Run:

pm2 restart wwwDEV

4. Observe that PM2 starts the application twice.

Actual behavior

This is the relevant PM2/application log from an unpatched restart:

PM2       | Stopping app:wwwDEV id:1
1|wwwDEV  | Shutdown Message Received
1|wwwDEV  | Starting Server Shutdown
1|wwwDEV  | Running Shutdown Functions...
1|wwwDEV  | Dev=>CronJobs: Shutting Down...
1|wwwDEV  | Done Running Shutdown Functions: 3.552ms
1|wwwDEV  | Server Shutdown Complete. Exiting by running process.exit(0)
PM2       | pid=9096 msg=process tree killed (1 pids)
PM2       | App [wwwDEV:1] starting in -fork mode-
PM2       | App [wwwDEV:1] exited with code [0] via signal [SIGINT]
PM2       | App [wwwDEV:1] will restart in 250ms
PM2       | App [wwwDEV:1] starting in -fork mode-
1|wwwDEV  | Dev=>Trying to Listen on Port 442
1|wwwDEV  | Initializing Robyn Modules...
1|wwwDEV  | Dev=>Trying to Listen on Port 442
1|wwwDEV  | Initializing Robyn Modules...

Both processes initialize concurrently. One subsequently fails to bind:

1|wwwDEV  | HTTPS Error:
1|wwwDEV  | Error: listen EADDRINUSE: address already in use :::442
1|wwwDEV  |     at Server.setupListenHandle [as _listen2] (node:net:2008:16)
1|wwwDEV  |     at listenInCluster (node:net:2065:12)
1|wwwDEV  |     at Server.listen (node:net:2170:7) {
1|wwwDEV  |   code: 'EADDRINUSE',
1|wwwDEV  |   errno: -4091,
1|wwwDEV  |   syscall: 'listen',
1|wwwDEV  |   address: '::',
1|wwwDEV  |   port: 442
1|wwwDEV  | }

The process churn can also cause PM2's metrics polling to query a process that has already disappeared:

PM2       | Error caught while calling pidusage
PM2       | Error: No matching pid found
PM2       |     at ...\pm2\node_modules\pidusage\lib\gwmi.js:79:21
PM2       |     at ChildProcess.<anonymous> (...\pidusage\lib\bin.js:44:5) {
PM2       |   code: 'ENOENT'
PM2       | }

Expected behavior

pm2 restart should:

1. Send the shutdown message to the old process.
2. Wait for it to exit.
3. Start exactly one replacement.
4. Ignore any delayed lifecycle event belonging to the old child.

A clean exit from the old process must not trigger an additional automatic restart after the explicit replacement has already started.

Suspected root cause

In lib/God.js, God.handleExit() receives the child object that emitted the exit event:

God.handleExit = function handleExit(clu, exit_code, kill_signal) {

It then discards that identity and looks up the active process using only the PM2 ID:

var proc = this.clusters_db[clu.pm2_env.pm_id];

On Windows, God.processIsDead() can detect that the PID has disappeared and invoke the restart callback before Node delivers the old child’s exit event.

The sequence is therefore:

1. restartProcessId() calls stopProcessId().
2. stopProcessId() calls killProcess().
3. killProcess() sends the Windows shutdown message.
4. processIsDead() polls until the old PID disappears.
5. The stop callback runs and restartProcessId() starts the replacement.
6. executeApp() stores the replacement in clusters_db under the same PM2 ID.
7. The old child finally emits its delayed exit event.
8. handleExit(oldChild) retrieves the replacement from clusters_db.
9. The replacement is launching/online, so stopping evaluates to false.
10. PM2 schedules another automatic restart and mutates the replacement’s state.

In addition to starting another process, the stale event can mark the replacement stopped and remove the PID file belonging to the current PM2 ID.

Suggested fix

Before processing an exit event, verify that the child which emitted it is still the child registered under that PM2 ID.

In lib/God.js, immediately after retrieving proc, add an object-identity check:

 God.handleExit = function handleExit(clu, exit_code, kill_signal) {
   console.log(`App [${clu.pm2_env.name}:${clu.pm2_env.pm_id}] exited with code [${exit_code}] via signal [${kill_signal || 'SIGINT'}]`)

   var proc = this.clusters_db[clu.pm2_env.pm_id];

   if (!proc) {
     console.error('Process undefined ? with process id ', clu.pm2_env.pm_id);
     return false;
   }

+  // processIsDead may complete and start a replacement before the old
+  // ChildProcess emits its exit event. Do not let an event from the old
+  // child mutate or restart the replacement occupying the same PM2 ID.
+  if (proc !== clu) {
+    console.log(
+      'Ignoring stale exit event for app [%s:%s] pid [%s]',
+      clu.pm2_env.name,
+      clu.pm2_env.pm_id,
+      clu.process && clu.process.pid
+    );
+    return false;
+  }
+
   var stopExitCodes = proc.pm2_env.stop_exit_codes !== undefined &&

The important part of the fix is:

if (proc !== clu) {
  return false;
}

The message is optional.

Results after applying the suggested fix

After applying the identity check locally to PM2 7.0.3 and restarting the PM2 daemon, repeated pm2 restart wwwDEV commands produced the following:

PM2      | Stopping app:wwwDEV id:1
1|wwwDEV | Shutdown Message Received
1|wwwDEV | Starting Server Shutdown
1|wwwDEV | Running Shutdown Functions...
1|wwwDEV |  - Running Shutdown Functions [ 'OpenConnections', 'OpenForbiddenConnections', 'CronJobs' ]
1|wwwDEV | Dev=>CronJobs: Shutting Down...
1|wwwDEV | Server Closed - HTTPSserver
1|wwwDEV |  - Shutdown Function Complete OpenForbiddenConnections: 1.056ms
1|wwwDEV |  - Shutdown Function Complete CronJobs: 1.115ms
1|wwwDEV |  - Shutdown Function Complete OpenConnections: 1.815ms
1|wwwDEV | Done Running Shutdown Functions: 4.479ms
1|wwwDEV | Server Shutdown Complete. Exiting by running process.exit(0)
PM2      | pid=12692 msg=process tree killed (1 pids)
PM2      | App [wwwDEV:1] starting in -fork mode-
PM2      | App [wwwDEV:1] exited with code [0] via signal [SIGINT]
PM2      | Ignoring stale exit event for app [wwwDEV:1] pid [0]
1|wwwDEV | Dev=>Trying to Listen on Port 442
1|wwwDEV | Initializing Robyn Modules...

With the patch:

• Only one replacement was started.
• Only one application initialization occurred.
• No EADDRINUSE occurred.
• The stale exit event was detected and ignored.
• Repeating the restart several times continued to work correctly.

The logged stale PID is 0 because stopProcessId() clears the old process record's PID before its delayed exit event is delivered. The object identity still distinguishes the old child from the active replacement.