Datanode add restart for single node / two nodes

[moesterheld]

Description

The rolling restart job can now also be used for 1 or 2 nodes. These will however, be restarted without a rolling restart.
In the case of two nodes, they will be restarted sequentially. In case there is a error, this should then already be visible when restarting the first node.
In both cases message processing is paused during restart.

The state machine has been updated for this with two new states PAUSING_PROCESSING and RESUMING_PROCESSING that are wired into the machine (replication states are skipped for 1/2 nodes)

/nocl part of the datanode upgrade process

Motivation and Context

resolves https://github.com/Graylog2/graylog-plugin-enterprise/issues/14881

How Has This Been Tested?

added tests, manually

Screenshots (if appropriate):

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Refactoring (non-breaking change)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have requested a documentation update.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.

[Copilot]

Pull request overview

This PR extends the DataNode rolling restart workflow to support small clusters (1–2 DataNodes) by pausing Graylog message processing for the duration of the restart, rather than relying on shard replication/relocation. It updates the rolling-restart state machine and execution job to branch between the “replication path” (>= 3 nodes) and the “pause processing path” (<= 2 nodes), and adds tests for the new behavior.

Changes:

• Add new rolling-restart states/triggers for pausing/resuming message processing and wire them into the state machine.
• Update the rolling restart job execution logic to pause/resume processing for 1–2 node clusters and skip replication-related steps.
• Add/extend unit tests covering the small-cluster path and add tests for the new pause/resume actions (including ephemeral token handling).

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
graylog2-server/src/test/java/org/graylog2/datanode/restart/RollingRestartStateMachineBuilderTest.java	Adds state machine transition tests for the new small-cluster pause/resume states and triggers.
graylog2-server/src/test/java/org/graylog2/datanode/restart/RollingRestartJobHandlerTest.java	Updates precondition/start behavior tests and validates that small clusters start in PAUSING_PROCESSING.
graylog2-server/src/test/java/org/graylog2/datanode/restart/RollingRestartExecutionJobTest.java	Adds execution-path tests for pausing/resuming processing and skipping replication steps in 1–2 node runs.
graylog2-server/src/test/java/org/graylog2/datanode/restart/RollingRestartActionsTest.java	New tests for ephemeral token minting + fan-out pause/resume + revocation behavior.
graylog2-server/src/main/java/org/graylog2/rest/resources/datanodes/RollingRestartResource.java	Uses getCurrentUser().getName() for triggeredBy instead of Shiro SecurityUtils principal access.
graylog2-server/src/main/java/org/graylog2/datanode/restart/RollingRestartTrigger.java	Adds triggers needed for the small-cluster branch (NO_MORE_NODES_RESUME, NODE_JOINED_NO_REPLICATION).
graylog2-server/src/main/java/org/graylog2/datanode/restart/RollingRestartStateMachineBuilder.java	Adds transitions for PAUSING_PROCESSING and RESUMING_PROCESSING, and small-cluster-specific trigger routing.
graylog2-server/src/main/java/org/graylog2/datanode/restart/RollingRestartState.java	Adds new states PAUSING_PROCESSING and RESUMING_PROCESSING.
graylog2-server/src/main/java/org/graylog2/datanode/restart/RollingRestartJobHandler.java	Allows starting runs with 1–2 DataNodes (fails only on 0 nodes) and initializes the correct initial state + pauseProcessing flag.
graylog2-server/src/main/java/org/graylog2/datanode/restart/RollingRestartExecutionJob.java	Implements the small-cluster execution branch (pause/resume processing; skip replication gates) and persists pause_processing.
graylog2-server/src/main/java/org/graylog2/datanode/restart/RollingRestartActions.java	Adds message-processing pause/resume fan-out using a short-lived access token minted + revoked per operation.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[todvora]

This seems like a valid point, @moesterheld. Is this a leftover from testing, or are we sure that 5min is good enough, even for big nodes/clusters?

[moesterheld]

I intentionally lowered this, this is the wait time for one node to start up and an error bubbling up. If a node doesn't rejoin after 5 minutes, I think we should consider this a problem

[moesterheld]

Sorry, ok I see what you mean... It didn't occur to me that reenabling allocation could take this long. Let me think about a better way for handling this, maybe based on cluster size. It doesn't make sense to wait 30 minutes before going into an error state if we don't need to