Fix runtime-rule (MAL/LAL) hot-update in no-init mode and k8s cluster node identity

.github/workflows/skywalking.yaml

@@ -399,8 +399,9 @@ jobs:
             env: ES_VERSION=8.18.8
           - name: Runtime Rule LAL Hot-Update
             config: test/e2e-v2/cases/runtime-rule/lal/e2e.yaml
-          - name: Runtime Rule Cluster Convergence
+          - name: Runtime Rule Cluster (kind)
             config: test/e2e-v2/cases/runtime-rule/cluster/e2e.yaml
+            runs-on: ubuntu-24.04
           - name: DSL Debug API — MAL
             config: test/e2e-v2/cases/dsl-debugging/mal/e2e.yaml
           - name: DSL Debug API — OAL

docs/en/changes/changes.md

@@ -248,6 +248,8 @@
   refcount-tracked and unregistered when the last declaring rule is removed. See
   [runtime-rule-hot-update.md#dynamic-layers](../concepts-and-designs/runtime-rule-hot-update.md)
   for the conflict rules and limitations.
+* Fix: runtime-rule (MAL/LAL hot-update) schema changes now work in `no-init` mode — the deployment mode every production cluster runs. Previously a runtime `addOrUpdate` that introduced a new metric blocked forever in the storage installer's init-node poll loop (`ModelInstaller.whenCreating`) on a `no-init` OAP, because the gate keyed off `RunningMode` rather than the operation's intent; the `/delete?mode=revertToBundled` recreate and BanyanDB in-place shape updates were dead the same way. The poll loop is now gated on a new `StorageManipulationOpt.Flags.deferDDLToInitNode` bit set only on the static boot-time `schemaCreateIfAbsent()` opt (DRYed into `ModelInstaller.deferDDLToInitNode(opt)` and reused by the BanyanDB shape-check / group-DDL gates), so the runtime-rule opts (`withSchemaChange` / `verifySchemaOnly` / `withoutSchemaChange`) are driven by their flags and by cluster main-ness — `no-init` and `default` no longer differ for DSL DDL; `init` mode stays the dedicated initializer. `DSLManager.tickStorageOpt` is collapsed accordingly (main → `withSchemaChange`, peer → `verifySchemaOnly` at boot / `withoutSchemaChange` on tick).
+* Fix: runtime-rule cross-node writes no longer fail with `HTTP 400 forward_self_loop` on a multi-replica Kubernetes cluster. Every OAP replica shared the cluster `selfNodeId` `0.0.0.0_11800` (derived from the `0.0.0.0` agent gRPC bind host via `TelemetryRelatedContext`), so the main's self-loop guard rejected a legitimate peer-to-peer Forward as if it had looped back. The runtime-rule node identity now prefers the unique per-pod `SKYWALKING_COLLECTOR_UID` (the pod UID injected by the helm chart / swck operator from `metadata.uid`), resolved in `start()` before any apply, and falls back to the telemetry id for non-k8s deployments. Adds a kind-based no-init cluster e2e (`test/e2e-v2/cases/runtime-rule/cluster`, deployed via skywalking-helm with `oap.replicas=2`) that drives the apply / STRUCTURAL / inactivate / delete lifecycle and the cross-node Forward path, replacing the prior docker-compose default-mode cluster case.
 * Fix: remove the redundant tags from the `envoy-ai-gateway.yaml` LAL configuration.
 * Add Zipkin Virtual GenAI e2e test. Use `zipkin_json` exporter to avoid protobuf dependency conflict
   between `opentelemetry-exporter-zipkin-proto-http` (protobuf~=3.12) and `opentelemetry-proto` (protobuf>=5.0).

oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/module/RuntimeRuleModuleProvider.java

@@ -219,6 +219,15 @@ public class RuntimeRuleModuleProvider extends ModuleProvider {
      */
     private static final long SCHEDULER_INITIAL_DELAY_SECONDS = 2L;
 
+    /**
+     * Env var carrying this OAP's unique per-node identity — the Kubernetes pod UID, injected
+     * by the skywalking-helm chart / swck operator from {@code metadata.uid}. Used as the
+     * runtime-rule cluster {@code selfNodeId} when present, because the telemetry-id fallback
+     * (gRPC {@code host_port}) collides across replicas under k8s where the bind host is
+     * {@code 0.0.0.0} (every pod reports {@code 0.0.0.0_11800}).
+     */
+    private static final String COLLECTOR_UID_ENV = "SKYWALKING_COLLECTOR_UID";
+
     private RuntimeRuleModuleConfig moduleConfig;
     private ScheduledExecutorService reconcilerExecutor;
     private DSLManager dslManager;
@@ -272,7 +281,12 @@ public void start() throws ServiceNotProvidedException, ModuleStartException {
         // cluster gRPC bus (default 11800). Privileged admin RPCs stay on the
         // admin-only port (default 17129) so a compromised node on the agent
         // network cannot reach Suspend/Resume/Forward.
-        final String selfNodeId = TelemetryRelatedContext.INSTANCE.getId();
+        // Resolve this node's stable, unique cluster identity HERE in start() — before
+        // notifyAfterCompleted() applies any rule — so the node knows who it is before it
+        // forwards a write to the main or broadcasts Suspend/Resume. Must be unique per
+        // replica: it is the Forward/Suspend/Resume sender id and the key the receiver's
+        // self-loop guard compares against. See resolveSelfNodeId().
+        final String selfNodeId = resolveSelfNodeId();
         final AdminClusterChannelManager adminPeerChannels =
             getManager().find(AdminServerModule.NAME).provider()
                         .getService(AdminClusterChannelManager.class);
@@ -343,24 +357,25 @@ public void notifyAfterCompleted() throws ModuleStartException {
         // applies under {@code withSchemaChange} if this node resolves as main. Backend DDL is
         // idempotent so the re-apply costs nothing.
         try {
-            // atBoot=true so a no-init OAP picks verifySchemaOnly and refuses to
-            // start with a missing or shape-mismatched backend (k8s pod backloop)
+            // atBoot=true so a cluster peer picks verifySchemaOnly and refuses to
+            // start against a missing or shape-mismatched backend (k8s pod backloop)
             // instead of silently registering local workers against schema that
-            // doesn't exist. Init / default-mode OAPs are unaffected — their boot
-            // opt mirrors the standard tick choice for those modes.
+            // doesn't exist; the main picks withSchemaChange and re-creates missing
+            // runtime schema. The choice is by cluster main-ness, not running mode
+            // (see DSLManager.tickStorageOpt); init mode is the lone exception.
             dslManager.tick(true);
             log.info("Runtime rule dslManager: synchronous first tick completed "
                 + "(runtime-only DB rows are now applied locally).");
         } catch (final RuntimeException re) {
-            // Boot pass under verifySchemaOnly re-throws missing/mismatch as a
-            // RuntimeException so module bootstrap aborts. Translate to
-            // ModuleStartException so the OAP exit message points the operator at
-            // the right place.
+            // The boot pass re-throws as a RuntimeException so module bootstrap aborts —
+            // a peer's verifySchemaOnly hitting a missing/mismatched backend, or a main's
+            // withSchemaChange failing to create it. Translate to ModuleStartException so
+            // the OAP exit message points the operator at the right place.
             throw new ModuleStartException(
-                "Runtime rule dslManager boot pass failed under verifySchemaOnly; "
-                    + "the backend schema is missing or diverges from the declared rule. "
-                    + "Bring up the init OAP first or align rule files with the backend, "
-                    + "then restart this node.",
+                "Runtime rule dslManager boot pass failed: backend schema is missing, "
+                    + "diverges from the declared rule, or could not be created. On a peer, "
+                    + "bring up the cluster main (or init OAP) first; on the main, align the "
+                    + "rule files with the backend, then restart this node.",
                 re);
         } catch (final Throwable t) {
             log.warn("Runtime rule dslManager: synchronous first tick failed — "
@@ -393,6 +408,32 @@ public void notifyAfterCompleted() throws ModuleStartException {
             SCHEDULER_INITIAL_DELAY_SECONDS, intervalSeconds);
     }
 
+    /**
+     * Resolve this node's unique, stable runtime-rule cluster identity. Prefers the Kubernetes
+     * pod UID ({@value #COLLECTOR_UID_ENV}, injected by the helm chart / swck operator from
+     * {@code metadata.uid}) because it is unique per replica; falls back to the telemetry id
+     * ({@code host_port}) for non-k8s deployments where each node already has a distinct host.
+     *
+     * <p>Why not the telemetry id directly: under Kubernetes the agent gRPC bind host is
+     * {@code 0.0.0.0}, so every replica's telemetry id is {@code 0.0.0.0_11800} — identical.
+     * That collision makes the receiver's self-loop guard (sender id == own id) reject a
+     * legitimate peer-to-peer Forward as if it had looped back, breaking cross-node writes on
+     * any multi-replica k8s cluster. {@code MainRouter} already routes correctly off the
+     * cluster peer addresses (pod IPs); only the self-identity used for loop suppression needs
+     * to be unique, which the pod UID guarantees.
+     */
+    private String resolveSelfNodeId() {
+        final String collectorUid = System.getenv(COLLECTOR_UID_ENV);
+        if (collectorUid != null && !collectorUid.trim().isEmpty()) {
+            log.info("Runtime rule: selfNodeId from {} (pod UID) = {}", COLLECTOR_UID_ENV, collectorUid);
+            return collectorUid;
+        }
+        final String telemetryId = TelemetryRelatedContext.INSTANCE.getId();
+        log.info("Runtime rule: {} not set; selfNodeId falls back to telemetry id = {} "
+            + "(ensure it is unique per node in a multi-node cluster).", COLLECTOR_UID_ENV, telemetryId);
+        return telemetryId;
+    }
+
     @Override
     public String[] requiredModules() {
         return new String[] {

oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/reconcile/DSLManager.java

@@ -229,15 +229,17 @@ public void tick() {
 
     /**
      * Variant invoked once at boot from {@code RuntimeRuleModuleProvider.notifyAfterCompleted}
-     * with {@code atBoot=true}. The boot pass on a no-init OAP picks
+     * with {@code atBoot=true}. The boot pass on a cluster <em>peer</em> picks
      * {@link StorageManipulationOpt#verifySchemaOnly()} so missing or shape-mismatched
      * backend schema fails the bootstrap (k8s pod backloop) instead of silently
-     * proceeding. The scheduled executor calls the no-arg overload so subsequent ticks
-     * stay on the lenient {@code withoutSchemaChange} retry path.
+     * proceeding; the <em>main</em> picks {@link StorageManipulationOpt#withSchemaChange()}
+     * so it re-creates any missing runtime schema. The scheduled executor calls the no-arg
+     * overload so subsequent peer ticks stay on the lenient {@code withoutSchemaChange}
+     * retry path.
      *
-     * <p>Boot semantics are scoped to no-init mode only — init-mode OAPs continue to
-     * pick {@link StorageManipulationOpt#schemaCreateIfAbsent()} (boot creates), and
-     * default-mode OAPs continue to pick by cluster main-ness.
+     * <p>The choice is by cluster main-ness, not running mode — no-init and default behave
+     * identically (see {@link #tickStorageOpt}). Init mode is the one exception: the
+     * dedicated initialiser picks {@link StorageManipulationOpt#schemaCreateIfAbsent()}.
      */
     public void tick(final boolean atBoot) {
         try {
@@ -708,44 +710,38 @@ private void invokeAlarmReset(final Set<String> affectedMetricNames) {
     /**
      * Pick the {@link StorageManipulationOpt} for a tick-driven apply.
      *
-     * <p>Two axes:
+     * <p>For runtime-rule (DSL) DDL the only axis that matters is <b>cluster main-ness</b> —
+     * <em>not</em> the init / no-init / default running mode. The running-mode axis governs
+     * <em>static</em> schema (the init OAP creates it, no-init OAPs wait); a runtime rule is
+     * created at runtime and the init OAP never knows about it, so gating DSL DDL on running
+     * mode would leave every production (no-init) cluster unable to apply rules. no-init and
+     * default therefore behave identically here.
      *
-     * <p><b>RunningMode (boot/init context).</b>
+     * <p><b>init mode</b> — the one exception. The dedicated initialiser picks
+     * {@link StorageManipulationOpt#schemaCreateIfAbsent()}, matching the static-rule install
+     * path (create-if-absent, idempotent against a backend that already holds the resource).
+     *
+     * <p><b>Everything else (no-init or default)</b> — branch on main-ness:
      * <ul>
-     *   <li>{@code init} mode — OAP is the dedicated initialiser; install schema if
-     *       absent. {@link StorageManipulationOpt#schemaCreateIfAbsent()} matches what the
-     *       rest of the static-rule install path does in init mode (idempotent against
-     *       backends that already hold the table).
-     *   <li>{@code no-init} mode — this OAP must NOT touch the backend; the init OAP
-     *       owns schema. The opt depends on whether this is the synchronous boot pass
-     *       or a scheduled tick:
+     *   <li>Self is main → {@link StorageManipulationOpt#withSchemaChange()}. The authority
+     *       creates / updates / drops backend schema. The boot pass uses this too, so a main
+     *       re-creates any missing runtime schema at startup.
+     *   <li>Peer (someone else is main):
      *     <ul>
      *       <li><b>Boot pass</b> ({@code atBoot=true}) →
-     *           {@link StorageManipulationOpt#verifySchemaOnly()}. Strict: backend
-     *           resources must already exist with the declared shape. A missing or
-     *           mismatched schema fails the bootstrap (k8s pod backloop) — operator must
-     *           bring up the init OAP first, or align rule files with the backend.
+     *           {@link StorageManipulationOpt#verifySchemaOnly()}. Strict: refuse to start
+     *           against a backend the main hasn't prepared (k8s pod backloop until the main
+     *           converges).
      *       <li><b>Scheduled tick</b> ({@code atBoot=false}) →
      *           {@link StorageManipulationOpt#withoutSchemaChange()}. Lenient: the timer
-     *           retries forever without raising errors so transient absence (init OAP
-     *           still catching up between ticks) self-heals.
+     *           retries without raising so transient absence (main still catching up between
+     *           ticks) self-heals.
      *     </ul>
-     *   <li>default mode (regular running OAP) — branch on cluster main-ness, see below.
-     * </ul>
-     *
-     * <p><b>Cluster main-ness (default mode only).</b>
-     * <ul>
-     *   <li>Self is main → {@link StorageManipulationOpt#withSchemaChange()}. The REST path
-     *       has the same shape; tick rarely runs on main because REST usually
-     *       converges the main's state first.
-     *   <li>Peer (someone else is main) → {@link StorageManipulationOpt#withoutSchemaChange()}.
-     *       Local MeterSystem + MetadataRegistry populate so the peer dispatches samples
-     *       correctly, but no server-side DDL fires.
      * </ul>
      *
-     * <p>When the cluster module isn't wired (embedded test topology), {@link
-     * MainRouter#isSelfMain} returns {@code true} and the default-mode branch falls
-     * through to {@code withSchemaChange} — single-process deployments are always main.
+     * <p>When the cluster module isn't wired (embedded / single-process topology),
+     * {@link MainRouter#isSelfMain} returns {@code true} so we fall through to
+     * {@code withSchemaChange} — a single process is always its own main.
      *
      * @param atBoot true for the synchronous one-shot pass invoked from
      *               {@code RuntimeRuleModuleProvider.notifyAfterCompleted}; false for
@@ -755,20 +751,21 @@ private StorageManipulationOpt tickStorageOpt(final boolean atBoot) {
         if (RunningMode.isInitMode()) {
             return StorageManipulationOpt.schemaCreateIfAbsent();
         }
-        if (RunningMode.isNoInitMode()) {
-            return atBoot
-                ? StorageManipulationOpt.verifySchemaOnly()
-                : StorageManipulationOpt.withoutSchemaChange();
-        }
+        final boolean selfMain;
         try {
             final AdminClusterChannelManager apm =
                 moduleManager.find(AdminServerModule.NAME).provider()
                              .getService(AdminClusterChannelManager.class);
-            return MainRouter.isSelfMain(apm)
-                ? StorageManipulationOpt.withSchemaChange()
-                : StorageManipulationOpt.withoutSchemaChange();
+            selfMain = MainRouter.isSelfMain(apm);
         } catch (final Throwable t) {
+            // Cluster module not wired (embedded / single-process) — always main.
+            return StorageManipulationOpt.withSchemaChange();
+        }
+        if (selfMain) {
             return StorageManipulationOpt.withSchemaChange();
         }
+        return atBoot
+            ? StorageManipulationOpt.verifySchemaOnly()
+            : StorageManipulationOpt.withoutSchemaChange();
     }
 }

oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/ModelInstaller.java

@@ -89,10 +89,15 @@ public void whenCreating(Model model, StorageManipulationOpt opt) throws Storage
             return;
         }
 
-        // Legacy poll loop for non-init OAPs that did not opt into the strict verify
-        // mode. Static models (boot-time) still take this path; runtime-rule reconciler
-        // explicitly chooses verify so this loop is bypassed.
-        if (RunningMode.isNoInitMode()) {
+        // Poll loop for the STATIC boot-time path on a non-init OAP: the init OAP owns
+        // schema creation, so this node waits until the resource appears rather than
+        // creating it. Gated on deferDDLToInitNode (set only on SCHEMA_CREATE_IF_ABSENT),
+        // NOT on RunningMode alone — a runtime-rule DSL apply (withSchemaChange) is the
+        // operator/main-driven authority and must fall through to createTable below
+        // regardless of no-init, because no init OAP knows about a metric created at
+        // runtime. Without this, a no-init OAP would block here forever waiting for a
+        // resource that only this very apply would ever create.
+        if (deferDDLToInitNode(opt)) {
             while (true) {
                 InstallInfo info = isExists(model, opt);
                 if (!info.isAllExist()) {
@@ -148,6 +153,23 @@ public void whenRemoving(Model model, StorageManipulationOpt opt) throws Storage
             StorageManipulationOpt.Outcome.DROPPED, null);
     }
 
+    /**
+     * True when this manipulation must defer all backend DDL to the dedicated init OAP and
+     * wait for it, rather than create / update / reshape the resource on this node. This is
+     * the single source of truth for the "no-init OAP doesn't own schema" rule across the
+     * base installer and every backend subclass — call it instead of re-checking
+     * {@link RunningMode#isNoInitMode()} inline, so the rule stays one decision.
+     *
+     * <p>True only for the static boot-time {@link StorageManipulationOpt#schemaCreateIfAbsent()}
+     * opt on a {@code no-init} OAP. The runtime-rule (DSL) opts leave
+     * {@link StorageManipulationOpt.Flags#isDeferDDLToInitNode() deferDDLToInitNode} unset, so
+     * an operator-driven apply is governed by the opt's own create / update / drop flags and
+     * by cluster main-ness — never by the init / no-init / default running mode.
+     */
+    protected static boolean deferDDLToInitNode(final StorageManipulationOpt opt) {
+        return RunningMode.isNoInitMode() && opt.getFlags().isDeferDDLToInitNode();
+    }
+
     public void start() {
     }