clusterer_controller: zero-config HA clusterer control module with automatic sharing tag management via encrypted UDP multicast#4074
Open
Lt-Flash wants to merge 21 commits into
Conversation
Lt-Flash
marked this pull request as draft
July 13, 2026 14:03
Member
|
Thank you very much for the contribution, I really like the idea behind it. I do see though that it is marked as a Draft - is it still work in progress, or has it reached to its final state? Let us know when it is ready to review. |
This was referenced Jul 14, 2026
Author
|
Hi, |
Lt-Flash
force-pushed
the
feature/clusterer-controller-devel
branch
4 times, most recently
from
July 14, 2026 11:07
9034536 to
2cf5b36
Compare
Port of the clusterer_controller module and its clusterer-module integration to the devel branch: zero-config HA clusterer control over encrypted UDP multicast, per-cluster node identity, hybrid native+controller topologies, cl_ctr_* MI/variables/functions, join flood-DoS hardening, and input validation. Devel adaptations: - cc_discover_bin_sockets() iterates protos[PROTO_BIN].listeners as struct socket_info_full (next/prev), reading the embedded socket_info. - The clusterer core edits are re-seated onto devel's clusterer.c / clusterer_mod.c / node_info.h, merging cleanly with the new bridges feature and the clusterer_enable_rerouting guard. Robustness / log hygiene: - cc_decrypt_pkt distinguishes bootstrap-key failures (real wrong-password / foreign-cluster / tampering -> WARN) from transient session-key mismatches during a (re)key or split-brain heal (-> DBG). - the split-brain defer budget resets on a fresh higher-IP JOIN_REQ so a lower-IP node waits for a live higher-IP peer instead of self-promoting (bounded by CC_JOIN_DEFER_HARDMAX); JOIN_REQ sends are rate-limited. - clusterer: the seed sync-fallback on a fresh start is downgraded from ERROR to DBG (the SYNC_IN_PROGRESS partial-sync failure path is unchanged). - cc_elect_master enforces "MASTER_ALIVE keepalive armed <=> I am the elected master": a node demoted purely by election now stops broadcasting MASTER_ALIVE, fixing an oscillation where a lower-IP node flapped between two masters. Validated end-to-end on a 3-node cluster: controller / native / hybrid modes, master failover and election stability (staggered + simultaneous starts converge to a single stable master), multiple controller clusters sharing one BIN socket (distinct multicast per cluster), MI outputs and error paths, buffer overrun/underrun fuzzing, wrong-password rejection, and config-mismatch. Docs and generated README included (admin guide + HA test appendix).
Lt-Flash
force-pushed
the
feature/clusterer-controller-devel
branch
from
July 14, 2026 11:46
2cf5b36 to
34f1042
Compare
Lt-Flash
marked this pull request as ready for review
July 14, 2026 11:46
Author
|
Now it's ready for review, thanks! |
Lt-Flash
force-pushed
the
feature/clusterer-controller-devel
branch
from
July 14, 2026 15:27
992d376 to
f93b82d
Compare
… consistency Two startup sanity checks for a previously silent misconfiguration: - clusterer_controller is only meaningful when the clusterer module has use_controller=1 (the global switch that pre-creates the controller-managed cluster stubs, sets each one's controller_managed flag so they never touch the DB, and arms the guard that stops the controller from hijacking a native cluster of the same id). If use_controller=0 there is no controller-managed cluster and those safety mechanisms are off, so mod_init now FAILS (refuse to start) rather than driving clusters clusterer never authorised. Hybrid setups keep use_controller=1 - only the per-cluster kind differs - so this never trips them. - The mirror case, use_controller=1 but no clusterer_controller module bound the controller API, logs an ERROR at clusterer child_init (the pre-created controller-managed stubs would never obtain an identity). clusterer does NOT abort here: its native/hybrid clusters still work; only the controller-stub clusters are dead. Plumbing: the clusterer_ctrl binds struct now carries clusterer's use_controller value, and load_clusterer_ctrl_binds() sets clusterer_ctrl_bound so clusterer can tell whether a controller registered. Verified on all permutations: pure controller, hybrid, native-only, and both mismatches.
Lt-Flash
force-pushed
the
feature/clusterer-controller-devel
branch
from
July 14, 2026 15:30
f93b82d to
4348580
Compare
Author
|
Follow-up commit Since this PR is already open for review, I added this as a distinct follow-up commit rather than squashing it into the main one, so the incremental change is easy to review and the existing review isn't disrupted by a force-push of the main commit. It enforces consistency between
Hybrid environments are unaffected. |
added 2 commits
July 15, 2026 02:18
…ples
Reviewed all config examples so every cluster is clearly and self-containedly
defined, per reviewer feedback:
- Show modparam("clusterer","cluster_id",N) for each controller-managed cluster
in the full-config and multi-cluster examples (previously several relied on
the implicit auto-create-on-capability-registration path, so the cluster
never appeared "defined" in the snippet).
- Hybrid DB example: note that native cluster 10 is defined by rows in the
clusterer DB table (not a modparam), and fix the misleading "this node id in
cluster 10" comment (my_node_id is a single global id across all DB clusters).
- Add the missing "Hybrid, no DB" (static) example to the admin guide, with the
required db_mode=0 (without it my_node_info/neighbor_node_info are ignored).
- Comment use_controller as the global switch throughout.
No behavior change - documentation only. README regenerated.
Add a per-cluster 'cluster_options' modparam - the same "key=value, key=value"
idiom as my_node_info - so a cluster is marked controller-managed with:
modparam("clusterer", "cluster_options", "cluster_id=1, use_controller=1")
cluster_id is required; use_controller is a 0/1 flag defaulting to 0 (native),
and only use_controller=1 pre-creates the controller-managed stub (which never
touches the DB and is guarded against hijacking a native cluster of the same
id). Native clusters need no cluster_options line at all. Every other clusterer
setting (db_mode, ping_*, my_node_id, sharing_tag, ...) stays a global modparam.
The controller-managed ids and the clusterer_controller 'cluster' entries must
match EXACTLY. clusterer_controller aborts at startup, naming the offending id,
if either side references a cluster the other does not:
- a managed id with no 'cluster' config has no BIN socket or crypto params;
- a 'cluster' config for an unmanaged id has nothing legitimate to drive.
The controller loads the clusterer's controller-managed id set through the ctrl
binds (managed_count/managed_ids) to run this check pre-fork.
The interim 'use_controller'/'cluster_id' int modparams (never released) are
kept registered only to fail with a migration hint.
Docs (admin guide, tests appendix, README) rewritten to the cluster_options
form; every example gives each cluster an explicit definition.
Lt-Flash
force-pushed
the
feature/clusterer-controller-devel
branch
from
July 14, 2026 17:11
82e123e to
b7a173e
Compare
Author
|
Config API for controller-managed clusters is now the per-cluster Same The controller-managed ids and the Verified locally (the build links wolfSSL, so the controller runs without a node): the new syntax loads, a managed id with no controller config aborts, a controller config for an unmanaged id aborts, and matching config starts. Docs (admin guide, tests appendix, README) and the PR description are updated to this form. |
…ude clusterer_controller by default
Make clusterer_controller an opt-in module and ensure the bundled clusterer
module is completely unchanged when it is not built.
Build wiring:
- Add clusterer_controller to exclude_modules in Makefile.conf.template, so a
stock build skips it (like the other modules with external-lib deps). Enable
it via include_modules.
- The top-level Makefile exports CLUSTERER_CTRL_SUPPORT=1 iff clusterer_controller
is in the configured build (mirrors the module-selection rule, so it is stable
across 'make all' and single-module rebuilds); clusterer/Makefile turns that
into -DCLUSTERER_CTRL_SUPPORT.
clusterer side, all under #ifdef CLUSTERER_CTRL_SUPPORT:
- the clusterer_ctrl API (clusterer_ctrl.c) and the cluster_options /
use_controller / cluster_id modparams + load_clusterer_ctrl_binds export;
- the controller stub pre-create loop, the child_init guard, the shm current_id
mirror, the on-demand stub, and the shtag_managed / controller_managed logic.
- The per-cluster identity and hybrid-db_mode accessors (cluster_self_id,
cl_db_mode, GET_CURRENT_ID, use_controller) get #else fallbacks to the stock
globals (current_id / db_mode / 0), so their call sites compile to the exact
upstream object code with no per-site #ifdef. add_node_info's internal self_id
parameter is likewise gated (falls back to current_id).
Result: a build without clusterer_controller produces the stock clusterer module -
no cluster_options parameter (rejected as unknown), no behavioural change, no added
exports. Verified with unifdef -UCLUSTERER_CTRL_SUPPORT against the base: no semantic
difference from upstream. Enabling the controller rebuilds clusterer with the hooks;
the two are a matched pair.
Docs: new "Building the Module" section (admin guide + README).
Author
|
Follow-up Previously the clusterer-side integration (the
So clusterer_controller can be completely omitted — a build without it produces the stock upstream clusterer module: no Verified with |
Author
Crypto is libsodium-only (see ce8e805)Worth stressing for reviewers: as of ce8e805 the module's cryptography settled on libsodium, and the earlier wolfSSL / OpenSSL-based paths were dropped entirely — there is no TLS-library fallback anymore. clusterer_controller now uses XChaCha20-Poly1305 + Argon2id for the shared-secret / at-rest key material and a Noise_NNpsk0 (Curve25519 / ChaCha20-Poly1305 / SHA-256) handshake for the join, all on libsodium primitives. Why the switch:
libsodium is therefore a hard build dependency of the module now; there is no |
Author
Why this PR also touches
|
added 4 commits
July 18, 2026 21:31
libsodium is a hard build requirement: the module is built sodium-only (XChaCha20-Poly1305 + Argon2id, plus X25519 / HKDF-SHA256 / RNG from libsodium), and the wolfSSL AES-256-GCM/scrypt fallback is not built. This keeps the module binary small and gives one consistent crypto suite across every build.
No behaviour change; a code-quality pass on the single-file module. - cc_seal_and_send(): every one of the ten packet senders repeated the same ~13-line tail (encrypt in place, build the multicast sockaddr, sendto, error log). Factor it into one helper. The multicast destination is now resolved once per cluster in mod_init (main process) into cl->mcast_dest and inherited by the forked workers - so it no longer rebuilds inet_addr()/htons() on every send, and mod_destroy's GOODBYE (which runs in the main process) uses the same path. Senders keep only their own success log via the helper's return value. - cc_peer_by_ip_locked(): the "find a peer entry by IP" scan was open-coded in many places; add a helper and use it where the lookup is standalone (cc_upsert_peer_locked, cc_update_peer_bin_locked). The fused election / member-list / prune loops are left as-is. - cc_recv_one(): reuse the is_bootstrap flag instead of re-memcmp'ing the packet magic three more times; drop the unreachable payload_len < 0 check (the minimum-length gate at the top already guarantees it is non-negative). Verified on a local two-node sodium cluster: join via KEY_GRANT + NODE_ASSIGN, sticky backup designation, master-death failover promotion, and graceful GOODBYE on shutdown - zero controller errors. Both build flavors (sodium-only and wolfSSL fallback) compile clean.
…se_NNpsk0
The join handshake used a bespoke construction: an ephemeral X25519 ECDH whose
shared secret was fed, with the password and a per-exchange nonce, into HKDF to
derive a key that XOR-wrapped the master_salt (cc_wrap_salt), plus a manual
nonce echo to bind the exchange. Replace it with a standard, analysable
handshake:
Noise_NNpsk0_25519_ChaChaPoly_SHA256, PSK = the Argon2id bootstrap key.
-> psk, e JOIN_REQ carries Noise message 1 (fresh ephemeral)
<- e, ee KEY_GRANT carries Noise message 2; its AEAD payload = master_salt
A compact Noise core (SymmetricState + CipherState + the NNpsk0 read/write) is
added on libsodium primitives (X25519, ChaCha20-Poly1305, SHA-256, HMAC-SHA256);
it was validated against the RFC 5869 HKDF vector and for self-consistency,
tamper-detection and wrong-PSK rejection in a standalone harness before wiring
in. The handshake hash binds the whole transcript, so the manual join_nonce
echo, the cc_peer_t.join_nonce field, and CC_JOIN_NONCE_SZ are removed - a stale
KEY_GRANT for a superseded JOIN_REQ now simply fails Noise msg-2 decryption.
KEY_HANDOFF (a one-shot, which NNpsk0's 2-message shape does not fit) switches
to an anonymous crypto_box_seal of the master_salt to the next master's
long-lived X25519 key (learned via ALIVE); sender authenticity still comes from
the session-key envelope. cc_wrap_salt and cc_ecdh_shared are deleted.
The multicast transport, outer AEAD framing (bootstrap/session magic), sequence
replay check and rate limiter are unchanged; wrong-password nodes are still
rejected at the bootstrap envelope before the handshake runs. Verified on a
local two-node cluster: join, sticky backup, master-death failover, GOODBYE, and
wrong-password shutdown - all clean.
Update the admin guide (and regenerated README) for the crypto changes:
- crypto is now libsodium-only (XChaCha20-Poly1305 + Argon2id); the wolfSSL
fallback and its AES-256-GCM/scrypt description are removed, and the build
now requires libsodium;
- the Phase 1 join handshake is described as Noise_NNpsk0_25519_ChaChaPoly_
SHA256 (PSK = bootstrap key) instead of the hand-rolled ECDH-and-XOR wrap;
- JOIN_REQ / KEY_GRANT carry Noise messages 1 / 2; KEY_HANDOFF uses a
crypto_box sealed to the next master's key;
- password / bootstrap-key wording updated (Argon2id, Noise PSK).
added 3 commits
July 18, 2026 21:31
The election result was only visible as a cluster-wide roles list; a node could not see its OWN transition on its own line - e.g. a rejoining higher-IP peer would silently demote this backup to a plain member with nothing marking the change. Track the last-known self role (enum cc_role) per cluster and, at the end of cc_elect_master(), emit a single "my role changed: X -> Y" line whenever it differs. This covers every election trigger (goodbye, join, timer) in one place. Zero-initialised to member, matching a not-yet-joined node. The post-goodbye "re-election complete" line drops its now-redundant role clause and just names the elected master.
tm_init_cluster() read this node cluster id once, at mod_init, and baked it into the ";cid=<id>" Via parameter (and into a cached tm_node_id used to decide whether an anycast reply is ours). That assumes the id is known and stable by mod_init, which holds for a statically configured clusterer but not for a controller-managed one: there the id is assigned at runtime, after the node joins, and can change on re-election. So mod_init froze the still-unassigned id (-1, printed as its unsigned form 18446744073709551615) into every outgoing Via, and every node compared incoming cids against -1 - t_anycast_replicate() could never route a reply to the owning node. Read the id live instead: pre-build only the fixed ";<param>=" prefix at init, and have tm_via_cid() append the current get_my_id() per request (returning no parameter while the node still has no id); compare incoming cids against the live get_my_id() too. cl_get_my_id() already returns the runtime id, so this needs nothing from the caller and self-corrects across re-elections.
The block flagged the sharing-tag override, its clearing, and the per-node shtag status as future work, but all three shipped: cl_ctr_shtag_force, cl_ctr_shtag_auto, and the shtag_mode field of cl_ctr_list_config. Reduce it to the one item still outstanding - node maintenance mode - and note that it would build on the shtag commands already in place, so the source no longer reads as unfinished where it is not.
Lt-Flash
force-pushed
the
feature/clusterer-controller-devel
branch
from
July 18, 2026 11:31
aad3a24 to
5dc976d
Compare
added 9 commits
July 19, 2026 17:56
Rename every internal identifier from the cc_ prefix to cl_ctr_, and the CC_ macros to CL_CTR_, so the whole module reads consistently with the public cl_ctr_* MI commands, pseudo-variables and script functions rather than carrying the older cc_ shorthand alongside them. This covers the static functions, the types (cl_ctr_cluster_t, cl_ctr_peer_t, ...), the module variables and the CL_CTR_ constants, across clusterer_controller.c and the controller-side hooks in clusterer (clusterer_ctrl.c/.h, clusterer_mod.c, node_info.h), plus the docs and the join-reject test tool. The wire magic bytes (the 0xCC prefix) are unchanged. Two key-derivation inputs are renamed along with everything else - the HKDF label "cc_session" -> "cl_ctr_session" and the bootstrap salt "opensips-cc-bootstrap-v1:" -> "opensips-cl-ctr-bootstrap-v1:" - which changes the derived session and bootstrap keys, so every node of a cluster must run this build; a mixed old/new cluster cannot interoperate.
KEY_GRANT is strictly 1:1 - the master answers one joiner's JOIN_REQ with the Noise msg 2 carrying that joiner's master_salt. It was multicast to the whole group, so every other member spent an AEAD decrypt only to find the packet was not addressed to it and discard it. Send it to the joiner's own address instead: the datagram source of its JOIN_REQ, threaded from recvfrom through to the sender as a generic struct sockaddr. The controller socket is bound to INADDR_ANY on the group port, so it already receives unicast to the node's own address on that port - no socket change is needed - and echoing the received sockaddr rather than rebuilding it from a string keeps this correct for IPv6 (it carries the scope id link-local traffic needs). Add cl_ctr_seal_and_send_to(), which takes an explicit destination; cl_ctr_seal_and_send() becomes a thin multicast wrapper over it for the group packets. The KEY_GRANT wire format is unchanged (target_ip is still carried), so this interoperates with peers that still multicast it. Turns the per-join KEY_GRANT cost from one AEAD decrypt on every member into one on the joiner alone.
The 1:1 handshake packets ride unreliable UDP: a lost KEY_GRANT left the
joiner waiting out its whole join window before re-trying. Add positive
acknowledgement with a bounded retransmit so a single loss is recovered
in milliseconds rather than seconds.
The receiver of a 1:1 packet replies with CL_CTR_PKT_ACK echoing the
packet's seq, sealed in the same key class as the packet it acks (a
KEY_GRANT is bootstrap-keyed, so its ACK is too - the joiner need not
already hold the session key). The sender keeps the sealed bytes in a
small worker-local queue and, if no ACK arrives, resends them verbatim;
after CL_CTR_RETX_MAX_RETRIES it gives up and lets the joiner's JOIN_REQ
retry restart the handshake. A joiner that already holds the key re-ACKs
a duplicate KEY_GRANT, so a lost ACK is recovered on the next retransmit
instead of burning the whole budget.
The budget is pinned to the join timers - CL_CTR_RETX_INTERVAL_US is
half of CL_CTR_JOIN_REQ_MIN_US and a compile-time check keeps the total
retransmit window shorter than one JOIN_REQ cycle - so the fast per-packet
ARQ nests inside the slower JOIN_REQ backstop instead of racing it.
Two invariants keep this strictly below the election layer, so it can
never cause split brain:
- ACKs are retransmit accounting only. They never gate membership or
election; a never-acked packet is simply dropped, and a node joins
through the normal JOIN_REQ -> ACTIVE -> ALIVE path regardless. So a
lost ACK or a departed joiner can stall nothing.
- The retransmit queue is flushed on loss of mastership and on
session-key rotation, so a demoted or re-keyed node never keeps
delivering a stale master-keyed KEY_GRANT that would push a joiner
onto a dead key.
Only the strictly 1:1 KEY_GRANT is wired to this here; the multicast
group packets are deliberately left unacked (acking them from every
member would be ack implosion - their loss is better healed by a
membership digest). This adds a new packet type, so all nodes of a
cluster must run this build; a node that does not understand
CL_CTR_PKT_ACK simply draws extra retransmits, which is harmless.
A peer's node_id and BIN mapping travel only in NODE_ASSIGN, which is multicast and so best-effort: a member that drops one knows the peer exists (from its ALIVE) but not its node_id, leaving its clusterer BIN mesh to that peer incomplete with no repair. Until now the only thing that fixed it was the master re-multicasting every peer's NODE_ASSIGN on the next join - an accident, not a mechanism, and useless in a stable cluster. Add an explicit anti-entropy repair. Every MASTER_ALIVE now carries a membership digest - the active-peer count plus an order-independent XOR of a per-peer FNV-1a over (node_id, ip). A member that computes a different digest from the master's has missed something - a whole peer (count differs) or just a peer's node_id (count matches, hash differs) - and sends a rate-limited RESYNC (new packet type). The master answers by re-broadcasting every NODE_ASSIGN plus the MEMBER_LIST, coalescing a burst of RESYNCs into one re-broadcast per interval. Roles are left out of the digest so transient master disagreement does not cause spurious resyncs. This is the multicast counterpart to the 1:1 ACK path: 1:1 handshake packets get positive-ack retransmit, group packets get digest-driven pull. It also unblocks unicasting the join-time NODE_ASSIGN burst to the joiner - a member that would have relied on the incidental re-multicast now self-heals through the digest instead. Adds a packet type, so all nodes of a cluster must run this build; a node that does not understand it simply advertises no digest and is never asked to resync.
On every join the master re-multicast a NODE_ASSIGN for each existing peer plus the MEMBER_LIST - one packet per member, decrypted by every member, of which only the joiner needed all but the newcomer's own assignment. So a join into an N-node cluster cost O(N) group packets and O(N^2) cluster-wide AEAD decrypts, nearly all wasted. Send the joiner-only state to the joiner. cl_ctr_send_node_assign() and cl_ctr_send_list_pkt() take an optional unicast destination (NULL keeps the multicast behaviour); the existing-peer NODE_ASSIGN burst and the MEMBER_LIST snapshot now go to the joiner's own address, while the newcomer's NODE_ASSIGN stays multicast so every member still learns it. Per join the group now sees one NODE_ASSIGN instead of N. Dropping the re-multicast removes the incidental repair it used to give a member that had missed a peer's original assignment; that is now covered by the membership digest added earlier - such a member notices the mismatch and pulls a RESYNC. The unicast snapshot is not acked: if the joiner loses it, it stays CL_CTR_NODE_NEW and its JOIN_REQ retry brings another, exactly as a lost multicast MEMBER_LIST behaved before.
…failover When a node departed (GOODBYE) or a master went silent, the newly elected master re-broadcast the full MEMBER_LIST to assert itself. That was redundant: every surviving member runs the same deterministic election locally off the very same GOODBYE or keepalive timeout, so they already agree on the winner; MASTER_ALIVE re-asserts it within a keepalive; and the membership digest reconciles anyone who missed the event. A node still joining reaches the new master through its own JOIN_REQ retry. Drop both re-broadcasts. This removes an O(N) packet - which fragments above ~80 nodes and is dropped wholesale by any middlebox that blocks fragmented UDP - from the failover path, where it bought nothing. Verified on a 3-node cluster: killing the master leaves the two survivors converging to the same new master (highest IP) with the other as backup, no split brain and no stuck state, purely via local re-election plus MASTER_ALIVE. The forced-shtag MEMBER_LIST is untouched - it carries the forced node_id, which nothing else propagates.
Every active node multicast an ALIVE every query_time carrying its pubkey and config, and every node ran an election off every one it received - so steady-state liveness cost O(N^2) packets and cluster-wide AEAD decrypts. Route it through the master instead. A settled non-master unicasts its ALIVE to the master (still multicast during formation, with no stable master/key yet, for discovery); the master learns every peer's last_seen, pubkey and config in O(N). The master then relays liveness to the whole group by appending a per-node-id bitmap to its MASTER_ALIVE - built from the ELECTION-window cutoff, not the purge window, so a departed node leaves the bitmap on the same schedule it leaves an election. A non-master refreshes last_seen for the peers the master reports alive, keeping its election window populated without hearing the peers' ALIVEs. Election itself is unchanged and stays local. The master keeps multicasting its own ALIVE so its liveness/config still reach everyone, and config-drift detection is preserved (the master sees every peer's config, non-masters see the master's). A settled non-master no longer hears a multicast loopback of its own ALIVE, so it must refresh its own last_seen locally in on_alive_tfd - otherwise it ages itself out of its own election window once the master is gone and briefly finds "no eligible peers" during failover. The KEY_HANDOFF pubkey a promoted backup needs is re-collected from the now-incoming unicast ALIVEs within a query_time; a handoff attempted in that brief window degrades to a rejoin, which the successor already handles. Validated on 3-node netns rigs: steady-state unicast+bitmap (no O(N^2)), single failover, chained failover, rapid double failure (master + backup killed together -> the survivor self-corrects through one extra watchdog round, converging to sole master), higher-IP join -> sticky backup, and simultaneous cold start -> deterministic single master, no split brain.
The unicast work assumed my_ip:multicast_port uniquely identifies a cluster's socket. It does not when several controller clusters run on one node: every socket binds INADDR_ANY:multicast_port, and the config allows two clusters to share a port with distinct multicast groups (the duplicate check only rejected an identical address+port). Multicast is unaffected there - IP group membership demultiplexes - but a unicast to my_ip:port is routed by port alone, so it can land on the wrong cluster's socket and be dropped by the cluster_id filter, losing the KEY_GRANT / ACK / NODE_ASSIGN / MEMBER_LIST / RESYNC / ALIVE. Detect the condition at mod_init (a per-cluster unicast_ok flag: cleared for any cluster that shares its multicast port with another local cluster) and redirect every unicast send to the group in cl_ctr_seal_and_send_to() when the flag is clear. Correct in all cases: single-cluster and distinct-port deployments keep unicast; shared-port clusters transparently use multicast for their 1:1 packets too, only losing the optimisation. A one-line warning at startup points operators at distinct ports. Hybrid (native + controller) nodes are unaffected: a native clusterer cluster uses the BIN protocol on its own TCP socket, not the controller's UDP port, so it never collides - only fellow controller clusters count. Verified: two controller clusters sharing port 3333 log the warning, both sockets come up, and their 1:1 traffic goes multicast.
When several controller clusters on one node share a multicast_port (distinct groups), every cluster socket binds INADDR_ANY:port, so the kernel demultiplexes multicast by group membership but unicast by port alone. A 1:1 reply meant for a co-located cluster can therefore be delivered to the wrong sibling's socket, where the cluster_id filter would drop it. fabcdd1 worked around this by making such clusters fall back to multicast for their 1:1 packets - correct, but it gives up the unicast scalability win exactly where several clusters coexist. Instead, recover the packet at the receiver: on a cluster_id mismatch the receiving worker looks up the local cluster the packet's cleartext cluster_id belongs to and forwards the still-encrypted datagram to that cluster's worker via ipc_send_rpc(); the target decrypts it with its own key and processes it exactly as if it had arrived on its own socket. Senders now always unicast; the multicast fallback and the per-cluster unicast_ok flag are gone. cl_ctr_recv_one() grows a forwarded-buffer path (fwd_buf) so the RPC handler reuses the full validate/dispatch logic without duplicating it. Only the original receiver forwards (a forwarded packet is never re-forwarded), so no loop is possible; the forward is size-bounded and rate-limited against the source to bound flood work. Single-cluster and distinct-port deployments never hit a cluster_id mismatch, so the forwarding path stays entirely inert for them. Tested on a netns rig: single-cluster 2-node failover unchanged; two clusters sharing port 3333 across two nodes both converge to full membership, with 61 misdelivered unicast packets correctly re-routed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
clusterer_controller — self-forming HA coordination for OpenSIPS
clusterer_controllersits on top of theclusterermodule and removes thestatic
my_node_info/neighbor_node_infowiring: nodes sharing a multicastgroup, a
cluster_idand a password self-organise — they discover eachother, elect a deterministic master (highest IP, sticky), assign
clusterernode-ids, hold sharing-tags on exactly one node, and fail over automatically —
with an encrypted control plane and zero per-node topology config.
This revision lands the unicast / scalability series (steady-state and
join cost cut from O(N²) → O(N), with anti-entropy repair and reliable
handshakes) and a module-wide
cc_→cl_ctr_rename so every symbolmatches the public
cl_ctr_*MI/pvar/function surface.Control plane at a glance
Encrypted UDP on the cluster's multicast group (default
239.0.10.x:3333).Only a small cleartext header is visible on the wire —
magic (2B)+cluster_id (2B)+nonce— everything else is AEAD-sealed(XChaCha20-Poly1305 + Argon2id, key agreement via Noise_NNpsk0).
Two key tiers, told apart by the magic byte:
0xCC01JOIN_REQ,KEY_GRANT,ACK,JOIN_REJECT,MASTER_BEACON0xCC00MASTER_ALIVE(+liveness bitmap +membership digest),ALIVE,NODE_ASSIGN,MEMBER_LIST,RESYNC,GOODBYE,KEY_HANDOFFLegend for the diagrams below:
-->>unicast (1:1),-)multicast (group).1. Cluster formation (cold start)
Nodes start together, discover over multicast, and converge on the highest-IP
node as master with no split brain — lower-IP nodes defer self-promotion while
a higher-IP peer is still joining.
sequenceDiagram autonumber participant N1 as .191 participant N2 as .192 participant N3 as .193 highest IP Note over N1,N3: simultaneous cold start, no master yet N1-)N3: JOIN_REQ (mcast, Noise msg1) N2-)N3: JOIN_REQ (mcast, Noise msg1) N3-)N1: JOIN_REQ (mcast, Noise msg1) Note over N1,N2: higher-IP peer still joining -> defer self-promotion Note over N3: highest IP -> self-promote, mint session key N3-->>N2: KEY_GRANT (unicast, Noise msg2 + salt) N2-->>N3: ACK N3-->>N1: KEY_GRANT (unicast) N1-->>N3: ACK N3-->>N2: NODE_ASSIGN + MEMBER_LIST (unicast) N3-->>N1: NODE_ASSIGN + MEMBER_LIST (unicast) Note over N1,N3: converged - .193 master, .192 backup, .191 member2. Member join into a running cluster
The join is unicast to the joiner: only the newcomer needs the full peer
list, so the master sends it 1:1 instead of multicasting N packets that every
member would decrypt and discard. Only the newcomer's own assignment is
multicast, so existing members learn it. A lost
KEY_GRANTis recovered byACK + retransmit (§5), a lost snapshot by the
JOIN_REQretry.sequenceDiagram autonumber participant J as .191 joiner participant E as .192 member participant M as .193 master J-)M: JOIN_REQ (mcast, Noise msg1 + BIN socket) M-->>J: KEY_GRANT (unicast, session key) J-->>M: ACK (unicast) M-)E: NODE_ASSIGN newcomer .191=id3 (mcast -> all learn it) M-->>J: NODE_ASSIGN .193=id1, .192=id2 (unicast, peers) M-->>J: MEMBER_LIST 3 members (unicast, snapshot) J-)M: ALIVE (joiner now participates) Note over J,M: per join the group sees ONE packet, not N3. Steady-state liveness — master-mediated (O(N²) → O(N))
Previously every node multicast an
ALIVEeveryquery_timeand every node ranan election off every one — O(N²) packets and cluster-wide decrypts. Now a settled
member unicasts its ALIVE to the master; the master folds all peers' liveness
into a per-node-id bitmap on its
MASTER_ALIVE, and members trust that bitmapto keep their election windows populated.
sequenceDiagram autonumber participant A as .191 member participant B as .192 backup participant M as .193 master A-->>M: ALIVE (unicast, ~query_time) B-->>M: ALIVE (unicast, ~query_time) M-)A: MASTER_ALIVE (mcast, ~1/s) + liveness bitmap + membership digest M-)B: MASTER_ALIVE (mcast) + bitmap + digest Note over A,B: refresh last_seen from the master's bitmap - no peer-to-peer ALIVE4. Anti-entropy — membership digest → RESYNC
Every
MASTER_ALIVEcarries a digest = active-peer count + order-independentXOR of
FNV-1a(node_id, ip). A member whose computed digest differs has missed aNODE_ASSIGN(whole peer, or just a node-id) and pulls a rate-limited RESYNC;the master coalesces a burst into one re-broadcast. This is the multicast
counterpart to the 1:1 ACK path.
sequenceDiagram autonumber participant Mem as .192 member participant M as .193 master M-)Mem: MASTER_ALIVE + digest = count + XOR hash Note over Mem: local digest != master's, missed a NODE_ASSIGN Mem-->>M: RESYNC (unicast, rate-limited) M-)Mem: NODE_ASSIGN (all peers) + MEMBER_LIST (re-broadcast) Note over Mem: digests match again5. Reliable handshake — ACK + bounded retransmit
The 1:1 handshake rides unreliable UDP. A lost
KEY_GRANTused to cost the joinerits whole join window; now the master retransmits until ACKed (bounded), so a
single loss heals in milliseconds.
sequenceDiagram autonumber participant J as .191 joiner participant M as .193 master J-)M: JOIN_REQ M-->>J: KEY_GRANT (unicast) x lost Note over M: no ACK within retransmit timeout M-->>J: KEY_GRANT (retransmit) J-->>M: ACK Note over J,M: recovered in ms, not a full join window6. Graceful leave (GOODBYE)
A departing node multicasts
GOODBYE. Survivors re-elect locally off the sameevent — no MEMBER_LIST re-broadcast (that O(N), fragmenting packet was dropped from
the failover path);
MASTER_ALIVEre-asserts and the digest reconciles stragglers.sequenceDiagram autonumber participant D as .191 leaving participant B as .192 backup participant M as .193 master D-)M: GOODBYE (mcast) D-)B: GOODBYE (mcast) Note over B,M: local re-election off the same GOODBYE - no re-broadcast Note over M: still master, 2 members - no role change M-)B: MASTER_ALIVE + digest (count now 2)7. Master failover (crash)
The master goes silent; members detect the missed
MASTER_ALIVEkeepalives and runthe same deterministic election locally — the backup (highest-IP survivor)
promotes and asserts via
MASTER_ALIVE.sequenceDiagram autonumber participant Me as .191 member participant B as .192 backup participant M as .193 master Note over M: master crashes x Note over Me,B: MASTER_ALIVE keepalives stop -> timeout Note over Me,B: identical local re-election -> highest-IP survivor wins Note over B: .192 promotes to master B-)Me: MASTER_ALIVE (new master asserts) + digest Me-->>B: ALIVE (unicast to new master) Note over Me,B: converged - .192 master, .191 member8. Graceful master handoff (KEY_HANDOFF)
On a planned master shutdown the outgoing master hands the session key to its
successor sealed to the successor's long-lived X25519 pubkey (
crypto_box_seal,learned from ALIVE) — so the successor takes over without a rejoin.
sequenceDiagram autonumber participant S as .192 successor participant M as .193 master leaving Note over M: planned shutdown M-->>S: KEY_HANDOFF (unicast, session key sealed to S's pubkey) M-)S: GOODBYE (mcast) Note over S: already holds the key -> promotes with no rejoin S-)S: MASTER_ALIVE (asserts as master)9. Wrong-password rejection
A node with the wrong password cannot produce a bootstrap-valid
JOIN_REQ. Afterrepeated failures from one source the master emits an authenticated
JOIN_REJECT;independently, a genuinely-wrong-password joiner self-terminates after its defer
budget rather than forming a lone split-brain master.
sequenceDiagram autonumber participant R as .200 wrong password participant M as .193 master R-)M: JOIN_REQ (bootstrap AEAD fails to authenticate) Note over M: repeated bootstrap-decrypt failures from .200 M-->>R: JOIN_REJECT (unicast, GCM-authenticated) Note over R: in NODE_NEW state -> exit(-1), no split brain10. Split-brain merge (MASTER_BEACON)
If a partition heals and two masters exist, each periodically emits a
bootstrap-keyed
MASTER_BEACON(readable across different session keys) carryingits
member_count. The inferior master (smaller count, IP tiebreak) yields andrejoins the superior one.
sequenceDiagram autonumber participant A as .191 master, 1 member participant B as .193 master, 3 members Note over A,B: partition heals - two masters B-)A: MASTER_BEACON (bootstrap-keyed, member_count=3) A-)B: MASTER_BEACON (member_count=1) Note over A: inferior (fewer members) -> demote A-)B: JOIN_REQ (rejoin the superior master) B-->>A: KEY_GRANT + NODE_ASSIGN + MEMBER_LIST (unicast) Note over A,B: single cluster, one master11. Shared-port unicast demux (multi-cluster on one node)
When several controller clusters on one node share a multicast port (distinct
groups), the kernel demuxes multicast by group but unicast by port alone — a 1:1
reply can land on the wrong sibling's socket. The receiver recovers it: on a
cluster_idmismatch it forwards the still-encrypted datagram to the correctlocal cluster's worker via
ipc_send_rpc(), which decrypts it with its own key.sequenceDiagram autonumber participant P as peer (cluster 2) participant W1 as worker cluster 1 participant W2 as worker cluster 2 P-->>W1: unicast reply for cluster 2 (arrives on wrong socket) Note over W1: cleartext cluster_id=2 != 1 W1->>W2: ipc_send_rpc(still-encrypted datagram) Note over W2: decrypt with cluster-2 key, process normally Note over W1,W2: forwarded once, never re-forwarded - no loopComplexity impact
Compatibility & deployment
RESYNCtype, ALIVE/MASTER_ALIVElayout, unicast routing). All nodes of a cluster must run this build — old and
new cannot interoperate; deploy with a coordinated cutover. A node that doesn't
understand the digest simply advertises none and is never asked to resync.
cc_→cl_ctr_/CC_→CL_CTR_rename (functions, types,macros); the KDF label and bootstrap salt were renamed too, which changes the
derived keys — another reason for the coordinated cutover. Wire magic bytes
(
0xCC…) are unchanged.Testing
Validated on netns rigs and a live 6-node test cluster (2 controller clusters):
cold start → deterministic single master; single / chained / rapid-double failover;
higher-IP join → sticky backup; two clusters sharing port 3333 → both converge with
misdelivered unicast correctly re-routed; wrong-password node rejected without split
brain; staged stop→rejoin captured on the wire confirming the unicast join path.