USHIFT-6790: MicroShift Cluster 2 Cluster Communication by pmtk · Pull Request #1970 · openshift/enhancements

pmtk · 2026-04-14T11:24:02Z

No description provided.

openshift-ci-robot · 2026-04-14T11:24:08Z

@pmtk: This pull request references USHIFT-6790 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2026-04-14T11:24:24Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign sdodson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

agullon · 2026-04-14T14:08:26Z

+1. Pod-to-Pod, Pod-to-Service (ClusterIP), and
+   Pod-to-Service (DNS) communication between MicroShift
+   clusters with non-overlapping CIDRs.


Pod-to-Pod, Pod-to-Service (ClusterIP), and Pod-to-Service (DNS) must be always bidirectional? I mean, is it possible to have communication in only 1 direction: for example, from Pod(cluster1)->Pod(cluster2) but NOT from Pod(cluster2)->Pod(cluster1).

It's certainly doable, but is it what we want? Maybe as an accidental by-product, but I would not support it. WDYT?

I was asking to learn if this was already discussed. I'd say keep it simple, I'm happy if we advertised in the docs /enhacement something like: "By default it's always biderectional. One direction only is not tested neither supported." WDYT?

agullon · 2026-04-14T14:11:18Z

+         domain: "cluster-b.remote"
+   ```
+
+4. The user restarts MicroShift.


Which microshift? If I understand correctly MicroShift must be restarted every time the config under c2cc changes, right?.

Yeah, as with any configuration changes - we don't have hot reloading.

agullon · 2026-04-14T14:13:02Z

+       - 10.46.0.0/16
+   ```
+
+3. On each cluster, the user adds a `c2cc` section


What will happen if c2cc is only set in only 1 cluster, but not the other? Is this something we will support? Is it mandatory to configure it always in both clusters?

I guess it's related to #1970 (comment)

What would happen? Provided that firewall ports are open on both hosts, I would assume that the C1 -> C2 would succeed, but I have not tested it.

As I said before, I prefer to keep it simple and add a note: we only support when c2cc is set in both cluster. I think it's a limitation @DanielFroehlich will agree about. I just want to keep the scope as small as possible.

@DanielFroehlich what do you think?

agullon · 2026-04-14T14:13:51Z

+1. The user removes the `c2cc` section from the
+   config and restarts MicroShift.


Should this be removed from both clusters at the same time, right?

Yeah... I haven't considered partial configuration of "Cluster TO Cluster communication" :D

So far, there's nothing in the enhancement about MicroShifts communicating with each other to ensure proper cross-config. Maybe it's a GA feature?

Is it easy to add a check and if not report an error in the log? Do you think is a good idea?

What kind of check? Do you mean like ask other clusters what their c2cc configuration is?

If Daniel agree only bidirectional is supported I was thinking on different checks, from easy to more complex:

Every MicroShift check if c2cc is defined in the remote cluster, if not report an error.

Every MicroShift check if the subnets defined in the c2cc match the network subnet of the remote cluster.

I guess this is more complicated when more than 2 clusters, it's mostly about user experience and probably not worth it.

agullon · 2026-04-14T14:16:39Z

+### Implementation Details/Notes/Constraints
+
+**Configuration**:
+- `C2CC` struct with `RemoteClusters []RemoteCluster`


because RemoteClusters []RemoteCluster is an array, does it means we can configure more than 1 remote cluster? How would the routing work then?

For TP, I think we'll want to support only 1 remote, but it's an array to keep the API open for the future (if we need it).

Each cluster would have different Cluster and Service CIDRs:

# Cluster B network: clusterNetwork: - 10.45.0.0/16 serviceNetwork: - 10.46.0.0/16 # Cluster C network: clusterNetwork: - 10.48.0.0/16 serviceNetwork: - 10.49.0.0/16 - # Cluster D network: clusterNetwork: - 10.51.0.0/16 serviceNetwork: - 10.52.0.0/16

# Cluster A config c2cc: remoteClusters: - nextHop: "192.168.122.101" clusterNetwork: - "10.45.0.0/16" serviceNetwork: - "10.46.0.0/16" domain: "cluster-b.remote" - nextHop: "192.168.122.102" clusterNetwork: - "10.48.0.0/16" serviceNetwork: - "10.49.0.0/16" domain: "cluster-c.remote" - nextHop: "192.168.122.103" clusterNetwork: - "10.51.0.0/16" serviceNetwork: - "10.52.0.0/16" domain: "cluster-d.remote"

It's great we can configure more than 1 cluster to comunicate multiple cluster between them. I really like it. I think to think about this in deep mostly about how can this be tested and all the unexpected and conflicts in configurations.

I think would be good to capture it explicitly in Goal section that this is for 1 remote cluster and that multiple remotes is a non goal. Can have a separate EP later for multiple remotes.

vthapar · 2026-04-15T12:31:02Z

+
+* As a MicroShift user, I want to reach services on a
+  remote cluster using DNS names (e.g.,
+  `myservice.mynamespace.svc.remote-cluster.local`).


remote-cluster here refers to remote cluster name or a generic replacement for cluster?

That would come from the config

c2cc: remoteClusters: - nextHop: "192.168.122.101" clusterNetwork: - "10.45.0.0/16" serviceNetwork: - "10.46.0.0/16" domain: "cluster-b.remote" <-----

I haven't decided on specific names yet (if we'll put remote-cluster.local or cluster-b.remote as an example config)

vthapar · 2026-04-15T12:34:01Z

+1. Pod-to-Pod, Pod-to-Service (ClusterIP), and
+   Pod-to-Service (DNS) communication between MicroShift
+   clusters with non-overlapping CIDRs.
+2. Declarative configuration via the MicroShift config


Does this include validating peer configuration to make sure using correct CIDRs for a given remote or onus on user to make sure it is correct?

Right now it's on user to make sure the configs are correct.
Mutual configuration check between MicroShift instances could be a goal for GA.

vthapar · 2026-04-15T12:42:49Z

+### Implementation Details/Notes/Constraints
+
+**Configuration**:
+- `C2CC` struct with `RemoteClusters []RemoteCluster`


I think would be good to capture it explicitly in Goal section that this is for 1 remote cluster and that multiple remotes is a non goal. Can have a separate EP later for multiple remotes.

vthapar · 2026-04-15T12:52:56Z

+
+**CoreDNS Integration**: Per-remote-cluster server
+block with domain rewrite and forwarding to the remote
+DNS IP (10th IP in service CIDR).


Would be good to provide sample CoreDNS config for a server block.

Added to the enhancement

other-cluster.local:5353 { bufsize 1232 errors rewrite name suffix .other-cluster.local .cluster.local answer auto forward . 10.46.0.10:53 }

vthapar · 2026-04-15T13:01:54Z

+   probe pod discover the remote probe pod's IP? DNS
+   is not reliable for this since the user will be
+   able to override CoreDNS configuration completely.
+   Maybe we can hardcode Cluster IP like the CoreDNS?


Can you clarify "user will be able to override CoreDNS configuration completely"?

One option can be to deploy probe pods as headless service with single pod and use same name for probe pod service on all clusters.

We have another feature (OCPSTRAT-2998) for complete override of the config.
Let me clarify the enhancement

And this is what the customer is planning to do, so the other-cluster.local:5353 { on our side is more for completeness at this time.

Probably unnecessary complexity

Let's only allow for traffic from Pods to Pods/Services, no traffic from host level

openshift-ci · 2026-04-17T09:30:11Z

@pmtk: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Apr 14, 2026

openshift-ci bot requested review from bparees and hasbro17 April 14, 2026 11:24

MicroShift Cluster 2 Cluster Communication

8814e87

pmtk force-pushed the microshift-c2cc branch from 0408293 to 8814e87 Compare April 14, 2026 11:30

agullon reviewed Apr 14, 2026

View reviewed changes

Comment thread enhancements/microshift/microshift-cluster-to-cluster-connectivity.md Outdated

agullon reviewed Apr 14, 2026

View reviewed changes

pmtk added 2 commits April 15, 2026 10:49

Clarify microshift configs and restarts

770657b

IP -> ClusterIP

7431dc2

vthapar reviewed Apr 15, 2026

View reviewed changes

pmtk added 10 commits April 15, 2026 17:09

Drop 'adaptive reconciliation time'

0bf1082

Probably unnecessary complexity

Event-driven reconciliation

911e141

Make sure headless Services also work

3ffb058

Test network policies

451454c

Default NetworkPolicy to block traffic from nextHop IP/CIDR

e5df9f0

Let's only allow for traffic from Pods to Pods/Services, no traffic from host level

Enhance IPSec strictness

c9a07f5

Prevent Host->Pod/Svc

6266f59

Ensure no regressions but running existing networking suites

78c4fee

Add example of CoreDNS block for remote cluster

9b1cd2d

Mention feature to override CoreDNS config completely

f5a3a46

		1. The user removes the `c2cc` section from the
		config and restarts MicroShift.

Conversation

pmtk commented Apr 14, 2026

Uh oh!

openshift-ci-robot commented Apr 14, 2026 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Apr 14, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

agullon Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Apr 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

openshift-ci-robot commented Apr 14, 2026 •

edited by openshift-ci bot

Loading

agullon Apr 15, 2026 •

edited

Loading