[BYDB-Replica] Add integration test for inter-cluster replication resilience

[hanahmily]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

We need to verify that BanyanDB’s global liaison remains available and routes queries correctly when one entire cluster fails. Specifically, data written in both clusters, and client queries against the global liaison, should succeed even if one cluster is down.

This issue tracks the creation of an automated integration test that:

• Brings up two independent BanyanDB clusters (A and B) with cross-cluster replication via the global liaison.
• Writes data to cluster A and B.
• Simulates a full cluster failure (cluster A or B down).
• Executes queries through the global liaison.
• Verifies query success and data correctness from the surviving cluster.

Use case

No response

Related issues

No response

Are you willing to submit a pull request to implement this on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct

[bitflicker64]

@wu-sheng @hanahmily I’m interested in taking this up and related work.

[UfBt12qJNZKcw]

When implementing the integration test, consider using the existing test_replication.py framework to ensure consistency in testing style and coverage. Additionally, how do you plan to simulate the full cluster failure — will you be using a specific failure mode or just shutting down the cluster process?

[bitflicker64]

@hanahmily @UfBt12qJNZKcw hey i did some digging into the test setup and the existing e2e framework uses docker-compose + yaml based tests not python so i think i can follow the same pattern for this one

ive worked on similar multi cluster docker setups before like apache/hugegraph#2952 which is a multi-node docker compose migration so this fits my stack pretty well

i can take this up if its still available

[hanahmily]

I recommand using kind k8s cluster to write the test. banyandb is k8s native, you can use helm to deploy the cluster easily.