From 32f8ada1d2864d753c5269a2bbf2a264f77df29a Mon Sep 17 00:00:00 2001 From: ktroller Date: Mon, 13 Jul 2026 18:17:35 -0300 Subject: [PATCH 01/17] W-23431158: Make hadr-guide the HA/DR hub with deployment-model routing Add a High Availability and Disaster Recovery by Deployment Model section with a decision matrix linking the CloudHub 2.0, CloudHub, Runtime Fabric, and Hybrid Standalone guides. Existing concept content is preserved. Co-authored-by: Cursor --- modules/ROOT/pages/hadr-guide.adoc | 33 +++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-) diff --git a/modules/ROOT/pages/hadr-guide.adoc b/modules/ROOT/pages/hadr-guide.adoc index 5be099f74..ab882d05d 100644 --- a/modules/ROOT/pages/hadr-guide.adoc +++ b/modules/ROOT/pages/hadr-guide.adoc @@ -221,10 +221,37 @@ System backups are a major component of a solid disaster recovery program. There |Zero data loss, 15 mins out of date? |=== -For deployment-specific disaster recovery details, see: +For deployment-specific high availability and disaster recovery details, see <>. -* xref:runtime-manager::cloudhub-hadr.adoc[]. -* xref:cloudhub-2::ch2-ha-dr.adoc[]. +[[ha-dr-by-deployment-model]] +== High Availability and Disaster Recovery by Deployment Model + +The high availability (HA) mechanisms and disaster recovery (DR) responsibilities differ depending on how you deploy Mule. Use the following table to identify the approach for your deployment model, and then see the deployment-specific guide for details. + +[%header,cols="20a,30a,35a,15a"] +|=== +|Deployment Model |Primary HA Mechanism |DR Strategy and Responsibility |Cross-Region + +|xref:cloudhub-2::ch2-ha-dr.adoc[CloudHub 2.0] +|Multiple replicas across availability zones, managed automatically by MuleSoft. +|Customer-owned multiregion strategy using a bring-your-own (BYO) global load balancer. MuleSoft manages in-region resilience. +|Yes, with the Global Deployment entitlement. + +|xref:cloudhub::cloudhub-hadr.adoc[CloudHub] +|Multiple workers across availability zones, managed automatically by MuleSoft. +|Customer-owned cross-region failover. MuleSoft manages in-region resilience. +|Customer-managed. + +|xref:runtime-fabric::rtf-ha-dr.adoc[Runtime Fabric] +|A minimum of two replicas across availability zones, with optional Horizontal Pod Autoscaling (HPA). +|Backup and restore, and failover to a standby cluster. Customer-owned. +|Customer-managed. + +|xref:hybrid-standalone::hybrid-standalone-hadr.adoc[Hybrid Standalone] +|Clustering with shared memory and automatic failover. +|Customer-owned backup, restore, and DR. MuleSoft provides control-plane availability and best practices. +|Customer-managed. +|=== == Keep Integrations Stateless From a0f5ab615a1b50649eb496cd8f0a73dc86c11a60 Mon Sep 17 00:00:00 2001 From: ktroller Date: Mon, 13 Jul 2026 18:44:56 -0300 Subject: [PATCH 02/17] W-23431158: Rewrite HA/DR hub for flow, parallelism, and active voice Co-authored-by: Cursor --- modules/ROOT/pages/hadr-guide.adoc | 148 ++++++++++++++--------------- 1 file changed, 73 insertions(+), 75 deletions(-) diff --git a/modules/ROOT/pages/hadr-guide.adoc b/modules/ROOT/pages/hadr-guide.adoc index ab882d05d..7ba80067a 100644 --- a/modules/ROOT/pages/hadr-guide.adoc +++ b/modules/ROOT/pages/hadr-guide.adoc @@ -3,11 +3,11 @@ ifndef::env-site,env-github[] include::_attributes.adoc[] endif::[] -High availability (HA) and disaster recovery (DR) strategies protect your Mule applications from downtime caused by infrastructure failures, disasters, or maintenance. Deploy Mule runtime in clustered topologies, configure active-active or standby environments, and define Service Level Agreements (SLAs) to meet your uptime requirements. Mule clustering provides automatic failover and shares transient state across nodes to ensure continuous operation. +High availability (HA) and disaster recovery (DR) strategies protect your Mule applications from downtime caused by infrastructure failures, disasters, or maintenance. To meet your uptime requirements, deploy Mule runtime engine in clustered topologies, configure active-active or standby environments, and define Service Level Agreements (SLAs). Mule clustering fails over automatically and shares transient state across nodes to keep operations continuous. -Planned or unplanned, infrastructure and application downtime can come at any time, from any direction, and in any form. The ability to keep an organization operational during a technology outage, facility destruction, loss of personnel, or loss of critical third party services is critical to preventing irreversible damage to a business. With the increasing global shift to e-commerce models and their reliance on 24/7 application uptime, high availability (HA) and disaster recovery (DR) impact the financial health of organizations. +Downtime strikes at any time, from any direction, and in any form, whether you plan for it or not. Keeping an organization operational during a technology outage, a facility loss, a personnel loss, or a third-party service failure prevents irreversible damage to the business. As organizations shift to e-commerce models that depend on 24/7 uptime, HA and DR increasingly determine their financial health. -This table shows how even the least amount of downtime can add up to negatively affect an organization: +The following table shows how even small amounts of downtime add up and hurt an organization: [%header,cols="25a,25a,25a,25a"] |=== @@ -20,48 +20,46 @@ This table shows how even the least amount of downtime can add up to negatively == High Availability Versus Disaster Recovery -* *High availability (HA)* - The measure of a system's ability to remain accessible in the event of a system component failure. Generally, HA is implemented by building multiple levels of fault tolerance and load balancing capabilities into a system. +* *High availability (HA)* - Measures a system's ability to remain accessible when a component fails. You implement HA by building multiple levels of fault tolerance and load balancing into a system. -* *Disaster recovery (DR)* - The process by which a system is restored to a previous acceptable state, after a natural (flooding, tornadoes, earthquakes, fires) or man-made (power failures, server failures, misconfigurations) disaster. +* *Disaster recovery (DR)* - Restores a system to an acceptable previous state after a natural disaster (flooding, tornadoes, earthquakes, fires) or a man-made disaster (power failures, server failures, misconfigurations). -While they both increase overall availability, the notable difference is that with HA there is generally no loss of service. HA retains the service and DR retains the data, but with DR, there is usually a slight loss of service while the DR plan executes and the system restores. +Both strategies increase overall availability, but they differ in one key way: HA generally preserves the service with no loss, whereas DR preserves the data but usually incurs a brief loss of service while the DR plan runs and the system restores. == Deploying Mule for HA and DR Strategies -You can deploy the Mule runtime in many different topologies to address your HA and DR strategies. One method is through the use of high availability clustering, which provides basic failover capability for Mule. +You can deploy Mule runtime engine in many topologies to support your HA and DR strategies. One common method, high-availability clustering, gives Mule basic failover capability. -When the primary Mule runtime becomes unavailable, for example, because of a fatal JVM or hardware failure or it's taken offline for maintenance, a backup Mule runtime immediately becomes the primary node and resumes processing where the failed instance left off. +When the primary Mule runtime becomes unavailable—because of a fatal JVM or hardware failure, or because you take it offline for maintenance—a backup Mule runtime immediately takes over as the primary node and resumes processing where the failed instance stopped. -After a system administrator recovers a failed Mule runtime server and puts it back online, that server automatically becomes the backup node. +After you recover the failed Mule runtime server and bring it back online, that server automatically becomes the backup node. image::hadr-cluster.png["A clustered server environment handling incoming and outgoing messages"] -Seamless failover is made possible by a distributed memory store that shares all transient state information among clustered Mule runtimes, such as: +A distributed memory store makes seamless failover possible by sharing all transient state across clustered Mule runtimes, including: * SEDA service event queues * In-memory message queues -As you build your Mule application, it's important to think critically about how best to architect your application to achieve the desired availability, fault tolerance, and performance characteristics. +As you build your Mule application, think critically about how to architect it for the availability, fault tolerance, and performance you need. == Creating Effective SLAs -The Service Level Agreement (SLA) outlines the specific definition of what is acceptable. +A Service Level Agreement (SLA) defines what counts as acceptable service. -The basic definition of high availability is that the service is up and -functioning and properly processing requests and responses. This doesn't -mean the service always operates at full capacity. Systems designed with high availability protect against interruptions and prevent outages from happening in the first place. The SLA identifies expectations for all stakeholders. +High availability means the service stays up, functions correctly, and processes requests and responses properly. It does not mean the service always runs at full capacity. Systems that you design for high availability prevent outages and protect against interruptions. The SLA sets expectations for every stakeholder. -SLA Requirements: +An effective SLA does the following: -* Single points of failure are eliminated -* Traffic/requests are redirected/handled -* Detection of failures +* Eliminates single points of failure +* Redirects and handles traffic +* Detects failures -For example a basic service may have these SLAs: +For example, a basic service might define these SLAs: -* The normal operation of the service handles 1000 transactions per sec with 1 sec response times. -* The total downtime per year is 0.5% or 1.83 days. -* Minimal acceptable service impact level is 100 transactions per sec with 1 sec response times for a period of 2 hours a week. +* Normal operation handles 1,000 transactions per second with 1-second response times. +* Total downtime per year stays at or below 0.5 percent, or 1.83 days. +* Minimum acceptable service handles 100 transactions per second with 1-second response times for up to 2 hours a week. See also: @@ -70,14 +68,14 @@ See also: == High Availability Options -You can achieve high availability through the use of clustering and load balancing of the nodes. Depending on the defined SLA, four HA options are possible with Mule: +You achieve high availability by clustering and load balancing nodes. Depending on your SLA, Mule supports four HA options: * <> * <> * <> * <> -NOTE: Often these options are associated with a disaster recovery strategy. +NOTE: These options often pair with a disaster recovery strategy. === Cold Standby @@ -85,8 +83,8 @@ NOTE: Often these options are associated with a disaster recovery strategy. |=== |Diagram |Description |Downtime |image:hadr-cold-standby.png["A cold standby configuration in a server setup"] -|The Mule environment is installed and configured, however one or more operating systems aren't running. This can be a backup of a production system/virtual machine. The environment of the operating system plus the Mule runtime has been started after an outage is detected. -|Some - The time it takes to start the environment and direct traffic. +|You install and configure the Mule environment but leave one or more operating systems stopped—for example, as a backup of a production system or virtual machine. You start the operating system and the Mule runtime after you detect an outage. +|Some. Covers the time to start the environment and route traffic. |=== === Warm Standby @@ -95,8 +93,8 @@ NOTE: Often these options are associated with a disaster recovery strategy. |=== |Diagram |Description |Downtime |image:hadr-warm-standby.png["A warm standby setup for server redundancy"] -|The Mule environment is installed and configured. However the Mule runtimes aren't running; only the operating systems. The Mule runtime instance is started after an outage is detected. -|Little - The time it takes for the Mule runtime instances to start and to route traffic to the environment. +|You install and configure the Mule environment and run the operating systems but leave the Mule runtimes stopped. You start the Mule runtime after you detect an outage. +|Little. Covers the time for the Mule runtime instances to start and route traffic to the environment. |=== === Hot Standby - Active-Passive @@ -105,8 +103,8 @@ NOTE: Often these options are associated with a disaster recovery strategy. |=== |Diagram |Description |Downtime |image:hadr-active-passive.png["An active-passive server setup in a high-availability configuration"] -|The Mule environment is installed, configured, and fully running. However, it's not processing requests until an outage is detected. -|Minimal to none - The time to route traffic to an environment. +|You install, configure, and fully run the Mule environment, but it does not process requests until you detect an outage. +|Minimal to none. Covers the time to route traffic to the environment. |=== === Active-Active @@ -115,21 +113,21 @@ NOTE: Often these options are associated with a disaster recovery strategy. |=== |Diagram |Description |Downtime |image:hadr-active-active1.png["A load balancer distributing traffic to a cluster of Mule instances in an active-active configuration"] -|Load Balanced clustered environments -There are two or more Mule environments (each environment has its own cluster) that are fully operational. The load balancer is directing traffic to all of the environments. -|None - There is no service downtime. +|Load-balanced clustered environments: +Two or more fully operational Mule environments run at the same time, each with its own cluster, and the load balancer distributes traffic across all of them. +|None. The service experiences no downtime. |image:hadr-active-active2.png["An active-active server configuration with a load balancer and clustering"] -|Load Balanced single clustered environment -There are two or more Mule environments, however they're part of the same clustered environment. To achieve this scenario, the network latency between environments must be less than 10ms. -|None - There is no service downtime. +|Load-balanced single clustered environment: +Two or more Mule environments belong to the same cluster. This scenario requires network latency between environments to stay below 10 ms. +|None. The service experiences no downtime. |=== == High-Availability for On-Premises Deployment Models [NOTE] ==== -When you deploy an application to an on-premises Mule cluster by using Runtime Manager, deployment artifacts are staged to cluster nodes sequentially. -Sequential staging doesn’t guarantee that the application starts on one node before startup begins on another. Application startup and readiness across cluster nodes aren’t ordered and can overlap. +When you deploy an application to an on-premises Mule cluster through Runtime Manager, Runtime Manager stages deployment artifacts to cluster nodes sequentially. +Sequential staging does not guarantee that the application starts on one node before startup begins on another. Application startup and readiness across cluster nodes follow no fixed order and can overlap. ==== * <> @@ -138,76 +136,76 @@ Sequential staging doesn’t guarantee that the application starts on one node b === Active-Active Clustering Deployment Model -It's plausible that two nodes in a clustered or load balanced environment can support 1,500 TPS with one-second responses. In this state, the normal operation of the SLA is being met. If a node fails, the service is impacted. However, the impact doesn't breach the SLA because the node is able to handle 700 TPS with one-second responses; well above the agreed upon acceptable impact level. +Two nodes in a clustered or load-balanced environment can plausibly support 1,500 TPS with 1-second responses, which meets the normal SLA. If one node fails, the service degrades but still handles 700 TPS with 1-second responses—well above the agreed acceptable impact—so the failure does not breach the SLA. image::hadr-aa-clustering.png["An active-active clustering setup with load balancers and multiple servers"] -Distribute the load evenly among multiple Mule nodes: +Distribute the load evenly across multiple Mule nodes: -* All nodes offer the same capabilities -* All nodes are active at the same time. +* All nodes offer the same capabilities. +* All nodes stay active at the same time. *Costs* -Vary depending on SLA requirements. This model needs two nodes to satisfy SLA. If the SLA's acceptable service impact changes to the terms stated in the normal operations then at a minimum the environment needs three nodes to accommodate one node failure. More nodes may be required depending on the probability of not having at least two nodes running. +Costs vary with your SLA requirements. This model needs two nodes to meet the SLA. If the acceptable service impact rises to match normal operations, the environment needs at least three nodes to absorb a single node failure. High-probability failure scenarios can require even more nodes. === Active-Active Clustering Fault Tolerance Deployment Model -Fault tolerance means a failure within the system doesn't impact the service at all. This differs from high availability as service impact and downtime are tolerated. +Fault tolerance means a failure within the system does not affect the service at all. It differs from high availability, which tolerates some service impact and downtime. image::hadr-fault-tolerant.png["A fault-tolerant system with redundant nodes and a load balancer"] -Fault tolerance differs from high availability by providing additional resources that allow an application to continue functioning after a component failure without interruption. Fault tolerant environments are more costly than high available environments. +Fault tolerance differs from high availability by adding resources that let an application keep running—without interruption—after a component fails. Fault-tolerant environments cost more than highly available environments. -The degree of fault tolerance requires the probability of system failures. Take the SLA example highlighted under high availability and make the minimal acceptable service impact level match the normal operation requirement. +The degree of fault tolerance you need depends on the probability of system failures. Take the earlier SLA example and raise the minimum acceptable service impact to match the normal operation requirement. -The new overall SLA now requires the system to be able to handle 1000 transactions per sec with 1 sec response times, zero downtime, and zero service impact. +The new SLA now requires the system to handle 1,000 transactions per second with 1-second response times, zero downtime, and zero service impact. -If the probability of having more than 1 node fail is low then the architecture would simply require 3 nodes. However, if the probability of more than 1 node is higher than acceptable, more than 3 nodes are required to accommodate multiple failures. +If the probability of losing more than one node stays low, the architecture needs just three nodes. If that probability rises above what you accept, the architecture needs more than three nodes to absorb multiple failures. *Costs* -More costly due to the required redundancy in order to meet defined SLA. +This model costs more because it requires the redundancy to meet the defined SLA. === Zero Downtime Deployment Model -The goal is to make changes to the environment without impacting the SLAs; including upgrading infrastructure and the applications running on the infrastructure. Typically zero downtime deployments leverage a side-by-side deployment, where the old and new coexist for a short period of time. This is in contrast to an in-place deployment where the service may experience reduced capacity to complete downtime. +This model lets you change the environment—including upgrading the infrastructure and the applications that run on it—without breaching your SLAs. Zero-downtime deployments typically use a side-by-side approach, where the old and new environments coexist briefly. An in-place deployment, by contrast, can reduce capacity or cause complete downtime. -https://www.gartner.com/en/information-technology/glossary/continuous-operations[Gartner] defines continuous operations as “those characteristics of a data-processing system that reduce or eliminate the need for planned downtime, such as scheduled maintenance. One element of 24-hour-a-day, seven-day-a-week operation”. +https://www.gartner.com/en/information-technology/glossary/continuous-operations[Gartner] defines continuous operations as "those characteristics of a data-processing system that reduce or eliminate the need for planned downtime, such as scheduled maintenance. One element of 24-hour-a-day, seven-day-a-week operation". -The baseline production environment is the current operating environment. A new environment is created with the changes (upgraded runtimes, configurations, new applications). A small percentage of traffic flows to the new environment and increases as the confidence in the new environment increases. The baseline production environment continues its use until the new environment is fully operational (it's handling 100% of the traffic). After the new environment is accepting all traffic, it becomes the new baseline production environment and the previous baseline production environment terminates. +The baseline production environment is your current operating environment. You create a new environment with the changes—upgraded runtimes, new configurations, and new applications. A small percentage of traffic flows to the new environment and increases as your confidence grows. The baseline environment keeps serving traffic until the new environment handles 100 percent of the traffic. After the new environment accepts all traffic, it becomes the new baseline production environment, and the previous baseline terminates. -This example is assuming each environment is using the same number of Mule runtimes and cores. It's plausible that the new environment has either more or fewer Mule runtimes and cores than the baseline environment. +This example assumes that each environment uses the same number of Mule runtimes and cores, but the new environment can use more or fewer. [%header,cols="60a,40a"] |=== |Deployment Step |Diagram -|New Production Environment deployed and a small percentage of traffic is routed to new environment. +|You deploy the new production environment and route a small percentage of traffic to it. |image:hadr-zero-downtime-1.png["A traffic distribution in a smaller percentage from current to new production environments in a high-availability deployment"] -|Confidence in the new environment continues to increase and more traffic is routed to it. +|Your confidence in the new environment grows, and you route more traffic to it. |image:hadr-zero-downtime-2.png["A traffic distribution in a higher percentage from current to new production environment in a high-availability deployment"] -|All traffic has been routed to the new environment. +|You route all traffic to the new environment. |image:hadr-zero-downtime-3.png["A zero-downtime deployment with traffic shifting from an current to a new environment"] -|All traffic has been routed to the new environment, which has been promoted to the baseline production environment; the previous baseline environment has been terminated. +|You promote the new environment to baseline production and terminate the previous baseline. |image:hadr-zero-downtime-4.png["A traffic routing to a new production environment completed in a high-availability environment"] |=== *Costs* -This deployment method may temporarily add capacity to the service (can be a few minutes, hours, or days). +This method can temporarily add capacity to the service for a few minutes, hours, or days. == Disaster Recovery *How quickly can your company get back to work after an IT emergency?* -Disaster recovery (DR) is the process by which a system is restored to a previous acceptable state, after a natural or man-made disaster. For DR, use measurable business requirements such as Recovery Time Objective (RTO) and Recovery Point Objective (RPO), to drive your DR plan. +Disaster recovery (DR) restores a system to an acceptable previous state after a natural or man-made disaster. To drive your DR plan, use measurable business requirements such as Recovery Time Objective (RTO) and Recovery Point Objective (RPO). -Disaster recovery is about your Recovery Point Objective (RPO) and your Recovery Time Objective (RTO). RPO is the "point" that you return to after an IT disaster. For example, if you back up the system every 24 hours, your RPO is a maximum of the past 24 hours. RTO, on the other hand, is how quickly you can restore to your RPO and get back to business. This includes activities like the time it takes to get your spare equipment to start running your backups if your primary equipment isn't working. +DR centers on your RPO and RTO. RPO is the point you return to after a disaster. For example, if you back up the system every 24 hours, your RPO is at most the previous 24 hours. RTO is how quickly you restore to your RPO and get back to business, which includes the time to bring spare equipment online to run your backups when your primary equipment fails. -System backups are a major component of a solid disaster recovery program. There are three types of recovery: cold, warm, and hot. +Backups form a major component of a solid disaster recovery program. Recovery comes in three types: cold, warm, and hot. [%header%autowidth.spread] |=== @@ -226,33 +224,33 @@ For deployment-specific high availability and disaster recovery details, see < Date: Mon, 13 Jul 2026 18:51:44 -0300 Subject: [PATCH 03/17] W-23431158: Optimize HA/DR hub for AI retrieval (prose recap of deployment matrix) Co-authored-by: Cursor --- modules/ROOT/pages/hadr-guide.adoc | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/modules/ROOT/pages/hadr-guide.adoc b/modules/ROOT/pages/hadr-guide.adoc index 7ba80067a..bd5956f56 100644 --- a/modules/ROOT/pages/hadr-guide.adoc +++ b/modules/ROOT/pages/hadr-guide.adoc @@ -224,7 +224,7 @@ For deployment-specific high availability and disaster recovery details, see < Date: Mon, 13 Jul 2026 19:05:47 -0300 Subject: [PATCH 04/17] W-23431158: Use contractions and remove 'the following' in HA/DR hub Co-authored-by: Cursor --- modules/ROOT/pages/hadr-guide.adoc | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/modules/ROOT/pages/hadr-guide.adoc b/modules/ROOT/pages/hadr-guide.adoc index bd5956f56..af81f66b4 100644 --- a/modules/ROOT/pages/hadr-guide.adoc +++ b/modules/ROOT/pages/hadr-guide.adoc @@ -7,7 +7,7 @@ High availability (HA) and disaster recovery (DR) strategies protect your Mule a Downtime strikes at any time, from any direction, and in any form, whether you plan for it or not. Keeping an organization operational during a technology outage, a facility loss, a personnel loss, or a third-party service failure prevents irreversible damage to the business. As organizations shift to e-commerce models that depend on 24/7 uptime, HA and DR increasingly determine their financial health. -The following table shows how even small amounts of downtime add up and hurt an organization: +This table shows how even small amounts of downtime add up and hurt an organization: [%header,cols="25a,25a,25a,25a"] |=== @@ -47,9 +47,9 @@ As you build your Mule application, think critically about how to architect it f A Service Level Agreement (SLA) defines what counts as acceptable service. -High availability means the service stays up, functions correctly, and processes requests and responses properly. It does not mean the service always runs at full capacity. Systems that you design for high availability prevent outages and protect against interruptions. The SLA sets expectations for every stakeholder. +High availability means the service stays up, functions correctly, and processes requests and responses properly. It doesn't mean the service always runs at full capacity. Systems that you design for high availability prevent outages and protect against interruptions. The SLA sets expectations for every stakeholder. -An effective SLA does the following: +An effective SLA does these things: * Eliminates single points of failure * Redirects and handles traffic @@ -103,7 +103,7 @@ NOTE: These options often pair with a disaster recovery strategy. |=== |Diagram |Description |Downtime |image:hadr-active-passive.png["An active-passive server setup in a high-availability configuration"] -|You install, configure, and fully run the Mule environment, but it does not process requests until you detect an outage. +|You install, configure, and fully run the Mule environment, but it doesn't process requests until you detect an outage. |Minimal to none. Covers the time to route traffic to the environment. |=== @@ -127,7 +127,7 @@ Two or more Mule environments belong to the same cluster. This scenario requires [NOTE] ==== When you deploy an application to an on-premises Mule cluster through Runtime Manager, Runtime Manager stages deployment artifacts to cluster nodes sequentially. -Sequential staging does not guarantee that the application starts on one node before startup begins on another. Application startup and readiness across cluster nodes follow no fixed order and can overlap. +Sequential staging doesn't guarantee that the application starts on one node before startup begins on another. Application startup and readiness across cluster nodes follow no fixed order and can overlap. ==== * <> @@ -136,7 +136,7 @@ Sequential staging does not guarantee that the application starts on one node be === Active-Active Clustering Deployment Model -Two nodes in a clustered or load-balanced environment can plausibly support 1,500 TPS with 1-second responses, which meets the normal SLA. If one node fails, the service degrades but still handles 700 TPS with 1-second responses—well above the agreed acceptable impact—so the failure does not breach the SLA. +Two nodes in a clustered or load-balanced environment can plausibly support 1,500 TPS with 1-second responses, which meets the normal SLA. If one node fails, the service degrades but still handles 700 TPS with 1-second responses—well above the agreed acceptable impact—so the failure doesn't breach the SLA. image::hadr-aa-clustering.png["An active-active clustering setup with load balancers and multiple servers"] @@ -151,7 +151,7 @@ Costs vary with your SLA requirements. This model needs two nodes to meet the SL === Active-Active Clustering Fault Tolerance Deployment Model -Fault tolerance means a failure within the system does not affect the service at all. It differs from high availability, which tolerates some service impact and downtime. +Fault tolerance means a failure within the system doesn't affect the service at all. It differs from high availability, which tolerates some service impact and downtime. image::hadr-fault-tolerant.png["A fault-tolerant system with redundant nodes and a load balancer"] @@ -262,4 +262,4 @@ In every model, MuleSoft keeps in-region infrastructure and the control plane re == Keep Integrations Stateless -As a general design principle, keep your integrations stateless. Do not share transactional information between client invocations or scheduled executions. When the middleware must retain data because of a system limitation, persist it in an external store, such as a database or a messaging queue, rather than in the middleware infrastructure or memory. As the system scales—especially in the cloud—keep each worker or node's state and resources independent of the other workers. This model delivers better performance, scalability, and reliability. +As a general design principle, keep your integrations stateless. Don't share transactional information between client invocations or scheduled executions. When the middleware must retain data because of a system limitation, persist it in an external store, such as a database or a messaging queue, rather than in the middleware infrastructure or memory. As the system scales—especially in the cloud—keep each worker or node's state and resources independent of the other workers. This model delivers better performance, scalability, and reliability. From 17da1c8f9be7dc49010ae642ae3ece13fa14d67b Mon Sep 17 00:00:00 2001 From: ktroller Date: Mon, 13 Jul 2026 20:09:01 -0300 Subject: [PATCH 05/17] W-23431158: Restructure HA/DR guide as deployment-neutral landing page Reorganize hadr-guide.adoc so concepts come first, followed by the deployment-model matrix and a consolidated On-Premises section (clustering, standby options, and on-premises deployment models). Add contractions and bulleted Best Practices, add an On-Premises row to the matrix linking to the in-page section, and promote the guide to a top-level nav entry. Co-authored-by: Cursor --- modules/ROOT/nav.adoc | 2 +- modules/ROOT/pages/hadr-guide.adoc | 196 +++++++++++++++-------------- 2 files changed, 106 insertions(+), 92 deletions(-) diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index c2d2b584a..217363a93 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -37,6 +37,7 @@ include::partial$nav-app-dev.adoc[] *** xref:tuning-app-design.adoc[App Design] ** xref:tuning-test-validations.adoc[Performance Test Validations] ** xref:tuning-monitoring.adoc[Performance Monitoring] +* xref:hadr-guide.adoc[High Availability and Disaster Recovery] * xref:mule-standalone.adoc[Run Mule Runtime Engine On-Premises] ** xref:mule-deployment-model.adoc[On-Premises Deployment Model] ** xref:hardware-and-software-requirements.adoc[Hardware and Software Requirements] @@ -47,7 +48,6 @@ include::partial$nav-app-dev.adoc[] ** xref:choosing-the-right-clustering-topology.adoc[Cluster Runtime Instances] *** xref:mule-high-availability-ha-clusters.adoc[Mule Runtime HA Cluster Overview] *** xref:creating-and-managing-a-cluster-manually.adoc[Create and Manage a Cluster Manually] - *** xref:hadr-guide.adoc[High Availability and Disaster Recovery] * xref:mule-upgrade-tool.adoc[Mule Upgrade Tool] * xref:mule-troubleshooting-plugin.adoc[Mule Troubleshooting Plugin] * xref:using-maven-with-mule.adoc[Maven Support in Mule] diff --git a/modules/ROOT/pages/hadr-guide.adoc b/modules/ROOT/pages/hadr-guide.adoc index af81f66b4..e887cb87c 100644 --- a/modules/ROOT/pages/hadr-guide.adoc +++ b/modules/ROOT/pages/hadr-guide.adoc @@ -3,7 +3,7 @@ ifndef::env-site,env-github[] include::_attributes.adoc[] endif::[] -High availability (HA) and disaster recovery (DR) strategies protect your Mule applications from downtime caused by infrastructure failures, disasters, or maintenance. To meet your uptime requirements, deploy Mule runtime engine in clustered topologies, configure active-active or standby environments, and define Service Level Agreements (SLAs). Mule clustering fails over automatically and shares transient state across nodes to keep operations continuous. +High availability (HA) and disaster recovery (DR) strategies protect your Mule applications from downtime caused by infrastructure failures, disasters, or maintenance. The right approach depends on how you deploy Mule runtime engine: managed platforms such as CloudHub 2.0 build in much of the resilience for you, whereas deployments that you operate require you to design clustering, failover, and backups yourself. This guide explains the core HA and DR concepts and then maps each deployment model to its HA mechanism and DR responsibilities. Downtime strikes at any time, from any direction, and in any form, whether you plan for it or not. Keeping an organization operational during a technology outage, a facility loss, a personnel loss, or a third-party service failure prevents irreversible damage to the business. As organizations shift to e-commerce models that depend on 24/7 uptime, HA and DR increasingly determine their financial health. @@ -26,23 +26,6 @@ This table shows how even small amounts of downtime add up and hurt an organizat Both strategies increase overall availability, but they differ in one key way: HA generally preserves the service with no loss, whereas DR preserves the data but usually incurs a brief loss of service while the DR plan runs and the system restores. -== Deploying Mule for HA and DR Strategies - -You can deploy Mule runtime engine in many topologies to support your HA and DR strategies. One common method, high-availability clustering, gives Mule basic failover capability. - -When the primary Mule runtime becomes unavailable—because of a fatal JVM or hardware failure, or because you take it offline for maintenance—a backup Mule runtime immediately takes over as the primary node and resumes processing where the failed instance stopped. - -After you recover the failed Mule runtime server and bring it back online, that server automatically becomes the backup node. - -image::hadr-cluster.png["A clustered server environment handling incoming and outgoing messages"] - -A distributed memory store makes seamless failover possible by sharing all transient state across clustered Mule runtimes, including: - -* SEDA service event queues -* In-memory message queues - -As you build your Mule application, think critically about how to architect it for the availability, fault tolerance, and performance you need. - == Creating Effective SLAs A Service Level Agreement (SLA) defines what counts as acceptable service. @@ -66,7 +49,96 @@ See also: * xref:api-manager::defining-sla-tiers.adoc[Defining SLA Tiers] * xref:api-manager::rate-limiting-and-throttling-sla-based-policies.adoc[Rate Limiting and Throttling - SLA-Based Policies] -== High Availability Options +== Recovery Objectives: RTO and RPO + +*How quickly can your company get back to work after an IT emergency?* + +Disaster recovery (DR) restores a system to an acceptable previous state after a natural or man-made disaster. To drive your DR plan, use measurable business requirements such as Recovery Time Objective (RTO) and Recovery Point Objective (RPO). + +DR centers on your RPO and RTO. RPO is the point you return to after a disaster. For example, if you back up the system every 24 hours, your RPO is at most the previous 24 hours. RTO is how quickly you restore to your RPO and get back to business, which includes the time to bring spare equipment online to run your backups when your primary equipment fails. + +Backups form a major component of a solid disaster recovery program. Recovery comes in three types: cold, warm, and hot. + +[%header%autowidth.spread] +|=== +|Term |Definition |Example +|Recovery Time Objective (RTO) +|How quickly do you need to recover this asset? +|1 min? 15 min? 1 hr? 4 hrs? 1 day? + +|Recovery Point Objective (RPO) +|How fresh must the recovery be for the asset? +|Zero data loss, 15 mins out of date? +|=== + +[[ha-dr-by-deployment-model]] +== High Availability and Disaster Recovery by Deployment Model + +HA mechanisms and DR responsibilities differ depending on how you deploy Mule. The deployment-model comparison table maps each model to its primary HA mechanism, its DR strategy and responsibility, and its cross-region support, so you can pick the right approach and then read the deployment-specific guide. + +[%header,cols="20a,30a,35a,15a"] +|=== +|Deployment Model |Primary HA Mechanism |DR Strategy and Responsibility |Cross-Region + +|xref:cloudhub-2::ch2-ha-dr.adoc[CloudHub 2.0] +|MuleSoft runs multiple replicas across availability zones automatically. +|You own the multiregion strategy and provide a bring-your-own (BYO) global load balancer. MuleSoft manages in-region resilience. +|Yes, with the Global Deployment entitlement. + +|xref:cloudhub::cloudhub-hadr.adoc[CloudHub] +|MuleSoft runs multiple workers across availability zones automatically. +|You own cross-region failover. MuleSoft manages in-region resilience. +|You manage it. + +|xref:runtime-fabric::rtf-ha-dr.adoc[Runtime Fabric] +|You run a minimum of two replicas across availability zones, with optional Horizontal Pod Autoscaling (HPA). +|You own backup, restore, and failover to a standby cluster. +|You manage it. + +|xref:hybrid-standalone::hybrid-standalone-hadr.adoc[Hybrid Standalone] +|You cluster runtimes to share memory and fail over automatically. +|You own backup, restore, and DR. MuleSoft provides control-plane availability and best practices. +|You manage it. + +|<> +|You cluster Mule runtime servers for automatic failover, using active-active, fault-tolerant, or zero-downtime topologies. +|You own the HA architecture, backup, restore, and DR for your servers and data. +|You manage it. +|=== + +To choose the guide that matches how you run Mule: + +* For fully managed, multi-replica resilience on AWS, read xref:cloudhub-2::ch2-ha-dr.adoc[CloudHub 2.0 High Availability and Disaster Recovery]. +* For fully managed, multi-worker resilience on AWS, read xref:cloudhub::cloudhub-hadr.adoc[CloudHub High Availability and Disaster Recovery]. +* For Kubernetes-based clusters that you operate, read xref:runtime-fabric::rtf-ha-dr.adoc[Runtime Fabric High Availability and Disaster Recovery]. +* For servers you control that you manage centrally through Anypoint Platform, read xref:hybrid-standalone::hybrid-standalone-hadr.adoc[Hybrid Standalone High Availability and Disaster Recovery]. +* For on-premises Mule clustering architectures that you operate yourself, see <> in this guide. + +MuleSoft keeps the Anypoint Platform control plane resilient. In the managed runtime models, MuleSoft also keeps in-region infrastructure resilient; in self-managed and on-premises models, you own the infrastructure. In every model, you own any strategy that spans regions. + +[[ha-on-premises]] +== High Availability and Disaster Recovery for On-Premises Deployments + +When you run Mule runtime engine on servers that you manage, you build HA and DR yourself. You cluster runtimes for failover, choose a standby topology that matches your SLA, and back up your environment and data. MuleSoft provides the runtime clustering capabilities and the architectural guidance in this section. + +=== On-Premises Clustering and Failover + +You can deploy Mule runtime engine in many topologies to support your HA and DR strategies. One common method, high-availability clustering, gives Mule basic failover capability. + +When the primary Mule runtime becomes unavailable—because of a fatal JVM or hardware failure, or because you take it offline for maintenance—a backup Mule runtime immediately takes over as the primary node and resumes processing where the failed instance stopped. + +After you recover the failed Mule runtime server and bring it back online, that server automatically becomes the backup node. + +image::hadr-cluster.png["A clustered server environment handling incoming and outgoing messages"] + +A distributed memory store makes seamless failover possible by sharing all transient state across clustered Mule runtimes, including: + +* SEDA service event queues +* In-memory message queues + +As you build your Mule application, think critically about how to architect it for the availability, fault tolerance, and performance you need. + +=== High Availability Options You achieve high availability by clustering and load balancing nodes. Depending on your SLA, Mule supports four HA options: @@ -77,7 +149,7 @@ You achieve high availability by clustering and load balancing nodes. Depending NOTE: These options often pair with a disaster recovery strategy. -=== Cold Standby +==== Cold Standby [%header%autowidth.spread] |=== @@ -87,7 +159,7 @@ NOTE: These options often pair with a disaster recovery strategy. |Some. Covers the time to start the environment and route traffic. |=== -=== Warm Standby +==== Warm Standby [%header%autowidth.spread] |=== @@ -97,7 +169,7 @@ NOTE: These options often pair with a disaster recovery strategy. |Little. Covers the time for the Mule runtime instances to start and route traffic to the environment. |=== -=== Hot Standby - Active-Passive +==== Hot Standby - Active-Passive [%header%autowidth.spread] |=== @@ -107,7 +179,7 @@ NOTE: These options often pair with a disaster recovery strategy. |Minimal to none. Covers the time to route traffic to the environment. |=== -=== Active-Active +==== Active-Active [%header%autowidth.spread] |=== @@ -122,7 +194,7 @@ Two or more Mule environments belong to the same cluster. This scenario requires |None. The service experiences no downtime. |=== -== High-Availability for On-Premises Deployment Models +=== On-Premises Deployment Models [NOTE] ==== @@ -134,7 +206,7 @@ Sequential staging doesn't guarantee that the application starts on one node bef * <> * <> -=== Active-Active Clustering Deployment Model +==== Active-Active Clustering Deployment Model Two nodes in a clustered or load-balanced environment can plausibly support 1,500 TPS with 1-second responses, which meets the normal SLA. If one node fails, the service degrades but still handles 700 TPS with 1-second responses—well above the agreed acceptable impact—so the failure doesn't breach the SLA. @@ -149,7 +221,7 @@ Distribute the load evenly across multiple Mule nodes: Costs vary with your SLA requirements. This model needs two nodes to meet the SLA. If the acceptable service impact rises to match normal operations, the environment needs at least three nodes to absorb a single node failure. High-probability failure scenarios can require even more nodes. -=== Active-Active Clustering Fault Tolerance Deployment Model +==== Active-Active Clustering Fault Tolerance Deployment Model Fault tolerance means a failure within the system doesn't affect the service at all. It differs from high availability, which tolerates some service impact and downtime. @@ -167,7 +239,7 @@ If the probability of losing more than one node stays low, the architecture need This model costs more because it requires the redundancy to meet the defined SLA. -=== Zero Downtime Deployment Model +==== Zero Downtime Deployment Model This model lets you change the environment—including upgrading the infrastructure and the applications that run on it—without breaching your SLAs. Zero-downtime deployments typically use a side-by-side approach, where the old and new environments coexist briefly. An in-place deployment, by contrast, can reduce capacity or cause complete downtime. @@ -197,69 +269,11 @@ This example assumes that each environment uses the same number of Mule runtimes This method can temporarily add capacity to the service for a few minutes, hours, or days. -== Disaster Recovery - -*How quickly can your company get back to work after an IT emergency?* - -Disaster recovery (DR) restores a system to an acceptable previous state after a natural or man-made disaster. To drive your DR plan, use measurable business requirements such as Recovery Time Objective (RTO) and Recovery Point Objective (RPO). - -DR centers on your RPO and RTO. RPO is the point you return to after a disaster. For example, if you back up the system every 24 hours, your RPO is at most the previous 24 hours. RTO is how quickly you restore to your RPO and get back to business, which includes the time to bring spare equipment online to run your backups when your primary equipment fails. - -Backups form a major component of a solid disaster recovery program. Recovery comes in three types: cold, warm, and hot. - -[%header%autowidth.spread] -|=== -|Term |Definition |Example -|Recovery Time Objective (RTO) -|How quickly do you need to recover this asset? -|1 min? 15 min? 1 hr? 4 hrs? 1 day? - -|Recovery Point Objective (RPO) -|How fresh must the recovery be for the asset? -|Zero data loss, 15 mins out of date? -|=== - -For deployment-specific high availability and disaster recovery details, see <>. - -[[ha-dr-by-deployment-model]] -== High Availability and Disaster Recovery by Deployment Model - -HA mechanisms and DR responsibilities differ depending on how you deploy Mule. The deployment-model comparison table maps each model to its primary HA mechanism, its DR strategy and responsibility, and its cross-region support, so you can pick the right approach and then read the deployment-specific guide. - -[%header,cols="20a,30a,35a,15a"] -|=== -|Deployment Model |Primary HA Mechanism |DR Strategy and Responsibility |Cross-Region - -|xref:cloudhub-2::ch2-ha-dr.adoc[CloudHub 2.0] -|MuleSoft runs multiple replicas across availability zones automatically. -|You own the multiregion strategy and provide a bring-your-own (BYO) global load balancer. MuleSoft manages in-region resilience. -|Yes, with the Global Deployment entitlement. - -|xref:cloudhub::cloudhub-hadr.adoc[CloudHub] -|MuleSoft runs multiple workers across availability zones automatically. -|You own cross-region failover. MuleSoft manages in-region resilience. -|You manage it. - -|xref:runtime-fabric::rtf-ha-dr.adoc[Runtime Fabric] -|You run a minimum of two replicas across availability zones, with optional Horizontal Pod Autoscaling (HPA). -|You own backup, restore, and failover to a standby cluster. -|You manage it. - -|xref:hybrid-standalone::hybrid-standalone-hadr.adoc[Hybrid Standalone] -|You cluster runtimes to share memory and fail over automatically. -|You own backup, restore, and DR. MuleSoft provides control-plane availability and best practices. -|You manage it. -|=== - -To choose the guide that matches how you run Mule: - -* For fully managed, multi-replica resilience on AWS, read xref:cloudhub-2::ch2-ha-dr.adoc[CloudHub 2.0 High Availability and Disaster Recovery]. -* For fully managed, multi-worker resilience on AWS, read xref:cloudhub::cloudhub-hadr.adoc[CloudHub High Availability and Disaster Recovery]. -* For Kubernetes-based clusters that you operate, read xref:runtime-fabric::rtf-ha-dr.adoc[Runtime Fabric High Availability and Disaster Recovery]. -* For self-managed servers that you cluster, read xref:hybrid-standalone::hybrid-standalone-hadr.adoc[Hybrid Standalone High Availability and Disaster Recovery]. - -In every model, MuleSoft keeps in-region infrastructure and the control plane resilient, and you own any strategy that spans regions. +== Best Practices -== Keep Integrations Stateless +Follow these general design principles for high availability and disaster recovery: -As a general design principle, keep your integrations stateless. Don't share transactional information between client invocations or scheduled executions. When the middleware must retain data because of a system limitation, persist it in an external store, such as a database or a messaging queue, rather than in the middleware infrastructure or memory. As the system scales—especially in the cloud—keep each worker or node's state and resources independent of the other workers. This model delivers better performance, scalability, and reliability. +* Keep your integrations stateless. +* Don't share transactional information between client invocations or scheduled executions. +* When the middleware must retain data because of a system limitation, persist it in an external store, such as a database or a messaging queue, rather than in the middleware infrastructure or memory. +* As the system scales—especially in the cloud—keep each worker or node's state and resources independent of the other workers, which delivers better performance, scalability, and reliability. From ff1562dd717b7cd659191ac14194c77dd36d154f Mon Sep 17 00:00:00 2001 From: ktroller Date: Mon, 13 Jul 2026 20:30:31 -0300 Subject: [PATCH 06/17] W-23431158: Move on-premises HA/DR to its own page Extract the On-Premises Deployment High Availability and Disaster Recovery section from the hadr-guide hub into a standalone on-premises-hadr.adoc page so every deployment model has a dedicated page in its own doc set. Link the new page from the deployment-model matrix and add it to the nav under Run Mule Runtime Engine On-Premises > Cluster Runtime Instances. Co-authored-by: Cursor --- modules/ROOT/nav.adoc | 1 + modules/ROOT/pages/hadr-guide.adoc | 194 +++-------------------- modules/ROOT/pages/on-premises-hadr.adoc | 171 ++++++++++++++++++++ 3 files changed, 197 insertions(+), 169 deletions(-) create mode 100644 modules/ROOT/pages/on-premises-hadr.adoc diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 217363a93..129e0f318 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -48,6 +48,7 @@ include::partial$nav-app-dev.adoc[] ** xref:choosing-the-right-clustering-topology.adoc[Cluster Runtime Instances] *** xref:mule-high-availability-ha-clusters.adoc[Mule Runtime HA Cluster Overview] *** xref:creating-and-managing-a-cluster-manually.adoc[Create and Manage a Cluster Manually] + *** xref:on-premises-hadr.adoc[On-Premises Deployment High Availability and Disaster Recovery] * xref:mule-upgrade-tool.adoc[Mule Upgrade Tool] * xref:mule-troubleshooting-plugin.adoc[Mule Troubleshooting Plugin] * xref:using-maven-with-mule.adoc[Maven Support in Mule] diff --git a/modules/ROOT/pages/hadr-guide.adoc b/modules/ROOT/pages/hadr-guide.adoc index e887cb87c..49444a343 100644 --- a/modules/ROOT/pages/hadr-guide.adoc +++ b/modules/ROOT/pages/hadr-guide.adoc @@ -49,13 +49,13 @@ See also: * xref:api-manager::defining-sla-tiers.adoc[Defining SLA Tiers] * xref:api-manager::rate-limiting-and-throttling-sla-based-policies.adoc[Rate Limiting and Throttling - SLA-Based Policies] -== Recovery Objectives: RTO and RPO +== Disaster Recovery *How quickly can your company get back to work after an IT emergency?* Disaster recovery (DR) restores a system to an acceptable previous state after a natural or man-made disaster. To drive your DR plan, use measurable business requirements such as Recovery Time Objective (RTO) and Recovery Point Objective (RPO). -DR centers on your RPO and RTO. RPO is the point you return to after a disaster. For example, if you back up the system every 24 hours, your RPO is at most the previous 24 hours. RTO is how quickly you restore to your RPO and get back to business, which includes the time to bring spare equipment online to run your backups when your primary equipment fails. +DR centers on your RPO and RTO. RPO is the point you return to after a disaster. For example, if you back up the system every 24 hours, your RPO is at most the previous 24 hours. RTO is how quickly you restore to your RPO and resume business, which includes the time to bring backup infrastructure or applications online when your primary environment fails. Backups form a major component of a solid disaster recovery program. Recovery comes in three types: cold, warm, and hot. @@ -76,198 +76,54 @@ Backups form a major component of a solid disaster recovery program. Recovery co HA mechanisms and DR responsibilities differ depending on how you deploy Mule. The deployment-model comparison table maps each model to its primary HA mechanism, its DR strategy and responsibility, and its cross-region support, so you can pick the right approach and then read the deployment-specific guide. -[%header,cols="20a,30a,35a,15a"] +[%header,cols="15a,24a,27a,11a,23a"] |=== -|Deployment Model |Primary HA Mechanism |DR Strategy and Responsibility |Cross-Region +|Deployment Model |Primary HA Mechanism |DR Strategy and Responsibility |Cross-Region |When to Use -|xref:cloudhub-2::ch2-ha-dr.adoc[CloudHub 2.0] +|CloudHub 2.0 |MuleSoft runs multiple replicas across availability zones automatically. |You own the multiregion strategy and provide a bring-your-own (BYO) global load balancer. MuleSoft manages in-region resilience. |Yes, with the Global Deployment entitlement. +|Fully managed, multi-replica resilience on AWS. -|xref:cloudhub::cloudhub-hadr.adoc[CloudHub] +See: xref:cloudhub-2::ch2-ha-dr.adoc[] + +|CloudHub |MuleSoft runs multiple workers across availability zones automatically. |You own cross-region failover. MuleSoft manages in-region resilience. |You manage it. +|Fully managed, multi-worker resilience on AWS. + +See: xref:cloudhub::cloudhub-hadr.adoc[] -|xref:runtime-fabric::rtf-ha-dr.adoc[Runtime Fabric] +|Runtime Fabric |You run a minimum of two replicas across availability zones, with optional Horizontal Pod Autoscaling (HPA). |You own backup, restore, and failover to a standby cluster. |You manage it. +|Kubernetes-based clusters that you operate. + +See: xref:runtime-fabric::rtf-ha-dr.adoc[] -|xref:hybrid-standalone::hybrid-standalone-hadr.adoc[Hybrid Standalone] +|Hybrid Standalone |You cluster runtimes to share memory and fail over automatically. |You own backup, restore, and DR. MuleSoft provides control-plane availability and best practices. |You manage it. +|Servers you control, managed through Anypoint Platform. -|<> +See: xref:hybrid-standalone::hybrid-standalone-hadr.adoc[] + +|On-Premises |You cluster Mule runtime servers for automatic failover, using active-active, fault-tolerant, or zero-downtime topologies. |You own the HA architecture, backup, restore, and DR for your servers and data. |You manage it. -|=== - -To choose the guide that matches how you run Mule: - -* For fully managed, multi-replica resilience on AWS, read xref:cloudhub-2::ch2-ha-dr.adoc[CloudHub 2.0 High Availability and Disaster Recovery]. -* For fully managed, multi-worker resilience on AWS, read xref:cloudhub::cloudhub-hadr.adoc[CloudHub High Availability and Disaster Recovery]. -* For Kubernetes-based clusters that you operate, read xref:runtime-fabric::rtf-ha-dr.adoc[Runtime Fabric High Availability and Disaster Recovery]. -* For servers you control that you manage centrally through Anypoint Platform, read xref:hybrid-standalone::hybrid-standalone-hadr.adoc[Hybrid Standalone High Availability and Disaster Recovery]. -* For on-premises Mule clustering architectures that you operate yourself, see <> in this guide. - -MuleSoft keeps the Anypoint Platform control plane resilient. In the managed runtime models, MuleSoft also keeps in-region infrastructure resilient; in self-managed and on-premises models, you own the infrastructure. In every model, you own any strategy that spans regions. - -[[ha-on-premises]] -== High Availability and Disaster Recovery for On-Premises Deployments - -When you run Mule runtime engine on servers that you manage, you build HA and DR yourself. You cluster runtimes for failover, choose a standby topology that matches your SLA, and back up your environment and data. MuleSoft provides the runtime clustering capabilities and the architectural guidance in this section. - -=== On-Premises Clustering and Failover - -You can deploy Mule runtime engine in many topologies to support your HA and DR strategies. One common method, high-availability clustering, gives Mule basic failover capability. - -When the primary Mule runtime becomes unavailable—because of a fatal JVM or hardware failure, or because you take it offline for maintenance—a backup Mule runtime immediately takes over as the primary node and resumes processing where the failed instance stopped. - -After you recover the failed Mule runtime server and bring it back online, that server automatically becomes the backup node. - -image::hadr-cluster.png["A clustered server environment handling incoming and outgoing messages"] - -A distributed memory store makes seamless failover possible by sharing all transient state across clustered Mule runtimes, including: - -* SEDA service event queues -* In-memory message queues - -As you build your Mule application, think critically about how to architect it for the availability, fault tolerance, and performance you need. - -=== High Availability Options - -You achieve high availability by clustering and load balancing nodes. Depending on your SLA, Mule supports four HA options: - -* <> -* <> -* <> -* <> - -NOTE: These options often pair with a disaster recovery strategy. - -==== Cold Standby - -[%header%autowidth.spread] -|=== -|Diagram |Description |Downtime -|image:hadr-cold-standby.png["A cold standby configuration in a server setup"] -|You install and configure the Mule environment but leave one or more operating systems stopped—for example, as a backup of a production system or virtual machine. You start the operating system and the Mule runtime after you detect an outage. -|Some. Covers the time to start the environment and route traffic. -|=== - -==== Warm Standby - -[%header%autowidth.spread] -|=== -|Diagram |Description |Downtime -|image:hadr-warm-standby.png["A warm standby setup for server redundancy"] -|You install and configure the Mule environment and run the operating systems but leave the Mule runtimes stopped. You start the Mule runtime after you detect an outage. -|Little. Covers the time for the Mule runtime instances to start and route traffic to the environment. -|=== - -==== Hot Standby - Active-Passive - -[%header%autowidth.spread] -|=== -|Diagram |Description |Downtime -|image:hadr-active-passive.png["An active-passive server setup in a high-availability configuration"] -|You install, configure, and fully run the Mule environment, but it doesn't process requests until you detect an outage. -|Minimal to none. Covers the time to route traffic to the environment. -|=== - -==== Active-Active - -[%header%autowidth.spread] -|=== -|Diagram |Description |Downtime -|image:hadr-active-active1.png["A load balancer distributing traffic to a cluster of Mule instances in an active-active configuration"] -|Load-balanced clustered environments: -Two or more fully operational Mule environments run at the same time, each with its own cluster, and the load balancer distributes traffic across all of them. -|None. The service experiences no downtime. -|image:hadr-active-active2.png["An active-active server configuration with a load balancer and clustering"] -|Load-balanced single clustered environment: -Two or more Mule environments belong to the same cluster. This scenario requires network latency between environments to stay below 10 ms. -|None. The service experiences no downtime. -|=== - -=== On-Premises Deployment Models - -[NOTE] -==== -When you deploy an application to an on-premises Mule cluster through Runtime Manager, Runtime Manager stages deployment artifacts to cluster nodes sequentially. -Sequential staging doesn't guarantee that the application starts on one node before startup begins on another. Application startup and readiness across cluster nodes follow no fixed order and can overlap. -==== - -* <> -* <> -* <> - -==== Active-Active Clustering Deployment Model - -Two nodes in a clustered or load-balanced environment can plausibly support 1,500 TPS with 1-second responses, which meets the normal SLA. If one node fails, the service degrades but still handles 700 TPS with 1-second responses—well above the agreed acceptable impact—so the failure doesn't breach the SLA. - -image::hadr-aa-clustering.png["An active-active clustering setup with load balancers and multiple servers"] - -Distribute the load evenly across multiple Mule nodes: - -* All nodes offer the same capabilities. -* All nodes stay active at the same time. - -*Costs* - -Costs vary with your SLA requirements. This model needs two nodes to meet the SLA. If the acceptable service impact rises to match normal operations, the environment needs at least three nodes to absorb a single node failure. High-probability failure scenarios can require even more nodes. - -==== Active-Active Clustering Fault Tolerance Deployment Model +|On-premises clustering that you operate yourself. -Fault tolerance means a failure within the system doesn't affect the service at all. It differs from high availability, which tolerates some service impact and downtime. - -image::hadr-fault-tolerant.png["A fault-tolerant system with redundant nodes and a load balancer"] - -Fault tolerance differs from high availability by adding resources that let an application keep running—without interruption—after a component fails. Fault-tolerant environments cost more than highly available environments. - -The degree of fault tolerance you need depends on the probability of system failures. Take the earlier SLA example and raise the minimum acceptable service impact to match the normal operation requirement. - -The new SLA now requires the system to handle 1,000 transactions per second with 1-second response times, zero downtime, and zero service impact. - -If the probability of losing more than one node stays low, the architecture needs just three nodes. If that probability rises above what you accept, the architecture needs more than three nodes to absorb multiple failures. - -*Costs* - -This model costs more because it requires the redundancy to meet the defined SLA. - -==== Zero Downtime Deployment Model - -This model lets you change the environment—including upgrading the infrastructure and the applications that run on it—without breaching your SLAs. Zero-downtime deployments typically use a side-by-side approach, where the old and new environments coexist briefly. An in-place deployment, by contrast, can reduce capacity or cause complete downtime. - -https://www.gartner.com/en/information-technology/glossary/continuous-operations[Gartner] defines continuous operations as "those characteristics of a data-processing system that reduce or eliminate the need for planned downtime, such as scheduled maintenance. One element of 24-hour-a-day, seven-day-a-week operation". - -The baseline production environment is your current operating environment. You create a new environment with the changes—upgraded runtimes, new configurations, and new applications. A small percentage of traffic flows to the new environment and increases as your confidence grows. The baseline environment keeps serving traffic until the new environment handles 100 percent of the traffic. After the new environment accepts all traffic, it becomes the new baseline production environment, and the previous baseline terminates. - -This example assumes that each environment uses the same number of Mule runtimes and cores, but the new environment can use more or fewer. - -[%header,cols="60a,40a"] +See: xref:on-premises-hadr.adoc[] |=== -|Deployment Step |Diagram -|You deploy the new production environment and route a small percentage of traffic to it. -|image:hadr-zero-downtime-1.png["A traffic distribution in a smaller percentage from current to new production environments in a high-availability deployment"] - -|Your confidence in the new environment grows, and you route more traffic to it. -|image:hadr-zero-downtime-2.png["A traffic distribution in a higher percentage from current to new production environment in a high-availability deployment"] -|You route all traffic to the new environment. -|image:hadr-zero-downtime-3.png["A zero-downtime deployment with traffic shifting from an current to a new environment"] - -|You promote the new environment to baseline production and terminate the previous baseline. -|image:hadr-zero-downtime-4.png["A traffic routing to a new production environment completed in a high-availability environment"] -|=== - -*Costs* +MuleSoft keeps the Anypoint Platform control plane resilient. In the managed runtime models, MuleSoft also keeps in-region infrastructure resilient; in self-managed and on-premises models, you own the infrastructure. In every model, you own any strategy that spans regions. -This method can temporarily add capacity to the service for a few minutes, hours, or days. +For on-premises deployments, see xref:on-premises-hadr.adoc[On-Premises Deployment High Availability and Disaster Recovery]. == Best Practices diff --git a/modules/ROOT/pages/on-premises-hadr.adoc b/modules/ROOT/pages/on-premises-hadr.adoc new file mode 100644 index 000000000..32e3bcd91 --- /dev/null +++ b/modules/ROOT/pages/on-premises-hadr.adoc @@ -0,0 +1,171 @@ += On-Premises Deployment High Availability and Disaster Recovery +ifndef::env-site,env-github[] +include::_attributes.adoc[] +endif::[] + +When you run Mule runtime engine on servers that you manage, you build HA and DR yourself. You cluster runtimes for failover, choose a standby topology that matches your SLA, and back up your environment and data. MuleSoft provides the runtime clustering capabilities and the architectural guidance on this page. + +For an overview of HA and DR across all Mule deployment models, and to compare on-premises deployments with managed options, see xref:hadr-guide.adoc[High Availability and Disaster Recovery]. + +== On-Premises Clustering and Failover + +You can deploy Mule runtime engine in many topologies to support your HA and DR strategies. One common method, high-availability clustering, gives Mule basic failover capability. + +When the primary Mule runtime becomes unavailable—because of a fatal JVM or hardware failure, or because you take it offline for maintenance—a backup Mule runtime immediately takes over as the primary node and resumes processing where the failed instance stopped. + +After you recover the failed Mule runtime server and bring it back online, that server automatically becomes the backup node. + +image::hadr-cluster.png["A clustered server environment handling incoming and outgoing messages"] + +A distributed memory store makes seamless failover possible by sharing all transient state across clustered Mule runtimes, including: + +* SEDA service event queues +* In-memory message queues + +As you build your Mule application, think critically about how to architect it for the availability, fault tolerance, and performance you need. + +== High Availability Options + +You achieve high availability by clustering and load balancing nodes. Depending on your SLA, Mule supports four HA options: + +* <> +* <> +* <> +* <> + +NOTE: These options often pair with a disaster recovery strategy. + +=== Cold Standby + +[%header%autowidth.spread] +|=== +|Diagram |Description |Downtime +|image:hadr-cold-standby.png["A cold standby configuration in a server setup"] +|You install and configure the Mule environment but leave one or more operating systems stopped—for example, as a backup of a production system or virtual machine. You start the operating system and the Mule runtime after you detect an outage. +|Some. Covers the time to start the environment and route traffic. +|=== + +=== Warm Standby + +[%header%autowidth.spread] +|=== +|Diagram |Description |Downtime +|image:hadr-warm-standby.png["A warm standby setup for server redundancy"] +|You install and configure the Mule environment and run the operating systems but leave the Mule runtimes stopped. You start the Mule runtime after you detect an outage. +|Little. Covers the time for the Mule runtime instances to start and route traffic to the environment. +|=== + +=== Hot Standby - Active-Passive + +[%header%autowidth.spread] +|=== +|Diagram |Description |Downtime +|image:hadr-active-passive.png["An active-passive server setup in a high-availability configuration"] +|You install, configure, and fully run the Mule environment, but it doesn't process requests until you detect an outage. +|Minimal to none. Covers the time to route traffic to the environment. +|=== + +=== Active-Active + +[%header%autowidth.spread] +|=== +|Diagram |Description |Downtime +|image:hadr-active-active1.png["A load balancer distributing traffic to a cluster of Mule instances in an active-active configuration"] +|Load-balanced clustered environments: +Two or more fully operational Mule environments run at the same time, each with its own cluster, and the load balancer distributes traffic across all of them. +|None. The service experiences no downtime. +|image:hadr-active-active2.png["An active-active server configuration with a load balancer and clustering"] +|Load-balanced single clustered environment: +Two or more Mule environments belong to the same cluster. This scenario requires network latency between environments to stay below 10 ms. +|None. The service experiences no downtime. +|=== + +== On-Premises Deployment Models + +[NOTE] +==== +When you deploy an application to an on-premises Mule cluster through Runtime Manager, Runtime Manager stages deployment artifacts to cluster nodes sequentially. +Sequential staging doesn't guarantee that the application starts on one node before startup begins on another. Application startup and readiness across cluster nodes follow no fixed order and can overlap. +==== + +* <> +* <> +* <> + +=== Active-Active Clustering Deployment Model + +Two nodes in a clustered or load-balanced environment can plausibly support 1,500 TPS with 1-second responses, which meets the normal SLA. If one node fails, the service degrades but still handles 700 TPS with 1-second responses—well above the agreed acceptable impact—so the failure doesn't breach the SLA. + +image::hadr-aa-clustering.png["An active-active clustering setup with load balancers and multiple servers"] + +Distribute the load evenly across multiple Mule nodes: + +* All nodes offer the same capabilities. +* All nodes stay active at the same time. + +*Costs* + +Costs vary with your SLA requirements. This model needs two nodes to meet the SLA. If the acceptable service impact rises to match normal operations, the environment needs at least three nodes to absorb a single node failure. High-probability failure scenarios can require even more nodes. + +=== Active-Active Clustering Fault Tolerance Deployment Model + +Fault tolerance means a failure within the system doesn't affect the service at all. It differs from high availability, which tolerates some service impact and downtime. + +image::hadr-fault-tolerant.png["A fault-tolerant system with redundant nodes and a load balancer"] + +Fault tolerance differs from high availability by adding resources that let an application keep running—without interruption—after a component fails. Fault-tolerant environments cost more than highly available environments. + +The degree of fault tolerance you need depends on the probability of system failures. Take the earlier SLA example and raise the minimum acceptable service impact to match the normal operation requirement. + +The new SLA now requires the system to handle 1,000 transactions per second with 1-second response times, zero downtime, and zero service impact. + +If the probability of losing more than one node stays low, the architecture needs just three nodes. If that probability rises above what you accept, the architecture needs more than three nodes to absorb multiple failures. + +*Costs* + +This model costs more because it requires the redundancy to meet the defined SLA. + +=== Zero Downtime Deployment Model + +This model lets you change the environment—including upgrading the infrastructure and the applications that run on it—without breaching your SLAs. Zero-downtime deployments typically use a side-by-side approach, where the old and new environments coexist briefly. An in-place deployment, by contrast, can reduce capacity or cause complete downtime. + +https://www.gartner.com/en/information-technology/glossary/continuous-operations[Gartner] defines continuous operations as "those characteristics of a data-processing system that reduce or eliminate the need for planned downtime, such as scheduled maintenance. One element of 24-hour-a-day, seven-day-a-week operation". + +The baseline production environment is your current operating environment. You create a new environment with the changes—upgraded runtimes, new configurations, and new applications. A small percentage of traffic flows to the new environment and increases as your confidence grows. The baseline environment keeps serving traffic until the new environment handles 100 percent of the traffic. After the new environment accepts all traffic, it becomes the new baseline production environment, and the previous baseline terminates. + +This example assumes that each environment uses the same number of Mule runtimes and cores, but the new environment can use more or fewer. + +[%header,cols="60a,40a"] +|=== +|Deployment Step |Diagram +|You deploy the new production environment and route a small percentage of traffic to it. +|image:hadr-zero-downtime-1.png["A traffic distribution in a smaller percentage from current to new production environments in a high-availability deployment"] + +|Your confidence in the new environment grows, and you route more traffic to it. +|image:hadr-zero-downtime-2.png["A traffic distribution in a higher percentage from current to new production environment in a high-availability deployment"] + +|You route all traffic to the new environment. +|image:hadr-zero-downtime-3.png["A zero-downtime deployment with traffic shifting from an current to a new environment"] + +|You promote the new environment to baseline production and terminate the previous baseline. +|image:hadr-zero-downtime-4.png["A traffic routing to a new production environment completed in a high-availability environment"] +|=== + +*Costs* + +This method can temporarily add capacity to the service for a few minutes, hours, or days. + +== Best Practices + +Follow these general design principles for high availability and disaster recovery: + +* Keep your integrations stateless. +* Don't share transactional information between client invocations or scheduled executions. +* When the middleware must retain data because of a system limitation, persist it in an external store, such as a database or a messaging queue, rather than in the middleware infrastructure or memory. +* As the system scales, keep each node's state and resources independent of the other nodes, which delivers better performance, scalability, and reliability. + +== See Also + +* xref:hadr-guide.adoc[High Availability and Disaster Recovery] +* xref:mule-high-availability-ha-clusters.adoc[Mule Runtime HA Cluster Overview] +* xref:creating-and-managing-a-cluster-manually.adoc[Create and Manage a Cluster Manually] From c6efbb6728e53beed1ba3f1f948b088579dc7132 Mon Sep 17 00:00:00 2001 From: ktroller Date: Mon, 13 Jul 2026 20:35:09 -0300 Subject: [PATCH 07/17] W-23431158: Align on-premises HA/DR page with shared spoke skeleton Restructure the on-premises page to follow the same skeleton as the CloudHub 2.0, CloudHub, Runtime Fabric, and Hybrid Standalone pages: HA vs DR with RTO/RPO, High Availability in On-Premises Deployments, Disaster Recovery in On-Premises Deployments, Considerations and Limitations, Best Practices, and See Also. Preserve the clustering, standby-topology, and deployment-model content and images, and add the DR and considerations sections. Co-authored-by: Cursor --- modules/ROOT/pages/on-premises-hadr.adoc | 88 ++++++++++++++++-------- 1 file changed, 59 insertions(+), 29 deletions(-) diff --git a/modules/ROOT/pages/on-premises-hadr.adoc b/modules/ROOT/pages/on-premises-hadr.adoc index 32e3bcd91..27c952970 100644 --- a/modules/ROOT/pages/on-premises-hadr.adoc +++ b/modules/ROOT/pages/on-premises-hadr.adoc @@ -3,28 +3,37 @@ ifndef::env-site,env-github[] include::_attributes.adoc[] endif::[] -When you run Mule runtime engine on servers that you manage, you build HA and DR yourself. You cluster runtimes for failover, choose a standby topology that matches your SLA, and back up your environment and data. MuleSoft provides the runtime clustering capabilities and the architectural guidance on this page. +When you run Mule runtime engine on servers that you manage, you build high availability (HA) and disaster recovery (DR) yourself. You achieve HA by clustering Mule runtime instances so that another node takes over when one fails, and you choose a standby topology that matches your SLA. You own DR under a shared-responsibility model: because you control the infrastructure, you back up, restore, and recover it with your own tooling, while MuleSoft provides the runtime clustering capabilities, the architectural guidance on this page, and control-plane availability through Anypoint Platform. -For an overview of HA and DR across all Mule deployment models, and to compare on-premises deployments with managed options, see xref:hadr-guide.adoc[High Availability and Disaster Recovery]. +For an overview of HA and DR across all Mule deployment models, see xref:hadr-guide.adoc[High Availability and Disaster Recovery]. -== On-Premises Clustering and Failover +== High Availability Versus Disaster Recovery -You can deploy Mule runtime engine in many topologies to support your HA and DR strategies. One common method, high-availability clustering, gives Mule basic failover capability. +High availability (HA) measures a system's ability to remain accessible when a component fails. On self-managed servers, you achieve HA by clustering and load balancing Mule runtime nodes, so that another node takes over when one fails. -When the primary Mule runtime becomes unavailable—because of a fatal JVM or hardware failure, or because you take it offline for maintenance—a backup Mule runtime immediately takes over as the primary node and resumes processing where the failed instance stopped. +Disaster recovery (DR) restores a system to an acceptable previous state after a natural or man-made disaster, such as flooding, power failures, server failures, or misconfigurations. -After you recover the failed Mule runtime server and bring it back online, that server automatically becomes the backup node. +Both strategies increase availability but differ in scope: on-premises, HA keeps your application serving requests when a node in the cluster fails with no loss of service, whereas DR restores service and data after an infrastructure disaster and usually incurs a brief loss of service while the DR plan runs. -image::hadr-cluster.png["A clustered server environment handling incoming and outgoing messages"] +Two measurable objectives drive your DR plan: + +Recovery Time Objective (RTO):: The maximum downtime a business tolerates. RTO is the time the system takes to recover after a disruption. + +Recovery Point Objective (RPO):: The maximum data loss a business accepts after a disaster. RPO drives how often you back up data. + +== High Availability in On-Premises Deployments + +You achieve high availability on self-managed servers by clustering and load balancing Mule runtime nodes. Depending on your SLA, you choose a standby topology and a deployment architecture that meet your availability and fault-tolerance requirements. -A distributed memory store makes seamless failover possible by sharing all transient state across clustered Mule runtimes, including: +Clustering and failover:: You can deploy Mule runtime engine in many topologies to support your HA and DR strategies. One common method, high-availability clustering, gives Mule basic failover capability. When the primary Mule runtime becomes unavailable—because of a fatal JVM or hardware failure, or because you take it offline for maintenance—a backup Mule runtime immediately takes over as the primary node and resumes processing where the failed instance stopped. After you recover the failed Mule runtime server and bring it back online, that server automatically becomes the backup node. -* SEDA service event queues -* In-memory message queues +Distributed memory:: A distributed memory store makes seamless failover possible by sharing all transient state across clustered Mule runtimes, including SEDA service event queues and in-memory message queues. + +image::hadr-cluster.png["A clustered server environment handling incoming and outgoing messages"] As you build your Mule application, think critically about how to architect it for the availability, fault tolerance, and performance you need. -== High Availability Options +=== High Availability Options You achieve high availability by clustering and load balancing nodes. Depending on your SLA, Mule supports four HA options: @@ -35,7 +44,7 @@ You achieve high availability by clustering and load balancing nodes. Depending NOTE: These options often pair with a disaster recovery strategy. -=== Cold Standby +==== Cold Standby [%header%autowidth.spread] |=== @@ -45,7 +54,7 @@ NOTE: These options often pair with a disaster recovery strategy. |Some. Covers the time to start the environment and route traffic. |=== -=== Warm Standby +==== Warm Standby [%header%autowidth.spread] |=== @@ -55,7 +64,7 @@ NOTE: These options often pair with a disaster recovery strategy. |Little. Covers the time for the Mule runtime instances to start and route traffic to the environment. |=== -=== Hot Standby - Active-Passive +==== Hot Standby - Active-Passive [%header%autowidth.spread] |=== @@ -65,7 +74,7 @@ NOTE: These options often pair with a disaster recovery strategy. |Minimal to none. Covers the time to route traffic to the environment. |=== -=== Active-Active +==== Active-Active [%header%autowidth.spread] |=== @@ -80,19 +89,15 @@ Two or more Mule environments belong to the same cluster. This scenario requires |None. The service experiences no downtime. |=== -== On-Premises Deployment Models +=== On-Premises Deployment Models -[NOTE] -==== -When you deploy an application to an on-premises Mule cluster through Runtime Manager, Runtime Manager stages deployment artifacts to cluster nodes sequentially. -Sequential staging doesn't guarantee that the application starts on one node before startup begins on another. Application startup and readiness across cluster nodes follow no fixed order and can overlap. -==== +You size your on-premises architecture to meet your SLA. Mule supports three common deployment models: * <> * <> * <> -=== Active-Active Clustering Deployment Model +==== Active-Active Clustering Deployment Model Two nodes in a clustered or load-balanced environment can plausibly support 1,500 TPS with 1-second responses, which meets the normal SLA. If one node fails, the service degrades but still handles 700 TPS with 1-second responses—well above the agreed acceptable impact—so the failure doesn't breach the SLA. @@ -107,7 +112,7 @@ Distribute the load evenly across multiple Mule nodes: Costs vary with your SLA requirements. This model needs two nodes to meet the SLA. If the acceptable service impact rises to match normal operations, the environment needs at least three nodes to absorb a single node failure. High-probability failure scenarios can require even more nodes. -=== Active-Active Clustering Fault Tolerance Deployment Model +==== Active-Active Clustering Fault Tolerance Deployment Model Fault tolerance means a failure within the system doesn't affect the service at all. It differs from high availability, which tolerates some service impact and downtime. @@ -125,7 +130,7 @@ If the probability of losing more than one node stays low, the architecture need This model costs more because it requires the redundancy to meet the defined SLA. -=== Zero Downtime Deployment Model +==== Zero Downtime Deployment Model This model lets you change the environment—including upgrading the infrastructure and the applications that run on it—without breaching your SLAs. Zero-downtime deployments typically use a side-by-side approach, where the old and new environments coexist briefly. An in-place deployment, by contrast, can reduce capacity or cause complete downtime. @@ -155,14 +160,39 @@ This example assumes that each environment uses the same number of Mule runtimes This method can temporarily add capacity to the service for a few minutes, hours, or days. +== Disaster Recovery in On-Premises Deployments + +Because you control the infrastructure, you own disaster recovery for on-premises deployments. You back up your servers, applications, and data, and you bring backup infrastructure online to meet your recovery objectives when your primary environment fails. + +Standby topologies:: Use a cold, warm, or hot standby topology to meet your RTO and RPO. A hot standby minimizes downtime but costs the most, whereas a cold standby costs the least but takes longest to recover. + +Backups:: Back up Mule applications, runtime configurations, and application data to external storage outside your primary environment. Schedule backups often enough to meet your RPO. + +Anypoint configuration:: Back up your platform configurations—such as connections and properties—with the Anypoint CLI or Anypoint Platform APIs. + +MuleSoft responsibility:: MuleSoft keeps the Anypoint Platform control plane available across multiple availability zones. The control plane stores only operational metadata and configuration, not your business data. + +== Considerations and Limitations + +Account for these factors when you design a clustered on-premises topology: + +Network latency:: Clustering across availability zones or regions can degrade performance because of network latency. A single clustered environment that spans locations requires network latency between nodes to stay below 10 ms. Where possible, place cluster nodes on the same subnet or availability zone. + +Simultaneous node restarts:: Don't restart multiple cluster nodes at the same time, such as during operating system patching. Concurrent restarts can trigger cluster-membership eviction and take applications offline. + +Sequential deployment staging:: When you deploy an application to an on-premises Mule cluster through Runtime Manager, Runtime Manager stages deployment artifacts to cluster nodes sequentially. Sequential staging doesn't guarantee that the application starts on one node before startup begins on another. Application startup and readiness across cluster nodes follow no fixed order and can overlap. + +Cost trade-offs:: Higher availability and fault tolerance require more nodes and infrastructure. Size your architecture to the SLA you must meet, rather than to the maximum possible resilience. + == Best Practices -Follow these general design principles for high availability and disaster recovery: +Follow these design principles for high availability and disaster recovery on self-managed servers: -* Keep your integrations stateless. -* Don't share transactional information between client invocations or scheduled executions. -* When the middleware must retain data because of a system limitation, persist it in an external store, such as a database or a messaging queue, rather than in the middleware infrastructure or memory. -* As the system scales, keep each node's state and resources independent of the other nodes, which delivers better performance, scalability, and reliability. +* Keep your integrations stateless, and externalize state to an external store, such as a database or a messaging queue. +* Run identical Mule runtime, Runtime Manager agent, and Java versions on all cluster nodes. +* Choose a standby topology and node count that meet your SLA rather than the maximum possible resilience. +* Schedule cluster changes during maintenance windows. +* Test your failover and restore procedures regularly. == See Also From 4cdc1f307f464ee6cdd66e40e9333db68a3a94fd Mon Sep 17 00:00:00 2001 From: ktroller Date: Mon, 13 Jul 2026 20:40:23 -0300 Subject: [PATCH 08/17] W-23431158: Remove redundant on-premises pointer sentence from HA/DR hub The deployment-model matrix already links to the On-Premises page, so the standalone pointer sentence is redundant. Co-authored-by: Cursor --- modules/ROOT/pages/hadr-guide.adoc | 2 -- 1 file changed, 2 deletions(-) diff --git a/modules/ROOT/pages/hadr-guide.adoc b/modules/ROOT/pages/hadr-guide.adoc index 49444a343..c971b927b 100644 --- a/modules/ROOT/pages/hadr-guide.adoc +++ b/modules/ROOT/pages/hadr-guide.adoc @@ -123,8 +123,6 @@ See: xref:on-premises-hadr.adoc[] MuleSoft keeps the Anypoint Platform control plane resilient. In the managed runtime models, MuleSoft also keeps in-region infrastructure resilient; in self-managed and on-premises models, you own the infrastructure. In every model, you own any strategy that spans regions. -For on-premises deployments, see xref:on-premises-hadr.adoc[On-Premises Deployment High Availability and Disaster Recovery]. - == Best Practices Follow these general design principles for high availability and disaster recovery: From a8d9f70b95f78e8744386f84432308b6fb5692cd Mon Sep 17 00:00:00 2001 From: ktroller Date: Mon, 13 Jul 2026 20:40:56 -0300 Subject: [PATCH 09/17] W-23431158: Drop colon after "See" in the When to Use column Co-authored-by: Cursor --- modules/ROOT/pages/hadr-guide.adoc | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/modules/ROOT/pages/hadr-guide.adoc b/modules/ROOT/pages/hadr-guide.adoc index c971b927b..1652e75ab 100644 --- a/modules/ROOT/pages/hadr-guide.adoc +++ b/modules/ROOT/pages/hadr-guide.adoc @@ -86,7 +86,7 @@ HA mechanisms and DR responsibilities differ depending on how you deploy Mule. T |Yes, with the Global Deployment entitlement. |Fully managed, multi-replica resilience on AWS. -See: xref:cloudhub-2::ch2-ha-dr.adoc[] +See xref:cloudhub-2::ch2-ha-dr.adoc[] |CloudHub |MuleSoft runs multiple workers across availability zones automatically. @@ -94,7 +94,7 @@ See: xref:cloudhub-2::ch2-ha-dr.adoc[] |You manage it. |Fully managed, multi-worker resilience on AWS. -See: xref:cloudhub::cloudhub-hadr.adoc[] +See xref:cloudhub::cloudhub-hadr.adoc[] |Runtime Fabric |You run a minimum of two replicas across availability zones, with optional Horizontal Pod Autoscaling (HPA). @@ -102,7 +102,7 @@ See: xref:cloudhub::cloudhub-hadr.adoc[] |You manage it. |Kubernetes-based clusters that you operate. -See: xref:runtime-fabric::rtf-ha-dr.adoc[] +See xref:runtime-fabric::rtf-ha-dr.adoc[] |Hybrid Standalone |You cluster runtimes to share memory and fail over automatically. @@ -110,7 +110,7 @@ See: xref:runtime-fabric::rtf-ha-dr.adoc[] |You manage it. |Servers you control, managed through Anypoint Platform. -See: xref:hybrid-standalone::hybrid-standalone-hadr.adoc[] +See xref:hybrid-standalone::hybrid-standalone-hadr.adoc[] |On-Premises |You cluster Mule runtime servers for automatic failover, using active-active, fault-tolerant, or zero-downtime topologies. @@ -118,7 +118,7 @@ See: xref:hybrid-standalone::hybrid-standalone-hadr.adoc[] |You manage it. |On-premises clustering that you operate yourself. -See: xref:on-premises-hadr.adoc[] +See xref:on-premises-hadr.adoc[] |=== MuleSoft keeps the Anypoint Platform control plane resilient. In the managed runtime models, MuleSoft also keeps in-region infrastructure resilient; in self-managed and on-premises models, you own the infrastructure. In every model, you own any strategy that spans regions. From 95fbf49895a58e813d82cb416d13be0a24941402 Mon Sep 17 00:00:00 2001 From: ktroller Date: Mon, 13 Jul 2026 20:42:35 -0300 Subject: [PATCH 10/17] W-23431158: Add See Also with links to all deployment HA/DR guides on hub Co-authored-by: Cursor --- modules/ROOT/pages/hadr-guide.adoc | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/modules/ROOT/pages/hadr-guide.adoc b/modules/ROOT/pages/hadr-guide.adoc index 1652e75ab..7eac04a39 100644 --- a/modules/ROOT/pages/hadr-guide.adoc +++ b/modules/ROOT/pages/hadr-guide.adoc @@ -131,3 +131,11 @@ Follow these general design principles for high availability and disaster recove * Don't share transactional information between client invocations or scheduled executions. * When the middleware must retain data because of a system limitation, persist it in an external store, such as a database or a messaging queue, rather than in the middleware infrastructure or memory. * As the system scales—especially in the cloud—keep each worker or node's state and resources independent of the other workers, which delivers better performance, scalability, and reliability. + +== See Also + +* xref:cloudhub-2::ch2-ha-dr.adoc[CloudHub 2.0 High Availability and Disaster Recovery] +* xref:cloudhub::cloudhub-hadr.adoc[CloudHub High Availability and Disaster Recovery] +* xref:runtime-fabric::rtf-ha-dr.adoc[Runtime Fabric High Availability and Disaster Recovery] +* xref:hybrid-standalone::hybrid-standalone-hadr.adoc[Hybrid Standalone High Availability and Disaster Recovery] +* xref:on-premises-hadr.adoc[On-Premises Deployment High Availability and Disaster Recovery] From f90fd9b57b01996eec36ac36f93f090974944b86 Mon Sep 17 00:00:00 2001 From: ktroller Date: Mon, 13 Jul 2026 20:44:05 -0300 Subject: [PATCH 11/17] W-23431158: Remove self-referential phrasing from HA/DR hub Drop "This guide explains", "This table shows", and the "comparison table maps" lead-in in favor of direct, deployment-neutral statements. Co-authored-by: Cursor --- modules/ROOT/pages/hadr-guide.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/modules/ROOT/pages/hadr-guide.adoc b/modules/ROOT/pages/hadr-guide.adoc index 7eac04a39..fc626cf88 100644 --- a/modules/ROOT/pages/hadr-guide.adoc +++ b/modules/ROOT/pages/hadr-guide.adoc @@ -3,11 +3,11 @@ ifndef::env-site,env-github[] include::_attributes.adoc[] endif::[] -High availability (HA) and disaster recovery (DR) strategies protect your Mule applications from downtime caused by infrastructure failures, disasters, or maintenance. The right approach depends on how you deploy Mule runtime engine: managed platforms such as CloudHub 2.0 build in much of the resilience for you, whereas deployments that you operate require you to design clustering, failover, and backups yourself. This guide explains the core HA and DR concepts and then maps each deployment model to its HA mechanism and DR responsibilities. +High availability (HA) and disaster recovery (DR) strategies protect your Mule applications from downtime caused by infrastructure failures, disasters, or maintenance. The right approach depends on how you deploy Mule runtime engine: managed platforms such as CloudHub 2.0 build in much of the resilience for you, whereas deployments that you operate require you to design clustering, failover, and backups yourself. Downtime strikes at any time, from any direction, and in any form, whether you plan for it or not. Keeping an organization operational during a technology outage, a facility loss, a personnel loss, or a third-party service failure prevents irreversible damage to the business. As organizations shift to e-commerce models that depend on 24/7 uptime, HA and DR increasingly determine their financial health. -This table shows how even small amounts of downtime add up and hurt an organization: +Even small amounts of downtime add up and hurt an organization: [%header,cols="25a,25a,25a,25a"] |=== @@ -74,7 +74,7 @@ Backups form a major component of a solid disaster recovery program. Recovery co [[ha-dr-by-deployment-model]] == High Availability and Disaster Recovery by Deployment Model -HA mechanisms and DR responsibilities differ depending on how you deploy Mule. The deployment-model comparison table maps each model to its primary HA mechanism, its DR strategy and responsibility, and its cross-region support, so you can pick the right approach and then read the deployment-specific guide. +HA mechanisms and DR responsibilities differ depending on how you deploy Mule. Each deployment model has a primary HA mechanism, a DR strategy and responsibility split, and a level of cross-region support. Use these differences to pick the right approach for how you run Mule. [%header,cols="15a,24a,27a,11a,23a"] |=== From c5a90fff5e26b14b4cdf6ec928c5fb128068c6f4 Mon Sep 17 00:00:00 2001 From: ktroller Date: Mon, 13 Jul 2026 20:45:45 -0300 Subject: [PATCH 12/17] W-23431158: Remove self-referential phrasing from on-premises HA/DR page Co-authored-by: Cursor --- modules/ROOT/pages/on-premises-hadr.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/on-premises-hadr.adoc b/modules/ROOT/pages/on-premises-hadr.adoc index 27c952970..c724e5878 100644 --- a/modules/ROOT/pages/on-premises-hadr.adoc +++ b/modules/ROOT/pages/on-premises-hadr.adoc @@ -3,7 +3,7 @@ ifndef::env-site,env-github[] include::_attributes.adoc[] endif::[] -When you run Mule runtime engine on servers that you manage, you build high availability (HA) and disaster recovery (DR) yourself. You achieve HA by clustering Mule runtime instances so that another node takes over when one fails, and you choose a standby topology that matches your SLA. You own DR under a shared-responsibility model: because you control the infrastructure, you back up, restore, and recover it with your own tooling, while MuleSoft provides the runtime clustering capabilities, the architectural guidance on this page, and control-plane availability through Anypoint Platform. +When you run Mule runtime engine on servers that you manage, you build high availability (HA) and disaster recovery (DR) yourself. You achieve HA by clustering Mule runtime instances so that another node takes over when one fails, and you choose a standby topology that matches your SLA. You own DR under a shared-responsibility model: because you control the infrastructure, you back up, restore, and recover it with your own tooling, while MuleSoft provides the runtime clustering capabilities and control-plane availability through Anypoint Platform. For an overview of HA and DR across all Mule deployment models, see xref:hadr-guide.adoc[High Availability and Disaster Recovery]. From 59767f045737f40846534c824349bf4d63376a4e Mon Sep 17 00:00:00 2001 From: ktroller Date: Mon, 13 Jul 2026 20:46:30 -0300 Subject: [PATCH 13/17] W-23431158: Add "The Cost of Downtime" heading for the downtime table Co-authored-by: Cursor --- modules/ROOT/pages/hadr-guide.adoc | 2 ++ 1 file changed, 2 insertions(+) diff --git a/modules/ROOT/pages/hadr-guide.adoc b/modules/ROOT/pages/hadr-guide.adoc index fc626cf88..c6d39ef48 100644 --- a/modules/ROOT/pages/hadr-guide.adoc +++ b/modules/ROOT/pages/hadr-guide.adoc @@ -5,6 +5,8 @@ endif::[] High availability (HA) and disaster recovery (DR) strategies protect your Mule applications from downtime caused by infrastructure failures, disasters, or maintenance. The right approach depends on how you deploy Mule runtime engine: managed platforms such as CloudHub 2.0 build in much of the resilience for you, whereas deployments that you operate require you to design clustering, failover, and backups yourself. +== The Cost of Downtime + Downtime strikes at any time, from any direction, and in any form, whether you plan for it or not. Keeping an organization operational during a technology outage, a facility loss, a personnel loss, or a third-party service failure prevents irreversible damage to the business. As organizations shift to e-commerce models that depend on 24/7 uptime, HA and DR increasingly determine their financial health. Even small amounts of downtime add up and hurt an organization: From 6ca8d3992b405ea6543b9383c9c86d3750e9cecb Mon Sep 17 00:00:00 2001 From: ktroller Date: Mon, 13 Jul 2026 20:47:08 -0300 Subject: [PATCH 14/17] W-23431158: Reword SLA cross-reference lead-in to "For details, refer to:" Co-authored-by: Cursor --- modules/ROOT/pages/hadr-guide.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/hadr-guide.adoc b/modules/ROOT/pages/hadr-guide.adoc index c6d39ef48..e1b886c30 100644 --- a/modules/ROOT/pages/hadr-guide.adoc +++ b/modules/ROOT/pages/hadr-guide.adoc @@ -46,7 +46,7 @@ For example, a basic service might define these SLAs: * Total downtime per year stays at or below 0.5 percent, or 1.83 days. * Minimum acceptable service handles 100 transactions per second with 1-second response times for up to 2 hours a week. -See also: +For details, refer to: * xref:api-manager::defining-sla-tiers.adoc[Defining SLA Tiers] * xref:api-manager::rate-limiting-and-throttling-sla-based-policies.adoc[Rate Limiting and Throttling - SLA-Based Policies] From 76c83298693d44037a4b2db69ffbad82220f2cfa Mon Sep 17 00:00:00 2001 From: ktroller Date: Mon, 13 Jul 2026 20:48:28 -0300 Subject: [PATCH 15/17] W-23431158: Rename downtime section heading to "Understanding Downtime" Co-authored-by: Cursor --- modules/ROOT/pages/hadr-guide.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/hadr-guide.adoc b/modules/ROOT/pages/hadr-guide.adoc index e1b886c30..0f46b859d 100644 --- a/modules/ROOT/pages/hadr-guide.adoc +++ b/modules/ROOT/pages/hadr-guide.adoc @@ -5,7 +5,7 @@ endif::[] High availability (HA) and disaster recovery (DR) strategies protect your Mule applications from downtime caused by infrastructure failures, disasters, or maintenance. The right approach depends on how you deploy Mule runtime engine: managed platforms such as CloudHub 2.0 build in much of the resilience for you, whereas deployments that you operate require you to design clustering, failover, and backups yourself. -== The Cost of Downtime +== Understanding Downtime Downtime strikes at any time, from any direction, and in any form, whether you plan for it or not. Keeping an organization operational during a technology outage, a facility loss, a personnel loss, or a third-party service failure prevents irreversible damage to the business. As organizations shift to e-commerce models that depend on 24/7 uptime, HA and DR increasingly determine their financial health. From bbc7979699bc740551083d736cae5ef6d448add0 Mon Sep 17 00:00:00 2001 From: ktroller Date: Mon, 13 Jul 2026 20:51:07 -0300 Subject: [PATCH 16/17] W-23431158: Clarify Cross-Region DR column in the deployment-model matrix Rename the column to "Cross-Region DR" and replace ambiguous "Yes" and "You manage it." cells with statements that say whether cross-region DR is supported and who provides it. Co-authored-by: Cursor --- modules/ROOT/pages/hadr-guide.adoc | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/modules/ROOT/pages/hadr-guide.adoc b/modules/ROOT/pages/hadr-guide.adoc index 0f46b859d..b68d88849 100644 --- a/modules/ROOT/pages/hadr-guide.adoc +++ b/modules/ROOT/pages/hadr-guide.adoc @@ -80,12 +80,12 @@ HA mechanisms and DR responsibilities differ depending on how you deploy Mule. E [%header,cols="15a,24a,27a,11a,23a"] |=== -|Deployment Model |Primary HA Mechanism |DR Strategy and Responsibility |Cross-Region |When to Use +|Deployment Model |Primary HA Mechanism |DR Strategy and Responsibility |Cross-Region DR |When to Use |CloudHub 2.0 |MuleSoft runs multiple replicas across availability zones automatically. |You own the multiregion strategy and provide a bring-your-own (BYO) global load balancer. MuleSoft manages in-region resilience. -|Yes, with the Global Deployment entitlement. +|Supported through the Global Deployment entitlement, which MuleSoft provides. |Fully managed, multi-replica resilience on AWS. See xref:cloudhub-2::ch2-ha-dr.adoc[] @@ -93,7 +93,7 @@ See xref:cloudhub-2::ch2-ha-dr.adoc[] |CloudHub |MuleSoft runs multiple workers across availability zones automatically. |You own cross-region failover. MuleSoft manages in-region resilience. -|You manage it. +|Not built in. You design and run cross-region failover. |Fully managed, multi-worker resilience on AWS. See xref:cloudhub::cloudhub-hadr.adoc[] @@ -101,7 +101,7 @@ See xref:cloudhub::cloudhub-hadr.adoc[] |Runtime Fabric |You run a minimum of two replicas across availability zones, with optional Horizontal Pod Autoscaling (HPA). |You own backup, restore, and failover to a standby cluster. -|You manage it. +|Not built in. You fail over to a standby cluster in another region. |Kubernetes-based clusters that you operate. See xref:runtime-fabric::rtf-ha-dr.adoc[] @@ -109,7 +109,7 @@ See xref:runtime-fabric::rtf-ha-dr.adoc[] |Hybrid Standalone |You cluster runtimes to share memory and fail over automatically. |You own backup, restore, and DR. MuleSoft provides control-plane availability and best practices. -|You manage it. +|Not built in. You design and run cross-region recovery. |Servers you control, managed through Anypoint Platform. See xref:hybrid-standalone::hybrid-standalone-hadr.adoc[] @@ -117,7 +117,7 @@ See xref:hybrid-standalone::hybrid-standalone-hadr.adoc[] |On-Premises |You cluster Mule runtime servers for automatic failover, using active-active, fault-tolerant, or zero-downtime topologies. |You own the HA architecture, backup, restore, and DR for your servers and data. -|You manage it. +|Not built in. You design and run cross-region recovery. |On-premises clustering that you operate yourself. See xref:on-premises-hadr.adoc[] From 06e7213082f37699e35ad105f0a41a31ffe96b2b Mon Sep 17 00:00:00 2001 From: ktroller Date: Mon, 13 Jul 2026 20:59:45 -0300 Subject: [PATCH 17/17] W-23431158: Mention all deployment models in HA/DR hub intro Co-authored-by: Cursor --- modules/ROOT/pages/hadr-guide.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/hadr-guide.adoc b/modules/ROOT/pages/hadr-guide.adoc index b68d88849..ec64de7a9 100644 --- a/modules/ROOT/pages/hadr-guide.adoc +++ b/modules/ROOT/pages/hadr-guide.adoc @@ -3,7 +3,7 @@ ifndef::env-site,env-github[] include::_attributes.adoc[] endif::[] -High availability (HA) and disaster recovery (DR) strategies protect your Mule applications from downtime caused by infrastructure failures, disasters, or maintenance. The right approach depends on how you deploy Mule runtime engine: managed platforms such as CloudHub 2.0 build in much of the resilience for you, whereas deployments that you operate require you to design clustering, failover, and backups yourself. +High availability (HA) and disaster recovery (DR) strategies protect your Mule applications from downtime caused by infrastructure failures, disasters, or maintenance. The right approach depends on how you deploy Mule runtime engine: managed platforms such as CloudHub 2.0 and CloudHub build in much of the resilience for you, whereas deployments that you operate—Runtime Fabric, Hybrid Standalone, and on-premises servers—require you to design clustering, failover, and backups yourself. == Understanding Downtime