diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index c2d2b584a..129e0f318 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -37,6 +37,7 @@ include::partial$nav-app-dev.adoc[] *** xref:tuning-app-design.adoc[App Design] ** xref:tuning-test-validations.adoc[Performance Test Validations] ** xref:tuning-monitoring.adoc[Performance Monitoring] +* xref:hadr-guide.adoc[High Availability and Disaster Recovery] * xref:mule-standalone.adoc[Run Mule Runtime Engine On-Premises] ** xref:mule-deployment-model.adoc[On-Premises Deployment Model] ** xref:hardware-and-software-requirements.adoc[Hardware and Software Requirements] @@ -47,7 +48,7 @@ include::partial$nav-app-dev.adoc[] ** xref:choosing-the-right-clustering-topology.adoc[Cluster Runtime Instances] *** xref:mule-high-availability-ha-clusters.adoc[Mule Runtime HA Cluster Overview] *** xref:creating-and-managing-a-cluster-manually.adoc[Create and Manage a Cluster Manually] - *** xref:hadr-guide.adoc[High Availability and Disaster Recovery] + *** xref:on-premises-hadr.adoc[On-Premises Deployment High Availability and Disaster Recovery] * xref:mule-upgrade-tool.adoc[Mule Upgrade Tool] * xref:mule-troubleshooting-plugin.adoc[Mule Troubleshooting Plugin] * xref:using-maven-with-mule.adoc[Maven Support in Mule] diff --git a/modules/ROOT/pages/hadr-guide.adoc b/modules/ROOT/pages/hadr-guide.adoc index 5be099f74..ec64de7a9 100644 --- a/modules/ROOT/pages/hadr-guide.adoc +++ b/modules/ROOT/pages/hadr-guide.adoc @@ -3,11 +3,13 @@ ifndef::env-site,env-github[] include::_attributes.adoc[] endif::[] -High availability (HA) and disaster recovery (DR) strategies protect your Mule applications from downtime caused by infrastructure failures, disasters, or maintenance. Deploy Mule runtime in clustered topologies, configure active-active or standby environments, and define Service Level Agreements (SLAs) to meet your uptime requirements. Mule clustering provides automatic failover and shares transient state across nodes to ensure continuous operation. +High availability (HA) and disaster recovery (DR) strategies protect your Mule applications from downtime caused by infrastructure failures, disasters, or maintenance. The right approach depends on how you deploy Mule runtime engine: managed platforms such as CloudHub 2.0 and CloudHub build in much of the resilience for you, whereas deployments that you operate—Runtime Fabric, Hybrid Standalone, and on-premises servers—require you to design clustering, failover, and backups yourself. -Planned or unplanned, infrastructure and application downtime can come at any time, from any direction, and in any form. The ability to keep an organization operational during a technology outage, facility destruction, loss of personnel, or loss of critical third party services is critical to preventing irreversible damage to a business. With the increasing global shift to e-commerce models and their reliance on 24/7 application uptime, high availability (HA) and disaster recovery (DR) impact the financial health of organizations. +== Understanding Downtime -This table shows how even the least amount of downtime can add up to negatively affect an organization: +Downtime strikes at any time, from any direction, and in any form, whether you plan for it or not. Keeping an organization operational during a technology outage, a facility loss, a personnel loss, or a third-party service failure prevents irreversible damage to the business. As organizations shift to e-commerce models that depend on 24/7 uptime, HA and DR increasingly determine their financial health. + +Even small amounts of downtime add up and hurt an organization: [%header,cols="25a,25a,25a,25a"] |=== @@ -20,212 +22,122 @@ This table shows how even the least amount of downtime can add up to negatively == High Availability Versus Disaster Recovery -* *High availability (HA)* - The measure of a system's ability to remain accessible in the event of a system component failure. Generally, HA is implemented by building multiple levels of fault tolerance and load balancing capabilities into a system. - -* *Disaster recovery (DR)* - The process by which a system is restored to a previous acceptable state, after a natural (flooding, tornadoes, earthquakes, fires) or man-made (power failures, server failures, misconfigurations) disaster. - -While they both increase overall availability, the notable difference is that with HA there is generally no loss of service. HA retains the service and DR retains the data, but with DR, there is usually a slight loss of service while the DR plan executes and the system restores. - -== Deploying Mule for HA and DR Strategies - -You can deploy the Mule runtime in many different topologies to address your HA and DR strategies. One method is through the use of high availability clustering, which provides basic failover capability for Mule. - -When the primary Mule runtime becomes unavailable, for example, because of a fatal JVM or hardware failure or it's taken offline for maintenance, a backup Mule runtime immediately becomes the primary node and resumes processing where the failed instance left off. - -After a system administrator recovers a failed Mule runtime server and puts it back online, that server automatically becomes the backup node. - -image::hadr-cluster.png["A clustered server environment handling incoming and outgoing messages"] - -Seamless failover is made possible by a distributed memory store that shares all transient state information among clustered Mule runtimes, such as: +* *High availability (HA)* - Measures a system's ability to remain accessible when a component fails. You implement HA by building multiple levels of fault tolerance and load balancing into a system. -* SEDA service event queues -* In-memory message queues +* *Disaster recovery (DR)* - Restores a system to an acceptable previous state after a natural disaster (flooding, tornadoes, earthquakes, fires) or a man-made disaster (power failures, server failures, misconfigurations). -As you build your Mule application, it's important to think critically about how best to architect your application to achieve the desired availability, fault tolerance, and performance characteristics. +Both strategies increase overall availability, but they differ in one key way: HA generally preserves the service with no loss, whereas DR preserves the data but usually incurs a brief loss of service while the DR plan runs and the system restores. == Creating Effective SLAs -The Service Level Agreement (SLA) outlines the specific definition of what is acceptable. +A Service Level Agreement (SLA) defines what counts as acceptable service. -The basic definition of high availability is that the service is up and -functioning and properly processing requests and responses. This doesn't -mean the service always operates at full capacity. Systems designed with high availability protect against interruptions and prevent outages from happening in the first place. The SLA identifies expectations for all stakeholders. +High availability means the service stays up, functions correctly, and processes requests and responses properly. It doesn't mean the service always runs at full capacity. Systems that you design for high availability prevent outages and protect against interruptions. The SLA sets expectations for every stakeholder. -SLA Requirements: +An effective SLA does these things: -* Single points of failure are eliminated -* Traffic/requests are redirected/handled -* Detection of failures +* Eliminates single points of failure +* Redirects and handles traffic +* Detects failures -For example a basic service may have these SLAs: +For example, a basic service might define these SLAs: -* The normal operation of the service handles 1000 transactions per sec with 1 sec response times. -* The total downtime per year is 0.5% or 1.83 days. -* Minimal acceptable service impact level is 100 transactions per sec with 1 sec response times for a period of 2 hours a week. +* Normal operation handles 1,000 transactions per second with 1-second response times. +* Total downtime per year stays at or below 0.5 percent, or 1.83 days. +* Minimum acceptable service handles 100 transactions per second with 1-second response times for up to 2 hours a week. -See also: +For details, refer to: * xref:api-manager::defining-sla-tiers.adoc[Defining SLA Tiers] * xref:api-manager::rate-limiting-and-throttling-sla-based-policies.adoc[Rate Limiting and Throttling - SLA-Based Policies] -== High Availability Options +== Disaster Recovery -You can achieve high availability through the use of clustering and load balancing of the nodes. Depending on the defined SLA, four HA options are possible with Mule: +*How quickly can your company get back to work after an IT emergency?* -* <> -* <> -* <> -* <> +Disaster recovery (DR) restores a system to an acceptable previous state after a natural or man-made disaster. To drive your DR plan, use measurable business requirements such as Recovery Time Objective (RTO) and Recovery Point Objective (RPO). -NOTE: Often these options are associated with a disaster recovery strategy. +DR centers on your RPO and RTO. RPO is the point you return to after a disaster. For example, if you back up the system every 24 hours, your RPO is at most the previous 24 hours. RTO is how quickly you restore to your RPO and resume business, which includes the time to bring backup infrastructure or applications online when your primary environment fails. -=== Cold Standby +Backups form a major component of a solid disaster recovery program. Recovery comes in three types: cold, warm, and hot. [%header%autowidth.spread] |=== -|Diagram |Description |Downtime -|image:hadr-cold-standby.png["A cold standby configuration in a server setup"] -|The Mule environment is installed and configured, however one or more operating systems aren't running. This can be a backup of a production system/virtual machine. The environment of the operating system plus the Mule runtime has been started after an outage is detected. -|Some - The time it takes to start the environment and direct traffic. -|=== - -=== Warm Standby +|Term |Definition |Example +|Recovery Time Objective (RTO) +|How quickly do you need to recover this asset? +|1 min? 15 min? 1 hr? 4 hrs? 1 day? -[%header%autowidth.spread] -|=== -|Diagram |Description |Downtime -|image:hadr-warm-standby.png["A warm standby setup for server redundancy"] -|The Mule environment is installed and configured. However the Mule runtimes aren't running; only the operating systems. The Mule runtime instance is started after an outage is detected. -|Little - The time it takes for the Mule runtime instances to start and to route traffic to the environment. +|Recovery Point Objective (RPO) +|How fresh must the recovery be for the asset? +|Zero data loss, 15 mins out of date? |=== -=== Hot Standby - Active-Passive - -[%header%autowidth.spread] -|=== -|Diagram |Description |Downtime -|image:hadr-active-passive.png["An active-passive server setup in a high-availability configuration"] -|The Mule environment is installed, configured, and fully running. However, it's not processing requests until an outage is detected. -|Minimal to none - The time to route traffic to an environment. -|=== +[[ha-dr-by-deployment-model]] +== High Availability and Disaster Recovery by Deployment Model -=== Active-Active +HA mechanisms and DR responsibilities differ depending on how you deploy Mule. Each deployment model has a primary HA mechanism, a DR strategy and responsibility split, and a level of cross-region support. Use these differences to pick the right approach for how you run Mule. -[%header%autowidth.spread] +[%header,cols="15a,24a,27a,11a,23a"] |=== -|Diagram |Description |Downtime -|image:hadr-active-active1.png["A load balancer distributing traffic to a cluster of Mule instances in an active-active configuration"] -|Load Balanced clustered environments -There are two or more Mule environments (each environment has its own cluster) that are fully operational. The load balancer is directing traffic to all of the environments. -|None - There is no service downtime. -|image:hadr-active-active2.png["An active-active server configuration with a load balancer and clustering"] -|Load Balanced single clustered environment -There are two or more Mule environments, however they're part of the same clustered environment. To achieve this scenario, the network latency between environments must be less than 10ms. -|None - There is no service downtime. -|=== - -== High-Availability for On-Premises Deployment Models - -[NOTE] -==== -When you deploy an application to an on-premises Mule cluster by using Runtime Manager, deployment artifacts are staged to cluster nodes sequentially. -Sequential staging doesn’t guarantee that the application starts on one node before startup begins on another. Application startup and readiness across cluster nodes aren’t ordered and can overlap. -==== - -* <> -* <> -* <> - -=== Active-Active Clustering Deployment Model - -It's plausible that two nodes in a clustered or load balanced environment can support 1,500 TPS with one-second responses. In this state, the normal operation of the SLA is being met. If a node fails, the service is impacted. However, the impact doesn't breach the SLA because the node is able to handle 700 TPS with one-second responses; well above the agreed upon acceptable impact level. +|Deployment Model |Primary HA Mechanism |DR Strategy and Responsibility |Cross-Region DR |When to Use -image::hadr-aa-clustering.png["An active-active clustering setup with load balancers and multiple servers"] +|CloudHub 2.0 +|MuleSoft runs multiple replicas across availability zones automatically. +|You own the multiregion strategy and provide a bring-your-own (BYO) global load balancer. MuleSoft manages in-region resilience. +|Supported through the Global Deployment entitlement, which MuleSoft provides. +|Fully managed, multi-replica resilience on AWS. -Distribute the load evenly among multiple Mule nodes: +See xref:cloudhub-2::ch2-ha-dr.adoc[] -* All nodes offer the same capabilities -* All nodes are active at the same time. +|CloudHub +|MuleSoft runs multiple workers across availability zones automatically. +|You own cross-region failover. MuleSoft manages in-region resilience. +|Not built in. You design and run cross-region failover. +|Fully managed, multi-worker resilience on AWS. -*Costs* +See xref:cloudhub::cloudhub-hadr.adoc[] -Vary depending on SLA requirements. This model needs two nodes to satisfy SLA. If the SLA's acceptable service impact changes to the terms stated in the normal operations then at a minimum the environment needs three nodes to accommodate one node failure. More nodes may be required depending on the probability of not having at least two nodes running. +|Runtime Fabric +|You run a minimum of two replicas across availability zones, with optional Horizontal Pod Autoscaling (HPA). +|You own backup, restore, and failover to a standby cluster. +|Not built in. You fail over to a standby cluster in another region. +|Kubernetes-based clusters that you operate. -=== Active-Active Clustering Fault Tolerance Deployment Model +See xref:runtime-fabric::rtf-ha-dr.adoc[] -Fault tolerance means a failure within the system doesn't impact the service at all. This differs from high availability as service impact and downtime are tolerated. +|Hybrid Standalone +|You cluster runtimes to share memory and fail over automatically. +|You own backup, restore, and DR. MuleSoft provides control-plane availability and best practices. +|Not built in. You design and run cross-region recovery. +|Servers you control, managed through Anypoint Platform. -image::hadr-fault-tolerant.png["A fault-tolerant system with redundant nodes and a load balancer"] +See xref:hybrid-standalone::hybrid-standalone-hadr.adoc[] -Fault tolerance differs from high availability by providing additional resources that allow an application to continue functioning after a component failure without interruption. Fault tolerant environments are more costly than high available environments. +|On-Premises +|You cluster Mule runtime servers for automatic failover, using active-active, fault-tolerant, or zero-downtime topologies. +|You own the HA architecture, backup, restore, and DR for your servers and data. +|Not built in. You design and run cross-region recovery. +|On-premises clustering that you operate yourself. -The degree of fault tolerance requires the probability of system failures. Take the SLA example highlighted under high availability and make the minimal acceptable service impact level match the normal operation requirement. - -The new overall SLA now requires the system to be able to handle 1000 transactions per sec with 1 sec response times, zero downtime, and zero service impact. - -If the probability of having more than 1 node fail is low then the architecture would simply require 3 nodes. However, if the probability of more than 1 node is higher than acceptable, more than 3 nodes are required to accommodate multiple failures. - -*Costs* - -More costly due to the required redundancy in order to meet defined SLA. - -=== Zero Downtime Deployment Model - -The goal is to make changes to the environment without impacting the SLAs; including upgrading infrastructure and the applications running on the infrastructure. Typically zero downtime deployments leverage a side-by-side deployment, where the old and new coexist for a short period of time. This is in contrast to an in-place deployment where the service may experience reduced capacity to complete downtime. - -https://www.gartner.com/en/information-technology/glossary/continuous-operations[Gartner] defines continuous operations as “those characteristics of a data-processing system that reduce or eliminate the need for planned downtime, such as scheduled maintenance. One element of 24-hour-a-day, seven-day-a-week operation”. - -The baseline production environment is the current operating environment. A new environment is created with the changes (upgraded runtimes, configurations, new applications). A small percentage of traffic flows to the new environment and increases as the confidence in the new environment increases. The baseline production environment continues its use until the new environment is fully operational (it's handling 100% of the traffic). After the new environment is accepting all traffic, it becomes the new baseline production environment and the previous baseline production environment terminates. - -This example is assuming each environment is using the same number of Mule runtimes and cores. It's plausible that the new environment has either more or fewer Mule runtimes and cores than the baseline environment. - -[%header,cols="60a,40a"] -|=== -|Deployment Step |Diagram -|New Production Environment deployed and a small percentage of traffic is routed to new environment. -|image:hadr-zero-downtime-1.png["A traffic distribution in a smaller percentage from current to new production environments in a high-availability deployment"] - -|Confidence in the new environment continues to increase and more traffic is routed to it. -|image:hadr-zero-downtime-2.png["A traffic distribution in a higher percentage from current to new production environment in a high-availability deployment"] - -|All traffic has been routed to the new environment. -|image:hadr-zero-downtime-3.png["A zero-downtime deployment with traffic shifting from an current to a new environment"] - -|All traffic has been routed to the new environment, which has been promoted to the baseline production environment; the previous baseline environment has been terminated. -|image:hadr-zero-downtime-4.png["A traffic routing to a new production environment completed in a high-availability environment"] +See xref:on-premises-hadr.adoc[] |=== -*Costs* - -This deployment method may temporarily add capacity to the service (can be a few minutes, hours, or days). - -== Disaster Recovery +MuleSoft keeps the Anypoint Platform control plane resilient. In the managed runtime models, MuleSoft also keeps in-region infrastructure resilient; in self-managed and on-premises models, you own the infrastructure. In every model, you own any strategy that spans regions. -*How quickly can your company get back to work after an IT emergency?* - -Disaster recovery (DR) is the process by which a system is restored to a previous acceptable state, after a natural or man-made disaster. For DR, use measurable business requirements such as Recovery Time Objective (RTO) and Recovery Point Objective (RPO), to drive your DR plan. - -Disaster recovery is about your Recovery Point Objective (RPO) and your Recovery Time Objective (RTO). RPO is the "point" that you return to after an IT disaster. For example, if you back up the system every 24 hours, your RPO is a maximum of the past 24 hours. RTO, on the other hand, is how quickly you can restore to your RPO and get back to business. This includes activities like the time it takes to get your spare equipment to start running your backups if your primary equipment isn't working. - -System backups are a major component of a solid disaster recovery program. There are three types of recovery: cold, warm, and hot. - -[%header%autowidth.spread] -|=== -|Term |Definition |Example -|Recovery Time Objective (RTO) -|How quickly do you need to recover this asset? -|1 min? 15 min? 1 hr? 4 hrs? 1 day? - -|Recovery Point Objective (RPO) -|How fresh must the recovery be for the asset? -|Zero data loss, 15 mins out of date? -|=== +== Best Practices -For deployment-specific disaster recovery details, see: +Follow these general design principles for high availability and disaster recovery: -* xref:runtime-manager::cloudhub-hadr.adoc[]. -* xref:cloudhub-2::ch2-ha-dr.adoc[]. +* Keep your integrations stateless. +* Don't share transactional information between client invocations or scheduled executions. +* When the middleware must retain data because of a system limitation, persist it in an external store, such as a database or a messaging queue, rather than in the middleware infrastructure or memory. +* As the system scales—especially in the cloud—keep each worker or node's state and resources independent of the other workers, which delivers better performance, scalability, and reliability. -== Keep Integrations Stateless +== See Also -As a general design principle it's important to ensure integrations are stateless in nature. This means that no transactional information is shared between various client invocations or the executions (in case of scheduled services). If the middleware needs to keep some data because of a system limitation, it should be persisted in an external store such as a database or a messaging queue. It shouldn't be stored within the middleware infrastructure or memory. It's critical to note that as the system scales, especially in the cloud, the state and resources used by each worker/node should be independent of the other workers. This model ensures better performance, scalability, and reliability. +* xref:cloudhub-2::ch2-ha-dr.adoc[CloudHub 2.0 High Availability and Disaster Recovery] +* xref:cloudhub::cloudhub-hadr.adoc[CloudHub High Availability and Disaster Recovery] +* xref:runtime-fabric::rtf-ha-dr.adoc[Runtime Fabric High Availability and Disaster Recovery] +* xref:hybrid-standalone::hybrid-standalone-hadr.adoc[Hybrid Standalone High Availability and Disaster Recovery] +* xref:on-premises-hadr.adoc[On-Premises Deployment High Availability and Disaster Recovery] diff --git a/modules/ROOT/pages/on-premises-hadr.adoc b/modules/ROOT/pages/on-premises-hadr.adoc new file mode 100644 index 000000000..c724e5878 --- /dev/null +++ b/modules/ROOT/pages/on-premises-hadr.adoc @@ -0,0 +1,201 @@ += On-Premises Deployment High Availability and Disaster Recovery +ifndef::env-site,env-github[] +include::_attributes.adoc[] +endif::[] + +When you run Mule runtime engine on servers that you manage, you build high availability (HA) and disaster recovery (DR) yourself. You achieve HA by clustering Mule runtime instances so that another node takes over when one fails, and you choose a standby topology that matches your SLA. You own DR under a shared-responsibility model: because you control the infrastructure, you back up, restore, and recover it with your own tooling, while MuleSoft provides the runtime clustering capabilities and control-plane availability through Anypoint Platform. + +For an overview of HA and DR across all Mule deployment models, see xref:hadr-guide.adoc[High Availability and Disaster Recovery]. + +== High Availability Versus Disaster Recovery + +High availability (HA) measures a system's ability to remain accessible when a component fails. On self-managed servers, you achieve HA by clustering and load balancing Mule runtime nodes, so that another node takes over when one fails. + +Disaster recovery (DR) restores a system to an acceptable previous state after a natural or man-made disaster, such as flooding, power failures, server failures, or misconfigurations. + +Both strategies increase availability but differ in scope: on-premises, HA keeps your application serving requests when a node in the cluster fails with no loss of service, whereas DR restores service and data after an infrastructure disaster and usually incurs a brief loss of service while the DR plan runs. + +Two measurable objectives drive your DR plan: + +Recovery Time Objective (RTO):: The maximum downtime a business tolerates. RTO is the time the system takes to recover after a disruption. + +Recovery Point Objective (RPO):: The maximum data loss a business accepts after a disaster. RPO drives how often you back up data. + +== High Availability in On-Premises Deployments + +You achieve high availability on self-managed servers by clustering and load balancing Mule runtime nodes. Depending on your SLA, you choose a standby topology and a deployment architecture that meet your availability and fault-tolerance requirements. + +Clustering and failover:: You can deploy Mule runtime engine in many topologies to support your HA and DR strategies. One common method, high-availability clustering, gives Mule basic failover capability. When the primary Mule runtime becomes unavailable—because of a fatal JVM or hardware failure, or because you take it offline for maintenance—a backup Mule runtime immediately takes over as the primary node and resumes processing where the failed instance stopped. After you recover the failed Mule runtime server and bring it back online, that server automatically becomes the backup node. + +Distributed memory:: A distributed memory store makes seamless failover possible by sharing all transient state across clustered Mule runtimes, including SEDA service event queues and in-memory message queues. + +image::hadr-cluster.png["A clustered server environment handling incoming and outgoing messages"] + +As you build your Mule application, think critically about how to architect it for the availability, fault tolerance, and performance you need. + +=== High Availability Options + +You achieve high availability by clustering and load balancing nodes. Depending on your SLA, Mule supports four HA options: + +* <> +* <> +* <> +* <> + +NOTE: These options often pair with a disaster recovery strategy. + +==== Cold Standby + +[%header%autowidth.spread] +|=== +|Diagram |Description |Downtime +|image:hadr-cold-standby.png["A cold standby configuration in a server setup"] +|You install and configure the Mule environment but leave one or more operating systems stopped—for example, as a backup of a production system or virtual machine. You start the operating system and the Mule runtime after you detect an outage. +|Some. Covers the time to start the environment and route traffic. +|=== + +==== Warm Standby + +[%header%autowidth.spread] +|=== +|Diagram |Description |Downtime +|image:hadr-warm-standby.png["A warm standby setup for server redundancy"] +|You install and configure the Mule environment and run the operating systems but leave the Mule runtimes stopped. You start the Mule runtime after you detect an outage. +|Little. Covers the time for the Mule runtime instances to start and route traffic to the environment. +|=== + +==== Hot Standby - Active-Passive + +[%header%autowidth.spread] +|=== +|Diagram |Description |Downtime +|image:hadr-active-passive.png["An active-passive server setup in a high-availability configuration"] +|You install, configure, and fully run the Mule environment, but it doesn't process requests until you detect an outage. +|Minimal to none. Covers the time to route traffic to the environment. +|=== + +==== Active-Active + +[%header%autowidth.spread] +|=== +|Diagram |Description |Downtime +|image:hadr-active-active1.png["A load balancer distributing traffic to a cluster of Mule instances in an active-active configuration"] +|Load-balanced clustered environments: +Two or more fully operational Mule environments run at the same time, each with its own cluster, and the load balancer distributes traffic across all of them. +|None. The service experiences no downtime. +|image:hadr-active-active2.png["An active-active server configuration with a load balancer and clustering"] +|Load-balanced single clustered environment: +Two or more Mule environments belong to the same cluster. This scenario requires network latency between environments to stay below 10 ms. +|None. The service experiences no downtime. +|=== + +=== On-Premises Deployment Models + +You size your on-premises architecture to meet your SLA. Mule supports three common deployment models: + +* <> +* <> +* <> + +==== Active-Active Clustering Deployment Model + +Two nodes in a clustered or load-balanced environment can plausibly support 1,500 TPS with 1-second responses, which meets the normal SLA. If one node fails, the service degrades but still handles 700 TPS with 1-second responses—well above the agreed acceptable impact—so the failure doesn't breach the SLA. + +image::hadr-aa-clustering.png["An active-active clustering setup with load balancers and multiple servers"] + +Distribute the load evenly across multiple Mule nodes: + +* All nodes offer the same capabilities. +* All nodes stay active at the same time. + +*Costs* + +Costs vary with your SLA requirements. This model needs two nodes to meet the SLA. If the acceptable service impact rises to match normal operations, the environment needs at least three nodes to absorb a single node failure. High-probability failure scenarios can require even more nodes. + +==== Active-Active Clustering Fault Tolerance Deployment Model + +Fault tolerance means a failure within the system doesn't affect the service at all. It differs from high availability, which tolerates some service impact and downtime. + +image::hadr-fault-tolerant.png["A fault-tolerant system with redundant nodes and a load balancer"] + +Fault tolerance differs from high availability by adding resources that let an application keep running—without interruption—after a component fails. Fault-tolerant environments cost more than highly available environments. + +The degree of fault tolerance you need depends on the probability of system failures. Take the earlier SLA example and raise the minimum acceptable service impact to match the normal operation requirement. + +The new SLA now requires the system to handle 1,000 transactions per second with 1-second response times, zero downtime, and zero service impact. + +If the probability of losing more than one node stays low, the architecture needs just three nodes. If that probability rises above what you accept, the architecture needs more than three nodes to absorb multiple failures. + +*Costs* + +This model costs more because it requires the redundancy to meet the defined SLA. + +==== Zero Downtime Deployment Model + +This model lets you change the environment—including upgrading the infrastructure and the applications that run on it—without breaching your SLAs. Zero-downtime deployments typically use a side-by-side approach, where the old and new environments coexist briefly. An in-place deployment, by contrast, can reduce capacity or cause complete downtime. + +https://www.gartner.com/en/information-technology/glossary/continuous-operations[Gartner] defines continuous operations as "those characteristics of a data-processing system that reduce or eliminate the need for planned downtime, such as scheduled maintenance. One element of 24-hour-a-day, seven-day-a-week operation". + +The baseline production environment is your current operating environment. You create a new environment with the changes—upgraded runtimes, new configurations, and new applications. A small percentage of traffic flows to the new environment and increases as your confidence grows. The baseline environment keeps serving traffic until the new environment handles 100 percent of the traffic. After the new environment accepts all traffic, it becomes the new baseline production environment, and the previous baseline terminates. + +This example assumes that each environment uses the same number of Mule runtimes and cores, but the new environment can use more or fewer. + +[%header,cols="60a,40a"] +|=== +|Deployment Step |Diagram +|You deploy the new production environment and route a small percentage of traffic to it. +|image:hadr-zero-downtime-1.png["A traffic distribution in a smaller percentage from current to new production environments in a high-availability deployment"] + +|Your confidence in the new environment grows, and you route more traffic to it. +|image:hadr-zero-downtime-2.png["A traffic distribution in a higher percentage from current to new production environment in a high-availability deployment"] + +|You route all traffic to the new environment. +|image:hadr-zero-downtime-3.png["A zero-downtime deployment with traffic shifting from an current to a new environment"] + +|You promote the new environment to baseline production and terminate the previous baseline. +|image:hadr-zero-downtime-4.png["A traffic routing to a new production environment completed in a high-availability environment"] +|=== + +*Costs* + +This method can temporarily add capacity to the service for a few minutes, hours, or days. + +== Disaster Recovery in On-Premises Deployments + +Because you control the infrastructure, you own disaster recovery for on-premises deployments. You back up your servers, applications, and data, and you bring backup infrastructure online to meet your recovery objectives when your primary environment fails. + +Standby topologies:: Use a cold, warm, or hot standby topology to meet your RTO and RPO. A hot standby minimizes downtime but costs the most, whereas a cold standby costs the least but takes longest to recover. + +Backups:: Back up Mule applications, runtime configurations, and application data to external storage outside your primary environment. Schedule backups often enough to meet your RPO. + +Anypoint configuration:: Back up your platform configurations—such as connections and properties—with the Anypoint CLI or Anypoint Platform APIs. + +MuleSoft responsibility:: MuleSoft keeps the Anypoint Platform control plane available across multiple availability zones. The control plane stores only operational metadata and configuration, not your business data. + +== Considerations and Limitations + +Account for these factors when you design a clustered on-premises topology: + +Network latency:: Clustering across availability zones or regions can degrade performance because of network latency. A single clustered environment that spans locations requires network latency between nodes to stay below 10 ms. Where possible, place cluster nodes on the same subnet or availability zone. + +Simultaneous node restarts:: Don't restart multiple cluster nodes at the same time, such as during operating system patching. Concurrent restarts can trigger cluster-membership eviction and take applications offline. + +Sequential deployment staging:: When you deploy an application to an on-premises Mule cluster through Runtime Manager, Runtime Manager stages deployment artifacts to cluster nodes sequentially. Sequential staging doesn't guarantee that the application starts on one node before startup begins on another. Application startup and readiness across cluster nodes follow no fixed order and can overlap. + +Cost trade-offs:: Higher availability and fault tolerance require more nodes and infrastructure. Size your architecture to the SLA you must meet, rather than to the maximum possible resilience. + +== Best Practices + +Follow these design principles for high availability and disaster recovery on self-managed servers: + +* Keep your integrations stateless, and externalize state to an external store, such as a database or a messaging queue. +* Run identical Mule runtime, Runtime Manager agent, and Java versions on all cluster nodes. +* Choose a standby topology and node count that meet your SLA rather than the maximum possible resilience. +* Schedule cluster changes during maintenance windows. +* Test your failover and restore procedures regularly. + +== See Also + +* xref:hadr-guide.adoc[High Availability and Disaster Recovery] +* xref:mule-high-availability-ha-clusters.adoc[Mule Runtime HA Cluster Overview] +* xref:creating-and-managing-a-cluster-manually.adoc[Create and Manage a Cluster Manually]