Problem
I will briefly explain a use case/requirement that the customer currently has:
The customer uses AI APIs (OpenAI Provider) that are connected to models deployed in Foundry. These models are deployed in multi-region mode to ensure high availability, disaster recovery, and reduced latency for their use cases.
Currently, WSO2 has a failover policy applicable to models (https://apim.docs.wso2.com/en/4.6.0/ai-gateway/multi-model-routing/failover/), but not to multi-region or multi-cluster within those models.
The requirement is a policy or guardrail that calls an enpoint/region within the same model, and if that region fails due to saturation (429) or downtime, switches to another failover region/endpoint and, at some point (either through a configurable time or ideally, when it automatically detects that the "primary" endpoint is ok again) call the primary region/endpoint again.
In a first proof of concept, this objective was achieved but in an inefficient way. The problem is that it calls the first endpoint/region, and if it fails due to any of the configured error codes, it is able to call an alternative one, but it will continue to call the first endpoint (Failed) while it is inactive, which generates too many calls.
Proposed Solution
Create a new policy similar to the current failover but that applies to Regions/endpoints.
Alternatives
You can create a customization, but it's not the cleanest approach, and if you create an out-of-the-box policy, more customers will likely be able to use it.
Version
APIM 4.6.0
Problem
I will briefly explain a use case/requirement that the customer currently has:
The customer uses AI APIs (OpenAI Provider) that are connected to models deployed in Foundry. These models are deployed in multi-region mode to ensure high availability, disaster recovery, and reduced latency for their use cases.
Currently, WSO2 has a failover policy applicable to models (https://apim.docs.wso2.com/en/4.6.0/ai-gateway/multi-model-routing/failover/), but not to multi-region or multi-cluster within those models.
The requirement is a policy or guardrail that calls an enpoint/region within the same model, and if that region fails due to saturation (429) or downtime, switches to another failover region/endpoint and, at some point (either through a configurable time or ideally, when it automatically detects that the "primary" endpoint is ok again) call the primary region/endpoint again.
In a first proof of concept, this objective was achieved but in an inefficient way. The problem is that it calls the first endpoint/region, and if it fails due to any of the configured error codes, it is able to call an alternative one, but it will continue to call the first endpoint (Failed) while it is inactive, which generates too many calls.
Proposed Solution
Create a new policy similar to the current failover but that applies to Regions/endpoints.
Alternatives
You can create a customization, but it's not the cleanest approach, and if you create an out-of-the-box policy, more customers will likely be able to use it.
Version
APIM 4.6.0