Back to blog
  • ENGINEERING
  • INSIGHTS
  • PRODUCT

Technical Deep Dive: Multi-Region and Multi-Cloud Resilience in Orkes Conductor

Alex Devlin Alex Devlin Content Engineer
JH James Harris Software Engineer
Last updated: · 14 mins read

How multi-region and multi-cloud resilience in Orkes Conductor keeps workflows running across regions and cloud providers, behind a single DNS endpoint.


A purple panel titled Multi-region Conductor, under the hood, beside a world map with
four Orkes Conductor clusters joined by replication links. Figures along the bottom read
10 min RTO, 90 s max RPO, and 99.99%
SLA.

As announced recently, multi-region and multi-cloud resilience is now available in Orkes Conductor as an add-on for enterprise deployments, backed by a 99.99% availability SLA. This follow-up post explains how the feature works on Orkes-managed deployments, the design decisions behind it, and what to keep in mind when building workflows that run across regions.

What multi-region resilience gives you

Regional cloud outages are rare, but when one does happen every workflow running in that region pauses with it: orders halfway through checkout, payments awaiting confirmation, and scheduled jobs waiting to fire.

Multi-region and multi-cloud resilience in Orkes Conductor addresses this by keeping a replica - a second Conductor cluster in another region or on another cloud provider entirely - ready to take over if the primary fails. When the replica takes over, workflows that were mid-flight continue; your workers and applications keep using the same multi-region DNS endpoint without any redeployment necessary.

The capability is available across AWS, Microsoft Azure, and Google Cloud. The replica can run with the same cloud provider as the primary, or a different one. Because the two clusters can sit with different providers, an outage at one provider does not have to take your workflows down with it: the replica already holds your workflows, their state, and the credentials your workers use.

Automatic failover is enabled by default on Orkes-managed deployments with no additional configuration required. Orkes SREs can also add a human approval step, so that a person confirms an outage is real before the replica takes over.

Designing resilience across clusters

Orkes Conductor is already resilient within a single region. Durable execution preserves workflow state through worker failures and restarts, and multi-AZ deployments protect a cluster against availability-zone outages. Multi-region and multi-cloud resilience takes that protection a step further, to the loss of an entire region or cloud provider. Getting there took careful design because a workflow engine is stateful.

Stateless services are generally simple to recover after a regional outage. The service can be redeployed in another region, the load balancer can point traffic at the new copy, and nothing is lost beyond the requests that were in flight at the moment of the outage.

For a workflow engine, the state is the critical part. At any moment a cluster is holding:

  • Partially-finished workflows.
  • Tasks handed out to workers that are pending execution.
  • Retry counters.
  • Scheduled work that has not been triggered yet.
  • The decision history that determines what happens next.
A checkout workflow for one order at the moment its region goes down, with 2 of 5
tasks complete. Validate cart and Check stock are completed; the recorded decision, in
stock, means the card is charged next. Charge card is in flight: handed to a worker,
result not returned yet, on attempt 2 of 3. Create shipment has not started. Follow-up
email is scheduled, with a timer set for tomorrow that has not triggered yet. Without
this workflow's state (its progress, decisions, retry counts, and timers) a second
cluster cannot resume the order. With that state replicated, execution continues on
the replica cluster from where it stopped: the Charge card task will be retried and
the final 2 tasks scheduled as expected.
One checkout order workflow, frozen mid-flight due to a regional outage. A second cluster just needs replicated state to continue execution.

The design question for multi-region was how to carry that state to a second cluster in another region or cloud, which may not share a storage technology with the first. That makes it a question about state rather than traffic: how to preserve it, how to share it, and who is allowed to change it.

How a pair works

A multi-region deployment is a pair: two Conductor clusters in different regions, or on different clouds, joined together as one logical cluster. At all times, one cluster is designated as the primary. It accepts writes, runs workflows, and hands out tasks. The other cluster is a read-only replica which continuously follows the state of the primary cluster. Replication runs one way, from primary to replica, and only one cluster holds the primary role at a time.

Clients never have to address either cluster directly because the pair has a single multi-region DNS endpoint: one name that always resolves to whichever cluster currently holds the primary role. Agents and workers poll it, applications start workflows against it, dashboards point at it, and users access Conductor through it. Workers pointed at the pair’s name follow the primary automatically, so they need no failover logic of their own.

A multi-region pair. Workers, apps, dashboards, and users all connect to one DNS
name, which resolves to the primary cluster in Region A. The primary accepts writes,
runs workflows, and hands out tasks, and replicates one way to a read-only replica in
Region B. Together the two clusters form one logical cluster.
Every client uses one DNS name, which resolves to the primary. The primary replicates one way to a read-only replica.

Roles change through a promotion: the replica becomes the primary, and the pair’s name follows the new primary. There are no client configuration changes needed, no connection strings are rewritten, and nothing needs to be redeployed. Since the replica already holds the replicated state, the pair simply continues to exist at the address it always had. Clients do see a brief interruption while the roles switch, which is covered in After a promotion.

Before and after a promotion. Before: one DNS name resolves to the primary in
Region A, which replicates to the replica in Region B. After: Region B is the primary,
the same DNS name now resolves to it, and Region A is now the replica, replicating
from Region B. Clients keep the same address, with no configuration changes, rewritten
connection strings, or redeploys.
On promotion, the replica becomes the primary, the former primary becomes its replica, and the name follows the new primary. Clients keep the same address.

Nothing extra in the request path

As stated above, clients reach a pair through a single DNS name. That name lives in a managed DNS service, which health-checks both clusters and always resolves the name to the cluster that is currently the healthy primary. From there, your client connects to that cluster exactly as it would in a single-region deployment.

In other words, pairing two clusters does not add a new network step between you and Conductor. DNS is a constant in any web service: your clients already resolve a name to reach a single-region cluster, and a pair uses that same lookup. There is no proxy, shared load balancer, routing tier, or global front door in the request path, so nothing new adds latency, saturates under load, or needs capacity planning. The DNS provider is the one component the two clusters share; it has an effective SLA of 100% and is served from a globally distributed network - not from any single region.

Both clusters also keep their own addresses - the pair’s DNS name is an additional layer on top of them. That matters during an outage because the replica stays reachable at its own address, ready to be promoted to primary. Clients never need to worry about the addresses of individual clusters: the pair’s name always resolves to the healthy primary so clients never need to be reconfigured.

Replicate state - not storage

The obvious way to keep two clusters in step is to replicate the storage. Put a cross-region read replica behind the second cluster, let the database vendor move the bytes, and treat the whole data pipeline as “someone else’s problem”.

We chose not to.

Instead, we designed multi-region resilience so that replication happens within Conductor. The primary cluster publishes its own changes as a stream, and the replica applies them through the same code paths Conductor uses for local execution. What crosses the region boundary is Conductor’s own understanding of what changes were made.

Comparison of two replication designs. Left: two databases from the same vendor
replicate underneath a primary Conductor and a standby Conductor, which ties both ends
to one vendor and can leave the replica in an unexpected state. Right: a primary
Conductor on AWS streams its own changes to a running replica Conductor on Google
Cloud; each side can use any storage engine Conductor supports, and the replica stays
in a valid state.
Replicating within Conductor is what lets each side of a pair use different storage, or a different cloud.

This decision carries three main advantages.

1. The two clusters do not have to share a storage technology, a version, or a vendor. Nothing in the path assumes both ends are the same database or managed storage from the same cloud provider. The only restriction is that each cluster must have a storage backend that Conductor supports, but this is already true for any single cluster.

2. Paired clusters can run on different clouds. For example, your primary could run on AWS while your replica runs on Google Cloud. This means you choose where your replica runs based on your own requirements, not on what your database vendor supports. A storage-level design would have tied a pair to one vendor’s replication feature and, with it, to one vendor.

3. The replica is a running Conductor instance. It holds the state as Conductor understands it, so promoting it is a change of role, with nothing to restore. All data arriving at the replica has already passed Conductor’s own validation and business logic on the primary, so the replica is always in a valid state that Conductor itself produced and verified. A storage-level design would not have this guarantee and could result in an unexpected state after a failover.

What carries over in a failover

Replication covers the state the replica needs to run your workflows as the new primary. That state falls into three groups.

  1. Execution state. Workflow executions and task state replicate as they change, so the replica tracks the work actually in flight - not just the definitions of what could run.
  2. Definitions and configuration artifacts, including:
    • Workflow and task definitions, every version of them.
    • Event handlers and webhook configurations.
    • Environment variables, secrets, and integration definitions.
    • AI prompt templates.
  3. Identity and access settings, including:
    • Access keys, with their hashed secrets, so a key issued against the pair keeps working after a promotion.
    • Users and service identities.
    • Role and permission grants, including tag-based ones.

Replication is encrypted end-to-end in transit between the two clusters, and data is encrypted at rest in both locations. That covers everything listed above, including secrets and access keys.

Defining the replication window

We deliberately designed definitions, configuration, and access settings to replicate synchronously. Workflow executions default to asynchronous replication, but can be set to replicate synchronously with a consistency override as described below.

Synchronous replication keeps two clusters in perfect lockstep. The primary cannot finish a data write until the replica, possibly thousands of miles away, confirms it has the write too. This is essential for keeping access controls aligned across a pair: a key that has been revoked on the primary must not start working again after a failover.

However, this is not practical for every workflow execution. Every write would wait on that cross-region round trip, slowing down all workflows on the primary. Worse, it would tie the primary’s health to the replica’s: a slow replica would slow the primary, and if the replica became unreachable, writes on the primary would stall too.

With asynchronous replication, the primary does not wait for confirmation. It completes each write and streams the change to the replica in the background. If the replica is slow or unreachable, it simply falls further behind while the primary keeps running. That lag is the trade-off: replication is eventually consistent, not instantaneous. This means that when the replica is promoted there may be a window of recently accepted or in-flight work that never reached it; this is the data-loss window that a recovery point objective (RPO) measures.

Timeline of asynchronous replication. Region A, the primary, accepts writes w1 to
w6. Region B, the replica, applies each write a little later and has reached w4 when
Region A is lost, so w5 and w6 fall in the RPO window and are completely lost in the
failover. Region B is then promoted and serves as the primary, and in-flight workflows
continue.
Workflow replication is asynchronous, so a promotion can miss the most recent writes. Here, writes w5 and w6 were in the RPO window when Region A went down. They never reached the replica, so the progress they recorded is lost.

How much can be lost depends on how a workflow is started. By default, we target a maximum RPO of 90 seconds, meaning a failover should not cause more than 90 seconds of execution loss.

Synchronous replication for individual workflows

Workflows that cannot tolerate losing recent progress can opt in to synchronous replication. When a workflow is started in synchronous execution mode with its consistency set to REGION_DURABLE, the request is replicated across regions before the workflow runs, and the workflow’s full execution state is then replicated synchronously as well. Synchronous execution mode alone does not change how a workflow replicates: without REGION_DURABLE, it still replicates asynchronously. REGION_DURABLE workflows have an RPO target of near zero, at the cost of the added latency described above.

After a promotion

The new primary carries on from the state it has received: workflows that were mid-flight continue, and tasks that were handed out but never returned are reclaimed under their existing timeout rules.

For workers and applications, a promotion looks like a temporary interruption. How long it lasts is what a recovery time objective (RTO) measures: the time it takes to restore service after the primary is lost. Our RTO is 10 minutes, and in practice recovery is typically faster. Where an approval step has been added, a promotion also waits for that approval.

Calls that were in flight when the primary went down can fail, so clients need to retry them; workers already do this, and applications should too. Once the pair’s name points at the new primary, the next attempt connects to it with the same access keys, so nothing needs reconfiguring. Because a retried call can repeat work the old primary had already accepted, those calls should be idempotent.

Who is primary? Ask DNS

Every failover design has to answer one question: how do the clusters agree on who is in charge, especially when parts of the system cannot talk to each other?

Most systems answer it with a dedicated coordinator, such as a consensus service or a quorum of observers. That works, but the coordinator becomes one more distributed system to operate, with an awkward requirement: it must keep working through the same outage it is meant to manage.

We took a different route. In a multi-region deployment, the DNS layer is the ultimate source of truth. The cluster that DNS reports as the healthy primary is the primary, for clients and for the failover process alike.

Each cluster exposes a health endpoint, which the DNS provider checks continuously from multiple geographic locations. The live primary answers that it is healthy; the replica answers that it is not. So the pair’s name always resolves to the healthy primary: if the primary fails, the name moves only after the replica has been promoted and reports healthy in its place.

DNS health checkers in multiple locations probe both clusters. Region A answers
healthy because it is the live primary. Region B answers not primary, which is the
correct answer for a replica. Workers and apps resolve the pair's name to Region A,
with Region B, the replica, used only once it has been promoted to primary.
Only the live primary reports healthy, so the name always resolves to whichever cluster is in charge.

That makes the health check more than a liveness probe. It reports each cluster’s role, not just whether the cluster is up. This means a replica failing its check provides the correct answer: it is not in charge. Because the pair’s name only ever resolves to the cluster passing that check, a DNS lookup tells anyone which cluster is primary at any time. The mechanism that routes your traffic and the mechanism that decides who is in charge are one and the same, so they cannot disagree.

Orchestrated failovers

Failover on Orkes-managed deployments is handled by a Conductor workflow that runs on a schedule within a centralized Orkes-owned Conductor instance. We run our control plane on our own product: the failover workflow runs on the same Conductor you use.

Each time the workflow runs, it reads the DNS provider’s health-check results for every pair and decides whether a replica needs to be promoted. DNS is the ultimate source of truth, so that is where the workflow takes its cue.

If a primary is down, the workflow promotes the replica and fences the old primary, so the two clusters cannot both believe they are in charge. Only then does the name move: the newly promoted cluster reports itself healthy, and DNS picks that up and points the pair’s name at it. The name never moves before the roles have changed.

The failover workflow. A scheduled health check leads to a decision. If the primary
is healthy, nothing changes. If it is unhealthy, the workflow promotes the replica,
optionally after a human approval, then fences the old primary, and the name follows.
Failover runs as an ordinary Conductor workflow, with an optional human approval before promotion.

Building it this way means that our SREs can utilize the following Conductor features:

  • Every failover has an execution history in the centralized Orkes-owned Conductor instance. We can open it and see each step, its inputs, how long it took, and what it returned. Post-incident review means reading a workflow execution instead of correlating logs.
  • Retries and timeouts are built in. They are declared the same way as in any other Conductor workflow, so a flaky step during an incident is retried instead of stopping the failover.
  • Orkes SREs can add a human step to guard against false positives. This ensures that if the primary is unreachable but not actually down, a person can confirm the outage is real before proceeding with a promotion.
  • Failovers can be triggered manually through exactly the same code path as the automatic case. The promotion our SREs rehearse is the same process that runs during a real outage.

We also use the same promotion mechanism outside of outages:

  • Migrating between regions or clouds. Because replication is not tied to a storage vendor, Orkes SREs can move an Orkes-managed cluster from one cloud to another with a promotion instead of an export and re-import of your data. There is far less room for error, and your credentials come with it.
  • Proving recovery time. Our SREs rehearse failovers on Orkes-managed deployments, and every rehearsal leaves an execution record. That lets us test the full failover cycle and back the RTO with evidence instead of an estimate.

Rehearsal only builds confidence if it exercises the real thing. A disaster recovery procedure that differs from the one that was rehearsed is not a procedure; it is a hope.

Caveats worth knowing

A multi-region deployment is built for teams whose workflows cannot stop when a region does: checkout flows, payment processing, and anything else with an uptime commitment behind it. It is not the answer to every reliability problem, however.

  • It does not split work across both regions. Only the primary accepts writes, while the replica stays ready to take over. A second region adds availability, not throughput.
  • It is not a backup. Replication faithfully propagates everything, including deletions and mistakes, to the replica. On Orkes-managed deployments, Orkes maintains backups separately for that purpose.
  • It does not remove the need for retries. A promotion is still an interruption that can last for the RTO window, and applications that treat a single failed call as fatal will see errors during one.
  • It does not prevent duplicate work. Some in-flight tasks may be retried during a promotion. Workers and endpoints that perform business actions, such as issuing a payment, should be idempotent - in other words, a retry must not repeat the action.

Why we run failovers in Conductor

Failovers are a well-established pattern in software engineering. What sets our design apart is where the decisions live: replication happens within Conductor, DNS is the ultimate source of truth for which cluster is primary, and the failover itself runs as a Conductor workflow.

We deliberately chose Conductor itself as our failover orchestrator. A failover is a multi-step process that has to keep going while parts of the system are failing: it must check health, promote the replica, fence the old primary, and move the name. Each step needs retries and timeouts, and sometimes a person to confirm the outage is real.

That is exactly the kind of process Conductor was built to run. A separate controller would have to rebuild those guarantees from scratch, in code that only runs during an emergency.

Two kinds of workflow running on the same engine. Your workflows, such as checkout,
payments, scheduled jobs, and AI agents, and the Orkes-managed multi-region failover
workflow (health check, promote, fence, name follows) both run on Orkes Conductor, and
share its execution history, retries and timeouts, human approval, and the ability to
rehearse the real path.
One engine for your workflows and for the failover that protects them.

With this design, your workflows and the failover that protects them run on the same engine, with the same execution history, retries, approvals, and rehearsals. Those are properties you already expect from your business processes, and there is no good reason your infrastructure should not have them too.

This is by far the strongest test we could give Conductor: the engine your business depends on is the same one we depend on when a region goes down!

Get started with multi-region and multi-cloud resilience

See how paired clusters keep your workflows running across regions and cloud providers in the event of an outage, with a 10-minute RTO.

Get a demo