Distributed Workflow Orchestration Platform  ·  View 30 of 31  ·  Assurance

Leader Election and Fencing

The one component that needs a single active coordinator, and why a lease alone is not enough.

Editable source SVG draw.io All views
Scheduler A
Scheduler A
Scheduler B
Scheduler B
Lease document (Cosmos)
Lease document (Cosmos)
Schedule store
Schedule store
Service Bus
Service Bus
Azure Monitor
Azure Monitor
1. conditional write if expired · token 41
1. conditional write if expired · token 41
2. acquired · fencing token 41 · TTL 30 s
2. acquired · fencing token 41 · TTL 30 s
3. renew every 10 s
3. renew every 10 s
4. claim due schedules with token 41
4. claim due schedules with token 41
5. enqueue execution starts
5. enqueue execution starts
6. 35 s stop-the-world pause · renewal missed
6. 35 s stop-the-world pause · renewal missed
7. lease expired · acquire · token 42
7. lease expired · acquire · token 42
8. acquired · fencing token 42
8. acquired · fencing token 42
9. claim due schedules with token 42
9. claim due schedules with token 42
10. resumes · writes with stale token 41
10. resumes · writes with stale token 41
11. rejected · 41 < 42
11. rejected · 41 < 42
12. self-demote · stop firing
12. self-demote · stop firing
13. LeaderChanged event · alert
13. LeaderChanged event · alert
Leader Election — Split Brain and the Fencing Token That Stops It
Leader Election — Split Brain and the Fencing Token That Stops It
The lease alone does not prevent split brain, because A cannot know it lost the lease until it tries to use it. The fencing token does: every write carries it, and the store rejects anything below the highest token it has seen.
The lease alone does not prevent split brain, because A cannot know it lost the lease until it tries to use it. The fencing token does: every write carries it, and the store rejects anything below the highest token it has seen.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display

The decision

  • Only the scheduler is leader-elected. Everything else is a competing consumer, because leader election is a liveness bottleneck and should be applied to the smallest possible surface.
  • The lease lives in Cosmos with a conditional write, not in Redis. Redis-based distributed locks are not safe for correctness-critical leases under pause and partition, and this lease decides whether a month-end job fires twice.
  • The fencing token is the actual protection. A lease alone cannot prevent split brain, because a paused leader has no way to know it lost the lease until it tries to use it — by which time it may already have acted.

How the trace resolves it

  • Scheduler A pauses for 35 seconds and the 30-second lease expires. B acquires it with token 42. A resumes believing it is still leader, which is the moment every naive design double-fires.
  • A's write carries token 41 and is rejected because the store has seen 42. A detects the rejection, self-demotes and stops firing — the failure is contained by the store rather than by A's good behaviour.
  • Lease TTL 30 seconds with renewal every 10 seconds gives two missed renewals of tolerance, and bounds worst-case scheduler unavailability at 30 seconds against a 60-second tick.

Risks

  • Every write path that a leader performs must carry and check the token. A path added later that forgets to is a silent reintroduction of split brain, so this is a code-review checklist item and a test.
  • Leader change during a fire window can delay schedules by up to one tick. That is accepted; the misfire policy in view 15 decides whether a delayed schedule catches up.
  • Cosmos availability bounds scheduler availability. Immediate and event-driven executions are unaffected, so a scheduler outage degrades one trigger mode rather than the platform.