Distributed Workflow Orchestration Platform  ·  View 09 of 31  ·  Data

Data Architecture & Storage Zones

What is authoritative, what is rebuildable, and what the backup plan therefore has to cover.

Editable source SVG draw.io All views
Authoritative — losing this loses the platform
Authoritative — losing this loses the platform
Cosmos DB · account cosmos-wfo-prod · session consistency
Cosmos DB · account cosmos-wfo-prod · session consistency
workflow-definitions
PK /tenantId · no TTL
workflow-definitions...
executions
PK /workflowExecutionId
executions...
execution-events
PK /workflowExecutionId · 90 d
execution-events...
dead-letters
PK /tenantId · 180 d
dead-letters...
leases
PK /leaseName · fencing token
leases...
idempotency-keys
PK /taskExecutionId · 7 d
idempotency-keys...
schedules
PK /tenantId
schedules...
Payload and artifact — referenced, never inlined
Payload and artifact — referenced, never inlined
Blob Storage · ZRS · immutable containers
Blob Storage · ZRS · immutable containers
task-payloads
claim-check > 64 KB · 30 d
task-payloads...
task-outputs
30 d · lifecycle to cool
task-outputs...
deadletter-payloads
180 d · legal hold capable
deadletter-payloads...
definition-archive
WORM · 7 y
definition-archive...
Derived — rebuildable from the event log
Derived — rebuildable from the event log
Reporting and cache
Reporting and cache
Azure SQL reporting
execution facts · 400 d
Azure SQL reporting...
Redis counters
concurrency · circuit state
Redis counters...
Status projections
hot read model
Status projections...
Telemetry — operational, not a system of record
Telemetry — operational, not a system of record
Azure Monitor
Azure Monitor
Log Analytics
30 d hot · 2 y archive
Log Analytics...
Application Insights
traces · 5% sampled
Application Insights...
Audit log export
immutable Blob · 7 y
Audit log export...
change feed
change feed
projection
projection
payload ref
payload ref
payload snapshot
payload snapshot
nightly export
nightly export
Data Architecture — Storage Zones by Ownership and Rebuildability
Data Architecture — Storage Zones by Ownership and Rebuildability
Data store
Data store
Security / platform
Security / platform
event / async
event / async
synchronous
synchronous
batch
batch
The rule that sizes the backup plan: only the authoritative zone is restored from backup. Everything in the derived zone is rebuilt by replaying execution-events, so its RPO is the log's, not its own.
The rule that sizes the backup plan: only the authoritative zone is restored from backup. Everything in the derived zone is rebuilt by replaying execution-events, so its RPO is the log's, not its own.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display

The zoning rule

  • Only the authoritative zone is restored from backup. Everything derived is rebuilt by replaying execution-events, which shrinks the recovery surface to one Cosmos account and one Blob account.
  • Payloads are never inlined into state or into messages beyond 64 KB. The claim-check pattern keeps message size bounded and keeps a 40 KB p99 payload from becoming a 40 KB RU charge on every state transition.
  • Telemetry is explicitly not a system of record. If an answer is only available in Log Analytics, it is not auditable — audit facts are execution events, exported to immutable Blob.

Partition keys and retention

  • executions and execution-events are partitioned on workflowExecutionId, which co-locates an execution's entire state and is what makes the transactional batch in view 13 possible.
  • workflow-definitions and dead-letters are partitioned on tenantId, because their access pattern is tenant-scoped listing rather than execution-scoped read.
  • Retention: events 90 days then archived, dead-letters 180 days, payloads 30 days, definitions forever, audit export 7 years. Every one of these is a cost decision as much as a compliance one.

Risks

  • A 500-task execution concentrates on one logical partition, bounded at 20 GB and 10,000 RU/s. With payloads externalised this is comfortable, but it is the design's hardest ceiling.
  • Session consistency is chosen over strong. A read from a different region or session may lag; every orchestration read that matters is a point read within the session that wrote it.
  • Blob lifecycle to cool storage after 30 days assumes dead-letter investigation happens inside that window. A slower operations process would silently raise rehydration cost.