Architecture Views

When a worker dies halfway through a five-step automation that has already created an invoice and sent an email, what happens next — and who pays for it?

Context

A run in this platform is a sequence of effects in systems the platform does not own and cannot roll back. If run state lives in the worker's memory, the only recovery from a lost worker is to re-run from the beginning, which means re-performing every step that already succeeded. For a pipeline of pure computation that is a performance problem. Here it is a wrong answer in a customer's accounting system. The same argument applies to every other interruption: a rate limit that needs waiting out, a credential that died mid-run, a provider returning 503. Each of them needs the run to stop and later continue from exactly where it was, and none of them can be served by a design whose only verb is 'start over'. At 12,000 step attempts a second the cost of externalising that state is not theoretical either, which is why this is a decision rather than an obvious default.

Decision

A run is a resumable state machine whose complete state lives in an append-only step ledger outside the worker. Every step attempt writes its intent to the ledger before the outbound call and its outcome after, carrying the run, step, attempt number, resolved inputs, effect key, outcome and the provider's response identifier where one is returned. Run status is a projection of the ledger, never an independently updated field. A worker holds only a lease; losing it releases the run for another worker to resume at the last committed step. No step may be attempted without a stable effect key (ADR-05).

How it is realised on AWS

DynamoDB holds the ledger, keyed (run_id, step_id, attempt), replicated synchronously across three availability zones, with on-demand capacity and sharding by run id. Step runners are Fargate tasks that hold no state between steps; a run lease is a short-TTL item whose expiry makes the run eligible for another runner. The intent write and the outcome write are separate items rather than an update, so the ledger is genuinely append-only and the ambiguous case — intent present, outcome absent — is directly observable rather than inferred. Run status in the author-facing run index is materialised from the ledger asynchronously and can be rebuilt from it in full.

Options weighed

• ChosenStep-level durability in an external append-only ledger: A park, a retry, a pause and a resume all become the same operation on the same structure, and no completed effect is ever repeated
• RejectedRun-level durability: the worker holds the run and retries it from the start: Dramatically simpler and needs no ledger, but re-runs step one to retry step four — unacceptable when step one created an invoice
• RejectedCheckpoint only at explicit author-declared boundaries: Pushes a correctness decision onto a non-engineer who has no way to reason about it, and leaves the default unsafe
• RejectedA single mutable run row updated in place per step: Loses the intent-without-outcome signal that makes an ambiguous effect detectable, which is the case the whole design exists to handle
• RejectedAn embedded workflow engine holding state in its own store: A reasonable alternative that moves the same decision behind a dependency; rejected because the effect-key semantics in ADR-05 are specific enough that owning the ledger is simpler than bending somebody else's

Consequences

What it buys

• A worker loss, a rate-limit park, a credential pause and a transient retry are one mechanism — moving a run between states — rather than four special cases with four recovery paths
• The ambiguous outcome is directly observable as an intent with no matching outcome, which is what makes ADR-06 implementable at all
• Run history, the repair inbox and author-initiated replay are all reads of one structure, so they cannot disagree with each other
• The run index can be dropped and rebuilt, which turns a schema change to author-facing history into a routine operation

What it costs

• Two synchronously replicated writes per step attempt, which at 12,000 steps/second is the platform's hottest path and its primary scaling constraint
• A priced component of every step: the ledger write is held to ≤ 18% of per-step cost, and that ceiling is a design constraint rather than a target
• Latency added to every step by the intent write before the call, paid even on the overwhelming majority of steps that succeed first time
• A lease mechanism with its own failure modes — a slow worker whose lease expires while its call is in flight is a duplicate risk that ADR-05 has to absorb

Choose differently when

Choose run-level retry instead when the effects are genuinely idempotent end to end, when runs are short enough that re-running from the start is cheap, or when the platform owns the systems the steps touch and can roll them back. A platform automating only its own internal services has all three properties, which is exactly why the adjacent distributed-workflow-orchestration-platform use case can make a different choice.

Why it holds up over time

Externalising long-running process state into a durable log is the same decision a transaction log, a saga and a workflow engine each make, and it has survived every generation of the technology underneath it. The store will be replaced; the property that a worker may die at any instant without repeating a completed effect follows from the effects being irreversible and somebody else's, which will not change.

When an author asks us to re-run last Tuesday's missed work, what do we read — our copy of the event, or the provider's current state?

Context

The providers are the system of record for what is true. They are not a reliable system of record for what happened: a webhook delivery is usually not retrievable afterwards, a polled change may have been superseded, and a deleted record simply is not there. If the platform keeps no copy, then a replay can only re-read the provider's current state, which means a repair performed a week after a credential died may run against data that has changed or vanished — or may be impossible because the credential is still dead, or because the provider is down. Keeping a copy makes replay exact and independent, and makes the platform the custodian of a month of its customers' data from every product they have connected.

Decision

Every accepted trigger event is durably committed to a partitioned, append-only event log before the provider is acknowledged, and retained for a declared replay window of 30 days. The log is the only thing a replay reads. Retention is therefore a recovery commitment — a promise about how far back a repair can reach — and not a storage preference.

How it is realised on AWS

Amazon MSK, partitioned by connection so per-connection ordering holds without requiring global ordering, with 30 days of retention using tiered storage so the window is affordable. Payload bodies above a size threshold are written to S3 and the log carries a reference, which keeps the log's partitions small and puts the residency and classification rules on object storage where they are easier to enforce. Admission reads the log continuously and deduplicates into at most one run per (automation version, event id).

Options weighed

• ChosenRetain the payload for a declared replay window; replay reads the log: Replay is exact and independent of the provider's availability, credential state and current data
• RejectedRetain only metadata and re-fetch the payload from the provider on replay: Holds almost nothing and materially reduces the breach surface, but makes replay silently run against changed data and impossible when the credential is dead — which is the most common reason a replay is needed
• RejectedRetain nothing; a missed trigger is simply missed: Coherent and honest, and would remove a whole class of risk, but it deletes the repair journey the product is largely sold on
• ChosenLet the workspace choose the window, including zero: Adopted as a per-workspace data-classification setting alongside the default, with the repair path's reduced capability stated plainly rather than discovered

Consequences

What it buys

• A repair performed days later runs against exactly what arrived, not against what the provider happens to hold now
• The accept path owes the provider durability and nothing else, which is what makes a 250 ms acknowledgement achievable
• Admission, dedup and fair scheduling can all be rebuilt or re-run from the log without involving any provider

What it costs

• Thirty days of customer data from every connected product concentrated in one platform — a residency, retention and breach surface far larger than the platform's own records
• Retention is now a throughput obligation too: the log must be readable fast enough that a bulk replay does not starve live admission
• A per-workspace no-retention setting creates a second, weaker repair path that has to be explained rather than hidden

Choose differently when

Choose metadata-plus-re-fetch when the payloads are highly sensitive, when providers offer a stable read-by-id that survives the retention period, and when replays are typically attempted within minutes rather than days. Choose to retain nothing when the product makes no repair promise at all.

Why it holds up over time

Separating 'what is true' from 'what happened' is a distinction every event-driven system eventually discovers, usually after trying to reconstruct history from current state and failing. The storage will change; the need for an independent record of the event, owned by the party that must act on it, will not.

The execution plane is down. Does the platform tell the provider to go away, or take the event anyway?

Context

Provider push deliveries have short timeouts and limited, inconsistent retry behaviour; many providers retry two or three times and then drop the event permanently. A rejected delivery is therefore usually gone for good, and no amount of later recovery brings it back. Execution, by contrast, can be arbitrarily delayed without losing anything, because the work is still described in the log. The two sides of the platform have fundamentally different costs of failure, and giving them one availability target would either over-engineer execution or under-protect ingest.

Decision

The accept path authenticates the delivery, commits it durably to the event log, and acknowledges — and does nothing else. No binding, no credential resolution, no quota check and no execution happens before the acknowledgement. Ingest and execution carry separate availability targets (≥ 99.99% and ≥ 99.95%), scale independently, and a total execution outage loses no trigger.

How it is realised on AWS

ALB in front of stateless Fargate push services; the only work before acknowledgement is signature verification against the per-connection secret and the MSK produce with acknowledgement from all in-sync replicas. Admission runs as a separate consumer group entirely, so its failure, backlog or redeployment is invisible to the provider. Poll workers follow the same contract: read, commit, then advance the cursor (ADR-11).

Options weighed

• ChosenAuthenticate, commit, acknowledge — nothing else on the accept path: The only design in which ingest availability is genuinely independent of everything downstream
• RejectedExecute the first step synchronously before acknowledging: Gives the author a faster perceived start and ties the acknowledgement to credential and provider availability, which is precisely the coupling that loses events
• RejectedValidate the payload against the automation's bindings before acknowledging: Catches author errors a few seconds earlier at the cost of making the accept path depend on the definition registry; the same error is caught at publish time instead (ADR-16)
• RejectedReject deliveries when the execution backlog is beyond a threshold: Protects the platform by discarding exactly the events it exists to not lose; back-pressure is declared through widened latency budgets instead

Consequences

What it buys

• A 250 ms p99 acknowledgement fits comfortably inside every common provider delivery timeout, while a run may legitimately take minutes or park for hours
• Execution can be redeployed, scaled, or fail entirely without a single trigger being lost
• Ingest is stateless apart from cursors, so it scales horizontally against burst without coordination

What it costs

• The author's first feedback moves from the acknowledgement to the run history, so perceived latency has to be managed explicitly as a product surface
• An automation whose definition is broken still accepts and logs events, producing runs that fail immediately and need a classified cause rather than a rejection
• Two availability targets mean two operational postures, and the ingest tier must be genuinely over-provisioned rather than merely autoscaled

Choose differently when

Collapse the two when deliveries are reliably retried by the source for hours, or when the platform owns the source. A system reading from its own durable queue has no reason to separate acceptance from execution, because nothing is lost by refusing.

Why it holds up over time

The principle that the acceptance of work and the performance of work have different costs of failure, and therefore different availability targets, outlives any particular queue. It is the same reason a payment gateway acknowledges before settling.

Can this step be retried safely? And who is supposed to know the answer — the platform, the connector developer, or the author?

Context

The platform's execution guarantees cannot be stronger than what the providers support, and support varies wildly across 40,000 actions. Some accept a caller-supplied idempotency key. Some offer a read-back by natural key that reliably shows whether the effect landed. Many — notably 'post this message' and 'send this email', among the most-used actions in the product — offer neither, and after a timeout the platform genuinely cannot know what happened. A single platform-wide retry policy must therefore be wrong for most of the catalogue: safe enough for one class, and either duplicating or silently dropping effects for another.

Decision

Every action carries a declared replay-safety class — idempotent, checkable or unsafe — as a versioned field of the connector manifest. The class is a hard release gate: an action cannot be published without it. Execution semantics are derived from the class rather than configured separately, and the author-facing behaviour of the unsafe class is stated in the product.

How it is realised on AWS

The class lives in the connector manifest in DynamoDB alongside the schemas and rate-limit profile, versioned with the connector and pinned by each automation to a compatible range. The release pipeline rejects a manifest without it, and contract tests run against provider sandboxes rather than mocks, because the failure being guarded against is the provider's behaviour, which a mock cannot reveal. The step runner reads the class at bind time and selects the ambiguity-resolution path in ADR-06.

Options weighed

• ChosenA declared, versioned per-action class, gated at release: Puts the answer where the knowledge is, and makes it reviewable, testable and pinnable
• RejectedOne platform-wide retry policy: Simple and wrong for most of the catalogue: necessarily either duplicating or dropping, depending on which way it is tuned
• RejectedInfer the class at runtime from provider responses: Attractive and unsound: the inference is only testable by causing the ambiguity it is meant to resolve, in a customer's system
• RejectedLet the author choose per step: Asks a non-engineer a question they have no basis to answer, and the wrong answer is invisible until an invoice is duplicated
• ChosenLet the author choose only the unsafe class's behaviour: Adopted in ADR-06: the author decides what to do about an ambiguity, never whether the ambiguity exists

Consequences

What it buys

• The platform can state a different, honest guarantee per action instead of one promise that is false for most of them
• The interface catalogue becomes organised by the property that actually matters, which makes the unsafe surface countable and reviewable
• Pinning a connector version pins the guarantee, so a provider change cannot silently alter execution semantics for a published automation

What it costs

• The class is self-declared by connector developers whose incentive is to ship, and a wrong declaration is indistinguishable from a right one until an effect is duplicated
• A per-action field across 40,000 actions is a large surface to review and to keep honest as providers change
• The product must explain a distinction users did not ask for, at the moment they are least interested in it

Choose differently when

Drop the classification when every integration target is under your control and can be made idempotent by fiat. An internal platform can mandate idempotency keys across its own services and needs none of this.

Why it holds up over time

For as long as a platform integrates systems it does not control, something must carry the answer to 'is a retry safe here'. The mechanism may one day be discovered rather than declared, but the dependency on the answer is permanent.

The author presses a button marked 'run it again'. Did they mean 'the first one may not have worked' or 'do it a second time on purpose'?

Context

Those two intentions produce opposite correct behaviours, and a platform that conflates them will sometimes duplicate an invoice and sometimes fail to send one. The distinction cannot live in a flag read at call time, because by then the two paths share the same code and the same provider call; it has to be built into how the key is derived, so that the difference is structural and cannot be got wrong by a caller.

Decision

Each step attempt carries a deterministic effect key derived from (run id, step id, logical attempt). A retry is a new physical attempt at the same logical attempt and therefore reuses the key, which is passed to the provider as an idempotency key wherever the action's class says it will be honoured. A replay is an author-initiated action that mints a new logical attempt and therefore a new key, is recorded in the ledger as a replay, and requires the duplicate consequence to be stated and confirmed first.

How it is realised on AWS

The key is a hash of the run id, step id and logical attempt number, computed by the runner before the intent write so that the ledger entry and the outbound call carry the same value. The governor forwards it in whatever header the connector manifest declares. The ledger's attempt key distinguishes physical attempts, while the logical attempt is a field on the entry — so 'how many times did we try' and 'how many times did the author ask for this' are separately countable.

Options weighed

• ChosenKey derived from (run, step, logical attempt); replay increments the logical attempt: Makes the distinction structural: the same code cannot deliver a replay when a retry was meant
• RejectedA random key per physical attempt: Defeats the entire purpose; every retry becomes a new effect at the provider
• RejectedA key derived from the payload's content hash: Collides across two legitimately identical actions — two identical messages the author genuinely wanted twice — and silently drops the second
• RejectedOne key per run: Cannot distinguish steps, so a provider honouring it would reject the second step of the same run
• RejectedLet the connector supply its own key derivation: Flexible, and makes the platform's correctness depend on 8,000 independent implementations of the same subtle rule

Consequences

What it buys

• A retry after a timeout, a lease expiry or a park cannot produce a second effect where the provider honours keys
• 'Try again' and 'do it again' are visibly different in the ledger, so the audit answers which one happened
• The key is transport-independent and survives a change of provider API or protocol

What it costs

• The key must be minted before the intent write, so the derivation sits on the hottest path and cannot depend on anything slow
• It is only useful where the provider honours it — the unsafe class carries the key and gets nothing for it, which has to be explained rather than assumed
• A lease expiry during an in-flight call produces two physical attempts at the same logical attempt, which is safe only because the key is reused — making lease handling a correctness concern, not just a scheduling one

Choose differently when

A single key per operation is enough when the system performs one effect per job and never needs an intentional repeat. The two-level scheme earns its complexity only where a deliberate re-do is a product feature.

Why it holds up over time

Caller-supplied operation identity is an old idea with new names each decade — message dedup id, idempotency key, request id. What endures is that the caller, not the callee, has to assert what counts as the same operation.

The request was sent, the connection dropped, and the provider offers no key and no read-back. Do we send it again or not?

Context

This is the one case where the platform genuinely cannot know what happened and cannot find out. Retrying risks a second email to a customer or a second message in a channel. Not retrying risks the thing never happening at all, silently. Both are wrong, and which is less wrong depends entirely on the action and the author's situation — a duplicate invoice is a serious problem, a duplicate internal reminder is not. The platform has no basis to choose, and choosing silently means being wrong without telling anybody.

Decision

For an action classified unsafe, an ambiguous outcome is never retried automatically. The run is parked in a state that names the ambiguity, the author is told exactly what may or may not have happened, and two explicit choices are offered: continue without retrying, accepting a possible gap, or retry and accept a possible duplicate. For checkable actions the platform reads back by natural key before retrying. For idempotent actions it simply retries under the same key.

How it is realised on AWS

The runner detects the ambiguity directly from the ledger's shape — an intent with no matching outcome — and branches on the manifest's class. The parked state carries a typed reason and the provider's last response if any, and surfaces in the repair inbox as its own category rather than mixed with ordinary failures. The run deadline still applies, so a parked-ambiguous run terminates into a named terminal state rather than waiting indefinitely for an author who will never look.

Options weighed

• ChosenPark and escalate to the author with two explicit choices: The platform refuses to guess on the author's behalf about an effect in their own business
• RejectedAlways retry: Produces duplicate invoices and duplicate customer emails, which is the failure customers escalate hardest and trust least
• RejectedNever retry: Produces a silent gap, which is worse because nobody finds out — the exact failure mode journey 05 is about
• RejectedLet the connector developer pick the default: Moves the guess one step away without improving the information available to make it
• DeferredRefuse to publish automations whose first write is unsafe without acknowledgement: Attractive for high-stakes actions and left to Core Architecture Question 3; it raises a friction cost at authoring time that has not been measured

Consequences

What it buys

• The platform never silently creates a duplicate effect in a customer's finance or communications system
• The ambiguity is named and visible rather than resolved by a default nobody chose
• The decision is made by the only party with the business context to make it

What it costs

• A parked run waits for a human, and many authors will not look — the run deadline then terminates it, which is itself a form of the silent gap this decision tried to avoid
• The product must explain a genuinely hard concept at a bad moment, and 'we do not know if your email sent' is a difficult sentence
• Parked-ambiguous runs accumulate and need their own inventory, alerting and expiry policy

Choose differently when

Pick a default and apply it when the actions are low-stakes and homogeneous — an internal notifier can always retry, and nobody will mind. The escalation is worth its friction only where the effects are irreversible and matter commercially.

Why it holds up over time

Distributed systems will not stop producing ambiguous outcomes, and no protocol removes the case where the acknowledgement is lost and the operation cannot be queried. The lasting decision is whose judgement resolves it.

Twelve thousand runners want to call the same provider at once. Who decides whether they may?

Context

Providers meter access per end-user account, per connector application credential, per region and per source address, and the penalties for overshoot escalate from a 429 to a suspended application credential that breaks every workspace at once. A decision made independently by each runner cannot respect a limit that is shared across runners, and the shared cases are the dangerous ones: a per-application limit is one scarce resource that every tenant is drawing on simultaneously. The same boundary is also the natural place to exchange a short-lived credential, because it is the last point before the call leaves the platform.

Decision

Every outbound third-party call passes a governance point that owns the quota decision; no step worker may call a provider on its own judgement. The governor enforces limits at each level the provider actually imposes them, honours provider backoff headers as authoritative over its own schedule, exchanges the short-lived single-connection token with custody, opens per-provider circuit breakers, and exits through the published egress range.

How it is realised on AWS

A Fargate service holding per-connection and per-connector leases in DynamoDB, in front of NAT gateways with a published Elastic IP range. Leases are checked out in blocks sized to the connection's observed rate so the common case is a local decision, with central arbitration reserved for per-application limits where tenants genuinely contend. Circuit breaker state is per provider and shared across the fleet, so one runner discovering an outage protects the rest.

Options weighed

• ChosenA governance point owning the decision, with leases checked out in blocks: Correct for shared limits, and degrades to local decisions with bounded overshoot when the governor is slow
• RejectedPer-worker local budgets with periodic reconciliation: Fastest and overshoots exactly when traffic is burstiest, which is when a provider is least forgiving
• RejectedA central broker consulted synchronously on every call: Correct and puts a hard synchronous dependency plus a network hop on the hottest path in the platform
• RejectedRely on the provider's 429 and back off reactively: Makes the provider's rate limiter our rate limiter, which works until the provider responds by suspending the application credential

Consequences

What it buys

• A limit shared across tenants can actually be respected and fairly apportioned, which no local scheme achieves
• Credential exchange, egress identity and the quota decision are one boundary, which is what keeps long-lived credentials out of the execution plane
• Provider health becomes a first-class platform signal rather than something each runner rediscovers

What it costs

• The governor is on the path of every outbound call and is the platform's hottest single dependency
• Leases add a tuning problem: too small and they are a central broker with extra steps, too large and they overshoot during a burst
• A cross-account token exchange per run adds latency, mitigated by caching for the run's duration at the cost of widening the revocation window

Choose differently when

Local budgets are sufficient when every limit is per end-user account, traffic is smooth, and the penalty for a modest overshoot is only a 429. Central arbitration earns its cost when tenants share one scarce credential.

Why it holds up over time

Providers will always meter, and the penalty for overshoot will always exceed the cost of pacing. Putting that decision in one place rather than in every caller survives any change of rate-limiting algorithm.

A provider says 'not now, try in ninety seconds'. Is that an error?

Context

At this scale a 429 is the ordinary case, not an incident — the platform makes billions of calls a month against quotas it did not set. Treating it as a failure has three bad consequences: it consumes the step's retry budget on something that is not a fault, it occupies a worker for the duration of a backoff, and it eventually exhausts the attempts and fails a run that was never actually broken. It also teaches the platform to treat the provider's own backoff signal as advice rather than instruction, which is how an application credential gets suspended.

Decision

A rate-limit response moves the run to a waiting state with a release time derived from the provider's own backoff header, which is authoritative in preference to the platform's schedule. A park consumes no worker and does not spend the step's retry budget. The connection's pacing adapts in response, so repeated parking feeds back into the lease size rather than simply repeating.

How it is realised on AWS

The governor returns a typed park outcome with a release timestamp; the runner writes it to the ledger and releases its lease. A dedicated parked-release queue class holds the run until its time, so parked work is isolated from interactive runs and cannot distort their queue-age signal. The platform's own SLO is that ≤ 0.1% of outbound calls receive a 429 at all — the governor's job is to make parking rare, not to make it cheap.

Options weighed

• ChosenPark with a provider-derived release time, outside the retry budget: Treats the ordinary case as ordinary, and keeps the retry budget for actual faults
• RejectedTreat 429 as a retryable error with exponential backoff: Spends the retry budget on capacity rather than faults, and holds a worker through every backoff
• RejectedFail the run and let the author retry: Converts a routine, self-resolving condition into manual work for someone who cannot influence it
• RejectedUse our own backoff schedule and ignore Retry-After: Substitutes our guess for the provider's statement about its own capacity, which is both rude and usually wrong

Consequences

What it buys

• Throughput recovers at the provider's stated time rather than at an arbitrary multiple of it
• The retry budget stays meaningful as a signal about faults, which keeps the failure taxonomy honest
• Parked runs occupy no compute, so a provider-wide slowdown costs storage and patience rather than capacity

What it costs

• A parked-release queue class with its own scheduling, inventory and alerting
• Parked runs can accumulate into a backlog that fires in a burst at release, which needs its own rate-limited release path
• The author experiences a slow automation with no error, which must be explained in the run history as 'running slowly' rather than left blank

Choose differently when

Treat rate limits as errors when they are genuinely exceptional and indicate misconfiguration rather than capacity — a well-provisioned internal API where a 429 means somebody has a bug.

Why it holds up over time

Backpressure from a dependency is information, not failure. Systems that learn to wait on it outlive systems that learn to retry through it, and that has been true of every protocol that has ever had a busy signal.

A customer's security team wants to allowlist us. What address do we give them, and what happens when a neighbour abuses it?

Context

Enterprise customers and many providers restrict inbound API access by source address, so a platform that cannot name its egress addresses cannot be adopted by them at all. Publishing a range makes it a commitment: it cannot change without notice, which constrains every future networking decision. It also makes the range a shared identity — providers rate-limit, throttle and block by source address, so a single runaway automation can degrade or sever access for every workspace behind it.

Decision

The platform egresses through NAT gateways with a published, stable address range that customers and providers may allowlist, and treats changes to it as a product change requiring notice. Because the range is shared, runaway detection, per-connection and per-connector quota leases, and the ability to move a workspace onto a separate egress range are part of the same decision rather than separate features.

How it is realised on AWS

NAT gateways with Elastic IPs per availability zone, published and versioned as documentation. Every outbound request carries a connector-identifying user agent and a correlation identifier a provider can quote in a support ticket. Residency-pinned workspaces egress from their own region's range. Runaway detection throttles an automation within 60 s of exceeding 10x its 7-day baseline, before a provider notices rather than after.

Options weighed

• ChosenA published stable NAT range, with runaway controls as part of the same decision: Enterprise adoption requires it, and the shared-reputation risk it creates has to be mitigated in the same breath
• RejectedEphemeral egress addresses: Operationally simplest and excludes every customer whose security policy requires an allowlist
• RejectedA dedicated egress address per workspace: Removes the shared-reputation risk entirely and does not scale to 2.5 M workspaces at any sane cost
• ChosenDedicated egress for workspaces that require or earn it: Adopted as both a plan feature and an incident response, on the same code path

Consequences

What it buys

• Enterprise customers can allowlist the platform, which is a precondition for a large part of the market
• A provider investigating traffic can identify the caller and quote a correlation id back, turning an opaque block into a conversation
• Residency pinning extends naturally, because the egress identity is already per-region

What it costs

• The range becomes immovable without a notice period, constraining future networking changes
• One workspace's abuse is every workspace's problem, which raises the stakes on runaway detection considerably
• Per-workspace egress as an escape hatch needs to exist before it is needed, not after

Choose differently when

Skip the published range when no customer requires allowlisting and providers do not restrict by address — a consumer-only product can egress from anywhere and avoid the shared-reputation problem entirely.

Why it holds up over time

Network identity as a trust signal has survived every change of protocol and will outlast the current generation of address-based controls, because the underlying need — naming who is calling — does not go away.

Four fifths of the catalogue will not tell us when something changes. How does the platform find out, and how does it notice that it has stopped finding out?

Context

Provider push is cheap, fast and available for roughly a fifth of the catalogue, and it brings subscription lifecycle, signature verification, duplicate deliveries and — most damagingly — silent subscription death, where a subscription expires or is revoked and the platform simply stops receiving events with no error anywhere. Polling works everywhere, costs linearly in connections, and at 2,000,000 connections on an average five-minute interval amounts to roughly 6,700 provider polls a second that somebody has to pay for and that providers have to absorb.

Decision

Both, treated as architecturally different rather than hidden behind one abstraction. Push is preferred wherever the provider supports it, and the platform owns the subscription lifecycle as a component: create, renew before expiry, detect silent death, re-create. Polling adapts its interval per connection to the provider's quota profile, the plan's entitlement and the observed change rate, with declared jitter, and the initial poll establishes a watermark rather than emitting an account's entire history.

How it is realised on AWS

A subscription manager tracks every push subscription's renewal deadline and renews ahead of it; a liveness expectation per connection raises a quiet-period alarm when a source has emitted nothing for longer than its declared quiet period, which is wired off the event log so it watches for the absence of data rather than the health of the poller. Poll workers are Fargate tasks reading cursors from DynamoDB with per-connection jitter, backing off for quiet connections and batching provider queries where the API allows.

Options weighed

• ChosenHybrid, with the subscription lifecycle owned explicitly: The only design that covers the catalogue while giving the fifth that can push the latency it deserves
• RejectedPoll everything, for uniformity: One mechanism and one failure taxonomy, at the cost of latency on the connectors where latency is most visible and a much larger polling bill
• RejectedPush only, and omit connectors that cannot push: Would cut the catalogue — the growth ceiling — by roughly four fifths
• RejectedTreat polling as a fallback inside one trigger abstraction: Hides two genuinely different failure taxonomies behind one interface, which is how a dead subscription gets mistaken for a quiet account

Consequences

What it buys

• Push connectors get seconds of latency and polled connectors get a declared, plan-appropriate interval, instead of both getting the worse of the two
• A subscription that expires is detected and re-created rather than becoming an invisible outage
• Jitter prevents the platform from aligning a million connections on the minute boundary against one provider

What it costs

• Two freshness models, two dedup mechanisms and two failure taxonomies behind one product promise
• Polling infrastructure is assumed at ≤ 25% of total platform compute cost, attributed to the connectors that lack push support
• Subscription lifecycle is ongoing work that scales with the catalogue and with provider API churn

Choose differently when

Poll everything when the catalogue is small and latency expectations are loose; push only when every integration target is modern and under contract. The hybrid is the price of a long tail.

Why it holds up over time

The split between systems that notify and systems that must be asked has outlived several generations of integration technology and shows no sign of closing, because it reflects the providers' priorities rather than a technical limitation.

What is the smallest piece of state in this platform, and what happens when we lose it?

Context

A polled connection's cursor — a watermark, an updated-at bound or a page token — is a few bytes per connection. It is also the only state in the architecture whose loss cannot be recovered from the two authoritative stores: the event log knows what arrived, and the ledger knows what was done, but neither knows what the provider would have returned. Lose a cursor forward and there is a permanent silent gap; lose it backward and there is a flood of duplicates. Worse, the ordinary failure here produces no error at all — a stuck cursor and a healthy quiet account look identical.

Decision

A cursor advances only after the events it covers are durably committed to the event log, never before. The cursor and subscription store is treated as a distinct data zone with its own durability posture rather than grouped with other operational state. Every connection carries a liveness expectation, and an unexpectedly quiet source raises an alarm — the platform monitors the absence of data, not only the health of the components that fetch it.

How it is realised on AWS

DynamoDB, one small item per connection, written after the MSK produce is acknowledged by all in-sync replicas. The quiet-period expectation is derived from the connection's own observed history rather than a global constant, so a genuinely low-traffic connection does not alarm. The alarm is evaluated off the event log, so it fires whether the cause is a dead subscription, a stuck cursor, a paused automation or a provider that has quietly stopped returning results.

Options weighed

• ChosenAdvance after commit, with a per-connection liveness expectation: Guarantees at-least-once at the cost of occasional duplicates, which ADR-05's effect key absorbs
• RejectedAdvance before commit: A one-line difference that produces a permanent, silent, unrecoverable gap — the worst failure mode available to this platform
• DeferredMake the cursor store as durable as the event log: Removes the residual risk and roughly doubles the cost of the hottest small store; left open as the second half of Core Architecture Question 1
• RejectedReconstruct cursors from the event log after a loss: Recovers the last event seen but not the provider's pagination position, so it narrows the gap without closing it

Consequences

What it buys

• A crash between read and commit costs a duplicate, which the effect key makes harmless, rather than a gap, which nothing makes harmless
• A stuck cursor, a dead subscription and a silently broken provider all surface through one alarm that watches for silence
• The riskiest store in the architecture is named as such, which is the precondition for anyone treating it carefully

What it costs

• Duplicates are now routine on the poll path and the dedup window has to be sized for them
• A per-connection liveness baseline is state about state, and it has to be learned rather than configured across two million connections
• The residual risk is real and unresolved: a cursor store failure still produces duplicates or a gap, and the choice between them is not yet made

Choose differently when

Advancing before commit is defensible only when the source can be fully re-read cheaply and idempotently at any time, which makes the cursor an optimisation rather than state.

Why it holds up over time

Checkpoint-after-commit is the same rule as a consumer offset in a log, a replication position, or a file-read watermark. It will outlive this platform because it follows from the ordering of durability, not from any technology.

Our execution plane is compromised. How much of our customers' other SaaS estate goes with it?

Context

The platform holds delegated grants into millions of customers' CRMs, mailboxes, file stores and finance systems. Its own data is not the prize; the access is. The execution plane is simultaneously the largest attack surface in the system — it binds untrusted provider responses, runs partner connector code, and makes outbound calls to arbitrary hosts — and the component that needs credentials most often. Those two facts pull in opposite directions, and resolving them by convenience means a single compromised task role reads every refresh token the platform holds.

Decision

Credential custody runs in a separate cloud account with its own key material, reachable only through a narrow exchange interface. No long-lived credential is ever available to a step worker. A worker receives a short-lived token scoped to one connection and one run, exchanged at the egress boundary (ADR-07); the refresh token never leaves custody. Material is encrypted under a per-workspace key, and plaintext never appears in logs, run history, error messages or the ledger.

How it is realised on AWS

A separate AWS account running the custody service on Fargate, with per-workspace KMS keys and envelope-encrypted ciphertext. Cross-account access is by a narrowly scoped role permitting only the exchange operation, never a read. The connection's state and scopes stay in the platform account so the control plane can query them; only the ciphertext and key reference live in custody. Every exchange is written to an audit record retained seven years independently of the workspace's retention setting.

Options weighed

• ChosenA separate account, with exchange-only access and run-scoped tokens: The blast-radius boundary matches the account boundary, which is the only boundary an attacker cannot talk their way across with a stolen role
• RejectedA managed secret store in the same account, read by runners: Far simpler and makes a compromised runner equivalent to a compromise of every customer's connected products
• RejectedA separate service in the same account: Better than nothing and still inside one IAM blast radius, which is the thing being defended against
• RejectedHold no credentials: ask the author to re-authorise each run: Eliminates the risk and the product, which exists precisely to run unattended

Consequences

What it buys

• A compromised execution plane yields run-scoped tokens for connections currently executing, not the estate
• Per-workspace keys make crypto-deletion of one workspace's credentials a key operation rather than a data sweep
• The audit of credential use is kept outside the workspace's own control, so it cannot be shortened by the party it holds accountable

What it costs

• A cross-account hop on the path of every step that touches a provider, mitigated by per-run caching at the cost of a wider revocation window
• Two accounts to operate, deploy and keep in step, with their own failure and permission-drift modes
• Custody becomes a 99.99% dependency — every step needs it — which is a higher bar than the execution plane it serves

Choose differently when

A same-account secret store is proportionate when credentials reach only systems you already own, so a compromise gains the attacker nothing they did not already have.

Why it holds up over time

Separating custody of a credential from use of a credential is the oldest idea in access control and is independent of the technology implementing either. The boundary will move between accounts, enclaves or HSMs; the separation will not go away.

A thousand runs on one connection hit an expired access token in the same second. What happens?

Context

Without coordination the answer is a thousand simultaneous refresh attempts against the provider. Many providers rotate the refresh token on use and invalidate the previous one, so concurrent refreshes race: one succeeds, the rest present a token that has just been invalidated, and the connection dies — a credential death entirely manufactured by the platform. The converse error is equally damaging: treating a genuine revocation as a transient fault and retrying produces a sustained authentication storm against a provider that has already said no, which is a reliable way to have an application credential suspended.

Decision

Access-token refresh is serialised per connection with single-flight semantics: one refresh proceeds, concurrent callers wait for its result. Credential death — revocation, expiry, password change, scope removal — is classified as non-retryable. Affected runs are parked, the automation is paused after a declared grace period, and the connection's owner is notified. The platform never retries an authentication failure as though it were transient.

How it is realised on AWS

Custody holds a per-connection lock for the duration of a refresh and returns the new token to all waiters. Failure classification distinguishes an expired access token (refresh and continue) from a dead grant (terminal), using the provider's error code as declared in the connector manifest rather than inferring from a status code alone. The terminal state feeds the automation health loop: classify, hold, notify, repair, observe.

Options weighed

• ChosenSingle-flight refresh per connection; credential death terminal: Prevents the platform from causing the failure it is trying to recover from
• RejectedLet each run refresh independently: Causes refresh-token rotation races and converts a routine expiry into a dead connection under load
• ChosenRefresh proactively on a schedule: Adopted alongside, as an optimisation that reduces how often the single-flight path is contended; it does not remove the need for it
• RejectedRetry authentication failures with backoff: Produces an authentication storm against a provider that has already declined, and risks suspension of the application credential

Consequences

What it buys

• A connection under heavy concurrent load refreshes once, not a thousand times
• A revoked credential fails fast and visibly instead of degrading slowly behind retries
• Authentication traffic to providers stays proportional to connections rather than to runs

What it costs

• A per-connection lock in custody, on the path of every step, with its own contention and timeout behaviour
• Waiters block on a refresh they did not initiate, adding tail latency at exactly the moment a connection is busiest
• Distinguishing 'expired' from 'revoked' depends on provider error semantics declared in the manifest, which is another field that can be wrong

Choose differently when

Independent refresh is harmless where providers do not rotate refresh tokens and tolerate concurrent refreshes. The single-flight requirement comes entirely from rotation semantics.

Why it holds up over time

Collapsing concurrent identical work into one operation is a pattern older than the web and applies wherever a shared resource is refreshed under load. Token rotation makes it a correctness requirement rather than an efficiency one, and rotation is becoming more common, not less.

One workspace imports fifty thousand rows. What happens to everybody else's two-step automation?

Context

The workload is extremely uneven: most workspaces run a handful of automations occasionally, and a few run bulk operations that enqueue tens of thousands of runs in a burst. On one shared pool with a single queue, the bulk workspace wins simply by arriving, and every other tenant's latency collapses. Worse, some of those bursts are not legitimate work at all but automations triggering themselves — an automation that writes to the sheet it is watching — which can consume a workspace's quota and a provider's goodwill in minutes.

Decision

Shared capacity with fair scheduling across workspaces and per-workspace concurrency leases, so no workspace holds more than 5% of a shared pool for longer than 60 s while others have queued work. Queue classes are separated by expected latency — interactive, retry, parked-release and bulk — so a backfill never shares a queue with a short automation. Runaway detection throttles an automation within 60 s of exceeding 10x its own 7-day baseline, then pauses it with notice.

How it is realised on AWS

Four SQS queues by latency class, with per-workspace concurrency leases in DynamoDB enforced at admission rather than at the runner, so an over-quota workspace never occupies a worker it will not be allowed to use. Runaway baselines are per automation and learned from its own history, because a rate that is pathological for one automation is normal for another. Dedicated capacity for workspaces that require it runs on the same code path, as a lease pool rather than a separate fleet.

Options weighed

• ChosenShared pool, fair scheduling, separated queue classes, runaway detection: Best utilisation with a bounded tail, and keeps idle cost near zero (ADR-15)
• RejectedOne shared pool, first-come-first-served: Simplest and gives the worst tail exactly when it is most visible
• RejectedPools per plan tier: Simple isolation that strands capacity and still does not protect tenants from each other within a tier
• RejectedDedicated workers per workspace: Ends noisy neighbours and breaks the near-zero idle cost that makes twelve million automations affordable
• ChosenDedicated capacity as a plan feature on the same code path: Adopted for the workspaces that require it, without creating a second execution path to maintain

Consequences

What it buys

• A bulk import is slowed rather than permitted to starve, and the guarantee is structural because the queues are physically separate
• A self-triggering automation is caught in a minute, before it spends a provider's goodwill or the workspace's quota
• Utilisation stays high, which is what keeps per-step cost inside its target

What it costs

• Fair scheduling is real machinery with its own tuning, and its failure mode is a latency regression that is hard to attribute
• Per-automation baselines are state that must be learned across twelve million automations and will misfire on genuinely spiky but legitimate work
• If plan-differentiated latency is ever sold, fairness and the commercial promise are in direct conflict and the architecture has to encode which wins

Choose differently when

First-come-first-served is adequate when the workload is homogeneous and no tenant can enqueue orders of magnitude more than another. Dedicated pools win when tenant isolation is a compliance requirement rather than a performance one.

Why it holds up over time

Fair queueing across tenants on shared infrastructure is as old as time-sharing and has survived every change in what the shared resource is. The specific scheduler will change; the need to stop one tenant monopolising a shared pool will not.

Twelve million automations exist. Most of them will not run today. What do they cost?

Context

The business is the long tail: the median workspace has a few automations that fire occasionally, and a large fraction of the twelve million are effectively dormant. If a dormant automation costs anything meaningful — a reserved worker, an open connection, a provisioned queue, a polling slot it does not need — the economics fail at a scale the product must reach to be viable. This is a genuine architectural constraint rather than a cost optimisation, because it rules out whole classes of design at the point where they are chosen.

Decision

An enabled but dormant automation holds no reserved compute, no open connection and no worker slot, at an assumed cost of ≤ $0.004 per automation per month. This rules out per-automation reserved capacity, dedicated workers by default, per-automation long-lived connections and any design where an automation's existence rather than its activity drives cost. Poll cost must be sub-linear in connection count, achieved through interval adaptation, change-rate backoff for quiet connections and batched provider queries.

How it is realised on AWS

Fargate tasks autoscaled on queue depth and age, with nothing allocated per automation. Automations exist as rows in the definition registry and subscriptions or cursors in a small key-value store — bytes, not capacity. Poll scheduling adapts per connection, so a connection that has not changed in a month is polled at its plan's floor rather than its ceiling.

Options weighed

• ChosenServerless workers, no per-automation allocation, adaptive polling: The only shape where twelve million mostly-dormant automations are economically viable
• RejectedA reserved worker or process per active automation: Simplest mental model and fails on cost by orders of magnitude at this scale
• RejectedLong-running connections per automation: Would reduce latency and makes dormant automations expensive in exactly the way the economics cannot absorb
• DeferredArchive dormant automations to cold storage: Worth doing if registry cost becomes material; it is not, because a dormant automation is already only bytes

Consequences

What it buys

• The long tail is viable, which is the business rather than a technical nicety
• Capacity tracks actual work, so a quiet weekend costs nothing and a Monday burst scales up
• No per-automation resource means no per-automation leak, exhaustion or cleanup path

What it costs

• Cold-start latency on scale-up, paid by whichever runs arrive first after a quiet period
• A per-task premium over long-running instances, accepted deliberately as the price of the constraint
• Adaptive polling adds state and complexity to the one tier that would otherwise be simple

Choose differently when

Per-tenant allocation is fine when tenants number in the thousands and each is commercially significant. The constraint comes from the tail's length, not from any dislike of reserved capacity.

Why it holds up over time

Making cost proportional to work rather than to configured entities is the economic principle under every generation of elastic infrastructure, and it will outlast the particular runtime that delivers it.

A non-engineer's automation failed. What do they see, and what can they do about it?

Context

The product's premise is that someone who cannot read a stack trace can build and own this. That premise turns failure presentation into an architectural concern rather than a UI one: if the platform's internal states cannot be reduced to a sentence an author can act on, then either the states are wrong or the product is undeliverable. The default alternative — forwarding the provider's error — is worse than useless, because provider errors are written for the developer who called the API, and often describe conditions the author has no way to influence.

Decision

Every failure is classified into a small, stable, author-facing taxonomy, each class carrying a named next action. An unclassified error is never presented as the primary message, and an unclassified share above 2% of author-facing failures is treated as a defect in the taxonomy. A repair inbox groups parked and failed runs by cause with bulk retry, replay, discard and reconnect actions. The classification exists in the execution path, not in the presentation layer.

How it is realised on AWS

The runner writes a typed cause onto the ledger entry at the moment of failure, where the provider response and the step context are both still available; the run index projects it, and the repair inbox groups on it. Provider-specific error codes are mapped to platform classes in the connector manifest, so the mapping is versioned with the connector that knows about it. Redaction of credential material and author-declared sensitive fields happens before anything reaches run history.

Options weighed

• ChosenA typed taxonomy written at failure time, with a named action per class: The only approach where the classification has access to the context needed to be accurate
• RejectedForward the provider's error message: Cheapest and describes conditions in a vocabulary the author cannot act on, often about systems they do not administer
• RejectedClassify in the presentation layer from stored error text: Brittle string matching, far from the context, and silently wrong whenever a provider rewords a message
• RejectedRoute every failure to human support: Does not scale past a tiny fraction of 1.8 billion monthly step attempts, and is slower than the author fixing it themselves

Consequences

What it buys

• An author can usually repair their own automation without support, which is the only workable path at this volume
• The unclassified rate becomes a measurable quality signal for the taxonomy itself, so gaps surface rather than accumulate
• Grouping by cause makes bulk repair possible: one reconnection fixes hundreds of parked runs at once

What it costs

• A mapping from provider errors to platform classes, per connector, which is ongoing work and another field that can be wrong
• The taxonomy must stay small to be legible and complete enough to cover reality, and those pull against each other continuously
• Redaction before storage means the raw provider response is not always available for operator debugging, which has its own cost

Choose differently when

Forwarding raw errors is correct when the users are developers who called the API themselves and have the context to interpret them. The taxonomy's cost is justified only by the non-engineer premise.

Why it holds up over time

The requirement that a system explain its failures in terms of the actions available to the person reading will outlast every interface technology, because it follows from who the user is rather than from how the message is rendered.