Architecture Views

When a read path wants to know whether someone may have an object, what does it ask?

Context

The cheapest thing to build is a container of files and a column on a row saying whether each one passed its scan. It works until the second read path is written. From then on, correctness depends on every caller — the download API, the preview generator, the mobile sync endpoint, the admin export tool, the migration script somebody writes at 2 a.m. — remembering to check the column, interpreting a null the same way, and re-checking it rather than caching the answer. Each of those is a place where a malicious object becomes reachable, and none of them is visible in a diagram. The question is not whether the check exists but whether it can be omitted, and in a flag-based design it can always be omitted.

Decision

A single lifecycle state machine is the only authority on access. A read path never inspects storage, never interprets a verdict, and never caches an access decision: it asks the state machine, which answers from the object's current state and the authorisation of the requesting principal at that moment. The state machine is also the only component that may move an object between storage planes, and every transition it makes is recorded with actor, cause and timestamp.

How it is realised on AWS

Object state lives in Cosmos DB with strong consistency inside the region. The download API resolves state and authorisation on every credential request — there is no cached grant and no long-lived link. The untrusted storage account has no identity entitled to issue a read credential for it, so the absence of access is enforced by the storage account's own access model and not only by the API's logic. Transitions append to an immutable blob log with legal hold.

Options weighed

• ChosenLifecycle state machine as the sole gatekeeper: One place to be right, one place to audit, and a read path that cannot accidentally skip the check.
• Rejectedis_clean flag on the object row, checked by each read path: Correct on the day it is written; one new read path away from wrong, with no structural signal that it has gone wrong.
• RejectedAccess decided by storage-layer ACLs alone, with no state model: Cannot express scanning, quarantined or revoked — and cannot explain afterwards why access was refused.
• RejectedSigned capability tokens issued at upload, valid until expiry: Fast and stateless, and unable to honour a revocation: the capability outlives the judgement that justified it.

Consequences

What it buys

• "Is this object reachable?" has one answer in one place, which a reviewer can verify without reading every caller.
• Revocation becomes expressible: changing state is sufficient to stop new access, with no cache to invalidate in the authorisation path.
• Adding a read path costs nothing in security review, because it has no choice but to ask.

What it costs

• Every download request pays a strongly-consistent metadata read, which is why credential issuance has an 80 ms p99 budget rather than being free.
• The state store is on the critical path for reads, so its availability target is the platform's download availability target.
• A client cannot be handed a durable link, which is a real product constraint on anything that wants to embed an object URL.

Choose differently when

If every read path were owned by one team in one codebase, permanently, and the product never needed to revoke access to an object already declared clean, a flag would be adequate and cheaper. The decision should also be revisited if the state store's availability ever becomes the dominant cause of download failures, in which case the answer is a read-through cache with a lifetime shorter than the revocation target — not abandoning the gatekeeper.

Why it holds up over time

This is a statement about which component is entitled to answer a question, not about a product. It survives replacing the metadata store, the credential format and the object store, because none of those changes who decides.

Where does an object live while nobody is allowed to read it, and what does becoming readable cost?

Context

Three arrangements are available. Keep everything in one container and distinguish by a flag or a metadata field: free promotion, instant availability, and an isolation claim that rests entirely on access control being correct everywhere. Keep two containers in one account: slightly better, still one identity boundary. Or keep two accounts with separate identity and network controls, and copy on promotion: an isolation claim a reviewer can verify by reading the access model rather than the code — and a copy whose cost at p99 object size is not small. At a mean of 2.4 MB the copy is noise. At the p99 of 240 MB it is visible. At 50 GB it is a minutes-long operation with its own failure mode sitting between the verdict and availability.

Decision

The untrusted plane and the serving plane are separate storage accounts with distinct managed identities, distinct network controls and no identity entitled to read the untrusted plane except the scan workers. An object becomes available by being copied into the serving plane under the state machine's direction, and the copy is treated as a real step with a latency budget, a failure path and a cost line — not as an implementation detail.

How it is realised on AWS

Two Azure Storage accounts behind private endpoints. The untrusted account permits write via user-delegation SAS scoped to one blob path, and read only to the scan plane's workload identity. The serving account is the only one Front Door is permitted to reach. Promotion is a server-side copy initiated by the control plane, with the object entering an explicit promoting sub-state so a failure mid-copy is visible to the reconciler rather than leaving an object that is clean and unreachable.

Options weighed

• ChosenTwo accounts, copy on promotion: Isolation is structural and auditable; the price is a copy that grows with object size and a new state to reconcile.
• RejectedOne container, state flag only: Free and instant promotion, and an isolation story that is only as good as every access-control rule in the estate.
• RejectedTwo containers in one account: Cheaper than two accounts and shares one identity and network boundary, which is the boundary that matters.
• rejected for nowIn-place promotion by rewriting access policy: Instant and free at any size; the fallback if the prototype shows promotion cost at 20 GB is prohibitive, accepted with the weaker claim stated.

Consequences

What it buys

• The claim "an unscanned object cannot be read" is enforced by the storage account's access model, so it survives a bug in the API layer.
• A compromised serving-plane credential yields no access to unscanned or quarantined objects at all.
• Quarantine and evidence live naturally in the untrusted account, with break-glass as a separate identity rather than a separate code path.

What it costs

• Promotion is priced per byte and takes time proportional to object size, which lands directly in the journey of the 40 GB upload.
• A second account doubles the lifecycle, key and network configuration surface.
• There is a window between verdict and availability in which an object is clean and unreachable — a state the reconciler must own.

Choose differently when

If the measured promotion cost at p99 and maximum object size exceeds what the product will pay in latency or money, switch to in-place promotion with an access-policy rewrite — and say plainly that isolation is then configured rather than structural, instead of continuing to make the stronger claim.

Why it holds up over time

The claim is that an isolation boundary a reviewer can verify is worth paying for. That outlives the specific cloud's notion of an account, though the price of the copy is a current-economics number and should be re-measured.

Do the bytes traverse platform-owned compute, or does the platform only hand out the right to write them?

Context

A streaming gateway is attractive because it can act on bytes as they arrive: enforce the real size rather than the declared one, determine the true content type before anything lands, reject mid-stream, and hide the storage topology entirely. It is also compute that must scale with aggregate ingress — 45 GB/s at the assumed peak — and that must itself implement resumability, parallel chunking and retry semantics that the object store already has. Direct-to-store removes that component, inherits block-level resumability, and halves the byte movement, at the price of admitting bytes before anything has looked at them and of expressing all enforcement in the store's own credentialing model.

Decision

Clients write directly to the object store using a short-lived credential scoped to exactly one object path and one verb. No platform component sits in the byte path on upload or on download. Everything the gateway would have enforced is moved either earlier — to admission, before a credential is minted — or later, to finalisation and scanning, where the bytes are already durable and can be inspected without holding a connection open.

How it is realised on AWS

User-delegation SAS derived from the platform's managed identity, write-only, single blob path, 60 minutes, renewable within an open session. Clients use block-blob uploads of 4 MB to 256 MB with up to 16 parallel connections. The declared length and type are checked at initiation; the real length is checked at finalisation against the committed block list, and the real type is determined by inspecting the stored bytes.

Options weighed

• ChosenDirect-to-store on a minted, path-scoped credential: No compute scaled to bytes, resumability for free, and enforcement relocated to admission and finalisation.
• RejectedStreaming gateway for all uploads: Can reject mid-stream and hide the topology; costs a tier sized to 45 GB/s and a reimplementation of resumable transfer.
• RejectedHybrid: gateway for small, direct for large: Defensible, and doubles the number of upload paths to secure, test and reason about for a benefit that admission already delivers.
• RejectedLong-lived storage key handed to trusted product backends: Simplest integration; a single leak exposes every object in the account, and nothing about it is revocable per object.

Consequences

What it buys

• The data path costs storage and network, not compute, which is where the pipeline's cost model earns most of its margin.
• Resumability, parallelism and retry are the store's problem and are already correct.
• A compromised platform control plane cannot read bytes it was never in the path of.

What it costs

• Bytes land before anything has inspected them, which is precisely why the untrusted plane and the state machine must exist.
• All credential scoping must be expressible in the store's model; anything it cannot express, the platform cannot enforce at write time.
• A client that ignores the declared length can waste storage up to the ceiling before finalisation refuses it — garbage-collected, and not free.

Choose differently when

If a policy ever genuinely requires acting on bytes before they are durable — a regulatory prohibition on storing a class of content even transiently, for example — a gateway becomes necessary for that scope, and the decision should be revisited for that scope alone rather than for the platform.

Why it holds up over time

Compute in a byte path costs proportionally to bytes, and objects are getting larger rather than smaller. The economics behind this decision strengthen with time.

When the scan tier is down, does the upload fail?

Context

If scanning is synchronous with upload, the pipeline's availability is the product of every engine's availability and the signature feed's. A vendor pushes a bad engine build and uploads stop estate-wide; a licence server has a bad afternoon and twelve product teams file incidents. Worse, the scan tier is not merely occasionally unavailable — it is permanently slower than the ingest tier, because inspecting a byte costs more than storing one. A design that couples them is therefore not just fragile in incidents, it is wrong at steady state: the uploader waits on the slowest component in the system by construction.

Decision

An object is acceptable, transferable and durably stored while the scan tier is wholly unavailable. It simply remains unreachable. Ingest and scanning have separate availability targets, separate scaling signals and separate failure handling, joined only by a durable queue. A scanner outage is a time-to-verdict incident, never an upload incident.

How it is realised on AWS

Finalisation publishes to Event Grid and returns 202 with state scanning; the Service Bus lanes hold work until workers exist. Ingest targets ≥ 99.99% monthly; the scan control plane ≥ 99.95%. The scan tier scales on oldest-unscanned age per lane, so a recovering scanner drains the backlog without any coordination with the ingest path.

Options weighed

• ChosenAsynchronous scanning behind a durable queue: Uploads survive any scanner failure; the cost is that an object exists in a state nobody may read.
• RejectedSynchronous scan inside the upload request: No quarantine, no state machine, no revocation — and upload latency equal to scan time at every object size.
• RejectedSynchronous for small objects, asynchronous above a threshold: Two correctness models for one platform, and the small-object path still fails when the engine does.
• RejectedAsynchronous, but reject new uploads while the scanner is down: Protects time-to-verdict by sacrificing the one promise the platform can always keep.

Consequences

What it buys

• A vendor engine outage degrades time-to-verdict and nothing else; members can still upload and nothing unsafe becomes readable.
• A 4× burst is a capacity question rather than a correctness one, because the accept path owes the caller only durability.
• The two tiers can be scaled, priced and operated on their own signals.

What it costs

• Objects exist that are durable and unreadable, which is a product state the UI must express honestly rather than as a spinner.
• A backlog is now possible, which makes oldest-unscanned age the platform's most important operational number.
• The queue becomes a correctness dependency: at-least-once delivery with idempotent verdict writes is mandatory, not a nicety.

Choose differently when

For a scope where no object may ever exist unscanned — some regulated intake flows genuinely require this — the right answer is to refuse the upload at initiation when the scanner is unavailable, not to make scanning synchronous. That is a per-scope policy, and it is still this architecture.

Why it holds up over time

Inspection will always cost more than storage. The asymmetry that motivates this decision is physical rather than technological.

What exactly is recorded when a scan finishes, and what happens when tomorrow's knowledge contradicts it?

Context

The natural model is a boolean on the object: scanned, clean, done. It cannot answer the two questions that are actually asked. An auditor asks which engine and which signature version declared a specific file clean on a specific date. An incident responder asks which files were declared clean by the signature set that turned out to be missing a family, and needs them re-judged. A boolean answers neither, and worse, it cannot be corrected without destroying the evidence that the earlier judgement was made — so "we were wrong yesterday" becomes an update that erases its own history.

Decision

A verdict is a row keyed by content hash and engine-version tuple, carrying the engine identity and build, the signature-set version and its publication time, the determined content type, the outcome, any detection identifier, the bound hit if one was, the scan duration, the decision time and a revocation field. Re-scanning under a new signature set writes a new verdict rather than overwriting the old one. An object's state points at its current verdict, and revoking that verdict moves the object out of the serving plane.

How it is realised on AWS

Verdicts in Cosmos DB, partitioned by content hash, immutable once written except for the revocation field. Engine and signature versions are recorded from the worker's own runtime rather than from configuration, so provenance reflects what actually ran. Objects clean within the last 30 days are re-judged within 24 hours of a material signature change; older objects on a rolling 180-day sweep. Revocation demotes the object within 60 seconds and lets in-flight read credentials expire within 5 minutes.

Options weighed

• ChosenVersioned, revocable verdict rows keyed by content and engine version: Answers the auditor and the responder, and makes being wrong a recordable event rather than a destructive edit.
• RejectedBoolean is_clean on the object: Cheapest; cannot express provenance, cannot be revoked without erasing history, cannot drive a re-scan.
• RejectedVerdict as a log entry only, with no current pointer: Perfect history and a read path that must reduce a log to a decision on every request.
• RejectedVerdict with a fixed expiry, requiring periodic re-scan of everything: Honest about staleness and prices a full-corpus re-read on a timer rather than on evidence.

Consequences

What it buys

• The platform can state, for any object and any date, which engine and signature version cleared it.
• New intelligence is actionable: the set of objects to re-judge is a query, not an archaeology project.
• Dedup reuse becomes safe to reason about, because the thing being reused names the exact conditions under which it was decided.

What it costs

• Verdict storage grows with re-scans as well as uploads, retained seven years.
• The read path must resolve current-verdict-not-revoked rather than reading one field.
• Re-scan is a real recurring cost in cold-storage reads and scan compute, and has to be funded rather than assumed.

Choose differently when

If detection ever became genuinely stable — signature sets no longer materially changing — the revocation machinery would be dead weight. Nothing suggests that, and a design that assumes it fails silently rather than loudly.

Why it holds up over time

Threat intelligence will always arrive after some uploads. A design in which clean is revocable remains correct however good detection becomes, because the arrival order of knowledge does not improve.

What is the verdict when the scanner runs out of time, memory, or patience with a nested archive?

Context

Every scan has limits, and a hostile object is designed to find them. A zip nested forty deep, an archive that expands a thousandfold, a container file whose index crashes a parser, a file crafted to take an engine ninety seconds instead of ninety milliseconds. The implementation that treats a timeout as a pass is both the most common and the most dangerous: it converts an attacker's ability to make scanning expensive into an ability to bypass scanning entirely. The implementation that treats a timeout as infected is safer and unusable, because legitimate large archives hit the same bound and the false-positive rate destroys the product.

Decision

Every scan attempt is bounded in wall-clock time, memory, archive recursion depth and total expanded size. Breaching any bound produces a typed indeterminate verdict naming the bound that was hit. Indeterminate is a first-class outcome with its own state, its own handling path and its own budget: it is never readable as clean, and every indeterminate object reaches a human or an explicit policy decision within 24 hours.

How it is realised on AWS

Depth 12, 200× expansion ratio, per-class wall-clock and memory ceilings enforced by the Container Apps job's own limits, and retry at most twice in a fresh sandbox before dead-lettering. The bound hit is recorded on the verdict row. The tenant's quarantine console shows indeterminate objects distinctly from infected ones, and the release path for indeterminate differs from the one for infected — which cannot be released through the ordinary API at all.

Options weighed

• ChosenTyped indeterminate verdict naming the bound: Safe and arguable: the admin can see why, and the platform has not pretended to an opinion it does not hold.
• RejectedTimeout treated as clean: Turns making scanning expensive into bypassing scanning, which is the cheapest attack on the whole pipeline.
• RejectedTimeout treated as infected: Safe and unusable: legitimate large archives hit the same bound and the block is unarguable.
• RejectedUnbounded scanning with no limits: A single crafted object consumes a lane indefinitely, which is a denial of service with a progress bar.

Consequences

What it buys

• A decompression bomb is a policy outcome with a named cause rather than a crashed worker and a blocked lane.
• The admin reviewing a block can distinguish "we found something" from "we could not finish looking", which is what makes an appeal possible.
• The indeterminate rate becomes a measurable quality signal for both the engines and the bounds.

What it costs

• A third outcome means a third path in the product, the console and the policy model.
• Legitimate deep archives are inconvenienced and need a human within 24 hours — a real operational load at 0.2% of 60 million objects a day.
• The bounds themselves are now a tuning surface with security consequences in both directions.

Choose differently when

If an engine family emerges that is provably bounded in time and memory for arbitrary input, the time and memory bounds become unnecessary — though depth and expansion bounds would remain, because those are properties of the content rather than the engine.

Why it holds up over time

Content formats keep gaining nesting, compression and indirection. A design that degrades to indeterminate rather than to clean gets safer as content gets more hostile.

What is assumed about a process whose entire job is to open files that may be designed to attack it?

Context

Scan engines are large C and C++ parsers for hundreds of formats, and parsers of hostile input are where remote code execution lives. Treating the worker as ordinary application compute — in the platform's network, with the platform's identity, in a long-lived pool with a shared filesystem — means a single engine vulnerability yields an attacker a foothold inside the system that decides what is safe. The worker is the one component in this architecture that should be designed on the assumption that it will be compromised, because its input is chosen by the adversary.

Decision

Scan workers run in their own subscription, in ephemeral jobs destroyed after one object whatever the outcome, with no inbound network reachability, an egress allow-list of exactly two destinations, no shared mutable filesystem between objects, and a workload identity that can read the untrusted plane and write verdicts and nothing else. In particular, no credential held by a worker can change an object's lifecycle state.

How it is realised on AWS

A separate Azure subscription with its own network boundary and private endpoints. Container Apps jobs, one object per job instance, no ingress. Egress restricted to the signature feed and the verdict sink. The worker's identity has read on the untrusted storage account and write on the verdict collection; it has no permission on the serving account, the object metadata store or the transition log.

Options weighed

• ChosenEphemeral per-object sandbox in an isolated subscription: A compromise sees one object, one tenant, two egress destinations, and cannot declare anything clean.
• RejectedWarm worker pool with per-object process isolation: Cheaper per scan and shares a kernel and a filesystem across tenants' objects.
• RejectedScanning inside the control plane: Simplest to build and places hostile-input parsing next to the component that grants access.
• RejectedThird-party scanning SaaS, objects shipped out: No sandbox to operate and breaks residency, since the bytes leave the tenant's declared geography.

Consequences

What it buys

• An engine zero-day is bounded to one object and one tenant, with no path to change a verdict or reach the serving plane.
• Seeing an object and deciding its fate are held by different identities, which is the assurance claim the security views rest on.
• Residency is preserved, because bytes never leave the boundary for scanning.

What it costs

• Per-object job start-up is real latency and real money against a warm pool, paid 60 million times a day.
• An extra subscription to govern, with its own identity, network and cost boundary.
• Engine images must be pulled and cached per boundary rather than fetched on demand.

Choose differently when

If job start-up latency ever threatens the interactive lane's p50, the right move is a warm pool per tenant with per-object process and filesystem isolation — accepting a weaker claim and saying so — not a shared pool across tenants.

Why it holds up over time

Parsers of adversarial input will remain the highest-risk code in any content platform. Designing their host as expendable is not a technology choice.

When the scan tier cannot keep up, which promise breaks first — and who decided?

Context

The scan tier is the bottleneck by construction, so saturation is a certainty rather than an incident. Four responses are available and each is a different product. Reject new uploads: protects time-to-verdict, visibly fails the user. Widen the verdict target: accepts everything and lets attachments sit in scanning with no published bound. Admit optimistically under load: trades security posture for throughput at exactly the moment an attacker would choose. Degrade to signature-only: keeps latency, lowers detection. A system that has not chosen in advance will choose under pressure, at 3 a.m., by whichever path happens to be least resistant — and that path is almost always the unbounded queue.

Decision

Three priority lanes — interactive, background and bulk — each with per-tenant admission limits, and a published shedding order: bulk initiations are refused first, then background work is throttled and its published target widened, and only then are interactive initiations rejected with a typed retry hint. Rejection always happens at initiation, where it costs the client nothing. The one promise never traded is that nothing reaches a reader without a current verdict.

How it is realised on AWS

Three Service Bus queues with per-tenant admission counters. The scan tier scales on oldest-unscanned age per lane, and the same signal drives the shedding bands: under 1× target is normal, 1–2× elevated, 2–5× saturated, above 5× critical. A widened target is published in the API response rather than implied. Interactive capacity is reserved; bulk capacity is interruptible.

Options weighed

• ChosenPublished lane-by-lane shedding order, rejection at initiation: The decision is made in daylight by the people who own the product promise, and the client is told the truth.
• RejectedUnbounded queue, widen targets silently: Nothing fails and nothing works; the backlog becomes the product and no number is true.
• RejectedAdmit optimistically under load: Trades the safety invariant precisely when load may itself be the attack.
• rejected as an automatic responseDegrade to signature-only scanning under load: Keeps latency and lowers detection without telling anyone; acceptable only as a declared per-tenant posture, never as an automatic fallback.

Consequences

What it buys

• A weekend migration cannot starve a Monday morning attachment, because the lanes have separate admission.
• The failure mode is a typed rejection with a retry hint rather than an object stuck in scanning indefinitely.
• Capacity planning has one number — oldest-unscanned age per lane — that is both the scaling trigger and the alarm.

What it costs

• Three lanes multiply the admission, quota and monitoring surface by three.
• A refused bulk initiation is visible unhappiness for a tenant mid-migration, which has to be a conversation rather than a surprise.
• Reserved interactive capacity is paid for whether or not it is used.

Choose differently when

If scan capacity ever became elastic enough that oldest-unscanned age never left the normal band at 4× burst, the lanes would still earn their place for cost reasons — bulk on interruptible capacity — but the shedding order would become theoretical. It should still be published.

Why it holds up over time

Any pipeline whose inspection stage is more expensive than its acceptance stage will face this choice. Naming the order in advance is a governance act, not a technical one, and it does not age.

If two tenants upload byte-identical files, may the second reuse the first's verdict?

Context

35% of objects in the assumed workload are duplicates: the same policy PDF, the same installer, the same meme, the same quarterly template. Platform-wide deduplication is therefore the single largest cost lever in the design — it removes a third of all scan compute for the price of an index lookup. It also turns the dedup index into an oracle. An attacker who can upload and observe behaviour can learn whether a specific file already exists in the platform, which across a tenant boundary means learning something about another organisation's contents: whether they hold a particular leaked document, a particular contract, a particular build. Encryption does not help; the hash is the disclosure.

Decision

Verdict reuse is a per-tenant policy with a declared isolation level: within a destination scope, within a tenant, or platform-wide. The default is within-tenant. Platform-wide reuse is available only as an explicit, knowing opt-in, and in every case reuse is applied on the platform's own ingest path and never made observable in a response — a caller cannot distinguish a reused verdict from a fresh scan by timing or by content of the reply.

How it is realised on AWS

The dedup index in Cosmos DB is partitioned by the tenant's declared isolation level, so a cross-boundary lookup is not merely forbidden but unroutable. Every reuse is recorded against the originating verdict so the audit trail shows how a decision propagated. Where reuse applies, the object still passes through the lane and still reports a time-to-verdict consistent with its class rather than returning instantly.

Options weighed

• ChosenDeclared isolation level, within-tenant default, reuse not observable: Keeps most of the cost benefit inside a tenant and closes the cross-tenant oracle by construction.
• RejectedPlatform-wide dedup by default: Maximum saving and a cross-tenant existence oracle that no later control removes.
• RejectedNo reuse at all; scan every object every time: Simplest to reason about and spends a third of the platform's scan budget re-deriving answers it already has.
• RejectedPlatform-wide reuse with rate limiting on the probe path: Raises the cost of the oracle without removing it, and makes the security argument a quantitative one nobody can audit.

Consequences

What it buys

• The expected ≥ 25% scan-compute saving is realised inside the trust boundary where it is free of disclosure risk.
• The side channel is closed structurally — a cross-boundary lookup cannot be issued, not merely should not be.
• Reuse is auditable: every reused verdict names the scan it came from.

What it costs

• A large multi-tenant platform re-scans the same popular installer once per tenant.
• Isolation level becomes another policy dimension a tenant has to understand, and a default that may be questioned at renewal.
• Hiding reuse means giving up the obvious latency win of answering instantly on a hash hit.

Choose differently when

A tenant may knowingly opt into platform-wide reuse for a scope where existence disclosure is not meaningful — public marketing assets, for instance. The default should change only if the oracle can be removed entirely rather than priced.

Why it holds up over time

Content-addressed storage will always create this tension between cost and existence disclosure. The resolution — declare the boundary, default to the narrow one, keep reuse unobservable — outlives any particular index technology.

Does a 50 KB screenshot go through the same pipeline as a 50 GB archive?

Context

They share an API and nothing else. The screenshot needs no assembly, no unpack stage, a few hundred milliseconds of engine time and a worker with a small memory footprint; a human is waiting for it. The archive needs assembly to scratch storage, a bounded container walk, tens of gigabytes of working space, minutes of engine time, and nobody is watching the screen. A single execution profile sized for the archive makes every screenshot expensive and wastes most of the platform's compute; one sized for the screenshot cannot scan the archive at all; one sized in the middle fails at both ends. The same applies to latency targets: a single published number is either a lie about large objects or a betrayal of small ones.

Decision

Three execution profiles by object size — small, large, and archive or container — each with its own memory and time ceilings, its own default lane, and its own published time-to-verdict target. The dispatcher routes on the finalised object's actual size and determined type rather than on the client's claim. Targets are published per size band, never as one platform number.

How it is realised on AWS

Container Apps job definitions per profile with distinct memory and CPU allocations. Small objects stream from storage and skip assembly entirely. Large and archive objects assemble to ephemeral scratch and are walked under the depth-12 and 200× expansion bounds. Published targets: p95 8 s under 10 MB, 180 s under 1 GB, 20 min under 50 GB.

Options weighed

• ChosenThree size-based execution profiles with per-band published targets: Median cost is set by the small profile, and no published number is a lie about either end of the range.
• RejectedOne profile sized for the largest object: Simplest to operate and spends worst-case resources on the 99% of objects that are small.
• RejectedOne profile with a size ceiling, large objects refused: Cheap and removes a product capability the heavy-media journey depends on.
• RejectedContinuous autoscaling of worker size per object: Theoretically optimal and adds a scheduling problem and an unbounded failure surface for a marginal gain over three buckets.

Consequences

What it buys

• The platform's unit cost is dominated by the cheap profile, which is what makes ≤ $0.85 per 1,000 objects plausible.
• Large objects get the resources they genuinely need without the small path paying for them.
• Published targets per band mean the interactive journey and the 40 GB journey can both be honest.

What it costs

• Three job definitions, three capacity pools and three sets of targets to operate and explain.
• A boundary case near a profile threshold can land in the wrong class and miss its target, which has to be tolerated rather than tuned away.
• Scratch storage for assembly is a real cost and a real cleanup obligation on the large and archive profiles.

Choose differently when

If the object size distribution ever lost its long tail — a product where nothing exceeds a few megabytes — a single profile would be correct and the dispatcher would be overhead. The distribution assumed here, with a mean of 2.4 MB and a maximum of 50 GB, is what makes three classes right.

Why it holds up over time

The gap between the median and the maximum object in a file platform has widened every decade. Treating size as a class rather than a parameter gets more right over time, not less.

How does the platform know a resumable upload of 10,000 blocks is complete and uncorrupted?

Context

With bytes written directly to the store by the client, the platform does not observe the transfer. It learns about it only from the client's claim that the transfer is finished. A design that treats that claim as true will, eventually and quietly, assemble and scan a truncated object: a sync client with a bug, a mobile client killed by the OS mid-upload, a flaky proxy that silently dropped a block, or a deliberately malformed finalise that commits a subset of blocks chosen so the malicious portion is absent from the scan and present in the delivered file.

Decision

Finalisation is an explicit call in which the client asserts the complete block set and the whole-object hash. The platform verifies the committed block list against the asserted set and the computed whole-object hash against the asserted hash, and refuses to finalise on any mismatch rather than assembling what it has. Per-chunk checksums are verified on receipt; an object that fails either check is a failed upload and never enters the scan queue.

How it is realised on AWS

Block-blob commit with an explicit block list, per-block checksum verification at write time, and a whole-object hash computed from the committed object after assembly rather than taken on trust from the client. Mismatches return a typed error naming which assertion failed, and the session remains resumable so a client can repair rather than restart.

Options weighed

• ChosenExplicit finalise asserting block set and whole-object hash, refusable: Nothing unverified enters the scan queue, and a buggy client gets a specific error instead of a corrupt file.
• RejectedImplicit finalisation on a timeout after the last block: No client change needed and silently assembles whatever arrived before the timer.
• RejectedTrust the client's declared length only: Catches truncation but not reordering, omission or substitution of blocks.
• RejectedVerify by re-reading and hashing asynchronously after finalisation: Cheaper on the critical path and allows a window in which an unverified object is already queued.

Consequences

What it buys

• The scan queue contains only objects whose bytes are known to be the bytes the client meant to send.
• Deliberate partial commits as a scan-evasion technique are closed off at the only point where they could be attempted.
• A client bug produces a named failure rather than a corrupt file discovered by a user months later.

What it costs

• Finalisation costs a whole-object hash computation, which at 50 GB is neither instant nor free.
• The client must track and assert its block set, which is a real client-side obligation and a source of integration friction.
• A mismatch strands storage until the session is repaired or garbage-collected.

Choose differently when

If the object store ever offered a verifiable whole-object digest computed at write time with the same guarantees, the platform could verify that instead of recomputing, which would remove the finalisation cost without weakening the check.

Why it holds up over time

Whenever a client writes to storage a platform does not observe, the completeness claim has to be verified rather than believed. That is a structural property of the direct-to-store decision.

What happens to the object that is in scanning and has been for nine hours?

Context

Every stage in this pipeline can lose an object without anything failing. A worker dies after reading bytes and before writing a verdict. A finalise event is published and the subscription is not yet healthy. A promotion copy fails halfway. An engine returns nothing and the lease simply expires repeatedly. In each case no error is raised anywhere, there is no failed request to retry, and the object sits in a non-terminal state looking exactly like an object that is merely busy. It is simultaneously a support ticket — the member sees scanning forever — and a security hole, because an object nobody is tracking is an object nobody is judging.

Decision

Every object has an expected dwell time for its state and size class. A reconciler sweeps for objects past that dwell time and re-drives, dead-letters or reports each one. Silence about an object is a defect by definition: an object either reaches a terminal state or appears, named, on a reconciliation report. Oldest dwell time past expectation is one of the four alarms that wake a human.

How it is realised on AWS

A scheduled Container Apps job reads object metadata for non-terminal states past their class dwell time, re-publishes the finalise event or re-enqueues the scan where that is safe and idempotent, and dead-letters with an indeterminate verdict where it is not. Queue delivery is at-least-once with idempotent verdict writes, so a re-drive cannot double-count. The reconciler's own output is a first-class operational signal, not a log line.

Options weighed

• ChosenReconciler as a first-class component with per-state dwell times: The only design in which losing an object is detectable rather than merely unlikely.
• RejectedRely on at-least-once delivery and queue redelivery: Covers the worker-death case and not the lost event, the failed promotion or the engine that returns nothing.
• RejectedClient-side polling and retry: Puts recovery in the hands of whichever client happens to still care, and leaves abandoned objects invisible.
• RejectedA periodic report with no automated re-drive: Honest and converts every lost object into manual work at a volume no team can absorb.

Consequences

What it buys

• An object stuck between the planes is found by the platform rather than by a user, which is the difference between an operational signal and a support escalation.
• The promotion window — clean but not yet available — has an owner.
• The safety invariant becomes checkable: the reconciler is the thing that can assert no object is unaccounted for.

What it costs

• A full sweep over 12 PB of object metadata is itself a cost and must be incremental and partitioned to stay affordable.
• Every re-drive path must be provably idempotent, which constrains how verdicts and promotions are written.
• Dwell times become a tuning surface: too tight produces noise, too loose hides the problem the component exists to find.

Choose differently when

Nothing short of exactly-once delivery across every hop, including the storage copy, would remove the need for this component — and that does not exist. The sweep's frequency and partitioning are tunable; its existence is not.

Why it holds up over time

Distributed systems lose work silently. A component whose job is to notice is permanent, whatever the queue technology underneath it.

How is a promise that a tenant's files stay in their geography made true during a failover?

Context

A residency promise is easy to make and easy to break by accident. The breaking mechanisms are all mundane: a geo-redundant storage pair configured with the wrong secondary, a global load balancer failing requests to the nearest healthy region regardless of boundary, a scan tier that bursts into whichever region has spare capacity, a disaster-recovery runbook written before the residency commitment existed. Each of those is a configuration away from a boundary crossing nobody intended and nobody notices until a regulator asks.

Decision

Each residency boundary is an independent stack — its own storage accounts, its own control plane, its own scan capacity, its own keys — with no replication relationship to any other boundary. Failover happens between regions inside a boundary, never across boundaries. Scanning never transfers an object's bytes outside its boundary, including to spare capacity elsewhere. There is no configuration that could be changed to make a cross-boundary copy happen, because there is no relationship to misconfigure.

How it is realised on AWS

Two regions per boundary: primary active, secondary warm with control scaled to zero. Zone-redundant storage in region, geo-redundant within the boundary only. Verdicts replicate with the bytes so a failover does not re-scan the corpus. Front Door routes within the boundary. Keys are per boundary in Key Vault, so even an erroneous copy would be unreadable at its destination.

Options weighed

• ChosenIndependent stacks per boundary, no cross-boundary relationship: The promise is kept by there being no mechanism to break it, rather than by configuration being correct.
• RejectedOne global stack with residency tags enforced in application logic: Cheapest to operate and one bug away from a reportable breach, with no structural signal that it happened.
• RejectedGlobal stack with policy-as-code guardrails: Better, and still a boundary enforced by the correctness of a rule rather than by the absence of a path.
• RejectedCross-boundary replication for disaster recovery only: Creates exactly the relationship this decision exists to remove, for an event the in-boundary secondary already covers.

Consequences

What it buys

• A failover cannot move bytes out of the declared geography, because there is nowhere out of the geography to move them to.
• Per-boundary keys mean an erroneous copy is unreadable even if one somehow occurred.
• The residency claim can be verified by reading the infrastructure definition rather than by auditing application logic.

What it costs

• Every boundary pays for its own warm secondary and its own reserved scan capacity; there is no pooling of spare capacity across boundaries.
• Three boundaries mean three of everything to deploy, patch and observe.
• A boundary-wide outage has no cross-boundary fallback, which is a deliberate availability sacrifice.

Choose differently when

If a tenant with no residency requirement at all represented enough volume, a shared no-residency boundary could pool capacity — as an additional boundary, never by relaxing an existing one.

Why it holds up over time

Jurisdictional boundaries are getting more numerous and more specific, not fewer. A design whose boundaries are structural absorbs that; one whose boundaries are configured accumulates risk with every new rule.