Architecture One-Pager

File Upload & Scanning Pipeline · Solution Architecture v1.0 · Microsoft Azure · Integration Platform Architecture · 2026-10

The object's lifecycle state — not the presence of its bytes in storage — is the sole authority on access; the untrusted plane and the serving plane are separate, and nothing crosses without a current verdict.

Someone drags a file into a conversation. That gesture crosses the hardest boundary a product has: a stranger's bytes entering a system that will later hand them to someone else who trusts the system more than they trust the stranger. The naive design — scan it inline, store it if it is clean — fails three ways at once. It cannot handle the 40 GB master from a hotel network, because an upload that must survive hours of bad wifi cannot also block on a scan. It cannot handle the scanner being slower than the uploaders, which it permanently is, so the pipeline's availability becomes the antivirus vendor's availability. And it has no answer at all when a signature published tomorrow matches a file declared clean today, because a design with no quarantine state has nowhere to put an object it has changed its mind about. Twelve product teams each solving this separately produces twelve different answers to "is this file safe", twelve support queues for attachments stuck in limbo, and one estate-wide liability that nobody owns.

Accept bytes directly into an untrusted storage plane on a short-lived, path-scoped, write-only credential, so platform compute never sits in the data path at 45 GB/s. Finalise with an explicit assertion of the chunk set and the whole-object hash, establish content identity by hashing and content type by inspecting the bytes, and enqueue the object exactly once onto a durable priority queue. Scan in ephemeral, egress-restricted sandboxes under hard bounds on time, memory, archive depth and expansion ratio, composing multiple engines' results into one versioned verdict that names every engine and signature version it rests on. Record the verdict, move the object's lifecycle state, and only then promote it into a separate serving plane from which a short read credential — minted after re-checking state and authorisation — lets the store serve the bytes directly. When the signature set moves, re-judge recent objects and revoke the verdicts that no longer hold.

What it is, and what it is not

• Lifecycle state as the sole authority on access — not Presence of bytes in a container, with a flag each read path must remember to check
• Two storage planes with separate identity and network controls — not One container with an is_clean column
• A verdict as a versioned, timestamped, revocable claim — not Clean as a permanent property of a file
• Ingest availability independent of scan availability — not An upload path that fails when the antivirus vendor does
• Direct-to-store transfer on a minted credential — not A streaming gateway sized to aggregate ingress
• Bounds breached yields indeterminate — not A timeout treated as a pass
• A published, lane-by-lane shedding order — not An unbounded queue and the word flake
• Content-hash dedup inside a declared isolation level — not Platform-wide reuse that makes the index an oracle

The decisions that are the architecture

1. State grants access, storage does not (ADR-01) — The lifecycle state machine is the only thing a read path consults. Bytes in the untrusted plane have no issuable read credential, so unreachability is structural rather than a convention every caller must honour.
2. Two storage planes, separate accounts (ADR-02) — Untrusted and serving are separate storage accounts with distinct identity and network controls, which makes the isolation claim verifiable — and makes promotion a priced copy at p99 object size.
3. Bytes go around the platform (ADR-03) — Clients write directly to the store on a write-only, path-scoped, 60-minute credential. No platform compute is sized to aggregate ingress, and resumability is inherited from the store.
4. Ingest availability is independent of scan availability (ADR-04) — An object is acceptable and durably stored while the scan tier is wholly unavailable, remaining unreachable until scanned. A scanner outage is a latency incident, not an availability one.
5. A verdict is versioned, timestamped and revocable (ADR-05) — Verdicts are keyed by content hash and engine version. Re-scan writes a new row; revocation moves the object out of the serving plane. Clean is never permanent.
6. Bounds breached yields indeterminate, never clean (ADR-06) — Every scan is bounded in wall-clock time, memory, archive depth and expansion ratio. A breach is a typed indeterminate verdict naming the bound, which reaches a human within 24 hours.
7. The scanner is the least-trusted compute in the platform (ADR-07) — Ephemeral per-object jobs in a separate subscription, no inbound reachability, egress allow-listed to two destinations, and no credential able to change an object's state.
8. Back-pressure is declared, not discovered (ADR-08) — Three priority lanes with per-tenant admission limits and a published shedding order: bulk refuses first, background throttles, interactive is rejected at initiation. Rejection never happens after bytes move.
9. Verdict reuse is bounded by a declared isolation level (ADR-09) — Content-hash dedup avoids re-scanning 35% of objects. Reuse defaults to within-tenant because platform-wide reuse turns the index into an oracle for whether another tenant holds a specific file.
10. Finalisation is an assertion the platform can refuse (ADR-10) — The client asserts the chunk set and the whole-object hash. A mismatch is a failed upload, not an assembled object: nothing unverified ever enters the scan queue.
11. Every object reaches a terminal state or a report (ADR-11) — A reconciler re-drives, dead-letters or reports any object past its expected dwell time. Silence about an object is a defect, because a stuck object is both a support ticket and a security hole.
12. Residency is enforced by absence of replication (ADR-12) — Each residency boundary is an independent stack with no replication relationship to any other, so a failover cannot move bytes out of the geography a tenant declared.
13. Size is a workload class (ADR-13) — Three execution profiles by object size rather than one sized for the worst case: a 50 KB screenshot and a 50 GB archive share an API, not a worker shape or a latency target.

Why this should still be right in ten years

Object stores, scan engines, container runtimes and credential formats will all be replaced inside a decade. What should survive is the set of claims about where authority lives, because none of them names a product.

• "State grants access" is not a technology. It is a claim about which component is allowed to answer the question "may this person have these bytes". It survives replacing the object store, the credential mechanism and the state store, because none of those changes who is entitled to decide.
• The seam between seeing and deciding. The worker reads bytes and cannot change state; the control plane changes state and never touches bytes. That separation outlives whatever sandbox technology currently implements it, and it is the claim a reviewer in 2036 will still want to check.
• Revocability is permanent. Threat intelligence will always arrive after some uploads. A design where clean is a revocable claim rather than a property will remain correct however good detection becomes, because the arrival order of knowledge does not improve.
• Bounds will matter more, not less. Formats will keep gaining nesting, compression and indirection. A design that treats a breached bound as indeterminate rather than clean degrades gracefully as content gets more hostile; one that treats a timeout as a pass degrades silently.
• The direct-to-store decision follows economics that are not changing. Compute in a byte path costs proportionally to bytes. As objects get larger, the case for keeping platform compute out of the data path strengthens rather than weakens.
• What will date. The engine mix, the dedup isolation default, and the re-scan window are all tied to current detection economics and current liability expectations. Expect all three to be revisited; none of them changes the boundary.

Non-functional targets

Every figure below is a stated assumption for this design, sized for the reference workload and intended to be argued with rather than believed. The view column points at the diagram where the figure is visible as a constraint.

Scope

In scope

• Upload initiation, admission and short-lived path-scoped credential minting for both end-user and server-to-server callers
• Resumable chunked transfer from 1 byte to 50 GB, with per-chunk and whole-object integrity verification
• Content identity by cryptographic hash, content type by byte inspection, and deduplicated verdict reuse within a declared isolation level
• Scan orchestration across a fast signature engine and a conditional deep engine, with bounded archive expansion
• The lifecycle state machine, quarantine with evidence retention, audited release, and verdict revocation
• Priority lanes, per-tenant admission limits and a published shedding order
• Authorised download credential issuance and edge delivery of available objects
• Per-tenant policy for ceilings, allowed types, required engines, reuse isolation and evidence retention
• Verdict and state-change events, the append-only transition log, and the operational signals that make the pipeline legible

Explicitly out of scope

• Preview, thumbnail and transcode generation — subscribers to the available event
• Data-loss-prevention classification and content indexing — consumers of verdicts, not gates on them
• Digital-rights management and watermarking
• The product surfaces that render attachments and the sharing model above them
• Long-term archival policy and legal-hold workflow beyond evidence retention
• Building or maintaining scan engines — they are licensed, pulled images

What a four-week prototype should prove

The prototype's job is to falsify the two claims the whole design rests on: that unreachability can be made structural rather than conventional, and that a verdict can be revoked fast enough to matter. Everything else in the design is comparatively ordinary engineering.

1. One tenant, two destination scopes, one fast engine, both storage planes as separate accounts
2. Resumable upload of a 20 GB object with three deliberate interruptions and one credential expiry
3. The full lifecycle state machine with promotion, quarantine, audited release and revocation
4. A durable queue with at-least-once delivery, idempotent verdict writes and a dead-letter path
5. A decompression bomb and a malformed container, to prove the bounds hold and the lane does not block
6. A revocation measured end to end: verdict revoked to last successful read

• A read credential is requested one second after a verdict is revoked, and is refused
• A read credential issued one second before revocation still works, and stops working within 5 minutes
• The scan tier is stopped entirely for an hour: uploads still succeed, nothing becomes readable, and the backlog drains without loss
• A worker is killed between reading the bytes and writing the verdict: the object is re-leased and the verdict is written once
• A 200× decompression bomb yields indeterminate naming the bound, and the next object in the lane is unaffected
• An upload credential is replayed against a different blob path, and is rejected by the store
• Promotion of a 20 GB object is measured, in wall-clock time and in money, to price the two-plane decision honestly

Open risks, carried rather than hidden

The reasoning behind every component and technology choice is in the Architecture Decision Record: 13 records across 5 areas, each with the alternatives that lost and what the choice costs.