File Upload & Scanning Pipeline  ·  View 10 of 21  ·  Data

Data Flow — One File to the First Read

Six stages, the 35% that never reaches a scanner, and the only arrow that runs backwards.

Editable source SVG draw.io All views
Declare Client Initiate size, type, scope, key Admission gate reject pre-bytes Land Path-scoped SAS write only, 60 min Blocks, 16 parallel per-chunk checksum Session state chunk set, 7 d Identify Finalise assert set + hash Type by inspection claim recorded apart Content identity hash, dedup lookup Judge Lane admission interactive Bounded scan time, memory, depth Composed verdict engine + sig version Promote State transition logged, immutable Serving plane available Available event Read Read credential state re-checked, 5 min Edge delivery no pre-promotion caching Another member verdict reuse on hash hit backlog sheds admission Data Flow — One File, Followed to the First Read External / third party Interface / broker Decision point Security / platform Data store Application we own Queue / topic Person or role batch failure / alternate The dotted path is the 35% that never reaches a scanner. The red path is the only arrow that runs backwards: saturation downstream is expressed as refusal at the front door. v 1.0 · dedup 35% assumed hash hit rate

Decisions

  • The declared content type is recorded as a claim and the determined type is established by inspecting the bytes; policy is applied to the determined type when they disagree
  • Content identity is a cryptographic hash, independent of filename, path and owning tenant — the filename is metadata and never influences storage layout or a scan decision
  • A hash hit short-circuits to an existing verdict within the tenant's declared reuse isolation level, and every reuse is recorded against the originating verdict

The backwards arrow

  • Scan saturation sheds admission at the gate, which is the only way to refuse work without lying about an object's state
  • The alternative — accept everything and let the backlog grow — leaves attachments sitting in scanning with no published bound, which is worse than a typed rejection with a retry hint
  • Which lane is shed first is published in advance; see the back-pressure view

Stated assumptions

  • 35% of objects deduplicate to an existing verdict
  • 4,000 objects/second enter the scan queue at peak
  • Read credential 5 minutes, re-checked against state on every issuance
◀  Interface Catalogue All views Storage Classes  ▶
Solution Architecture v1.0 · Microsoft Azure · Integration Platform Architecture · 2026-1010-data-flow