File Upload & Scanning Pipeline  ·  View 05 of 21  ·  People and journeys

Journey — A 40 GB Master from a Bad Network

The case that kills naive designs: hours of transfer, an interruption that is certain, and a verdict measured in minutes.

Editable source SVG draw.io All views
Heavy-media creator desktop, hotel wifi Goal — Get the finished edit into the shared folder overnight Trigger — A deadline tomorrow and one evening of bandwidth Done when — It is there in the morning, whole, and playable 1 · Start declare size 2 · Transfer hours 3 · Interrupted ◆ moment of truth 4 · Finalise ◆ moment of truth 5 · Scanned minutes What they do Drops a 40 GB file Leaves it running Lift, VPN flap, sleep Comes back next morning Shares the link What the platform does Ceiling check, 60 min SAS 256 MB blocks, 16 parallel Per-chunk checksum Session held 7 days SAS renewed in session Chunk-set + hash assert Large-object profile scan Archive bounds enforced How it feels Confident Uneasy Despairing Where it hurts No idea if 40 GB is allowed Fear of starting over Credential expired Silent truncation 20 min of "scanning" What we do about it Ceiling rejected at initiate, pre-bytes A drop costs one chunk Query stored chunks, send the gap Refuse to assemble a mismatch State and reason shown, not a spinner Journey — A 40 GB Master from a Bad Network v 1.0 · lane background · target p95 verdict 20 min at 50 GB

What the journey proves

  • Resumability is not a feature here, it is the only way the journey completes at all: at 40 GB over hotel wifi, an interruption is certain and a restart is a lost evening
  • Querying which blocks landed, and sending only the gap, makes a dropped connection cost one chunk — the single highest-value requirement in §3 of the ask
  • The size ceiling is enforced at initiation because a rejection after six hours of transfer is indistinguishable from contempt

The structural consequence

  • Credential expiry mid-transfer is a normal event, not a failure: the session outlives the credential and renewal is in-session, which is why the SAS is 60 minutes and the session is 7 days
  • Time-to-verdict at this size is dominated by whether the object must be assembled before it can be judged — Question 5, and the reason the large-object profile is a separate execution class
  • A 40 GB object in the untrusted plane that then has to be copied to the serving plane is a real, priced step; it is why in-place promotion is on the table at all

Stated assumptions

  • Chunk size chosen by the client between 4 MB and 256 MB, up to 16 parallel
  • Unfinalised sessions resumable for 7 days, then collected
  • p95 verdict 20 minutes at 50 GB on the background lane
◀  Journey — Attach a File to a Conversation All views Journey — Review and Release a Blocked File  ▶
Solution Architecture v1.0 · Microsoft Azure · Integration Platform Architecture · 2026-1005-journey-large-upload