Progressive Media Materialisation

User-generated media follows an extreme popularity distribution: a small fraction of uploads receives almost all views, and most are viewed rarely or never. Generating a full rendition ladder for every upload therefore spends most of the transcoding and storage budget on content nobody watches.

Progressive materialisation produces one baseline rendition on upload and generates higher renditions on first request, caching them afterwards.

Why it matters

For a cost-sensitive media platform, transcoding compute and storage of unwatched renditions are two of the largest line items, and both scale with uploads rather than with views. Decoupling them from upload volume changes the unit economics directly.

Implementation patterns

• Always generate one immediately playable rendition on upload, so the first view is never blocked on transcoding.
• Trigger higher renditions on demand, with the first requester served the baseline while the better version is produced.
• Cache aggressively once materialised, with immutable URLs and long TTLs, so the cost is paid once.
• Prioritise the transcoding queue by live demand, not upload order. When something goes viral it is often still transcoding, and demand-driven priority is what makes the first minutes survivable.
• Tier storage by access recency — recent and popular content on standard storage, the tail on cold storage, accepting retrieval latency for a first view of something two years old.
• Negotiate format and resolution with the client, since serving a rendition higher than the device can display is pure waste at exactly the scale where it matters.

Industry example

Platforms such as ShareChat serving very large volumes of user-generated media to a cost-sensitive market face this most acutely, and video platforms such as Vimeo and Loom face the same trade with a different mix — Loom's content is typically viewed by a small known audience shortly after creation, which argues for eager transcoding of recent uploads and lazy handling of the archive.

The right policy is derived from the viewing distribution, and it differs between products that look similar.

Failure scenarios

• Eager full-ladder transcoding, spending the majority of compute on unwatched content.
• No baseline rendition, so the first view waits for transcoding.
• Upload-ordered transcoding queues, so a viral item is stuck behind thousands of routine uploads.
• Renditions materialised but never cached, paying the cost repeatedly.
• A rendition ladder that grew by accretion, with variants nobody requests still being produced.

Trade-offs

The first viewer of an unpopular item gets a lower-quality experience, or a brief delay, and that is a genuine product cost. It is small in aggregate because the affected content is by definition rarely viewed, but it is concentrated on the creator checking their own upload — which is the worst possible person to give a degraded experience.

The usual mitigation is to treat the uploader's own first view as eager, generating the full ladder for content the creator is actively looking at, and lazy for everyone else's tail.

Interview question

"Ninety percent of your uploads get fewer than ten views. Design your transcoding and storage policy, and tell me who gets a worse experience and whether you are comfortable with that."