A machine-learning platform must ingest telemetry from many concurrent training runs. What are the workload's distinguishing characteristics?

The distinguishing characteristics

• Extremely bursty, high-cardinality metric ingestion. A distributed training run emits metrics from many workers, at high frequency, with dimensions per worker, per layer, per step. Cardinality is the cost driver, and a naive design multiplies series count by worker count.
• Large artefacts alongside small metrics. Model checkpoints are gigabytes; metrics are bytes. These are different storage problems and forcing them into one system serves neither — artefacts belong in object storage with metadata in a database, and the metadata is what must be queryable.
• Write-heavy during training, read-heavy afterwards, with a sharp transition. The read pattern is comparison across runs, which is a different access shape from the write pattern.
• Long-lived, mostly-cold data. Most runs are never looked at again, and the ones that matter are looked at repeatedly. An extreme popularity skew, which makes tiering disproportionately effective.
• Correctness requirements are unusual: losing a metric point is acceptable; losing a checkpoint is not. Those different guarantees justify different paths.

The design consequences

• Buffer and batch ingestion at the client, since per-step synchronous writes from every worker would dominate the training job's own runtime — and an observability system that slows the workload it observes will be disabled.
• Separate the artefact path from the metric path, with different durability, different storage and different cost profiles.
• Deduplicate artefacts by content hash, since checkpoints across runs and steps share large amounts of data and the storage saving is substantial.
• Tier aggressively by age and access, given the skew.
• Make the client resilient to the platform being unavailable, buffering locally and continuing, because a tracking platform that can fail a training run is worse than no tracking platform.

The organisational value

The platform's real product is comparability: being able to say which change caused which result, across runs, months apart, by different people. That requires capturing the full configuration and code version alongside the metrics — which is a discipline problem more than a technical one, and it is what distinguishes a useful experiment record from a chart.