Accelerators in CI and the Test Budget

GitHub's hosted Linux 2-core x64 runner bills $0.006 per minute. Its Linux 4-core GPU runner bills $0.052, a little under nine times as much, and every job's runtime rounds up to the whole minute (GitHub, Actions runner pricing). A 25-minute GPU suite therefore costs about $1.30 a run. Fire it on every push at 50 pushes a day and it is $65 a day, roughly $1,430 over a 22-day month, against $165 if the same wall-clock ran on CPU runners. That ratio, not any principle about test coverage, is what decides the shape of an ML pipeline.

What genuinely needs a device

A surprising amount does not. Tensor shapes, configuration validation, data transforms, tokeniser round-trips, checkpoint serialisation and whole-graph construction on PyTorch's meta device all run on a CPU runner in seconds, and anything that can be moved there should be.

What cannot: custom CUDA or Triton kernels, where the test is the kernel; mixed-precision numerics and overflow behaviour, which depend on the hardware's accumulate semantics; multi-device collectives, sharding and checkpoint resharding, where the bug is in the communication pattern; memory-ceiling tests that assert the model and its KV cache fit at the maximum configured sequence length; and any claim about throughput or latency, which is a measurement and not an assertion.

Trigger design, not willpower

The pattern that holds up is four tiers with mechanical triggers.

The load-bearing word is path filter. A tier triggered by a reviewer's judgement about whether a change is risky degrades to never, or to always, within a quarter. A tier triggered by paths: ["kernels/**", "serving/**"] keeps working after the person who set it up leaves. Caching helps on top of that: key the expensive job on a manifest hash covering the model identifier, kernel sources and dependency lock, and an unchanged component is not re-billed.

Why GPU tests flake differently

Numerical defaults live in the image. PyTorch's torch.backends.cuda.matmul.allow_tf32 has defaulted to False since 1.12, while the cuDNN flag torch.backends.cudnn.allow_tf32 defaults to True; TF32 rounds inputs to a 10-bit mantissa and accumulates in FP32 (PyTorch, CUDA semantics). NVIDIA's NGC containers then set TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1, which flips the matmul default back on (MONAI, Precision and Accelerating). Bumping a base image tag can therefore change the numbers a tolerance was tuned against, with no diff in the code under test.

Tolerances are per-architecture. A library picks different kernels for different compute capabilities, so an atol that holds on an A100 can fail on an L4 or an H100. A tolerance that was tightened until the one available runner went green is a tolerance fitted to that runner.

The device is shared. A co-tenant job's allocation turns your test into an out-of-memory error that looks exactly like a leak in your own code.

Preemption is a failure mode of the plan, not the code. Spot and preemptible GPU capacity is the obvious way to afford the nightly tier, and vendor surveys of AWS's published trailing-30-day Spot interruption rates put high-end GPU instance families in the double digits of percent, varying by region and over time (Thunder Compute, 2025, Should I use cloud GPU spot instances?). Those are vendor readings of a vendor dashboard rather than an independent measurement, but the order of magnitude is enough to plan against: a 40-minute job on that capacity needs checkpointing and automatic retry, or it needs on-demand.

When it breaks

Queue time, not run time, is the cost. A four-GPU self-hosted pool serving 50 pushes a day has a p95 wait measured in tens of minutes, and a merge gate whose p95 latency is 40 minutes changes how people work regardless of its price.

Per-minute rounding punishes small jobs. Twenty GPU seconds bills a full minute, so eight separate 20-second device jobs cost four times what one 100-second job costs. Batch device work into one job.

One runner type, three production devices. Because the tier is expensive it runs on whatever was cheapest to provision, and the kernels that get exercised are that device's. The weekly matrix exists for this and is the first thing cut.

Single-device green says nothing about eight. No collective, no sharding path, no rank-dependent bug is reachable on one GPU. A single two-GPU nightly job covers more of that surface than any number of single-device runs.

Self-hosted runners drift. They are the one environment in the pipeline not rebuilt from an image on every run, so their driver and CUDA versions age independently of what the Dockerfile says. Reimaging them on a schedule, and asserting the driver version in the job, turns a confusing numerical failure into a clear one.