Your pipeline takes 45 minutes and fails randomly twice a week. Which do you fix first and why?

What is being tested

Whether you recognise that trustworthiness is a precondition for speed mattering at all.

Why flakiness first

A flaky pipeline trains the team to re-run rather than investigate. Once that habit exists, a genuine failure is dismissed as flakiness — which is a production incident waiting to happen.

At that point the pipeline provides no signal, so making it faster provides no faster signal. The speed work is wasted until the trust is restored.

Fix flakiness first: quarantine immediately into a tracked, visible list, then fix or delete within a defined window. Deleting is a legitimate outcome — a test nobody trusts provides no confidence at any cost.

The usual causes: shared mutable state between tests, timing assumptions and sleeps, test-order dependence, real external dependencies, non-deterministic data.

Then duration, which matters enormously

Beyond about ten minutes for the commit stage, behaviour changes: people batch changes, stop running it locally, and eventually merge with it red. That reintroduces the batch-size problem the pipeline existed to solve.

How to reduce it: run stages in parallel; fail fast with the cheapest checks first so an obvious error reports in seconds; split the commit stage from the longer acceptance stage; cache dependencies and build outputs; and run only what the change affects.

The staging that works

• Commit stage — build, unit tests, static analysis, dependency scan. Minutes. Blocks everything.
• Acceptance stage — integration and contract tests. Longer, still blocking.
• Deployment, gated by automated verification rather than by a human.
• Post-deployment — canary analysis and automatic rollback.

What to check alongside

The same artefact promoted through environments, never rebuilt per environment, or what was tested is not what ships.

Automated rollback, so a bad deployment costs minutes. This should exist before deployment frequency is increased, because it is what makes frequent deployment safe rather than reckless.

Manual approval gates that add latency and no information — an approver who cannot meaningfully assess the change is contributing delay only.