advanced 3 min answer

A platform team standardises every service on OTLP push to a collector and switches off Prometheus scraping. Metrics still arrive and dashboards still work. What has the team given up, and when does that bill arrive?

opentelemetryprometheuspushscrapecollectoralerting
Show the full answer Hide the answer

What is gained

  • Short-lived work can report at all. A job that lives for eight seconds is never scraped; it can push.
  • No service discovery to configure. The sender knows where the collector is; nothing has to enumerate targets.
  • It crosses boundaries. Push works out of NAT, out of a customer's network, out of an edge site the monitoring system cannot reach inbound.
  • One pipeline for three signals, with one agent, one auth mechanism and one set of processors.

What is paid

The up signal disappears. A scrape synthesises a per-target sample every interval whether the target answered or not, so "this instance stopped reporting" is a first-class, alertable fact produced by the monitoring system itself. Under push, a silent instance and a healthy idle instance are the same observation: no data. Alerting on absence is possible but you now have to build and maintain it, and it is the rule teams skip.

The target list stops being an inventory. Scrape configuration is a statement of what should exist. Push only shows what did report, so the question "which of my 600 pods is not reporting" has no answer without a second source of truth about what should be running.

Ad-hoc debugging gets harder. A scrape endpoint can be curled from a laptop to see what one instance believes right now. With push you see what arrived at the collector after batching, sampling and processors, which is a different artefact.

Back-pressure moves into your application. A scrape is paced by the collector. A pushing fleet sets its own rate, so a fleet-wide configuration change can flood the collector, and the collector's queue and drop policy are now in the critical path of every service's memory. The failure mode is an exporter queue filling in-process during an incident.

When the bill arrives

During the incident where an instance is wedged rather than dead — the process is up, the container is passing its probes, and it has stopped exporting. Under scrape this pages in one interval. Under push, nothing fires, the aggregate looks slightly better because a struggling instance stopped contributing errors, and the instance is found by a human eventually.

How to keep the option

  • Emit a per-instance heartbeat gauge and alert on its absence, which reconstructs up explicitly.
  • Keep /metrics exposed for debugging even when push is the production pipeline; it costs nothing and it is the thing you want at 3 AM.
  • Set the exporter's queue size and drop policy deliberately, and export a metric for dropped points so telemetry loss is itself visible.
  • Maintain an expected-instance list from the orchestrator, so absence can be evaluated against what should exist.

When not to agonise over it

For serverless functions, batch jobs, mobile clients and anything behind a customer's firewall, push is the only option that works and the trade is not a trade. The discussion only matters for a long-lived fleet where instance inventory carries operational meaning. Running both — scrape the fleet, push the ephemera — is the common and defensible answer, and the reason the collector supports receivers for both.