Usage Metering & Billing Pipeline  ·  View 09 of 22  ·  Data

Metering Data Flow

From an emitted event to a served quantity, including the two paths that are not the happy one.

Editable source SVG draw.io All views
Emit Producer SDK durable buffer Edge agent minute rollup Accept Validate schema + refs Dedup index 7-day TTL Quarantine reason kept Record Event stream Kinesis Ledger append Firehose Event ledger S3 Iceberg Window Watermark 72 h lateness Event-time windows hourly, daily Aggregate Counter & gauge Unique entity Aggregate store DynamoDB Serve Rating service Usage API freshness stamped Baseline detector rejected event id late event baseline Metering Data Flow — Emit to Served Quantity Application we own Data store Queue / topic Interface / broker failure / alternate synchronous event / async batch Quantities are provisional until the watermark clears the period. Nothing here is authoritative: every box right of the ledger is rebuildable from it. v 1.0 · owner Architecture Practice · date 2026-10

Decisions

  • Aggregation is on event time with an explicit watermark, not on arrival time. Usage belongs to the period in which it happened, and a mobile or edge producer that was offline for an hour must not move revenue between months.
  • Rejected events are quarantined with their reason and replayed after the defect is fixed, never discarded. A silent discard is indistinguishable from usage that never happened.
  • The baseline detector reads aggregates, not events, so it can see an account's shape change without re-scanning the ledger.

Assumptions

  • Lateness bound of 72 hours (stated assumption), chosen to cover an offline edge or mobile producer and a long weekend, and short enough that the period can close within the invoice run's window.
  • Deduplication window of 7 days (stated assumption), covering the longest expected producer retry.

Risks

  • Unique-entity counting is the one aggregate that is not cheaply incremental; at 400 meters it is the likeliest source of aggregation cost surprise, which is why cost-per-meter is a watched signal in view 18.
  • A watermark that advances on a quiet partition will close a period early for that partition. Idle-partition handling is a Flink configuration detail with a revenue consequence.
◀  Integration and Interface Catalogue All views Storage Zones  ▶
Solution Architecture v1.0 · Amazon Web Services · Billing Platform Architecture · 2026-1009-metering-data-flow