Why did workflow 123 take 47 seconds? Answering it needs one trace spanning an HTTP request, a database write, a queue hop, a worker on a different pod, two retries and a downstream call — which is only possible because traceparent is propagated in Service Bus application properties.
Six identifiers travel together on every log line, span and message: traceId, spanId, correlationId, tenantId, workflowExecutionId and taskExecutionId. Any one of them reconstructs the whole picture.
Business events are separated from logs. ExecutionCompleted is a fact with a retention and an SLA attached; a log line is diagnostic output that may be sampled away.
Sampling and cost
5% head sampling with 100% tail retention on errors. At 1.8M tasks per day, full trace retention would dominate the platform's running cost while adding nothing to the p50 case.
Logs are 30 days hot and 2 years in archive; metrics 93 days; business facts 400 days in Azure SQL for annual comparison. Every retention figure is a cost decision as much as a compliance one.
No payload content is ever logged. Log lines carry references and identifiers, which keeps the observability estate outside the data-protection boundary.
Alerting discipline
Every alert links a runbook. An alert without one is deleted at the next review, because an unactionable page trains people to ignore pages.
Alert on burn rate, dead-letter arrival rate, oldest-message-age and projection lag — symptoms a user would feel. CPU and memory are dashboards, not pages.
Per-tenant SLO views are a launch requirement. Without them a single tenant's degradation is invisible inside a healthy aggregate.
