Scheduler Control Plane
The orchestrator's own state and availability, which becomes a critical dependency for every pipeline it runs.
An orchestrator holds the state that matters: which runs happened, which succeeded, where each pipeline is. That makes its database and its scheduler process a single point of failure for the entire data platform — and one that is frequently deployed with less care than the pipelines it runs.
Three properties are worth designing for explicitly. Recovery: if the control plane is lost, can it be rebuilt and can it work out what already ran? If the answer depends on a database nobody backs up, that is the real risk. Idempotency of tasks: after a control plane failure, some tasks will be re-run, and a task that is not safe to repeat turns a scheduler restart into a data incident. Isolation: a runaway task that consumes the workers starves every other pipeline, so worker pools and concurrency limits per team are not optional at scale.
The related choice is where the work executes. Schedulers that run the transformation inside the orchestrator couple its resource needs to the pipeline's; those that dispatch to an external engine keep the control plane small, which is the more robust arrangement and the one that survives growth.