Every task in the data platform sits in a queued state. Worker CPU is around 6%, the metadata database is healthy, the task logs show nothing, and no task has failed. Someone proposes adding workers. What do you look at first, and what is probably happening?

The first three things to look at

1. The concurrency limits, top down. Global parallelism, per-DAG concurrency, per-pool slots. Queued means "admitted but not dispatched", so the constraint is almost always a counter, not capacity.
2. What currently holds the slots. Sort running tasks by duration. If the longest-running are sensors or wait-for-file steps, you have found it.
3. The scheduler's own loop time. If the loop is taking minutes because it is parsing thousands of definition files each pass, tasks queue behind the scheduler rather than behind the workers.

The diagnosis

Long-running sensors hold worker slots while doing nothing. A sensor that polls for an upstream file every 60 seconds occupies a slot for its entire wait — six hours of waiting is six hours of a slot consumed by a process that is asleep almost all of it. Thirty such sensors on a pool of 32 slots and the platform is fully booked while using no CPU at all.

The idle CPU is the misleading clue. It says "not compute-bound", which most people read as "needs more workers", when it actually says "the work being blocked is not CPU work". Adding workers helps only if the limiting counter is raised too, and if the limiting counter is raised without fixing the sensors, the same wall arrives at the next multiple.

The fix, in order

1. Move sensors to a deferrable or reschedule mode, where waiting releases the slot and the task is resumed on an event or a poll timer. This usually returns most of the pool immediately.
2. Give sensors their own pool with a hard cap, so that even in reschedule mode they cannot crowd out execution.
3. Replace polling with event-driven triggering where the upstream can publish: an object store notification instead of a directory poll.
4. Only then consider more workers, sized against measured concurrent execution rather than the task count.

The alert that would have caught it

Alert on time in the queued state — for example, any task queued longer than 10 minutes — and on pool slot utilisation above 90% for 15 minutes. Task duration and worker CPU both look healthy during this incident, which is why dashboards built around them stay green while the platform delivers nothing.

When this is the wrong diagnosis

If running tasks are at high CPU and queue time tracks the number of submitted tasks, the pool is genuinely undersized. The distinguishing test takes one minute: are the slot-holders working or waiting?