intermediate 3 min answer

Review this design. An order-to-cash flow runs in a workflow engine as a single process definition with 17 service tasks and about 340 process variables; pricing and credit rules are written as inline script tasks inside the definition. Over the last quarter 61% of production incidents were stuck process instances resolved by hand. What would you remove, what would you change, and what would you leave alone?

workflow-engineorchestrationprocess-designincidentsscript-tasks
Show the full answer Hide the answer

What is actually required

An order-to-cash flow has genuine long-lived state. It waits for a credit decision, a pick confirmation, a delivery event and a payment that may arrive days later, and it must survive restarts and be answerable for where any order currently sits. A workflow engine is the right tool for that and should not be the thing you remove. The question is what has been put inside it that does not belong there.

What I would remove

The script tasks. Pricing and credit rules in a scripting language inside a process definition form a second codebase with no unit tests, no code review that engineers take seriously, no static analysis and no local run. They are also the usual cause of the stuck instances: a script throws on an unexpected null, the instance parks in an incident state, and the only recovery is an operator editing a variable in a console. That is a correctness hazard as much as a toil problem, because manual variable edits are untracked writes to business state.

Most of the 340 process variables. A process variable is state the engine persists and locks on every transition. Carrying the full order payload inside the instance means the engine is now a slow, poorly indexed copy of the order database. Carry identifiers and the few values that drive routing; read the rest.

The one change that matters

Move the rules into the services that own them and let the process call them. Pricing becomes a call to a pricing service with a versioned contract and its own tests; credit becomes a call to the credit service. The process definition keeps sequence, waiting and compensation, which is what it is good at.

Then do the thing that actually cuts the 61%: every service task gets a timeout, a bounded retry and a defined failure branch. An instance should never be able to park indefinitely. If the credit service does not answer in 30 seconds across three attempts, the process takes the documented path, which may be "hold for manual review" as a modelled state with an owner and an SLA, not as an incident.

What I would leave alone

The engine itself, the single process definition, and the fact that one team owns the flow. Splitting one coherent business process across three engines to make it look like microservices is the change people reach for next and it makes the state harder to see, not easier. One business process, one definition, is a feature.

How I would argue this in the review

Bring the incident numbers, not the architecture opinion. 61% of incidents in one failure class, each resolved by a human editing state by hand, is a figure the operations owner already feels. Propose the timeout-and-failure-branch change first, because it is two weeks and it moves the number; propose the rule extraction second, sequenced by rule change frequency, with the pricing rules going first because they change weekly and credit rules change twice a year.

When this is the wrong answer

If the flow has no long-lived waits and every case completes inside one synchronous call in under 30 seconds, the engine is overhead and the correct change is to delete it and write the sequence in code. A workflow engine earns its licence and its operational surface only when the process waits for events that arrive hours or days later. Plenty of teams that adopted one around 2019 run a flow that never waits, and they are paying a second runtime to host a state machine that fits in a few hundred lines. Choose the engine for durable waiting, not for the diagram.