advanced 3 min answer

Review this design. A finance team's invoice processor uses a planner agent that decomposes each invoice into subtasks, a critic agent that reviews the plan, a vector store of the last 50000 processed invoices as episodic memory, a 40-step tool loop, and a human queue for anything the critic flags. The task is to extract six fields and post them to the ledger. What would you remove and what would you keep?

agentsover-engineeringstructured-outputdeterminismreview
Show the full answer Hide the answer

What is actually required

Six fields from a document, validated, posted to a ledger. The requirement that matters is not intelligence: it is that a wrong number must never reach the ledger silently, and that every posting can be explained to an auditor a year later.

That requirement is met by a deterministic check and an audit record. None of it requires planning.

What I would remove, and why it is safe

  • The planner. There is no plan to make. The steps are fixed: read the document, extract fields, validate, post. A planner on a fixed workflow adds a model call, a failure mode and non-determinism, and buys nothing. Write the four steps as code and call the model once, with a strict output schema.
  • The critic. A second model reviewing the first model's work shares the first model's blind spots and correlates with its errors. For structured extraction the real check is arithmetic: does the line-item total equal the invoice total, does the supplier id exist, is the date within a plausible range. A deterministic validator catches what a critic cannot and never hallucinates an approval.
  • The episodic memory of 50,000 past invoices. The most expensive item to keep and the most dangerous. It makes behaviour depend on the system's own history, so the same invoice is processed differently in March and in June and no test reproduces a production failure. It also creates a data path between suppliers. Per-supplier extraction hints belong in a small reviewed configuration table.
  • The 40-step loop. With the planner gone the step budget is 1.

The one change that matters

Route on the validator's output rather than on the critic's judgement. Post automatically when every arithmetic check passes and confidence on the supplier match is high; send everything else to the human queue. That converts a fuzzy escalation rule into a rule you can state in a sentence, audit, and tune with a number.

What I would keep, even though it looks odd

  • The human queue. Reversibility is the test: a wrong ledger posting takes accounting work to unwind, so the escalation path is not optional. Size it honestly, because the residual cases are the hard ones and reviewer throughput on them is much lower than on average invoices.
  • The full input and output transcript per invoice, including the model version and prompt version. It looks like over-logging until the first dispute.

What this costs and where it flips

The simplified system is roughly one model call per invoice instead of five or more, which is a large cost reduction and a similar latency reduction. It flips when the task stops being extraction: if the system must chase a missing purchase order across three systems, the next step genuinely depends on the last result, and a bounded tool loop becomes the right shape. Even then the planner and the critic remain separable questions.

Common weak answers

  • "Keep the critic but make its prompt stricter." A second call from the same model family agrees with the first far more often than an independent check would, so the measured catch rate is much lower than it looks. Test it: feed the critic 100 invoices with a deliberately corrupted total and count how many it flags.
  • "Keep the memory but scope it per supplier." Better, and still the wrong mechanism. If the useful content is "this supplier puts the VAT number in the footer", that is a reviewable fact for a table, not an embedding retrieved by similarity.
  • "Remove the human queue since accuracy is 98%." Two percent of a thousand invoices a day is twenty wrong ledger postings a day. The escalation path is sized by the cost of a miss, not by the accuracy figure.

How I would argue this in the review

Not as "this is over-engineered", which invites defence. As a measurement: run both on 500 real invoices and compare field-level accuracy, cost and the share sent to humans. If the elaborate version does not win on accuracy, its complexity has no argument left.