Review Change Rate

An architecture board reports 212 submissions, 97% approved, a 3-day median to decision and reviewer satisfaction of 4.2 out of 5. Every number looks healthy and none of them answers the question leadership actually asked, which is whether the board changes anything.

Approval rate cannot answer it, because a good review and an ornamental one both approve almost everything. A board that publishes its bar sees designs that already meet it; a board that waves things through sees the same 97%. The two are indistinguishable from the outcome column.

Review change rate is the number that separates them: of the designs reviewed, what fraction shipped different from the version submitted, in a way a reader can attribute to the review — a datastore swapped, a boundary moved, a rollback path added, a dependency removed. It is measured by sampling, not by self-report, because a board asked to score its own influence will score it high.

Why it matters

Architecture governance is expensive and the cost is paid in the currency teams notice most, which is calendar time. A board of five senior people meeting weekly consumes on the order of 400 to 600 engineer-hours a year before any preparation, and each submission costs the submitting team a document and a wait.

With no measure of effect, the board's defence of its own existence is its throughput, which is why boards drift toward optimising the queue: faster decisions, higher satisfaction, more submissions. Those are the three things that improve when scrutiny falls.

The second reason is that change rate tells you which repair is needed. A low rate and a high rate are both problems, and they have opposite fixes.

Implementation patterns

• Sample, do not survey. Take 20 approved submissions from the last year, read the submitted design and the as-built system, and classify each as unchanged, cosmetically changed, or materially changed. Two reviewers, independently, then reconcile.
• Pair it with coverage, counted from production evidence rather than submissions: new datastores created, new destinations for personal data, new public endpoints, new production runtimes. 212 submissions against an organisation making 600 significant changes a year means the board sees a third of the surface, and change rate on a third is a weak lever.
• Record submission timing. A design submitted after implementation started cannot be changed by the review, so those rows must be excluded from the denominator or the rate is understated.
• Publish both numbers and nothing else. Dashboards of throughput and satisfaction invite the behaviour they measure.

Industry example

Consider a travel marketplace of the kind Expedia operates: dozens of product teams, a central board, and the four figures above. Pulling change rate on a 20-submission sample typically produces one of three pictures. Under about 10% the board is advisory in practice and should be made advisory in name, with central sign-off reserved for a short published list. Between 20% and 40% it is doing real work. Above 60% the bar is being discovered in the room rather than published, and the fix is review-readiness criteria so teams can meet it before they arrive.

This is written as an archetype rather than a published case: the four reported figures are the ones governance programmes of that size habitually collect, and the thresholds come from the economics rather than from any one company's disclosure.

Failure scenarios

• The rate is measured by asking reviewers, who report the designs they remember influencing. Recall is biased toward the dramatic sessions.
• Coverage is ignored, so a board with an excellent change rate on the 30% of work it sees is declared effective while the risky work routes around it.
• The rate is turned into a target. Reviewers then request changes to demonstrate value, the bar becomes unpredictable, and teams respond by submitting late — which degrades coverage and timing at once.
• Cosmetic changes are counted as material, usually diagram edits and naming, which inflates the figure without any design consequence.

Trade-offs

Measuring change rate also surfaces a political cost: the first honest number is usually low, and the person who produced it owns the consequence.

When not to use it

In a newly merged organisation, a near-100% approval rate with a near-zero change rate can be correct for a year. The board's output then is a shared map and a shared vocabulary, not design changes, and the right measure is whether reviewers can describe each other's systems without help. Switch to change rate once the map exists. The same applies to a board whose stated purpose is recording decisions rather than altering them — but then say so, and stop calling it a gate.

Interview question

Q: You inherit an architecture board reporting 97% approval, a 3-day median and 4.2 satisfaction. Leadership wants to cut it. What two numbers do you produce in a week to defend or close it, and what would make you recommend closing it?

What a strong answer covers: change rate from a built-versus-submitted sample and coverage counted from infrastructure state; the exclusion of late submissions from the denominator; a recommendation to go advisory under roughly 10% change rate with central sign-off kept for a short published list; and the observation that satisfaction and median time measure the queue, so neither can defend the board.

Quick check

Quiz: A board approves 97% of submissions and teams rate it 4.2 out of 5. Which number tells you whether it is a gate? — Neither; material change rate on a sample of built-versus-submitted designs does, paired with coverage of the real change set.

Flashcard: Why can approval rate never show whether a review board works? — A board with a published bar and a board that stamps everything both approve nearly all submissions, so the outcome column cannot separate them.