Evidence Transferability

Engineering practice is shaped heavily by published accounts: conference talks, blog posts, incident reports and architecture write-ups from organisations with recognisable names. They are the profession's main source of shared experience, and they are systematically misapplied.

Evidence transferability is the question of whether a finding from another system applies to yours, and the answer depends on similarity of workload shape, scale, constraints and organisational context — not on the prestige of the source.

Why it matters

The most influential technical opinions in circulation are generalisations from cases whose specifics were discarded. A conclusion valid for a workload with particular characteristics becomes a rule applied to workloads with none of them, and the resulting architecture is wrong in a way that is difficult to argue against because it carries a famous name.

The reverse error is equally costly: dismissing a genuinely applicable finding because it came from an organisation operating at a different scale, when the mechanism it describes is scale-independent.

The discipline is asking what specifically made it true there, and whether those conditions hold here.

Implementation patterns

• Extract the mechanism, not the conclusion. "Orchestration billed per state transition becomes dominant when the unit of work is finer than the request" transfers; "serverless is expensive" does not.
• Identify the workload shape explicitly: payload size, interaction frequency, read/write ratio, tail sensitivity, cardinality, correlation of arrivals. Two systems with the same architecture diagram and different workload shapes have almost nothing in common.
• Check the scale distance. A finding from an organisation three orders of magnitude larger frequently describes a problem you will never have, and the solution carries costs you cannot amortise.
• Ask what is not being said: what the new design cost, what was lost, what emerged later, and whether the original design was well chosen.
• Ask what the author wants. Recruiting, credibility, a vendor narrative and genuine sharing usually coexist, and knowing the mix explains the emphasis.
• Weight by similarity, deliberately — evidence from a comparable organisation with a comparable workload is worth far more than a more famous case that is less alike.
• Prefer the question to the answer. The durable value of a case study is the question it teaches you to ask at design time.

Industry example

The Prime Video video-quality monitoring post is the clearest recent case. One team moved one internal component from a serverless step-function pipeline to a single long-running service, reporting a large cost reduction driven by per-transition orchestration billing at frame granularity and large intermediate payloads crossing object storage.

It was widely reported as "Amazon abandons microservices" — a conclusion the post did not state, that the evidence did not support, and that is contradicted by the organisation continuing to run an enormous number of services.

The transferable finding is narrow and genuinely useful: function-per-step decomposition is wrong when the steps are chatty and the payloads are large, and boundaries should follow the data rather than the verbs. The untransferable claims — about serverless, about microservices, about monoliths — are the ones that circulated.

Failure scenarios

• Adopting an architecture because a large company uses it, without the scale, workload or team that made it appropriate.
• Rejecting an approach because a famous post criticised it, in a context where the criticism does not apply.
• Copying an organisational structure — team topology, platform model, review process — from a company with a different size and product.
• Taking a reversal as a refutation of a pattern, when it is one team's finding about one workload.
• Ignoring evidence from smaller organisations, which is frequently more applicable than evidence from larger ones.
• Reading a vendor case study as neutral evidence.
• Building for a scale you will not reach, on the authority of a post written by someone who did.

Trade-offs

Rigorous evaluation of transferability costs time and slows decisions, and there is a real value in convention: adopting what large organisations do is a reasonable heuristic when you have no better information, and it is defensible in a way that an original decision is not.

There is also a failure mode in the other direction — exceptionalism. A team that believes its situation is unique dismisses all external evidence and rediscovers well-known failures at its own expense. "Our requirements are unique" is usually wrong, and the discipline cuts both ways.

The trade is decision speed and organisational defensibility against the risk of importing an architecture suited to someone else's problem. The practical resolution is proportionality: spend the analysis on decisions that are expensive to reverse, and follow convention on the rest.

Interview question

"Someone links a post from a very large company arguing for the opposite of what we have built, and the room is now uncertain. Walk me through how you would evaluate whether it applies to us, and tell me what you would need to know about their system before changing anything about ours."