Executable Example

An integrator follows the quickstart, and step four returns a 422 for a field the page says is optional. They now know two things: this call does not work, and nothing on this site can be trusted without verification. The second is far more expensive than the first.

Documentation drift is not a discipline problem to be solved with reminders. A hand-written reference and a changing API diverge by default, because only one of them has a test. An executable example closes the gap mechanically: every request and response in the documentation is run in CI against a real environment, and a change that breaks an example fails the build that made it.

The reframe is the whole value. Documentation stops being something a person remembers to update and becomes something the pipeline refuses to let you break.

Why it matters

Wrong reference documentation is worse than none, because it destroys the reader's ability to calibrate. With no documentation, an integrator reads the schema or asks. With documentation that is sometimes wrong, they cannot tell which half to believe, so they verify everything - and the support inbox carries the cost.

The quickstart is the highest-stakes page by a wide margin: it is the first thing a new integrator touches, and a failure there is read as a statement about the whole product. An example that is a test is the only durable way to keep that page honest across three years of releases.

Implementation patterns

• Extract examples from the documentation source and run them in CI, or generate the documentation from the tests. Either direction works; what does not work is maintaining the two separately. Budget for it honestly: a suite covering the main flows typically adds 5 minutes to 10 minutes of CI time, which is why it belongs in the pipeline rather than in a nightly job nobody reads.
• Run against a real environment, not a mock. A mock asserts that your documentation agrees with your assumptions, which is not the claim being made.
• Seed deterministic fixtures so examples are reproducible, and keep responses stable by redacting what varies - identifiers, timestamps - rather than by loosening the assertion to nothing.
• Generate the endpoint reference from the specification, and verify the implementation against that specification with a schema-driven test. Then delete the hand-written reference instead of keeping both.
• Generate the changelog from release metadata, so it is a by-product rather than a chore - a changelog that stops eighteen months ago tells integrators that nobody is writing anything down.
• Let the support inbox choose what to write. Six recurring questions are the specification for six task-shaped pages, and that is the highest-return documentation work available.

Industry example

The practice is standard in library documentation and under-used in API documentation. Go's testable examples and Python's doctest make documented examples part of the test suite by design, and Rust compiles and runs the code in documentation comments as part of cargo test - so a library whose documentation compiles is a library whose examples are at least not stale.

On the HTTP side the equivalent is schema-driven verification: tools that exercise an API against its OpenAPI specification and fail when the implementation and the specification disagree. The pattern is the same and the adoption is lower, which is why the drift is so much worse in HTTP APIs than in the libraries of the same organisations.

Failure scenarios

• The three-year-old quickstart that fails at step four, teaching every new integrator to distrust the portal.
• Examples tested against a mock, passing happily while the real API returns a different shape.
• A specification maintained by hand beside the implementation, drifting silently because neither is generated from the other.
• Examples so loosely asserted - status code only - that they pass while the response body has changed completely.
• An errors page that does not exist, because generated references rarely include error semantics and nobody wrote them by hand.
• A documentation build decoupled from the release, so correct documentation is published weeks after the change it describes.

Trade-offs

Executable examples need a test environment, fixtures and maintenance - in effect, a test suite with a second audience. For an API with two internal consumers that is more machinery than the problem justifies.

They also constrain how examples are written: anything requiring a human step, a third-party sandbox or a long-running approval is awkward to automate, and the temptation is to exclude exactly the complicated flows that integrators most need documented. Excluding them is the right call - with a visible marker that those examples are unverified, so the reader knows which parts carry the guarantee.

When not to use it

For an internal API with a handful of consumers who can ask in a chat channel, the honest answer is a short page and a schema. The investment scales with the number of readers you cannot talk to.

And where the real problem is the API's design, documentation is being used to paper over it. Recurring questions about inconsistent naming, surprising error semantics or an authentication flow nobody gets right first time are telling you about the API, not the pages. Fix the two worst inconsistencies first; some of the questions disappear without a single page being written.

Interview question

Q: Your developer portal has 140 pages of hand-written endpoint reference, a quickstart that no longer works, and a support inbox with six recurring questions. What do you delete, what do you generate, and how do you keep it correct in two years' time?

What a strong answer covers: deleting the hand-written reference rather than keeping it beside a generated one, because wrong reference is worse than none; generating the reference from the specification and verifying the implementation against it in CI; executable examples run against a real environment as the mechanism that stops drift; writing one task-shaped page per recurring support question; generating the changelog from release metadata; an errors page as the highest-value hand-written artefact; and using the support inbox as the measurable argument that wins the work against a feature.

Quick check

Quiz: Why is drifted reference documentation worse than having none at all? Because a reader who finds one documented field that does not exist cannot calibrate which parts to trust, so they verify everything - and the verification lands in your support inbox.

Flashcard: What single practice keeps API documentation from rotting? Running every documented example in CI against a real environment, so a change that invalidates an example fails the build that made it.