Architecture Decision Records — Multimodal RAG Platform
Solution Architecture v1.0 · Enterprise Architecture · 2026-08
Each record states the decision, what was rejected, and what would make us revisit it.
ADR-001 — Normalise every modality into one chunk record
Status Accepted
Context Six input modalities with nothing structurally in common. Retrieval, ranking, citation and access control each need a uniform object to operate on.
Decision Every extraction path emits the same record: a text projection, a modality tag,
a provenance pointer and an optional parent reference. Modality-specific detail lives in the
asset entity, not in the retrieval path.
Rejected Per-modality indexes with a federated ranker. Rejected because ranking across independently-scored indexes requires score calibration that drifts, and because citation and ACL logic would be implemented six times.
Consequences Retrieval, reranking and generation are modality-agnostic. The cost is that each new modality must be reduced to text well enough to be findable — which is a real constraint on, for example, purely visual content with no OCR-able text.
Revisit if a modality arrives that genuinely cannot be projected to text without losing the thing users search for.
ADR-002 — Index images as generated descriptions plus a native image vector
Status Accepted
Context Images carry meaning that OCR alone misses — chart trends, diagram topology, photographs. A joint image-text embedding retrieves visually similar images but poorly captures "what does this chart say".
Decision Two channels. A vision model writes a description; that description plus any OCR text is embedded as text and becomes the primary retrieval channel. A joint image-text embedding is stored as a secondary channel for visual-similarity queries. The channels are fused at rerank.
Rejected Image embeddings only — fails the dominant question type. Descriptions only — fails "find the diagram that looks like this".
Consequences Indexing cost per image is higher and includes a model call. Description quality is now permanent index quality: a bad description makes an image permanently unretrievable, so this call uses the strong model, not the cheap one.
Revisit if joint embedding models close the semantic gap enough to make descriptions redundant.
ADR-003 — Store tables twice: linearised for retrieval, typed for SQL
Status Accepted
Context Users ask both "what does the pricing table say about tier 3" (retrieval) and "what is the total across all regions" (aggregation). The second has no answer in any chunk.
Decision Tables are linearised to markdown and indexed as ordinary chunks, and separately loaded as typed rows into an analytical store. Numeric and aggregate questions are routed to generated SQL against those rows.
Rejected Retrieval-only over linearised tables. Rejected because it produces confident arithmetic that is wrong — the worst possible failure mode for a system whose value proposition is groundedness.
Consequences A routing decision is now required at query time, and generated SQL must be sandboxed and validated. Duplicate storage of table content is accepted.
Revisit if table sizes make row-level loading impractical at corpus scale.
ADR-004 — Hybrid retrieval with reciprocal rank fusion and cross-encoder reranking
Status Accepted
Context Enterprise queries contain identifiers — part numbers, error codes, policy references — where dense retrieval reliably fails. Pure lexical retrieval fails on paraphrase.
Decision Dense and sparse retrieval run concurrently, each returning 50 candidates. Results merge by reciprocal rank fusion, then a cross-encoder reranks to the top 8.
Rejected Weighted score fusion — requires calibration between incomparable score distributions and drifts as either index changes. RRF operates on ranks and needs none.
Consequences Two indexes to maintain and keep consistent. A GPU reranker on the latency-critical path, budgeted at 400 ms for search plus rerank.
Revisit if measured recall from a single channel matches the hybrid within tolerance on the golden set.
ADR-005 — Prepend generated context to each chunk at index time
Status Accepted
Context An isolated chunk often lacks the terms a user would search by. "The rate increased to 4.2%" does not mention which policy or which year.
Decision At index time, generate one short line situating the chunk in its document and prepend it before embedding.
Rejected Larger chunks — improves context at a direct cost to retrieval precision. Query-time expansion only — does not fix a chunk that is unfindable in the first place.
Consequences One model call per chunk at index time. Mitigated by using Claude Haiku 4.5 via the Batch API at half rate. Reindexing is required if the enrichment prompt changes, which is why the derived zone exists.
ADR-006 — Small-to-big retrieval via parent references
Status Accepted
Context Precision favours small chunks; generation quality favours large context.
Decision Embed small chunks; on a hit, return the parent section to the model.
Consequences chunk.parent_chunk_id is load-bearing. Deduplication is needed when several
child hits share a parent.
ADR-007 — Enforce grounding before an answer is released
Status Accepted
Context The requirement is grounded, cited answers. Generation with retrieved context raises groundedness but does not guarantee it.
Decision A verification step checks each claim against a retrieved span. Failure triggers one retry with a rewritten query; a second failure returns an explicit abstention.
Rejected Trusting the generation prompt. Rejected because an unenforced instruction is not a control, and because groundedness then cannot be measured as a release gate.
Consequences Added latency and an extra model call on the read path. Abstention rate becomes a tracked quality metric — a rising rate means retrieval degraded, which is exactly the signal wanted.
ADR-008 — All model calls through a single gateway
Status Accepted
Context Model choice, cost control, redaction, caching, fallback and retry policy would otherwise be reimplemented in every service that calls a model.
Decision One gateway owns routing by task, semantic caching, per-tenant budget caps, payload redaction, provider fallback and retry. No service holds a provider credential.
Consequences The gateway is on the critical path and must be highly available. In exchange, a model swap, a price change or a redaction rule is a single-place change, and egress is auditable at one point.
ADR-009 — Separate the asynchronous write path from the synchronous read path
Status Accepted
Context Ingestion is bursty and minutes-scale; querying is steady and seconds-scale.
Decision Two independently scaled domains sharing only storage. Ingestion is queue-driven with idempotent, resumable workers; querying is a synchronous request path.
Consequences Eventual consistency between upload and queryability, bounded at 15 minutes and stated as an NFR. Users are shown ingestion status rather than left to guess.
ADR-010 — Enforce tenant isolation inside the retriever
Status Accepted
Context Multi-tenant corpora in a shared index. A filter supplied by the caller is a filter an attacker can omit.
Decision The retriever derives tenant and ACL filters from verified token claims and applies them itself. Vector namespaces are partitioned per tenant; metadata uses row-level security.
Consequences The retriever cannot be called in an unfiltered mode, including by internal batch jobs, which must present a service identity scoped to a tenant. Cross-tenant leakage is asserted in automated tests as a release gate.
ADR-011 — Treat prompts, chunking configuration and eval sets as versioned code
Status Accepted
Context A prompt edit or a chunk-size change moves answer quality as much as a code change, but conventionally bypasses every release control.
Decision These artefacts live in the repository, are versioned with the code, and pass the same CI gates — including the RAG eval gate and the cost gate.
Consequences No hot-editing prompts in production. The eval harness must be fast enough to run per pull request, which caps golden-set size and argues for a tiered set: a fast tier in CI, a full tier nightly.
ADR-012 — Rebuild indexes on failover rather than replicate them
Status Accepted
Context RTO is 4 hours. Cross-region replication of vector and lexical indexes roughly doubles their cost.
Decision Replicate the object store and the metadata database. Rebuild vector and lexical indexes in the DR region from the derived zone on failover.
Consequences Failover is slower but within RTO. The rebuild path must be exercised, not assumed — measured quarterly, and the decision reverts to replication if rebuild time approaches the RTO.