Multimodal RAG Platform

Version 1.0 · Status For review · Date 2026-08 · Owner Enterprise Architecture

1. Purpose and scope

Build a platform that ingests heterogeneous enterprise content — text, PDFs, images, tables, audio and video — and answers natural-language questions against it with grounded, cited responses.

In scope: multimodal ingestion and understanding, embedding and indexing, hybrid retrieval, LLM generation with citation enforcement, a web interface for upload and query, multi-tenancy, security, observability and the evaluation loop.

Out of scope for v1: real-time streaming ingestion of live media, agentic multi-step tool use beyond retrieval, fine-tuning of foundation models, and write-back into source systems.

2. Drivers

3. Architecture overview

The platform separates a write path (asynchronous, minutes-scale, bursty) from a read path (synchronous, seconds-scale, latency-sensitive). They share only storage. This is the single most important structural decision: the two have different scaling profiles, different failure semantics and different cost drivers.

Write path   sources → intake → raw store → per-modality understanding →
             chunk + enrich → embed → vector | lexical | typed tables

Read path    question → guard → rewrite/route → hybrid search →
             fuse + rerank → parent expand → generate → grounding gate → answer

See view 02-high-level-architecture for the picture, 05 and 06 for the write path, 07 and 15 for the read path.

4. Key design decisions

4.1 One chunk record for every modality

Every source, regardless of type, normalises into a common record:

This is what makes a question answerable across modalities at all: the retrieval layer only ever sees one kind of thing. It is also what makes a citation clickable — the provenance columns point at a specific page, region or moment.

4.2 Text-projection-first retrieval, with a native image channel alongside

Images are indexed twice. A vision model writes a description of the image, and any text in it is OCR'd; both are embedded as text. Separately, a joint image-text embedding is stored.

Rationale: for the dominant question type — "what does the Q3 revenue chart show" — a generated description retrieves far better than a joint image embedding, because it captures semantics the embedding compresses away. The native image vector is retained for the minority case: "find the diagram that looks like this". Two channels, fused at rerank.

4.3 Tables are stored twice

A linearised markdown chunk enters the vector and lexical indexes. The actual typed rows land in an analytical table addressable by SQL.

Rationale: retrieval over a linearised table cannot answer "what was the total across all regions" — the number is not in any chunk. Numeric questions route to generated SQL instead. Without this, the platform silently produces confident wrong arithmetic.

4.4 Hybrid retrieval, not vector-only

Dense and sparse searches run concurrently and are merged with reciprocal rank fusion, then reranked by a cross-encoder.

Rationale: dense retrieval fails on part numbers, error codes, policy identifiers and personal names — exactly the tokens enterprise users search for. RRF needs no score calibration between the channels, which is why it is preferred over weighted score fusion.

4.5 Contextual chunk enrichment

At index time, each chunk is prefixed with a short generated line situating it in its document. This is a one-off indexing cost that materially raises retrieval accuracy, because a chunk that says "the rate increased to 4.2%" becomes findable by a query about the specific policy it belongs to. It runs on Claude Haiku 4.5 through the Batch API at half the standard token rate.

4.6 Small-to-big retrieval

Small chunks are embedded for retrieval precision; the larger parent section is what reaches the model. Precision where it is searched, context where it is read.

4.7 The grounding gate

After generation, every claim is checked against a retrieved span. Unsupported claims trigger one retry with a rewritten query; failing that, the platform abstains and says so.

Rationale: the requirement is grounded answers. Grounding that is hoped for is not grounding. Abstention is a logged, measured outcome — the abstention rate is a quality metric, not a defect.

4.8 A model gateway, not direct SDK calls

Every model call leaves through one gateway that owns routing, fallback, semantic caching, per-tenant budget caps, payload redaction and retry policy.

Rationale: without it, model choice, cost control and redaction are scattered across a dozen services and cannot be changed or audited in one place.

4.9 Isolation enforced at the retriever

Tenant and ACL filtering is applied inside the retriever, using claims from the verified token — never by a filter the caller supplies. A caller that can request an unfiltered search is a caller that can leak another tenant's corpus.

5. Model selection

Every generation call goes through the model gateway, which routes by task:

Adaptive thinking (thinking: {type: "adaptive"}) is on for generation, with output_config.effort tuned per route — high for answer synthesis, low for classification. Prompt caching holds the stable system prompt and tool definitions ahead of the volatile retrieved context, so the cacheable prefix is not invalidated per request.

Embeddings are not an Anthropic surface; a dedicated embedding provider supplies the text and image vectors, called through the same gateway.

6. Security architecture

See view 10-security-trust-zones.

Five zones in decreasing exposure: Internet, perimeter, application, data, control plane.

Threats explicitly modelled:

• Malicious upload. Files are quarantined and scanned before any parser touches them. Parsers run with no network access and a strict resource budget.
• Prompt injection inside document text. This is the defining threat of RAG. Retrieved content is fenced and labelled as data in the prompt; the content guard strips instruction-shaped text at index time; the generation prompt states that retrieved content is never an instruction. Defence is layered because no single layer is reliable.
• Cross-tenant leakage. Namespace per tenant in the vector index, row-level security on metadata, filtering enforced inside the retriever.

Controls: TLS 1.3 inbound, mTLS between services, OIDC with JWKS verification at the gateway, customer-managed encryption keys, secrets rotated every 90 days, write-once audit log forwarded to the SIEM, egress restricted to an allow-list.

7. Non-functional requirements

8. Cost model

Cost is dominated by two lines: generation tokens on the read path and enrichment plus embedding on the write path.

Per-tenant cost is metered at the model gateway and enforced as a budget cap. Cost per query is a CI gate (view 12), so a change that improves quality by tripling spend does not ship unnoticed.

9. Risks

10. Assumptions

• Source systems expose delta or change APIs; full re-crawls are not required per cycle.
• Document ACLs are available at ingest and are authoritative for retrieval filtering.
• A corporate OIDC provider exists and can issue tokens carrying tenant and group claims.
• Initial corpus is on the order of 10^6 chunks per tenant, not 10^9.

11. Roadmap

Phase 1 is deliberately a full vertical slice rather than a horizontal layer. It proves the grounding gate and the eval harness — the two things most likely to invalidate the design — before the modality-specific work multiplies the cost of changing them.

12. Traceability

Each record states the decision, what was rejected, and what would make us revisit it.

ADR-001 — Normalise every modality into one chunk record

Status Accepted

Context Six input modalities with nothing structurally in common. Retrieval, ranking, citation and access control each need a uniform object to operate on.

Decision Every extraction path emits the same record: a text projection, a modality tag, a provenance pointer and an optional parent reference. Modality-specific detail lives in the asset entity, not in the retrieval path.

Rejected Per-modality indexes with a federated ranker. Rejected because ranking across independently-scored indexes requires score calibration that drifts, and because citation and ACL logic would be implemented six times.

Consequences Retrieval, reranking and generation are modality-agnostic. The cost is that each new modality must be reduced to text well enough to be findable — which is a real constraint on, for example, purely visual content with no OCR-able text.

Revisit if a modality arrives that genuinely cannot be projected to text without losing the thing users search for.

ADR-002 — Index images as generated descriptions plus a native image vector

Status Accepted

Context Images carry meaning that OCR alone misses — chart trends, diagram topology, photographs. A joint image-text embedding retrieves visually similar images but poorly captures "what does this chart say".

Decision Two channels. A vision model writes a description; that description plus any OCR text is embedded as text and becomes the primary retrieval channel. A joint image-text embedding is stored as a secondary channel for visual-similarity queries. The channels are fused at rerank.

Rejected Image embeddings only — fails the dominant question type. Descriptions only — fails "find the diagram that looks like this".

Consequences Indexing cost per image is higher and includes a model call. Description quality is now permanent index quality: a bad description makes an image permanently unretrievable, so this call uses the strong model, not the cheap one.

Revisit if joint embedding models close the semantic gap enough to make descriptions redundant.

ADR-003 — Store tables twice: linearised for retrieval, typed for SQL

Status Accepted

Context Users ask both "what does the pricing table say about tier 3" (retrieval) and "what is the total across all regions" (aggregation). The second has no answer in any chunk.

Decision Tables are linearised to markdown and indexed as ordinary chunks, and separately loaded as typed rows into an analytical store. Numeric and aggregate questions are routed to generated SQL against those rows.

Rejected Retrieval-only over linearised tables. Rejected because it produces confident arithmetic that is wrong — the worst possible failure mode for a system whose value proposition is groundedness.

Consequences A routing decision is now required at query time, and generated SQL must be sandboxed and validated. Duplicate storage of table content is accepted.

Revisit if table sizes make row-level loading impractical at corpus scale.

ADR-004 — Hybrid retrieval with reciprocal rank fusion and cross-encoder reranking

Status Accepted

Context Enterprise queries contain identifiers — part numbers, error codes, policy references — where dense retrieval reliably fails. Pure lexical retrieval fails on paraphrase.

Decision Dense and sparse retrieval run concurrently, each returning 50 candidates. Results merge by reciprocal rank fusion, then a cross-encoder reranks to the top 8.

Rejected Weighted score fusion — requires calibration between incomparable score distributions and drifts as either index changes. RRF operates on ranks and needs none.

Consequences Two indexes to maintain and keep consistent. A GPU reranker on the latency-critical path, budgeted at 400 ms for search plus rerank.

Revisit if measured recall from a single channel matches the hybrid within tolerance on the golden set.

ADR-005 — Prepend generated context to each chunk at index time

Status Accepted

Context An isolated chunk often lacks the terms a user would search by. "The rate increased to 4.2%" does not mention which policy or which year.

Decision At index time, generate one short line situating the chunk in its document and prepend it before embedding.

Rejected Larger chunks — improves context at a direct cost to retrieval precision. Query-time expansion only — does not fix a chunk that is unfindable in the first place.

Consequences One model call per chunk at index time. Mitigated by using Claude Haiku 4.5 via the Batch API at half rate. Reindexing is required if the enrichment prompt changes, which is why the derived zone exists.

ADR-006 — Small-to-big retrieval via parent references

Status Accepted

Context Precision favours small chunks; generation quality favours large context.

Decision Embed small chunks; on a hit, return the parent section to the model.

Consequences chunk.parent_chunk_id is load-bearing. Deduplication is needed when several child hits share a parent.

ADR-007 — Enforce grounding before an answer is released

Status Accepted

Context The requirement is grounded, cited answers. Generation with retrieved context raises groundedness but does not guarantee it.

Decision A verification step checks each claim against a retrieved span. Failure triggers one retry with a rewritten query; a second failure returns an explicit abstention.

Rejected Trusting the generation prompt. Rejected because an unenforced instruction is not a control, and because groundedness then cannot be measured as a release gate.

Consequences Added latency and an extra model call on the read path. Abstention rate becomes a tracked quality metric — a rising rate means retrieval degraded, which is exactly the signal wanted.

ADR-008 — All model calls through a single gateway

Status Accepted

Context Model choice, cost control, redaction, caching, fallback and retry policy would otherwise be reimplemented in every service that calls a model.

Decision One gateway owns routing by task, semantic caching, per-tenant budget caps, payload redaction, provider fallback and retry. No service holds a provider credential.

Consequences The gateway is on the critical path and must be highly available. In exchange, a model swap, a price change or a redaction rule is a single-place change, and egress is auditable at one point.

ADR-009 — Separate the asynchronous write path from the synchronous read path

Status Accepted

Context Ingestion is bursty and minutes-scale; querying is steady and seconds-scale.

Decision Two independently scaled domains sharing only storage. Ingestion is queue-driven with idempotent, resumable workers; querying is a synchronous request path.

Consequences Eventual consistency between upload and queryability, bounded at 15 minutes and stated as an NFR. Users are shown ingestion status rather than left to guess.

ADR-010 — Enforce tenant isolation inside the retriever

Status Accepted

Context Multi-tenant corpora in a shared index. A filter supplied by the caller is a filter an attacker can omit.

Decision The retriever derives tenant and ACL filters from verified token claims and applies them itself. Vector namespaces are partitioned per tenant; metadata uses row-level security.

Consequences The retriever cannot be called in an unfiltered mode, including by internal batch jobs, which must present a service identity scoped to a tenant. Cross-tenant leakage is asserted in automated tests as a release gate.

ADR-011 — Treat prompts, chunking configuration and eval sets as versioned code

Status Accepted

Context A prompt edit or a chunk-size change moves answer quality as much as a code change, but conventionally bypasses every release control.

Decision These artefacts live in the repository, are versioned with the code, and pass the same CI gates — including the RAG eval gate and the cost gate.

Consequences No hot-editing prompts in production. The eval harness must be fast enough to run per pull request, which caps golden-set size and argues for a tiered set: a fast tier in CI, a full tier nightly.

ADR-012 — Rebuild indexes on failover rather than replicate them

Status Accepted

Context RTO is 4 hours. Cross-region replication of vector and lexical indexes roughly doubles their cost.

Decision Replicate the object store and the metadata database. Rebuild vector and lexical indexes in the DR region from the derived zone on failover.

Consequences Failover is slower but within RTO. The rebuild path must be exercised, not assumed — measured quarterly, and the decision reverts to replication if rebuild time approaches the RTO.