Production RAG checklist

Use this before a pilot, design review, or production handoff. A checked box should point to evidence—not confidence.

Data and access

• Source owners and freshness expectations are named.
• Deletions and access changes propagate to the index.
• Document and chunk metadata retain tenant and permission boundaries.
• Retrieved content is authorized for the requesting user.
• Sensitive data handling and retention are documented.
• Embedding model, version, dimensions, input modes, and distance function are pinned with the index.

Retrieval

• Representative questions have labeled supporting passages.
• Chunking preserves headings, lists, tables, and source identity.
• Retrieval metrics include the value of k and an explicit denominator.
• Filters, hybrid search, or reranking have evidence that they improve a baseline.
• Operators can inspect retrieved passages for one request.

Generation and citations

• The model is told what to do when evidence is insufficient.
• Claims that affect user decisions require source support.
• Citations resolve to content the user can access.
• Prompt injection in retrieved content is treated as untrusted input.
• Authorization and external-sharing rules are enforced outside the model.
• Consequential tool calls show exact arguments for approval where required.
• Model and prompt versions are recorded with evaluation results.

Reliability and operations

• Each external call has a timeout and bounded retry policy.
• Retried side effects are idempotent or explicitly guarded.
• Latency, cost, error, and quality signals are observable per stage.
• A safe degraded path exists when retrieval or the model is unavailable.
• An owner can reproduce a bad answer from request traces.
• Semantic-cache entries, if any, include tenant, permission, prompt/model, and knowledge versions.
• Cache tests measure false hits, stale hits, and cross-scope hits—not only savings.

Customer handoff

• Success criteria and release thresholds are written down.
• Known limitations are visible to users and support staff.
• Runbooks cover stale data, bad retrieval, provider failure, and data isolation.
• The customer can add evaluation cases from real failures.
• Ownership after launch is unambiguous.