Semantic caching without serving the wrong answer

• Lesson
• Intermediate
• 13 min read
• Checked 15 Aug 2026

In one sentence

A semantic cache saves a previous answer for a nearby question. That can reduce work, but a false hit can confidently return the wrong user’s stale answer.

First, name the cache

Cache	Match	What happens on a hit
Exact response cache	Identical governed key	Return the stored answer
Provider prompt cache	Identical prompt prefix	Reuse provider computation, then generate a new answer
Semantic response cache	Nearby query embedding	Return a previously generated answer

Semantic response caching makes the strongest assumption: “these two questions can share one answer.” Similarity alone does not prove that.

“Can I return order 1042?”
“Can I cancel order 1042?”

The wording and vectors may be close, but the allowed action, policy, and live order state can differ.

A narrow safe starting point

Start with stable, read-only, non-personal FAQ routes. Avoid semantic response reuse for:

• user- or tenant-specific answers;
• rapidly changing facts;
• multi-turn questions that depend on hidden history;
• authorization decisions;
• tool calls or side effects such as sending an email;
• answers whose source or policy version is unknown.

Redis’s agent guidance specifically warns about later conversational turns, cached errors or function calls, and non-idempotent operations.¹

Scope is part of the key

tenant + authorization scope + route + locale
+ policy version + prompt version + model version
+ knowledge/index version + time window

Two users asking the same sentence must not share an answer when their permissions or underlying data differ. Azure’s semantic-cache policy supports varying entries by identity-related dimensions for this reason.²

src/examples/llmops/semantic_cache.py

"""A deliberately conservative semantic response-cache policy."""

from dataclasses import dataclass
from datetime import UTC, datetime, timedelta
from typing import Callable, Sequence


Vector = Sequence[float]


@dataclass(frozen=True)
class CacheScope:
    tenant_id: str
    authorization_scope: str
    route: str
    locale: str
    policy_version: str
    prompt_version: str
    model_version: str
    knowledge_version: str


@dataclass(frozen=True)
class CacheEntry:
    scope: CacheScope
    query_vector: Vector
    answer: str
    created_at: datetime


def safe_to_cache(*, route: str, has_personal_data: bool, has_side_effect: bool) -> bool:
    """Cache only stable, read-only FAQ responses in this example."""
    return route == "public_faq" and not has_personal_data and not has_side_effect


def add_cached_answer(
    entries: list[CacheEntry],
    *,
    scope: CacheScope,
    query_vector: Vector,
    answer: str,
    has_personal_data: bool,
    has_side_effect: bool,
    now: datetime | None = None,
) -> bool:
    """Insert only after the route-level cache policy accepts the response."""
    if not safe_to_cache(
        route=scope.route,
        has_personal_data=has_personal_data,
        has_side_effect=has_side_effect,
    ):
        return False

    entries.append(
        CacheEntry(
            scope=scope,
            query_vector=query_vector,
            answer=answer,
            created_at=now or datetime.now(UTC),
        )
    )
    return True


def find_cached_answer(
    entries: list[CacheEntry],
    *,
    scope: CacheScope,
    query_vector: Vector,
    similarity: Callable[[Vector, Vector], float],
    threshold: float,
    max_age: timedelta,
    now: datetime | None = None,
) -> str | None:
    """Return a scoped, fresh answer only when similarity clears the tested threshold."""
    current_time = now or datetime.now(UTC)
    candidates = [
        entry
        for entry in entries
        if entry.scope == scope and current_time - entry.created_at <= max_age
    ]
    if not candidates:
        return None

    best = max(candidates, key=lambda entry: similarity(query_vector, entry.query_vector))
    if similarity(query_vector, best.query_vector) < threshold:
        return None
    return best.answer


# Production code also needs encryption, deletion, audit records, concurrency
# control, and an evaluation of false hits and stale hits.

The code is intentionally conservative. Production code also needs encrypted storage, deletion, concurrency control, invalidation events, and audit records.

Thresholds are local

A threshold such as 0.92 has no portable meaning across embedding models, distance functions, and corpora. Some APIs return similarity where higher is better; others return distance where lower is better.

Build labeled pairs:

safe reuse       “How do I install VPN?” / “Steps to set up VPN”
false-hit trap   “Can I return this?” / “Can enterprise users return this?”
stale-hit trap   old policy answer after a source revision
isolation trap   same question from two permission scopes

Measure hit rate, false-hit rate, stale-hit rate, cross-scope hits, saved model calls, and added cache-lookup latency. The useful operating point is the one that saves work while remaining below the product’s error budget.

Invalidation is part of correctness

Expire or invalidate entries when a source, prompt, model, policy, locale, or authorization rule changes. A longer TTL may increase hits and stale answers at the same time.

Begin with an exact cache. Add semantic matching only after repeated traffic shows a real opportunity and labeled pairs show that the reuse decision is safe.

Footnotes

1. Redis, semantic caching guidance for agents. ↩

2. Microsoft, semantic cache lookup policy, documents identity- and request-based cache variation. ↩