Semantic or fuzzy caching is implemented for similar queries

Users phrase semantically identical questions in dozens of different ways: "What is the refund policy?" and "How do I get a refund?" produce identical answers but fail an exact-match cache check. Semantic caching, which compares query embeddings for similarity rather than string equality, captures this overlap. Research and production deployments consistently show that semantic caching captures 20–40% additional cache-eligible traffic that exact matching misses — traffic that would otherwise generate fresh API calls. ISO 25010 performance-efficiency classifies redundant recomputation of equivalent responses as an efficiency defect.

Implement semantic caching using a vector index with a similarity threshold. Upstash Semantic Cache provides this out-of-box; for custom implementations, embed the query and run a nearest-neighbor search before calling the AI API.

import { SemanticCache } from "@upstash/semantic-cache";
import { Index } from "@upstash/vector";

const index = new Index();
const semanticCache = new SemanticCache({ index, minProximity: 0.95 });

export async function semanticallyCachedGenerate(prompt: string): Promise<string> {
  const cached = await semanticCache.get(prompt);
  if (cached) return cached;

  const { text } = await generateText({
    model: openai("gpt-4o-mini"),
    prompt,
    maxTokens: 500,
  });

  await semanticCache.set(prompt, text);
  return text;
}

Verify by sending two semantically equivalent but differently worded queries — the second should return a cache hit without triggering a new API call.