A semantic cache answers a question from memory when a question with the same meaning has been asked before. That sounds like a small trick, yet in AI applications it can save a large share of model calls. The topic keeps coming back to me because many chatbot and FAQ projects show the same picture: hundreds of people ask the same twenty questions, just with different words each time. Even so, each time the language model computes the answer from scratch.
In this article, I build a semantic cache in front of Amazon Bedrock, using ElastiCache Serverless for Valkey as the vector store, a Lambda function and the Serverless Framework. For example, I use an FAQ assistant for a fictional bike rental. Just as important as the code, though, is the question of when such a cache becomes dangerous. After all, a false hit delivers a wrong answer with full confidence.
In short
A semantic cache stores a language model’s answers together with the embedding of the question and returns them again when a new question is close enough. It pays off most for recurring, general questions, and an exact cache level in front of it even saves the embedding. The biggest pitfall is false hits: questions that sound almost the same but need a different answer. Hard filters before the vector search and a carefully calibrated threshold protect you from them.
What you will learn in this article
- How a semantic cache works and how it differs from prompt caching
- When the cache pays off and which questions never belong in it
- Preventing false hits with hard filters and a calibrated threshold
- Building the cache with Lambda, ElastiCache Serverless and the Serverless Framework
- Keeping an eye on hit rate and cost
What is a semantic cache?
A classic cache works with exact keys. If someone asks “How long can I rent a bike?”, the answer is stored under exactly that text. When the next person writes “What is the maximum rental period?”, the cache finds nothing, even though both want to know the same thing.
A semantic cache therefore compares meaning instead of wording. To do so, an embedding model turns every question into a vector, a long list of numbers. As a result, questions with a similar meaning end up close to each other. Then the cache looks up the nearest stored vector for the new question. If the distance is small enough, it returns the stored answer. Otherwise it asks the language model and stores that answer for next time.
In practice, the distance is usually measured as cosine distance. For normalized vectors it ranges from 0 to 2, where 0 means “same direction”. Many articles talk about similarity instead, which is simply 1 minus the distance. So a distance of 0.1 corresponds to a similarity of 0.9.
How it differs from prompt caching
The name is easy to confuse, so here is a quick distinction. Prompt caching in Bedrock remembers a long, always identical beginning of a prompt, for example an extensive system prompt. However, the model is still called for every request; only the repeated part is processed more cheaply and quickly. A semantic cache, by contrast, skips the model call entirely. The two don’t exclude each other: prompt caching helps with every miss of the semantic cache.
When a semantic cache pays off
At the end of 2025, AWS published an evaluation on its Database Blog with roughly 63,000 real chatbot queries, placing a semantic cache on ElastiCache in front of Bedrock. With a similarity threshold of 0.75, daily costs in that experiment dropped by about 86 percent, and average response time fell from 4.35 to 0.51 seconds. Still, these are vendor figures from a single dataset with many rephrasings. That is why I find the table behind them more interesting: the more generous the threshold, the higher the hit rate, but the lower the accuracy of the answers served.
This leads to a simple rule of thumb. A semantic cache pays off when many people ask similar questions and the right answer does not depend on who is asking. Typical candidates are:
- FAQ assistants for opening hours, rules, processes and terms
- internal knowledge bots, for example about vacation requests or travel expenses
- explanations of products, features and technical terms
- classifications where the same text should always get the same category
Which questions never belong in the cache
The opposite list matters just as much. Personal questions such as “Where is my booking 4711815?” must never end up in the cache, otherwise the next person might see another customer’s data. Similarly, time-sensitive questions such as “Is a cargo bike still free today?” go stale within minutes. Multi-turn dialogs are tricky as well, because the meaning of a question depends on the conversation so far. For such cases, the example contains a rule that bypasses the cache on purpose.
The architecture with two cache levels
The solution consists of three levels that are asked one after the other. Level 1 is a perfectly normal exact cache: the question is normalized, which means lowercased and stripped of punctuation, and looked up as a hash key. This costs only a single read and no embedding. Only when nothing is found there does level 2 create an embedding and look for the nearest neighbor in the vector index. Finally, level 3 is the language model itself. Its answer is then stored in both cache levels, each with a time to live.
🎬 How a request flows through the semantic cache – this animated diagram only runs on my site: watch it on sascha-ms.de
All three levels run in a single Lambda function behind an HTTP API. As in the article on ElastiCache vector search, I run the cache with a public endpoint. As a result, the function needs no VPC and no NAT gateway, while the connection is still encrypted with TLS 1.3 and secured with IAM.
Avoiding false hits in a semantic cache
Now to the part that many tutorials skip. Take the questions “Do I need a helmet for the e-bike?” and “Do I need a helmet for the cargo bike?”. For an embedding model, these are almost identical sentences: same structure, same topic, just one word different. The answers, however, can differ considerably. A generous threshold would happily serve the wrong answer here.
Schematic: a threshold alone does not reliably separate similar questions with different answers.
The threshold alone does not solve this problem, because it is always a compromise for all questions at once. That is why I also rely on hard filters before the vector search. Valkey supports this directly: before the arrow => comes a filter on TAG fields, followed by the KNN search. In other words, only entries that match the filter exactly are searched. In the example, there are three such fields:
- Topic such as “rental” or “prices”, which the user interface sends along
- Knowledge version (kbv), a version label for your content, so that old answers no longer match after a price change
- Bike type, which a small list of key terms and synonyms detects in the question
The last point in particular carries over to other domains. Almost every domain has a handful of terms that flip an answer: plan names, product variants, countries, contract types. If you treat these terms as an exact filter, the vector search can only find similar entries within the matching group. Admittedly, that costs a few hits, but it prevents exactly the embarrassing mix-ups.
Building a semantic cache step by step
The following nine steps take you from an empty account to the first answer from the cache. You will also find the complete code in the example repository, linked at the end of the article.
Prerequisites
- An AWS account with permissions for ElastiCache, Lambda, API Gateway, IAM and Bedrock
- AWS CLI version 2, configured for eu-central-1
- Node.js 24 and npm 10 or newer
- The Serverless Framework v4 (
npm i -g serverless) - Access to Amazon Titan Text Embeddings V2 and a text model in Bedrock, Amazon Nova Lite in this example
Step 1: Create the ElastiCache Serverless cache
You create the cache once via the CLI. What matters are Valkey version 9 or later, the connection type public and the system-managed user group for IAM authentication. With --cache-usage-limits you cap storage and compute, which is a good safeguard against surprises in test accounts.
aws elasticache create-serverless-cache \ --serverless-cache-name bike-station-cache \ --engine valkey \ --major-engine-version 9 \ --connection-type public \ --user-group-id default.iam-user-group \ --cache-usage-limits "DataStorage={Maximum=1,Unit=GB},ECPUPerSecond={Maximum=5000}" \ --region eu-central-1 # Query status and address (repeat until "available" appears) aws elasticache describe-serverless-caches \ --serverless-cache-name bike-station-cache \ --query "ServerlessCaches[0].[Status,Endpoint.Address]" \ --output text --region eu-central-1 Note the address as CACHE_HOST. Also check which cross-region inference profiles your account offers for Nova Lite. You will need the profile ID as the model ID in a moment:
aws bedrock list-inference-profiles --region eu-central-1 \ --type-equals SYSTEM_DEFINED \ --query "inferenceProfileSummaries[?contains(inferenceProfileId, 'nova-lite')].inferenceProfileId" \ --output text Step 2: Set up the project
As the client, I use Valkey GLIDE again. It comes with IAM authentication including automatic token renewal and has its own module for search commands. In addition, you need the Bedrock SDK for embeddings and answers, plus Vitest for the tests.
{ "name": "bike-station-faq", "version": "1.0.0", "private": true, "type": "commonjs", "scripts": { "build": "tsc", "test": "vitest run", "setup-index": "tsx scripts/setup-index.ts", "calibrate": "tsx scripts/calibrate.ts", "deploy": "npm run build && serverless deploy --param=\"cacheHost=$CACHE_HOST\"" }, "dependencies": { "@aws-sdk/client-bedrock-runtime": "^3.1169.0", "@valkey/valkey-glide": "^2.6.0" }, "devDependencies": { "@types/aws-lambda": "^8.10.164", "@types/node": "^24.19.2", "tsx": "^4.23.15", "typescript": "^7.0.2", "vitest": "^5.1.0" } } { "compilerOptions": { "target": "ES2023", "module": "nodenext", "moduleResolution": "nodenext", "types": ["node"], "strict": true, "esModuleInterop": true, "skipLibCheck": true, "outDir": "dist", "rootDir": "." }, "include": ["src", "scripts", "test"] } I take the cache connection and the embedding almost unchanged from the vector search article. The only difference: the request timeout is set to 500 milliseconds. After all, a cache that takes longer than that no longer saves anything.
// src/valkey.ts import { GlideClusterClient, ServiceType } from "@valkey/valkey-glide"; let pending: Promise<GlideClusterClient> | undefined; /** One client per Lambda instance; GLIDE refreshes the IAM token itself. */ export function getClient(): Promise<GlideClusterClient> { pending ??= GlideClusterClient.createClient({ addresses: [{ host: required("CACHE_HOST"), port: 6379 }], useTLS: true, requestTimeout: 500, credentials: { username: "default.iam-user", iamConfig: { clusterName: required("CACHE_NAME").toLowerCase(), service: ServiceType.Elasticache, region: required("AWS_REGION"), }, }, }).catch((error: unknown) => { pending = undefined; // reconnect on the next call throw error; }); return pending; } export function required(name: string): string { const value = process.env[name]; if (!value) throw new Error(`Environment variable ${name} is missing`); return value; } // src/embed.ts import { BedrockRuntimeClient, InvokeModelCommand } from "@aws-sdk/client-bedrock-runtime"; export const EMBEDDING_DIM = 512; const bedrock = new BedrockRuntimeClient({}); /** Creates a normalized embedding with Titan Text Embeddings V2. */ export async function embed(text: string): Promise<number[]> { const response = await bedrock.send( new InvokeModelCommand({ modelId: "amazon.titan-embed-text-v2:0", contentType: "application/json", accept: "application/json", body: JSON.stringify({ inputText: text, dimensions: EMBEDDING_DIM, normalize: true }), }), ); const parsed = JSON.parse(new TextDecoder().decode(response.body)) as { embedding: number[] }; return parsed.embedding; } /** Valkey expects FLOAT32 values as little-endian bytes. */ export function toVectorBuffer(vector: number[]): Buffer { const bytes = Buffer.alloc(vector.length * 4); vector.forEach((value, i) => bytes.writeFloatLE(value, i * 4)); return bytes; } Step 3: Create the vector index
The index watches all hashes with the prefix answer:. Furthermore, it contains the three filter fields as TAG and the embedding as a vector with 512 dimensions and cosine distance. Question and answer themselves don’t need to be indexed; they are only returned with the result.
// scripts/setup-index.ts import { GlideFt } from "@valkey/valkey-glide"; import { getClient } from "../src/valkey"; import { EMBEDDING_DIM } from "../src/embed"; import { INDEX_NAME, KEY_PREFIX } from "../src/semantic-cache"; async function main(): Promise<void> { const client = await getClient(); const existing = (await GlideFt.list(client)).map(String); if (existing.includes(INDEX_NAME)) { console.log(`Index ${INDEX_NAME} already exists`); } else { await GlideFt.create( client, INDEX_NAME, [ { type: "TAG", name: "topic" }, { type: "TAG", name: "kbv" }, { type: "TAG", name: "bike" }, { type: "VECTOR", name: "embedding", attributes: { algorithm: "HNSW", dimensions: EMBEDDING_DIM, distanceMetric: "COSINE", type: "FLOAT32" }, }, ], { dataType: "HASH", prefixes: [KEY_PREFIX] }, ); console.log(`Index ${INDEX_NAME} created`); } client.close(); } main().catch((error: unknown) => { console.error(error); process.exit(1); }); npm install export AWS_REGION=eu-central-1 export CACHE_NAME=bike-station-cache export CACHE_HOST=bike-station-cache-abc123.public.serverless.euc1.cache.amazonaws.com npm run setup-index You should see the line “Index idx:answers created”. On a second run, the script reports that the index already exists.
Step 4: The rules as pure functions
Every decision that can go wrong lives in one file without dependencies on AWS. This includes normalization, the rule for personal questions, bike type detection, the key for the exact level and the query for the vector search. Because tag values end up in the query language, assertSafeTag only lets lowercase letters and digits through. Therefore, nobody can inject their own filters with crafted input. One English-specific detail: compounds like “cargo bike” are often written as two words, so the function joins them before the lookup.
// src/policy.ts import { createHash } from "node:crypto"; export const TOPICS = ["rental", "prices", "repair", "general"] as const; export type Topic = (typeof TOPICS)[number]; /** Key terms that change an answer. They must match exactly. */ const BIKE_TYPES: Record<string, string> = { "e-bike": "ebike", ebike: "ebike", pedelec: "ebike", "cargo-bike": "cargo", cargobike: "cargo", "kids-bike": "kids", citybike: "city", "city-bike": "city", tandem: "tandem", }; export function isTopic(value: unknown): value is Topic { return typeof value === "string" && (TOPICS as readonly string[]).includes(value); } /** Lowercase, drop punctuation, collapse whitespace. */ export function normalizeQuestion(question: string): string { return question .toLowerCase() .replace(/[^\p{L}\p{N}\s-]/gu, " ") .replace(/\s+/g, " ") .trim(); } /** * Personal or time-sensitive questions never go into the cache: * booking numbers, email addresses, “my booking”, “today”, “right now” etc. */ export function isCacheable(question: string): boolean { const q = question.toLowerCase(); if (/\d{5,}/.test(q)) return false; // booking or customer number if (/[\w.+-]+@[\w-]+\.[\w.]+/.test(q)) return false; // email address if (/\b(my|mine)\b/.test(q)) return false; if (/\b(today|tomorrow|right now|now|still free|available)\b/.test(q)) return false; return true; } /** Detects bike types; the result is stored as a TAG and must match exactly. */ export function detectBikeType(question: string): string { // English often writes compounds as two words: "cargo bike" becomes "cargo-bike". const words = normalizeQuestion(question) .replace(/\b(cargo|kids|city) bike\b/g, "$1-bike") .split(" "); const found = new Set<string>(); for (const word of words) { const type = BIKE_TYPES[word]; if (type) found.add(type); } return found.size === 0 ? "none" : [...found].sort().join("and"); } /** Key for the exact level: same question in the same context. */ export function exactKey(scope: CacheScope, question: string): string { const hash = createHash("sha256").update(normalizeQuestion(question)).digest("hex"); return `exact:${scope.topic}:${scope.kbVersion}:${hash}`; } export interface CacheScope { topic: Topic; kbVersion: string; bikeType: string; } /** Lowercase letters and digits only: these values end up in the query language. */ export function assertSafeTag(value: string): string { if (!/^[a-z0-9]{1,40}$/.test(value)) throw new Error(`Invalid tag value: ${value}`); return value; } /** Pre-filter on topic, knowledge version and bike type, then the nearest neighbor. */ export function buildLookupQuery(scope: CacheScope): string { const filter = [ `@topic:{${assertSafeTag(scope.topic)}}`, `@kbv:{${assertSafeTag(scope.kbVersion)}}`, `@bike:{${assertSafeTag(scope.bikeType)}}`, ].join(" "); return `(${filter})=>[KNN 1 @embedding $vec AS distance]`; } /** Hit only if the cosine distance is small enough (0 = identical). */ export function isHit(distance: number | undefined, maxDistance: number): boolean { return distance !== undefinedSource:https://dev.to/sascha-ms/semantic-cache-for-bedrock-with-elasticache-and-lambda-27poRelated posts

electrician services Singapore
Build High-Converting WordPress Sites Faster — Black Friday Sale 2026

